<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2106.10860] Multiplying Matrices Without Multiplying</title><meta property="og:description" content="Multiplying matrices is among the most fundamental and compute-intensive operations in machine learning. Consequently, there has been significant work on efficiently approximating matrix multiplies.
We introduce a lear…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Multiplying Matrices Without Multiplying">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Multiplying Matrices Without Multiplying">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2106.10860">

<!--Generated on Sat Mar  2 02:54:20 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="keywords" lang="en" content="Vector Quantization,  Approximate Algorithms,  Matrix Multiplication">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_pruned_first">
<h1 class="ltx_title ltx_title_document">Multiplying Matrices Without Multiplying</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Davis Blalock
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">John Guttag
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.2" class="ltx_p">Multiplying matrices is among the most fundamental and compute-intensive operations in machine learning. Consequently, there has been significant work on efficiently approximating matrix multiplies.
We introduce a learning-based algorithm for this task that greatly outperforms existing methods.
Experiments using hundreds of matrices from diverse domains show that it often runs <math id="id1.1.m1.1" class="ltx_math_unparsed" alttext="100\times" display="inline"><semantics id="id1.1.m1.1a"><mrow id="id1.1.m1.1b"><mn id="id1.1.m1.1.1">100</mn><mo lspace="0.222em" id="id1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="id1.1.m1.1c">100\times</annotation></semantics></math> faster than exact matrix products and <math id="id2.2.m2.1" class="ltx_math_unparsed" alttext="10\times" display="inline"><semantics id="id2.2.m2.1a"><mrow id="id2.2.m2.1b"><mn id="id2.2.m2.1.1">10</mn><mo lspace="0.222em" id="id2.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="id2.2.m2.1c">10\times</annotation></semantics></math> faster than current approximate methods. In the common case that one matrix is known ahead of time,
our method also has the interesting property that it requires zero multiply-adds.
These results suggest that a mixture of hashing, averaging, and byte shuffling—–the core operations of our method—–could be a more promising building block for machine learning than the sparsified, factorized, and/or scalar quantized matrix products that have recently been the focus of substantial research and hardware investment.</p>
</div>
<div class="ltx_keywords">Vector Quantization, Approximate Algorithms, Matrix Multiplication
</div>
<div class="ltx_keywords">Vector Quantization, Approximate Algorithms, Matrix Multiplication
</div>
<div id="p2" class="ltx_para">
<br class="ltx_break">
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Matrix multiplication is among the most fundamental subroutines used in machine learning and scientific computing. As a result, there has been a great deal of work on implementing high-speed matrix multiplication libraries <cite class="ltx_cite ltx_citemacro_citep">(Paszke et al., <a href="#bib.bib60" title="" class="ltx_ref">2017</a>; Guennebaud et al., <a href="#bib.bib32" title="" class="ltx_ref">2010</a>; Abadi et al., <a href="#bib.bib1" title="" class="ltx_ref">2016</a>)</cite>, designing custom hardware to accelerate multiplication of certain classes of matrices <cite class="ltx_cite ltx_citemacro_citep">(Han et al., <a href="#bib.bib34" title="" class="ltx_ref">2016</a>; Chen et al., <a href="#bib.bib16" title="" class="ltx_ref">2016</a>; Parashar et al., <a href="#bib.bib59" title="" class="ltx_ref">2017</a>; Jouppi et al., <a href="#bib.bib43" title="" class="ltx_ref">2017</a>)</cite>,
speeding up distributed matrix multiplication <cite class="ltx_cite ltx_citemacro_citep">(Yu et al., <a href="#bib.bib70" title="" class="ltx_ref">2017</a>; Dutta et al., <a href="#bib.bib24" title="" class="ltx_ref">2016</a>; Yu et al., <a href="#bib.bib71" title="" class="ltx_ref">2020</a>; Irony et al., <a href="#bib.bib40" title="" class="ltx_ref">2004</a>)</cite>,
and designing efficient Approximate Matrix Multiplication (AMM) algorithms under various assumptions.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.3" class="ltx_p">We focus on the AMM task under the assumptions that the matrices are tall, relatively dense, and resident in a single machine’s memory. In this setting, the primary challenge is minimizing the amount of compute time required to approximate linear operations with a given level of fidelity.
This setting arises naturally in machine learning and data mining when one has a data matrix <math id="S1.p2.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S1.p2.1.m1.1a"><mi id="S1.p2.1.m1.1.1" xref="S1.p2.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S1.p2.1.m1.1b"><ci id="S1.p2.1.m1.1.1.cmml" xref="S1.p2.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.1.m1.1c">\bm{A}</annotation></semantics></math> whose rows are samples and a linear operator <math id="S1.p2.2.m2.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S1.p2.2.m2.1a"><mi id="S1.p2.2.m2.1.1" xref="S1.p2.2.m2.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S1.p2.2.m2.1b"><ci id="S1.p2.2.m2.1.1.cmml" xref="S1.p2.2.m2.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.2.m2.1c">\bm{B}</annotation></semantics></math> one wishes to apply to these samples. <math id="S1.p2.3.m3.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S1.p2.3.m3.1a"><mi id="S1.p2.3.m3.1.1" xref="S1.p2.3.m3.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S1.p2.3.m3.1b"><ci id="S1.p2.3.m3.1.1.cmml" xref="S1.p2.3.m3.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.3.m3.1c">\bm{B}</annotation></semantics></math> could be a linear classifier, linear regressor, or an embedding matrix, among other possibilities.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.4" class="ltx_p">As a concrete example, consider the task of approximating a softmax classifier trained to predict image labels given embeddings derived from a neural network. Here, the rows of <math id="S1.p3.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S1.p3.1.m1.1a"><mi id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><ci id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">\bm{A}</annotation></semantics></math> are the embeddings for each image, and the columns of <math id="S1.p3.2.m2.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S1.p3.2.m2.1a"><mi id="S1.p3.2.m2.1.1" xref="S1.p3.2.m2.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S1.p3.2.m2.1b"><ci id="S1.p3.2.m2.1.1.cmml" xref="S1.p3.2.m2.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.2.m2.1c">\bm{B}</annotation></semantics></math> are the weight vectors for each class. Classification is performed by computing the product <math id="S1.p3.3.m3.1" class="ltx_Math" alttext="\bm{A}\bm{B}" display="inline"><semantics id="S1.p3.3.m3.1a"><mrow id="S1.p3.3.m3.1.1" xref="S1.p3.3.m3.1.1.cmml"><mi id="S1.p3.3.m3.1.1.2" xref="S1.p3.3.m3.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.p3.3.m3.1.1.1" xref="S1.p3.3.m3.1.1.1.cmml">​</mo><mi id="S1.p3.3.m3.1.1.3" xref="S1.p3.3.m3.1.1.3.cmml">𝑩</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p3.3.m3.1b"><apply id="S1.p3.3.m3.1.1.cmml" xref="S1.p3.3.m3.1.1"><times id="S1.p3.3.m3.1.1.1.cmml" xref="S1.p3.3.m3.1.1.1"></times><ci id="S1.p3.3.m3.1.1.2.cmml" xref="S1.p3.3.m3.1.1.2">𝑨</ci><ci id="S1.p3.3.m3.1.1.3.cmml" xref="S1.p3.3.m3.1.1.3">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.3.m3.1c">\bm{A}\bm{B}</annotation></semantics></math> and taking the argmax within each row of the result.
In Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we see the results of approximating <math id="S1.p3.4.m4.1" class="ltx_Math" alttext="\bm{A}\bm{B}" display="inline"><semantics id="S1.p3.4.m4.1a"><mrow id="S1.p3.4.m4.1.1" xref="S1.p3.4.m4.1.1.cmml"><mi id="S1.p3.4.m4.1.1.2" xref="S1.p3.4.m4.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.p3.4.m4.1.1.1" xref="S1.p3.4.m4.1.1.1.cmml">​</mo><mi id="S1.p3.4.m4.1.1.3" xref="S1.p3.4.m4.1.1.3.cmml">𝑩</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p3.4.m4.1b"><apply id="S1.p3.4.m4.1.1.cmml" xref="S1.p3.4.m4.1.1"><times id="S1.p3.4.m4.1.1.1.cmml" xref="S1.p3.4.m4.1.1.1"></times><ci id="S1.p3.4.m4.1.1.2.cmml" xref="S1.p3.4.m4.1.1.2">𝑨</ci><ci id="S1.p3.4.m4.1.1.3.cmml" xref="S1.p3.4.m4.1.1.3">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.4.m4.1c">\bm{A}\bm{B}</annotation></semantics></math> using our method and its best-performing rivals <cite class="ltx_cite ltx_citemacro_citep">(Dasgupta et al., <a href="#bib.bib17" title="" class="ltx_ref">2010</a>; Mairal et al., <a href="#bib.bib52" title="" class="ltx_ref">2009</a>)</cite> on the CIFAR-10 and CIFAR-100 datasets.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2106.10860/assets/x1.png" id="S1.F1.1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="438" height="405" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Our method achieves a dramatically better speed-accuracy tradeoff than the best existing methods when approximating two linear classifiers.</figcaption>
</figure>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Our method represents a significant methodological departure from most traditional approaches to this problem. Traditional AMM methods construct matrices <math id="S1.p4.1.m1.2" class="ltx_Math" alttext="\bm{V}_{A},\bm{V}_{B}\in\mathbb{R}^{D\times d},d\ll D" display="inline"><semantics id="S1.p4.1.m1.2a"><mrow id="S1.p4.1.m1.2.2.2" xref="S1.p4.1.m1.2.2.3.cmml"><mrow id="S1.p4.1.m1.1.1.1.1" xref="S1.p4.1.m1.1.1.1.1.cmml"><mrow id="S1.p4.1.m1.1.1.1.1.2.2" xref="S1.p4.1.m1.1.1.1.1.2.3.cmml"><msub id="S1.p4.1.m1.1.1.1.1.1.1.1" xref="S1.p4.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S1.p4.1.m1.1.1.1.1.1.1.1.2" xref="S1.p4.1.m1.1.1.1.1.1.1.1.2.cmml">𝑽</mi><mi id="S1.p4.1.m1.1.1.1.1.1.1.1.3" xref="S1.p4.1.m1.1.1.1.1.1.1.1.3.cmml">A</mi></msub><mo id="S1.p4.1.m1.1.1.1.1.2.2.3" xref="S1.p4.1.m1.1.1.1.1.2.3.cmml">,</mo><msub id="S1.p4.1.m1.1.1.1.1.2.2.2" xref="S1.p4.1.m1.1.1.1.1.2.2.2.cmml"><mi id="S1.p4.1.m1.1.1.1.1.2.2.2.2" xref="S1.p4.1.m1.1.1.1.1.2.2.2.2.cmml">𝑽</mi><mi id="S1.p4.1.m1.1.1.1.1.2.2.2.3" xref="S1.p4.1.m1.1.1.1.1.2.2.2.3.cmml">B</mi></msub></mrow><mo id="S1.p4.1.m1.1.1.1.1.3" xref="S1.p4.1.m1.1.1.1.1.3.cmml">∈</mo><msup id="S1.p4.1.m1.1.1.1.1.4" xref="S1.p4.1.m1.1.1.1.1.4.cmml"><mi id="S1.p4.1.m1.1.1.1.1.4.2" xref="S1.p4.1.m1.1.1.1.1.4.2.cmml">ℝ</mi><mrow id="S1.p4.1.m1.1.1.1.1.4.3" xref="S1.p4.1.m1.1.1.1.1.4.3.cmml"><mi id="S1.p4.1.m1.1.1.1.1.4.3.2" xref="S1.p4.1.m1.1.1.1.1.4.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S1.p4.1.m1.1.1.1.1.4.3.1" xref="S1.p4.1.m1.1.1.1.1.4.3.1.cmml">×</mo><mi id="S1.p4.1.m1.1.1.1.1.4.3.3" xref="S1.p4.1.m1.1.1.1.1.4.3.3.cmml">d</mi></mrow></msup></mrow><mo id="S1.p4.1.m1.2.2.2.3" xref="S1.p4.1.m1.2.2.3a.cmml">,</mo><mrow id="S1.p4.1.m1.2.2.2.2" xref="S1.p4.1.m1.2.2.2.2.cmml"><mi id="S1.p4.1.m1.2.2.2.2.2" xref="S1.p4.1.m1.2.2.2.2.2.cmml">d</mi><mo id="S1.p4.1.m1.2.2.2.2.1" xref="S1.p4.1.m1.2.2.2.2.1.cmml">≪</mo><mi id="S1.p4.1.m1.2.2.2.2.3" xref="S1.p4.1.m1.2.2.2.2.3.cmml">D</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.2b"><apply id="S1.p4.1.m1.2.2.3.cmml" xref="S1.p4.1.m1.2.2.2"><csymbol cd="ambiguous" id="S1.p4.1.m1.2.2.3a.cmml" xref="S1.p4.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S1.p4.1.m1.1.1.1.1.cmml" xref="S1.p4.1.m1.1.1.1.1"><in id="S1.p4.1.m1.1.1.1.1.3.cmml" xref="S1.p4.1.m1.1.1.1.1.3"></in><list id="S1.p4.1.m1.1.1.1.1.2.3.cmml" xref="S1.p4.1.m1.1.1.1.1.2.2"><apply id="S1.p4.1.m1.1.1.1.1.1.1.1.cmml" xref="S1.p4.1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S1.p4.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S1.p4.1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S1.p4.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S1.p4.1.m1.1.1.1.1.1.1.1.2">𝑽</ci><ci id="S1.p4.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S1.p4.1.m1.1.1.1.1.1.1.1.3">𝐴</ci></apply><apply id="S1.p4.1.m1.1.1.1.1.2.2.2.cmml" xref="S1.p4.1.m1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S1.p4.1.m1.1.1.1.1.2.2.2.1.cmml" xref="S1.p4.1.m1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S1.p4.1.m1.1.1.1.1.2.2.2.2.cmml" xref="S1.p4.1.m1.1.1.1.1.2.2.2.2">𝑽</ci><ci id="S1.p4.1.m1.1.1.1.1.2.2.2.3.cmml" xref="S1.p4.1.m1.1.1.1.1.2.2.2.3">𝐵</ci></apply></list><apply id="S1.p4.1.m1.1.1.1.1.4.cmml" xref="S1.p4.1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S1.p4.1.m1.1.1.1.1.4.1.cmml" xref="S1.p4.1.m1.1.1.1.1.4">superscript</csymbol><ci id="S1.p4.1.m1.1.1.1.1.4.2.cmml" xref="S1.p4.1.m1.1.1.1.1.4.2">ℝ</ci><apply id="S1.p4.1.m1.1.1.1.1.4.3.cmml" xref="S1.p4.1.m1.1.1.1.1.4.3"><times id="S1.p4.1.m1.1.1.1.1.4.3.1.cmml" xref="S1.p4.1.m1.1.1.1.1.4.3.1"></times><ci id="S1.p4.1.m1.1.1.1.1.4.3.2.cmml" xref="S1.p4.1.m1.1.1.1.1.4.3.2">𝐷</ci><ci id="S1.p4.1.m1.1.1.1.1.4.3.3.cmml" xref="S1.p4.1.m1.1.1.1.1.4.3.3">𝑑</ci></apply></apply></apply><apply id="S1.p4.1.m1.2.2.2.2.cmml" xref="S1.p4.1.m1.2.2.2.2"><csymbol cd="latexml" id="S1.p4.1.m1.2.2.2.2.1.cmml" xref="S1.p4.1.m1.2.2.2.2.1">much-less-than</csymbol><ci id="S1.p4.1.m1.2.2.2.2.2.cmml" xref="S1.p4.1.m1.2.2.2.2.2">𝑑</ci><ci id="S1.p4.1.m1.2.2.2.2.3.cmml" xref="S1.p4.1.m1.2.2.2.2.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.2c">\bm{V}_{A},\bm{V}_{B}\in\mathbb{R}^{D\times d},d\ll D</annotation></semantics></math> such that</p>
<table id="A6.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S1.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S1.E1.m1.1" class="ltx_Math" alttext="\displaystyle\bm{A}\bm{B}\approx(\bm{A}\bm{V}_{A})(\bm{V}_{B}^{\top}\bm{B})." display="inline"><semantics id="S1.E1.m1.1a"><mrow id="S1.E1.m1.1.1.1" xref="S1.E1.m1.1.1.1.1.cmml"><mrow id="S1.E1.m1.1.1.1.1" xref="S1.E1.m1.1.1.1.1.cmml"><mrow id="S1.E1.m1.1.1.1.1.4" xref="S1.E1.m1.1.1.1.1.4.cmml"><mi id="S1.E1.m1.1.1.1.1.4.2" xref="S1.E1.m1.1.1.1.1.4.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.E1.m1.1.1.1.1.4.1" xref="S1.E1.m1.1.1.1.1.4.1.cmml">​</mo><mi id="S1.E1.m1.1.1.1.1.4.3" xref="S1.E1.m1.1.1.1.1.4.3.cmml">𝑩</mi></mrow><mo id="S1.E1.m1.1.1.1.1.3" xref="S1.E1.m1.1.1.1.1.3.cmml">≈</mo><mrow id="S1.E1.m1.1.1.1.1.2" xref="S1.E1.m1.1.1.1.1.2.cmml"><mrow id="S1.E1.m1.1.1.1.1.1.1.1" xref="S1.E1.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.E1.m1.1.1.1.1.1.1.1.2" xref="S1.E1.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.E1.m1.1.1.1.1.1.1.1.1" xref="S1.E1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S1.E1.m1.1.1.1.1.1.1.1.1.2" xref="S1.E1.m1.1.1.1.1.1.1.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.E1.m1.1.1.1.1.1.1.1.1.1" xref="S1.E1.m1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S1.E1.m1.1.1.1.1.1.1.1.1.3" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S1.E1.m1.1.1.1.1.1.1.1.1.3.2" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3.2.cmml">𝑽</mi><mi id="S1.E1.m1.1.1.1.1.1.1.1.1.3.3" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3.3.cmml">A</mi></msub></mrow><mo stretchy="false" id="S1.E1.m1.1.1.1.1.1.1.1.3" xref="S1.E1.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S1.E1.m1.1.1.1.1.2.3" xref="S1.E1.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S1.E1.m1.1.1.1.1.2.2.1" xref="S1.E1.m1.1.1.1.1.2.2.1.1.cmml"><mo stretchy="false" id="S1.E1.m1.1.1.1.1.2.2.1.2" xref="S1.E1.m1.1.1.1.1.2.2.1.1.cmml">(</mo><mrow id="S1.E1.m1.1.1.1.1.2.2.1.1" xref="S1.E1.m1.1.1.1.1.2.2.1.1.cmml"><msubsup id="S1.E1.m1.1.1.1.1.2.2.1.1.2" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.cmml"><mi id="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.2" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.2.cmml">𝑽</mi><mi id="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.3" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.3.cmml">B</mi><mo id="S1.E1.m1.1.1.1.1.2.2.1.1.2.3" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.3.cmml">⊤</mo></msubsup><mo lspace="0em" rspace="0em" id="S1.E1.m1.1.1.1.1.2.2.1.1.1" xref="S1.E1.m1.1.1.1.1.2.2.1.1.1.cmml">​</mo><mi id="S1.E1.m1.1.1.1.1.2.2.1.1.3" xref="S1.E1.m1.1.1.1.1.2.2.1.1.3.cmml">𝑩</mi></mrow><mo stretchy="false" id="S1.E1.m1.1.1.1.1.2.2.1.3" xref="S1.E1.m1.1.1.1.1.2.2.1.1.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S1.E1.m1.1.1.1.2" xref="S1.E1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.E1.m1.1b"><apply id="S1.E1.m1.1.1.1.1.cmml" xref="S1.E1.m1.1.1.1"><approx id="S1.E1.m1.1.1.1.1.3.cmml" xref="S1.E1.m1.1.1.1.1.3"></approx><apply id="S1.E1.m1.1.1.1.1.4.cmml" xref="S1.E1.m1.1.1.1.1.4"><times id="S1.E1.m1.1.1.1.1.4.1.cmml" xref="S1.E1.m1.1.1.1.1.4.1"></times><ci id="S1.E1.m1.1.1.1.1.4.2.cmml" xref="S1.E1.m1.1.1.1.1.4.2">𝑨</ci><ci id="S1.E1.m1.1.1.1.1.4.3.cmml" xref="S1.E1.m1.1.1.1.1.4.3">𝑩</ci></apply><apply id="S1.E1.m1.1.1.1.1.2.cmml" xref="S1.E1.m1.1.1.1.1.2"><times id="S1.E1.m1.1.1.1.1.2.3.cmml" xref="S1.E1.m1.1.1.1.1.2.3"></times><apply id="S1.E1.m1.1.1.1.1.1.1.1.1.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1"><times id="S1.E1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1.1.1"></times><ci id="S1.E1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1.1.2">𝑨</ci><apply id="S1.E1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.E1.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S1.E1.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3.2">𝑽</ci><ci id="S1.E1.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S1.E1.m1.1.1.1.1.1.1.1.1.3.3">𝐴</ci></apply></apply><apply id="S1.E1.m1.1.1.1.1.2.2.1.1.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1"><times id="S1.E1.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.1"></times><apply id="S1.E1.m1.1.1.1.1.2.2.1.1.2.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2"><csymbol cd="ambiguous" id="S1.E1.m1.1.1.1.1.2.2.1.1.2.1.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2">superscript</csymbol><apply id="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2"><csymbol cd="ambiguous" id="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.1.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2">subscript</csymbol><ci id="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.2.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.2">𝑽</ci><ci id="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.3.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.2.3">𝐵</ci></apply><csymbol cd="latexml" id="S1.E1.m1.1.1.1.1.2.2.1.1.2.3.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.2.3">top</csymbol></apply><ci id="S1.E1.m1.1.1.1.1.2.2.1.1.3.cmml" xref="S1.E1.m1.1.1.1.1.2.2.1.1.3">𝑩</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.E1.m1.1c">\displaystyle\bm{A}\bm{B}\approx(\bm{A}\bm{V}_{A})(\bm{V}_{B}^{\top}\bm{B}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S1.p4.5" class="ltx_p">Often, <math id="S1.p4.2.m1.1" class="ltx_Math" alttext="\bm{V}_{A}" display="inline"><semantics id="S1.p4.2.m1.1a"><msub id="S1.p4.2.m1.1.1" xref="S1.p4.2.m1.1.1.cmml"><mi id="S1.p4.2.m1.1.1.2" xref="S1.p4.2.m1.1.1.2.cmml">𝑽</mi><mi id="S1.p4.2.m1.1.1.3" xref="S1.p4.2.m1.1.1.3.cmml">A</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p4.2.m1.1b"><apply id="S1.p4.2.m1.1.1.cmml" xref="S1.p4.2.m1.1.1"><csymbol cd="ambiguous" id="S1.p4.2.m1.1.1.1.cmml" xref="S1.p4.2.m1.1.1">subscript</csymbol><ci id="S1.p4.2.m1.1.1.2.cmml" xref="S1.p4.2.m1.1.1.2">𝑽</ci><ci id="S1.p4.2.m1.1.1.3.cmml" xref="S1.p4.2.m1.1.1.3">𝐴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.2.m1.1c">\bm{V}_{A}</annotation></semantics></math> and <math id="S1.p4.3.m2.1" class="ltx_Math" alttext="\bm{V}_{B}" display="inline"><semantics id="S1.p4.3.m2.1a"><msub id="S1.p4.3.m2.1.1" xref="S1.p4.3.m2.1.1.cmml"><mi id="S1.p4.3.m2.1.1.2" xref="S1.p4.3.m2.1.1.2.cmml">𝑽</mi><mi id="S1.p4.3.m2.1.1.3" xref="S1.p4.3.m2.1.1.3.cmml">B</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p4.3.m2.1b"><apply id="S1.p4.3.m2.1.1.cmml" xref="S1.p4.3.m2.1.1"><csymbol cd="ambiguous" id="S1.p4.3.m2.1.1.1.cmml" xref="S1.p4.3.m2.1.1">subscript</csymbol><ci id="S1.p4.3.m2.1.1.2.cmml" xref="S1.p4.3.m2.1.1.2">𝑽</ci><ci id="S1.p4.3.m2.1.1.3.cmml" xref="S1.p4.3.m2.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.3.m2.1c">\bm{V}_{B}</annotation></semantics></math> are sparse, embody some sort of sampling scheme, or have other structure such that these projection operations are faster than a dense matrix multiply. In short, these methods use linear functions to preprocess <math id="S1.p4.4.m3.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S1.p4.4.m3.1a"><mi id="S1.p4.4.m3.1.1" xref="S1.p4.4.m3.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S1.p4.4.m3.1b"><ci id="S1.p4.4.m3.1.1.cmml" xref="S1.p4.4.m3.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.4.m3.1c">\bm{A}</annotation></semantics></math> and <math id="S1.p4.5.m4.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S1.p4.5.m4.1a"><mi id="S1.p4.5.m4.1.1" xref="S1.p4.5.m4.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S1.p4.5.m4.1b"><ci id="S1.p4.5.m4.1.1.cmml" xref="S1.p4.5.m4.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.5.m4.1c">\bm{B}</annotation></semantics></math> and reduce the problem to exact matrix multiplication in a lower-dimensional space.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Our proposed method, <span id="S1.p5.1.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note"><span id="footnote1.1.1.1" class="ltx_text ltx_font_upright">1</span></span><span id="footnote1.5" class="ltx_text ltx_font_upright">Multiply-ADDitioN-lESS</span></span></span></span></span>, instead employs a <span id="S1.p5.1.2" class="ltx_text ltx_font_italic">nonlinear</span> preprocessing function and reduces the problem to table lookups. Moreover, in the case that <math id="S1.p5.1.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S1.p5.1.m1.1a"><mi id="S1.p5.1.m1.1.1" xref="S1.p5.1.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S1.p5.1.m1.1b"><ci id="S1.p5.1.m1.1.1.cmml" xref="S1.p5.1.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.1.m1.1c">\bm{B}</annotation></semantics></math> is known ahead of time—which happens when applying a trained linear model to new data, among other situations—<span id="S1.p5.1.3" class="ltx_text ltx_font_smallcaps">Maddness<span id="S1.p5.1.3.1" class="ltx_text ltx_font_upright"> </span></span>does not require any multiply-add operations.
Our method is most closely related to vector quantization methods used for similarity search (e.g., <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>; André et al., <a href="#bib.bib4" title="" class="ltx_ref">2017</a>, <a href="#bib.bib5" title="" class="ltx_ref">2019</a>; Jegou et al., <a href="#bib.bib41" title="" class="ltx_ref">2011</a>; Ge et al., <a href="#bib.bib29" title="" class="ltx_ref">2014</a>)</cite>). However, instead of using an expensive quantization function that requires many multiply-adds, we introduce a family of quantization functions that require no multiply-adds.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Our contributions can be summarized as follows:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">An efficient family of learned vector quantization functions that can encode over 100GB of data per second in a single CPU thread.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;padding-top:-2.8pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">A high-speed summation algorithm for low-bitwidth integers that avoids upcasting, saturation, and overflow.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;padding-top:-2.8pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">An algorithm based on these functions for approximate matrix multiplication. Experiments across hundreds of diverse matrices demonstrate that this algorithm significantly outperforms existing alternatives. It also features theoretical quality guarantees.</p>
</div>
</li>
</ul>
</div>
<section id="S1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">1.1 </span>Problem Formulation</h3>

<div id="S1.SS1.p1" class="ltx_para">
<p id="S1.SS1.p1.9" class="ltx_p">Let <math id="S1.SS1.p1.1.m1.1" class="ltx_Math" alttext="\bm{A}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="S1.SS1.p1.1.m1.1a"><mrow id="S1.SS1.p1.1.m1.1.1" xref="S1.SS1.p1.1.m1.1.1.cmml"><mi id="S1.SS1.p1.1.m1.1.1.2" xref="S1.SS1.p1.1.m1.1.1.2.cmml">𝑨</mi><mo id="S1.SS1.p1.1.m1.1.1.1" xref="S1.SS1.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S1.SS1.p1.1.m1.1.1.3" xref="S1.SS1.p1.1.m1.1.1.3.cmml"><mi id="S1.SS1.p1.1.m1.1.1.3.2" xref="S1.SS1.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S1.SS1.p1.1.m1.1.1.3.3" xref="S1.SS1.p1.1.m1.1.1.3.3.cmml"><mi id="S1.SS1.p1.1.m1.1.1.3.3.2" xref="S1.SS1.p1.1.m1.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S1.SS1.p1.1.m1.1.1.3.3.1" xref="S1.SS1.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S1.SS1.p1.1.m1.1.1.3.3.3" xref="S1.SS1.p1.1.m1.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.1.m1.1b"><apply id="S1.SS1.p1.1.m1.1.1.cmml" xref="S1.SS1.p1.1.m1.1.1"><in id="S1.SS1.p1.1.m1.1.1.1.cmml" xref="S1.SS1.p1.1.m1.1.1.1"></in><ci id="S1.SS1.p1.1.m1.1.1.2.cmml" xref="S1.SS1.p1.1.m1.1.1.2">𝑨</ci><apply id="S1.SS1.p1.1.m1.1.1.3.cmml" xref="S1.SS1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S1.SS1.p1.1.m1.1.1.3.1.cmml" xref="S1.SS1.p1.1.m1.1.1.3">superscript</csymbol><ci id="S1.SS1.p1.1.m1.1.1.3.2.cmml" xref="S1.SS1.p1.1.m1.1.1.3.2">ℝ</ci><apply id="S1.SS1.p1.1.m1.1.1.3.3.cmml" xref="S1.SS1.p1.1.m1.1.1.3.3"><times id="S1.SS1.p1.1.m1.1.1.3.3.1.cmml" xref="S1.SS1.p1.1.m1.1.1.3.3.1"></times><ci id="S1.SS1.p1.1.m1.1.1.3.3.2.cmml" xref="S1.SS1.p1.1.m1.1.1.3.3.2">𝑁</ci><ci id="S1.SS1.p1.1.m1.1.1.3.3.3.cmml" xref="S1.SS1.p1.1.m1.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.1.m1.1c">\bm{A}\in\mathbb{R}^{N\times D}</annotation></semantics></math> and <math id="S1.SS1.p1.2.m2.1" class="ltx_Math" alttext="\bm{B}\in\mathbb{R}^{D\times M}" display="inline"><semantics id="S1.SS1.p1.2.m2.1a"><mrow id="S1.SS1.p1.2.m2.1.1" xref="S1.SS1.p1.2.m2.1.1.cmml"><mi id="S1.SS1.p1.2.m2.1.1.2" xref="S1.SS1.p1.2.m2.1.1.2.cmml">𝑩</mi><mo id="S1.SS1.p1.2.m2.1.1.1" xref="S1.SS1.p1.2.m2.1.1.1.cmml">∈</mo><msup id="S1.SS1.p1.2.m2.1.1.3" xref="S1.SS1.p1.2.m2.1.1.3.cmml"><mi id="S1.SS1.p1.2.m2.1.1.3.2" xref="S1.SS1.p1.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S1.SS1.p1.2.m2.1.1.3.3" xref="S1.SS1.p1.2.m2.1.1.3.3.cmml"><mi id="S1.SS1.p1.2.m2.1.1.3.3.2" xref="S1.SS1.p1.2.m2.1.1.3.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S1.SS1.p1.2.m2.1.1.3.3.1" xref="S1.SS1.p1.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S1.SS1.p1.2.m2.1.1.3.3.3" xref="S1.SS1.p1.2.m2.1.1.3.3.3.cmml">M</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.2.m2.1b"><apply id="S1.SS1.p1.2.m2.1.1.cmml" xref="S1.SS1.p1.2.m2.1.1"><in id="S1.SS1.p1.2.m2.1.1.1.cmml" xref="S1.SS1.p1.2.m2.1.1.1"></in><ci id="S1.SS1.p1.2.m2.1.1.2.cmml" xref="S1.SS1.p1.2.m2.1.1.2">𝑩</ci><apply id="S1.SS1.p1.2.m2.1.1.3.cmml" xref="S1.SS1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S1.SS1.p1.2.m2.1.1.3.1.cmml" xref="S1.SS1.p1.2.m2.1.1.3">superscript</csymbol><ci id="S1.SS1.p1.2.m2.1.1.3.2.cmml" xref="S1.SS1.p1.2.m2.1.1.3.2">ℝ</ci><apply id="S1.SS1.p1.2.m2.1.1.3.3.cmml" xref="S1.SS1.p1.2.m2.1.1.3.3"><times id="S1.SS1.p1.2.m2.1.1.3.3.1.cmml" xref="S1.SS1.p1.2.m2.1.1.3.3.1"></times><ci id="S1.SS1.p1.2.m2.1.1.3.3.2.cmml" xref="S1.SS1.p1.2.m2.1.1.3.3.2">𝐷</ci><ci id="S1.SS1.p1.2.m2.1.1.3.3.3.cmml" xref="S1.SS1.p1.2.m2.1.1.3.3.3">𝑀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.2.m2.1c">\bm{B}\in\mathbb{R}^{D\times M}</annotation></semantics></math> be two matrices, with <math id="S1.SS1.p1.3.m3.1" class="ltx_Math" alttext="N\gg D\geq M" display="inline"><semantics id="S1.SS1.p1.3.m3.1a"><mrow id="S1.SS1.p1.3.m3.1.1" xref="S1.SS1.p1.3.m3.1.1.cmml"><mi id="S1.SS1.p1.3.m3.1.1.2" xref="S1.SS1.p1.3.m3.1.1.2.cmml">N</mi><mo id="S1.SS1.p1.3.m3.1.1.3" xref="S1.SS1.p1.3.m3.1.1.3.cmml">≫</mo><mi id="S1.SS1.p1.3.m3.1.1.4" xref="S1.SS1.p1.3.m3.1.1.4.cmml">D</mi><mo id="S1.SS1.p1.3.m3.1.1.5" xref="S1.SS1.p1.3.m3.1.1.5.cmml">≥</mo><mi id="S1.SS1.p1.3.m3.1.1.6" xref="S1.SS1.p1.3.m3.1.1.6.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.3.m3.1b"><apply id="S1.SS1.p1.3.m3.1.1.cmml" xref="S1.SS1.p1.3.m3.1.1"><and id="S1.SS1.p1.3.m3.1.1a.cmml" xref="S1.SS1.p1.3.m3.1.1"></and><apply id="S1.SS1.p1.3.m3.1.1b.cmml" xref="S1.SS1.p1.3.m3.1.1"><csymbol cd="latexml" id="S1.SS1.p1.3.m3.1.1.3.cmml" xref="S1.SS1.p1.3.m3.1.1.3">much-greater-than</csymbol><ci id="S1.SS1.p1.3.m3.1.1.2.cmml" xref="S1.SS1.p1.3.m3.1.1.2">𝑁</ci><ci id="S1.SS1.p1.3.m3.1.1.4.cmml" xref="S1.SS1.p1.3.m3.1.1.4">𝐷</ci></apply><apply id="S1.SS1.p1.3.m3.1.1c.cmml" xref="S1.SS1.p1.3.m3.1.1"><geq id="S1.SS1.p1.3.m3.1.1.5.cmml" xref="S1.SS1.p1.3.m3.1.1.5"></geq><share href="#S1.SS1.p1.3.m3.1.1.4.cmml" id="S1.SS1.p1.3.m3.1.1d.cmml" xref="S1.SS1.p1.3.m3.1.1"></share><ci id="S1.SS1.p1.3.m3.1.1.6.cmml" xref="S1.SS1.p1.3.m3.1.1.6">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.3.m3.1c">N\gg D\geq M</annotation></semantics></math>. Given a computation time budget <math id="S1.SS1.p1.4.m4.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S1.SS1.p1.4.m4.1a"><mi id="S1.SS1.p1.4.m4.1.1" xref="S1.SS1.p1.4.m4.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.4.m4.1b"><ci id="S1.SS1.p1.4.m4.1.1.cmml" xref="S1.SS1.p1.4.m4.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.4.m4.1c">\tau</annotation></semantics></math>, our task is to
construct three functions <math id="S1.SS1.p1.5.m5.1" class="ltx_Math" alttext="g(\cdot)" display="inline"><semantics id="S1.SS1.p1.5.m5.1a"><mrow id="S1.SS1.p1.5.m5.1.2" xref="S1.SS1.p1.5.m5.1.2.cmml"><mi id="S1.SS1.p1.5.m5.1.2.2" xref="S1.SS1.p1.5.m5.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.5.m5.1.2.1" xref="S1.SS1.p1.5.m5.1.2.1.cmml">​</mo><mrow id="S1.SS1.p1.5.m5.1.2.3.2" xref="S1.SS1.p1.5.m5.1.2.cmml"><mo stretchy="false" id="S1.SS1.p1.5.m5.1.2.3.2.1" xref="S1.SS1.p1.5.m5.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.5.m5.1.1" xref="S1.SS1.p1.5.m5.1.1.cmml">⋅</mo><mo stretchy="false" id="S1.SS1.p1.5.m5.1.2.3.2.2" xref="S1.SS1.p1.5.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.5.m5.1b"><apply id="S1.SS1.p1.5.m5.1.2.cmml" xref="S1.SS1.p1.5.m5.1.2"><times id="S1.SS1.p1.5.m5.1.2.1.cmml" xref="S1.SS1.p1.5.m5.1.2.1"></times><ci id="S1.SS1.p1.5.m5.1.2.2.cmml" xref="S1.SS1.p1.5.m5.1.2.2">𝑔</ci><ci id="S1.SS1.p1.5.m5.1.1.cmml" xref="S1.SS1.p1.5.m5.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.5.m5.1c">g(\cdot)</annotation></semantics></math>, <math id="S1.SS1.p1.6.m6.1" class="ltx_Math" alttext="h(\cdot)" display="inline"><semantics id="S1.SS1.p1.6.m6.1a"><mrow id="S1.SS1.p1.6.m6.1.2" xref="S1.SS1.p1.6.m6.1.2.cmml"><mi id="S1.SS1.p1.6.m6.1.2.2" xref="S1.SS1.p1.6.m6.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.6.m6.1.2.1" xref="S1.SS1.p1.6.m6.1.2.1.cmml">​</mo><mrow id="S1.SS1.p1.6.m6.1.2.3.2" xref="S1.SS1.p1.6.m6.1.2.cmml"><mo stretchy="false" id="S1.SS1.p1.6.m6.1.2.3.2.1" xref="S1.SS1.p1.6.m6.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.6.m6.1.1" xref="S1.SS1.p1.6.m6.1.1.cmml">⋅</mo><mo stretchy="false" id="S1.SS1.p1.6.m6.1.2.3.2.2" xref="S1.SS1.p1.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.6.m6.1b"><apply id="S1.SS1.p1.6.m6.1.2.cmml" xref="S1.SS1.p1.6.m6.1.2"><times id="S1.SS1.p1.6.m6.1.2.1.cmml" xref="S1.SS1.p1.6.m6.1.2.1"></times><ci id="S1.SS1.p1.6.m6.1.2.2.cmml" xref="S1.SS1.p1.6.m6.1.2.2">ℎ</ci><ci id="S1.SS1.p1.6.m6.1.1.cmml" xref="S1.SS1.p1.6.m6.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.6.m6.1c">h(\cdot)</annotation></semantics></math>, and <math id="S1.SS1.p1.7.m7.1" class="ltx_Math" alttext="f(\cdot)" display="inline"><semantics id="S1.SS1.p1.7.m7.1a"><mrow id="S1.SS1.p1.7.m7.1.2" xref="S1.SS1.p1.7.m7.1.2.cmml"><mi id="S1.SS1.p1.7.m7.1.2.2" xref="S1.SS1.p1.7.m7.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.7.m7.1.2.1" xref="S1.SS1.p1.7.m7.1.2.1.cmml">​</mo><mrow id="S1.SS1.p1.7.m7.1.2.3.2" xref="S1.SS1.p1.7.m7.1.2.cmml"><mo stretchy="false" id="S1.SS1.p1.7.m7.1.2.3.2.1" xref="S1.SS1.p1.7.m7.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.7.m7.1.1" xref="S1.SS1.p1.7.m7.1.1.cmml">⋅</mo><mo stretchy="false" id="S1.SS1.p1.7.m7.1.2.3.2.2" xref="S1.SS1.p1.7.m7.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.7.m7.1b"><apply id="S1.SS1.p1.7.m7.1.2.cmml" xref="S1.SS1.p1.7.m7.1.2"><times id="S1.SS1.p1.7.m7.1.2.1.cmml" xref="S1.SS1.p1.7.m7.1.2.1"></times><ci id="S1.SS1.p1.7.m7.1.2.2.cmml" xref="S1.SS1.p1.7.m7.1.2.2">𝑓</ci><ci id="S1.SS1.p1.7.m7.1.1.cmml" xref="S1.SS1.p1.7.m7.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.7.m7.1c">f(\cdot)</annotation></semantics></math>, along with constants <math id="S1.SS1.p1.8.m8.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="S1.SS1.p1.8.m8.1a"><mi id="S1.SS1.p1.8.m8.1.1" xref="S1.SS1.p1.8.m8.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.8.m8.1b"><ci id="S1.SS1.p1.8.m8.1.1.cmml" xref="S1.SS1.p1.8.m8.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.8.m8.1c">\alpha</annotation></semantics></math> and <math id="S1.SS1.p1.9.m9.1" class="ltx_Math" alttext="\bm{\beta}" display="inline"><semantics id="S1.SS1.p1.9.m9.1a"><mi id="S1.SS1.p1.9.m9.1.1" xref="S1.SS1.p1.9.m9.1.1.cmml">𝜷</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.9.m9.1b"><ci id="S1.SS1.p1.9.m9.1.1.cmml" xref="S1.SS1.p1.9.m9.1.1">𝜷</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.9.m9.1c">\bm{\beta}</annotation></semantics></math>, such that</p>
<table id="A6.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S1.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S1.E2.m1.5" class="ltx_Math" alttext="\displaystyle\lVert\alpha f(g(\bm{A}),h(\bm{B}))+\bm{\beta}-\bm{A}\bm{B}\rVert_{F}&lt;\varepsilon(\tau)\lVert\bm{A}\bm{B}\rVert_{F}" display="inline"><semantics id="S1.E2.m1.5a"><mrow id="S1.E2.m1.5.5" xref="S1.E2.m1.5.5.cmml"><msub id="S1.E2.m1.4.4.1" xref="S1.E2.m1.4.4.1.cmml"><mrow id="S1.E2.m1.4.4.1.1.1" xref="S1.E2.m1.4.4.1.1.2.cmml"><mo fence="true" rspace="0em" id="S1.E2.m1.4.4.1.1.1.2" xref="S1.E2.m1.4.4.1.1.2.1.cmml">∥</mo><mrow id="S1.E2.m1.4.4.1.1.1.1" xref="S1.E2.m1.4.4.1.1.1.1.cmml"><mrow id="S1.E2.m1.4.4.1.1.1.1.2" xref="S1.E2.m1.4.4.1.1.1.1.2.cmml"><mrow id="S1.E2.m1.4.4.1.1.1.1.2.2" xref="S1.E2.m1.4.4.1.1.1.1.2.2.cmml"><mi id="S1.E2.m1.4.4.1.1.1.1.2.2.4" xref="S1.E2.m1.4.4.1.1.1.1.2.2.4.cmml">α</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.4.4.1.1.1.1.2.2.3" xref="S1.E2.m1.4.4.1.1.1.1.2.2.3.cmml">​</mo><mi id="S1.E2.m1.4.4.1.1.1.1.2.2.5" xref="S1.E2.m1.4.4.1.1.1.1.2.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.4.4.1.1.1.1.2.2.3a" xref="S1.E2.m1.4.4.1.1.1.1.2.2.3.cmml">​</mo><mrow id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.3.cmml"><mo stretchy="false" id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.3" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.3.cmml">(</mo><mrow id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml"><mi id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.1" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml">(</mo><mi id="S1.E2.m1.1.1" xref="S1.E2.m1.1.1.cmml">𝑨</mi><mo stretchy="false" id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.3.2.2" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.4" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.3.cmml">,</mo><mrow id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.cmml"><mi id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.2" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.1" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.1.cmml">​</mo><mrow id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.3.2" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.cmml"><mo stretchy="false" id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.3.2.1" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.cmml">(</mo><mi id="S1.E2.m1.2.2" xref="S1.E2.m1.2.2.cmml">𝑩</mi><mo stretchy="false" id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.3.2.2" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.5" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S1.E2.m1.4.4.1.1.1.1.2.3" xref="S1.E2.m1.4.4.1.1.1.1.2.3.cmml">+</mo><mi id="S1.E2.m1.4.4.1.1.1.1.2.4" xref="S1.E2.m1.4.4.1.1.1.1.2.4.cmml">𝜷</mi></mrow><mo id="S1.E2.m1.4.4.1.1.1.1.3" xref="S1.E2.m1.4.4.1.1.1.1.3.cmml">−</mo><mrow id="S1.E2.m1.4.4.1.1.1.1.4" xref="S1.E2.m1.4.4.1.1.1.1.4.cmml"><mi id="S1.E2.m1.4.4.1.1.1.1.4.2" xref="S1.E2.m1.4.4.1.1.1.1.4.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.4.4.1.1.1.1.4.1" xref="S1.E2.m1.4.4.1.1.1.1.4.1.cmml">​</mo><mi id="S1.E2.m1.4.4.1.1.1.1.4.3" xref="S1.E2.m1.4.4.1.1.1.1.4.3.cmml">𝑩</mi></mrow></mrow><mo fence="true" lspace="0em" id="S1.E2.m1.4.4.1.1.1.3" xref="S1.E2.m1.4.4.1.1.2.1.cmml">∥</mo></mrow><mi id="S1.E2.m1.4.4.1.3" xref="S1.E2.m1.4.4.1.3.cmml">F</mi></msub><mo id="S1.E2.m1.5.5.3" xref="S1.E2.m1.5.5.3.cmml">&lt;</mo><mrow id="S1.E2.m1.5.5.2" xref="S1.E2.m1.5.5.2.cmml"><mi id="S1.E2.m1.5.5.2.3" xref="S1.E2.m1.5.5.2.3.cmml">ε</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.5.5.2.2" xref="S1.E2.m1.5.5.2.2.cmml">​</mo><mrow id="S1.E2.m1.5.5.2.4.2" xref="S1.E2.m1.5.5.2.cmml"><mo stretchy="false" id="S1.E2.m1.5.5.2.4.2.1" xref="S1.E2.m1.5.5.2.cmml">(</mo><mi id="S1.E2.m1.3.3" xref="S1.E2.m1.3.3.cmml">τ</mi><mo stretchy="false" id="S1.E2.m1.5.5.2.4.2.2" xref="S1.E2.m1.5.5.2.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S1.E2.m1.5.5.2.2a" xref="S1.E2.m1.5.5.2.2.cmml">​</mo><msub id="S1.E2.m1.5.5.2.1" xref="S1.E2.m1.5.5.2.1.cmml"><mrow id="S1.E2.m1.5.5.2.1.1.1" xref="S1.E2.m1.5.5.2.1.1.2.cmml"><mo fence="true" rspace="0em" id="S1.E2.m1.5.5.2.1.1.1.2" xref="S1.E2.m1.5.5.2.1.1.2.1.cmml">∥</mo><mrow id="S1.E2.m1.5.5.2.1.1.1.1" xref="S1.E2.m1.5.5.2.1.1.1.1.cmml"><mi id="S1.E2.m1.5.5.2.1.1.1.1.2" xref="S1.E2.m1.5.5.2.1.1.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.E2.m1.5.5.2.1.1.1.1.1" xref="S1.E2.m1.5.5.2.1.1.1.1.1.cmml">​</mo><mi id="S1.E2.m1.5.5.2.1.1.1.1.3" xref="S1.E2.m1.5.5.2.1.1.1.1.3.cmml">𝑩</mi></mrow><mo fence="true" lspace="0em" id="S1.E2.m1.5.5.2.1.1.1.3" xref="S1.E2.m1.5.5.2.1.1.2.1.cmml">∥</mo></mrow><mi id="S1.E2.m1.5.5.2.1.3" xref="S1.E2.m1.5.5.2.1.3.cmml">F</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.E2.m1.5b"><apply id="S1.E2.m1.5.5.cmml" xref="S1.E2.m1.5.5"><lt id="S1.E2.m1.5.5.3.cmml" xref="S1.E2.m1.5.5.3"></lt><apply id="S1.E2.m1.4.4.1.cmml" xref="S1.E2.m1.4.4.1"><csymbol cd="ambiguous" id="S1.E2.m1.4.4.1.2.cmml" xref="S1.E2.m1.4.4.1">subscript</csymbol><apply id="S1.E2.m1.4.4.1.1.2.cmml" xref="S1.E2.m1.4.4.1.1.1"><csymbol cd="latexml" id="S1.E2.m1.4.4.1.1.2.1.cmml" xref="S1.E2.m1.4.4.1.1.1.2">delimited-∥∥</csymbol><apply id="S1.E2.m1.4.4.1.1.1.1.cmml" xref="S1.E2.m1.4.4.1.1.1.1"><minus id="S1.E2.m1.4.4.1.1.1.1.3.cmml" xref="S1.E2.m1.4.4.1.1.1.1.3"></minus><apply id="S1.E2.m1.4.4.1.1.1.1.2.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2"><plus id="S1.E2.m1.4.4.1.1.1.1.2.3.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.3"></plus><apply id="S1.E2.m1.4.4.1.1.1.1.2.2.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2"><times id="S1.E2.m1.4.4.1.1.1.1.2.2.3.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.3"></times><ci id="S1.E2.m1.4.4.1.1.1.1.2.2.4.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.4">𝛼</ci><ci id="S1.E2.m1.4.4.1.1.1.1.2.2.5.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.5">𝑓</ci><interval closure="open" id="S1.E2.m1.4.4.1.1.1.1.2.2.2.3.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2"><apply id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1"><times id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1.cmml" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.1"></times><ci id="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2.cmml" xref="S1.E2.m1.4.4.1.1.1.1.1.1.1.1.1.2">𝑔</ci><ci id="S1.E2.m1.1.1.cmml" xref="S1.E2.m1.1.1">𝑨</ci></apply><apply id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2"><times id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.1.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.1"></times><ci id="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.2.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.2.2.2.2.2">ℎ</ci><ci id="S1.E2.m1.2.2.cmml" xref="S1.E2.m1.2.2">𝑩</ci></apply></interval></apply><ci id="S1.E2.m1.4.4.1.1.1.1.2.4.cmml" xref="S1.E2.m1.4.4.1.1.1.1.2.4">𝜷</ci></apply><apply id="S1.E2.m1.4.4.1.1.1.1.4.cmml" xref="S1.E2.m1.4.4.1.1.1.1.4"><times id="S1.E2.m1.4.4.1.1.1.1.4.1.cmml" xref="S1.E2.m1.4.4.1.1.1.1.4.1"></times><ci id="S1.E2.m1.4.4.1.1.1.1.4.2.cmml" xref="S1.E2.m1.4.4.1.1.1.1.4.2">𝑨</ci><ci id="S1.E2.m1.4.4.1.1.1.1.4.3.cmml" xref="S1.E2.m1.4.4.1.1.1.1.4.3">𝑩</ci></apply></apply></apply><ci id="S1.E2.m1.4.4.1.3.cmml" xref="S1.E2.m1.4.4.1.3">𝐹</ci></apply><apply id="S1.E2.m1.5.5.2.cmml" xref="S1.E2.m1.5.5.2"><times id="S1.E2.m1.5.5.2.2.cmml" xref="S1.E2.m1.5.5.2.2"></times><ci id="S1.E2.m1.5.5.2.3.cmml" xref="S1.E2.m1.5.5.2.3">𝜀</ci><ci id="S1.E2.m1.3.3.cmml" xref="S1.E2.m1.3.3">𝜏</ci><apply id="S1.E2.m1.5.5.2.1.cmml" xref="S1.E2.m1.5.5.2.1"><csymbol cd="ambiguous" id="S1.E2.m1.5.5.2.1.2.cmml" xref="S1.E2.m1.5.5.2.1">subscript</csymbol><apply id="S1.E2.m1.5.5.2.1.1.2.cmml" xref="S1.E2.m1.5.5.2.1.1.1"><csymbol cd="latexml" id="S1.E2.m1.5.5.2.1.1.2.1.cmml" xref="S1.E2.m1.5.5.2.1.1.1.2">delimited-∥∥</csymbol><apply id="S1.E2.m1.5.5.2.1.1.1.1.cmml" xref="S1.E2.m1.5.5.2.1.1.1.1"><times id="S1.E2.m1.5.5.2.1.1.1.1.1.cmml" xref="S1.E2.m1.5.5.2.1.1.1.1.1"></times><ci id="S1.E2.m1.5.5.2.1.1.1.1.2.cmml" xref="S1.E2.m1.5.5.2.1.1.1.1.2">𝑨</ci><ci id="S1.E2.m1.5.5.2.1.1.1.1.3.cmml" xref="S1.E2.m1.5.5.2.1.1.1.1.3">𝑩</ci></apply></apply><ci id="S1.E2.m1.5.5.2.1.3.cmml" xref="S1.E2.m1.5.5.2.1.3">𝐹</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.E2.m1.5c">\displaystyle\lVert\alpha f(g(\bm{A}),h(\bm{B}))+\bm{\beta}-\bm{A}\bm{B}\rVert_{F}&lt;\varepsilon(\tau)\lVert\bm{A}\bm{B}\rVert_{F}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S1.SS1.p1.16" class="ltx_p">for as small an error <math id="S1.SS1.p1.10.m1.1" class="ltx_Math" alttext="\varepsilon(\tau)" display="inline"><semantics id="S1.SS1.p1.10.m1.1a"><mrow id="S1.SS1.p1.10.m1.1.2" xref="S1.SS1.p1.10.m1.1.2.cmml"><mi id="S1.SS1.p1.10.m1.1.2.2" xref="S1.SS1.p1.10.m1.1.2.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.10.m1.1.2.1" xref="S1.SS1.p1.10.m1.1.2.1.cmml">​</mo><mrow id="S1.SS1.p1.10.m1.1.2.3.2" xref="S1.SS1.p1.10.m1.1.2.cmml"><mo stretchy="false" id="S1.SS1.p1.10.m1.1.2.3.2.1" xref="S1.SS1.p1.10.m1.1.2.cmml">(</mo><mi id="S1.SS1.p1.10.m1.1.1" xref="S1.SS1.p1.10.m1.1.1.cmml">τ</mi><mo stretchy="false" id="S1.SS1.p1.10.m1.1.2.3.2.2" xref="S1.SS1.p1.10.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.10.m1.1b"><apply id="S1.SS1.p1.10.m1.1.2.cmml" xref="S1.SS1.p1.10.m1.1.2"><times id="S1.SS1.p1.10.m1.1.2.1.cmml" xref="S1.SS1.p1.10.m1.1.2.1"></times><ci id="S1.SS1.p1.10.m1.1.2.2.cmml" xref="S1.SS1.p1.10.m1.1.2.2">𝜀</ci><ci id="S1.SS1.p1.10.m1.1.1.cmml" xref="S1.SS1.p1.10.m1.1.1">𝜏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.10.m1.1c">\varepsilon(\tau)</annotation></semantics></math> possible. The constants <math id="S1.SS1.p1.11.m2.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="S1.SS1.p1.11.m2.1a"><mi id="S1.SS1.p1.11.m2.1.1" xref="S1.SS1.p1.11.m2.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.11.m2.1b"><ci id="S1.SS1.p1.11.m2.1.1.cmml" xref="S1.SS1.p1.11.m2.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.11.m2.1c">\alpha</annotation></semantics></math> and <math id="S1.SS1.p1.12.m3.1" class="ltx_Math" alttext="\bm{\beta}" display="inline"><semantics id="S1.SS1.p1.12.m3.1a"><mi id="S1.SS1.p1.12.m3.1.1" xref="S1.SS1.p1.12.m3.1.1.cmml">𝜷</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.12.m3.1b"><ci id="S1.SS1.p1.12.m3.1.1.cmml" xref="S1.SS1.p1.12.m3.1.1">𝜷</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.12.m3.1c">\bm{\beta}</annotation></semantics></math> are separated from <math id="S1.SS1.p1.13.m4.2" class="ltx_Math" alttext="f(\cdot,\cdot)" display="inline"><semantics id="S1.SS1.p1.13.m4.2a"><mrow id="S1.SS1.p1.13.m4.2.3" xref="S1.SS1.p1.13.m4.2.3.cmml"><mi id="S1.SS1.p1.13.m4.2.3.2" xref="S1.SS1.p1.13.m4.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.13.m4.2.3.1" xref="S1.SS1.p1.13.m4.2.3.1.cmml">​</mo><mrow id="S1.SS1.p1.13.m4.2.3.3.2" xref="S1.SS1.p1.13.m4.2.3.3.1.cmml"><mo stretchy="false" id="S1.SS1.p1.13.m4.2.3.3.2.1" xref="S1.SS1.p1.13.m4.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.13.m4.1.1" xref="S1.SS1.p1.13.m4.1.1.cmml">⋅</mo><mo rspace="0em" id="S1.SS1.p1.13.m4.2.3.3.2.2" xref="S1.SS1.p1.13.m4.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.13.m4.2.2" xref="S1.SS1.p1.13.m4.2.2.cmml">⋅</mo><mo stretchy="false" id="S1.SS1.p1.13.m4.2.3.3.2.3" xref="S1.SS1.p1.13.m4.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.13.m4.2b"><apply id="S1.SS1.p1.13.m4.2.3.cmml" xref="S1.SS1.p1.13.m4.2.3"><times id="S1.SS1.p1.13.m4.2.3.1.cmml" xref="S1.SS1.p1.13.m4.2.3.1"></times><ci id="S1.SS1.p1.13.m4.2.3.2.cmml" xref="S1.SS1.p1.13.m4.2.3.2">𝑓</ci><interval closure="open" id="S1.SS1.p1.13.m4.2.3.3.1.cmml" xref="S1.SS1.p1.13.m4.2.3.3.2"><ci id="S1.SS1.p1.13.m4.1.1.cmml" xref="S1.SS1.p1.13.m4.1.1">⋅</ci><ci id="S1.SS1.p1.13.m4.2.2.cmml" xref="S1.SS1.p1.13.m4.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.13.m4.2c">f(\cdot,\cdot)</annotation></semantics></math> so that <math id="S1.SS1.p1.14.m5.2" class="ltx_Math" alttext="f(\cdot,\cdot)" display="inline"><semantics id="S1.SS1.p1.14.m5.2a"><mrow id="S1.SS1.p1.14.m5.2.3" xref="S1.SS1.p1.14.m5.2.3.cmml"><mi id="S1.SS1.p1.14.m5.2.3.2" xref="S1.SS1.p1.14.m5.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.14.m5.2.3.1" xref="S1.SS1.p1.14.m5.2.3.1.cmml">​</mo><mrow id="S1.SS1.p1.14.m5.2.3.3.2" xref="S1.SS1.p1.14.m5.2.3.3.1.cmml"><mo stretchy="false" id="S1.SS1.p1.14.m5.2.3.3.2.1" xref="S1.SS1.p1.14.m5.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.14.m5.1.1" xref="S1.SS1.p1.14.m5.1.1.cmml">⋅</mo><mo rspace="0em" id="S1.SS1.p1.14.m5.2.3.3.2.2" xref="S1.SS1.p1.14.m5.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S1.SS1.p1.14.m5.2.2" xref="S1.SS1.p1.14.m5.2.2.cmml">⋅</mo><mo stretchy="false" id="S1.SS1.p1.14.m5.2.3.3.2.3" xref="S1.SS1.p1.14.m5.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.14.m5.2b"><apply id="S1.SS1.p1.14.m5.2.3.cmml" xref="S1.SS1.p1.14.m5.2.3"><times id="S1.SS1.p1.14.m5.2.3.1.cmml" xref="S1.SS1.p1.14.m5.2.3.1"></times><ci id="S1.SS1.p1.14.m5.2.3.2.cmml" xref="S1.SS1.p1.14.m5.2.3.2">𝑓</ci><interval closure="open" id="S1.SS1.p1.14.m5.2.3.3.1.cmml" xref="S1.SS1.p1.14.m5.2.3.3.2"><ci id="S1.SS1.p1.14.m5.1.1.cmml" xref="S1.SS1.p1.14.m5.1.1">⋅</ci><ci id="S1.SS1.p1.14.m5.2.2.cmml" xref="S1.SS1.p1.14.m5.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.14.m5.2c">f(\cdot,\cdot)</annotation></semantics></math> can produce low-bitwidth outputs (e.g., in the range <math id="S1.SS1.p1.15.m6.2" class="ltx_Math" alttext="[0,255]" display="inline"><semantics id="S1.SS1.p1.15.m6.2a"><mrow id="S1.SS1.p1.15.m6.2.3.2" xref="S1.SS1.p1.15.m6.2.3.1.cmml"><mo stretchy="false" id="S1.SS1.p1.15.m6.2.3.2.1" xref="S1.SS1.p1.15.m6.2.3.1.cmml">[</mo><mn id="S1.SS1.p1.15.m6.1.1" xref="S1.SS1.p1.15.m6.1.1.cmml">0</mn><mo id="S1.SS1.p1.15.m6.2.3.2.2" xref="S1.SS1.p1.15.m6.2.3.1.cmml">,</mo><mn id="S1.SS1.p1.15.m6.2.2" xref="S1.SS1.p1.15.m6.2.2.cmml">255</mn><mo stretchy="false" id="S1.SS1.p1.15.m6.2.3.2.3" xref="S1.SS1.p1.15.m6.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.15.m6.2b"><interval closure="closed" id="S1.SS1.p1.15.m6.2.3.1.cmml" xref="S1.SS1.p1.15.m6.2.3.2"><cn type="integer" id="S1.SS1.p1.15.m6.1.1.cmml" xref="S1.SS1.p1.15.m6.1.1">0</cn><cn type="integer" id="S1.SS1.p1.15.m6.2.2.cmml" xref="S1.SS1.p1.15.m6.2.2">255</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.15.m6.2c">[0,255]</annotation></semantics></math>) even when the entries of <math id="S1.SS1.p1.16.m7.1" class="ltx_Math" alttext="\bm{A}\bm{B}" display="inline"><semantics id="S1.SS1.p1.16.m7.1a"><mrow id="S1.SS1.p1.16.m7.1.1" xref="S1.SS1.p1.16.m7.1.1.cmml"><mi id="S1.SS1.p1.16.m7.1.1.2" xref="S1.SS1.p1.16.m7.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S1.SS1.p1.16.m7.1.1.1" xref="S1.SS1.p1.16.m7.1.1.1.cmml">​</mo><mi id="S1.SS1.p1.16.m7.1.1.3" xref="S1.SS1.p1.16.m7.1.1.3.cmml">𝑩</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.p1.16.m7.1b"><apply id="S1.SS1.p1.16.m7.1.1.cmml" xref="S1.SS1.p1.16.m7.1.1"><times id="S1.SS1.p1.16.m7.1.1.1.cmml" xref="S1.SS1.p1.16.m7.1.1.1"></times><ci id="S1.SS1.p1.16.m7.1.1.2.cmml" xref="S1.SS1.p1.16.m7.1.1.2">𝑨</ci><ci id="S1.SS1.p1.16.m7.1.1.3.cmml" xref="S1.SS1.p1.16.m7.1.1.3">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p1.16.m7.1c">\bm{A}\bm{B}</annotation></semantics></math> do not fall in this range.</p>
</div>
<div id="S1.SS1.p2" class="ltx_para">
<p id="S1.SS1.p2.3" class="ltx_p">We assume the existence of a training set <math id="S1.SS1.p2.1.m1.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S1.SS1.p2.1.m1.1a"><mover accent="true" id="S1.SS1.p2.1.m1.1.1" xref="S1.SS1.p2.1.m1.1.1.cmml"><mi id="S1.SS1.p2.1.m1.1.1.2" xref="S1.SS1.p2.1.m1.1.1.2.cmml">𝑨</mi><mo id="S1.SS1.p2.1.m1.1.1.1" xref="S1.SS1.p2.1.m1.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S1.SS1.p2.1.m1.1b"><apply id="S1.SS1.p2.1.m1.1.1.cmml" xref="S1.SS1.p2.1.m1.1.1"><ci id="S1.SS1.p2.1.m1.1.1.1.cmml" xref="S1.SS1.p2.1.m1.1.1.1">~</ci><ci id="S1.SS1.p2.1.m1.1.1.2.cmml" xref="S1.SS1.p2.1.m1.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p2.1.m1.1c">\tilde{\bm{A}}</annotation></semantics></math>, whose rows are drawn from the same distribution as the rows of <math id="S1.SS1.p2.2.m2.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S1.SS1.p2.2.m2.1a"><mi id="S1.SS1.p2.2.m2.1.1" xref="S1.SS1.p2.2.m2.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p2.2.m2.1b"><ci id="S1.SS1.p2.2.m2.1.1.cmml" xref="S1.SS1.p2.2.m2.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p2.2.m2.1c">\bm{A}</annotation></semantics></math>. This is a natural assumption in the case that rows of <math id="S1.SS1.p2.3.m3.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S1.SS1.p2.3.m3.1a"><mi id="S1.SS1.p2.3.m3.1.1" xref="S1.SS1.p2.3.m3.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.p2.3.m3.1b"><ci id="S1.SS1.p2.3.m3.1.1.cmml" xref="S1.SS1.p2.3.m3.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.p2.3.m3.1c">\bm{A}</annotation></semantics></math> represent examples in training data, or structured subsets thereof (such as patches of images).</p>
</div>
</section>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">Because our work draws on ideas from randomized algorithms, approximate matrix multiplication, vector quantization, and other fields, the body of work related to our own is vast. Here, we provide only a high-level overview, and refer the interested reader to <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib66" title="" class="ltx_ref">2016a</a>, <a href="#bib.bib64" title="" class="ltx_ref">2014a</a>; Desai et al., <a href="#bib.bib20" title="" class="ltx_ref">2016</a>)</cite> for more detailed surveys. We also defer discussion of related vector quantization methods to the following sections.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Linear Approximation</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.2" class="ltx_p">Most AMM methods work by projecting <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mi id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><ci id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">\bm{A}</annotation></semantics></math> and <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mi id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><ci id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">\bm{B}</annotation></semantics></math> into lower-dimensional spaces and then performing an exact matrix multiply.
One simple option for choosing the projection matrices is to use matrix sketching algorithms. The most prominent deterministic matrix sketching methods are the Frequent Directions algorithm <cite class="ltx_cite ltx_citemacro_citep">(Liberty, <a href="#bib.bib49" title="" class="ltx_ref">2012</a>; Ghashami et al., <a href="#bib.bib31" title="" class="ltx_ref">2016</a>)</cite> and its many variations <cite class="ltx_cite ltx_citemacro_citep">(Teng &amp; Chu, <a href="#bib.bib63" title="" class="ltx_ref">2019</a>; Francis &amp; Raimond, <a href="#bib.bib28" title="" class="ltx_ref">2018b</a>; Ye et al., <a href="#bib.bib69" title="" class="ltx_ref">2016</a>; Huang, <a href="#bib.bib38" title="" class="ltx_ref">2019</a>; Luo et al., <a href="#bib.bib51" title="" class="ltx_ref">2019</a>; Francis &amp; Raimond, <a href="#bib.bib27" title="" class="ltx_ref">2018a</a>)</cite>. There are also many randomized sketching methods <cite class="ltx_cite ltx_citemacro_citep">(Sarlos, <a href="#bib.bib61" title="" class="ltx_ref">2006</a>; Kyrillidis et al., <a href="#bib.bib48" title="" class="ltx_ref">2014</a>; Pagh, <a href="#bib.bib58" title="" class="ltx_ref">2013</a>; Dasgupta et al., <a href="#bib.bib17" title="" class="ltx_ref">2010</a>; Nelson &amp; Nguyên, <a href="#bib.bib57" title="" class="ltx_ref">2013</a>)</cite> and sampling methods <cite class="ltx_cite ltx_citemacro_citep">(Drineas et al., <a href="#bib.bib22" title="" class="ltx_ref">2006b</a>, <a href="#bib.bib23" title="" class="ltx_ref">c</a>)</cite>.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.2" class="ltx_p">A weakness of matrix sketching methods in the context of matrix multiplication is that they consider each matrix in isolation. To exploit information about both matrices simultaneously, <cite class="ltx_cite ltx_citemacro_citet">Drineas et al. (<a href="#bib.bib21" title="" class="ltx_ref">2006a</a>)</cite> sample columns of <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mi id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><ci id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">\bm{A}</annotation></semantics></math> and rows of <math id="S2.SS1.p2.2.m2.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S2.SS1.p2.2.m2.1a"><mi id="S2.SS1.p2.2.m2.1.1" xref="S2.SS1.p2.2.m2.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.2.m2.1b"><ci id="S2.SS1.p2.2.m2.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.2.m2.1c">\bm{B}</annotation></semantics></math> according to a sampling distribution dependent upon both matrices. Later work by <cite class="ltx_cite ltx_citemacro_citet">Manne &amp; Pal (<a href="#bib.bib53" title="" class="ltx_ref">2014</a>)</cite> reduces approximation of the matrices to an optimization problem, which is solved by steepest descent. <cite class="ltx_cite ltx_citemacro_citet">Mroueh et al. (<a href="#bib.bib56" title="" class="ltx_ref">2016</a>)</cite>, <cite class="ltx_cite ltx_citemacro_citet">Ye et al. (<a href="#bib.bib69" title="" class="ltx_ref">2016</a>)</cite>, and <cite class="ltx_cite ltx_citemacro_citet">Francis &amp; Raimond (<a href="#bib.bib27" title="" class="ltx_ref">2018a</a>)</cite> introduce variations of the Frequent Directions algorithm that take into account both matrices.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.1" class="ltx_p">All of the above methods differ from our own not merely in specifics, but also in problem formulation. These methods all assume that there is no training set <math id="S2.SS1.p3.1.m1.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S2.SS1.p3.1.m1.1a"><mover accent="true" id="S2.SS1.p3.1.m1.1.1" xref="S2.SS1.p3.1.m1.1.1.cmml"><mi id="S2.SS1.p3.1.m1.1.1.2" xref="S2.SS1.p3.1.m1.1.1.2.cmml">𝑨</mi><mo id="S2.SS1.p3.1.m1.1.1.1" xref="S2.SS1.p3.1.m1.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.1.m1.1b"><apply id="S2.SS1.p3.1.m1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1"><ci id="S2.SS1.p3.1.m1.1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1.1">~</ci><ci id="S2.SS1.p3.1.m1.1.1.2.cmml" xref="S2.SS1.p3.1.m1.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.1.m1.1c">\tilde{\bm{A}}</annotation></semantics></math> and nearly all focus on large matrices, where provably reduced asymptotic complexity for a given level of error is the goal.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Hashing to Avoid Linear Operations</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">In the neural network acceleration literature, there have been several efforts to accelerate dense linear layers using some form of hashing <cite class="ltx_cite ltx_citemacro_citep">(Spring &amp; Shrivastava, <a href="#bib.bib62" title="" class="ltx_ref">2017</a>; Chen et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>; Bakhtiary et al., <a href="#bib.bib9" title="" class="ltx_ref">2015</a>; Dean et al., <a href="#bib.bib19" title="" class="ltx_ref">2013</a>; Chen et al., <a href="#bib.bib15" title="" class="ltx_ref">2015</a>)</cite>. These methods differ from our own in the hash functions chosen, in not exploiting a training set, and in the overall goal of the algorithm. While we seek to approximate the entire output matrix, these methods seek to either sample outputs <cite class="ltx_cite ltx_citemacro_citep">(Spring &amp; Shrivastava, <a href="#bib.bib62" title="" class="ltx_ref">2017</a>; Chen et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite>, approximate only the largest outputs <cite class="ltx_cite ltx_citemacro_citep">(Bakhtiary et al., <a href="#bib.bib9" title="" class="ltx_ref">2015</a>; Dean et al., <a href="#bib.bib19" title="" class="ltx_ref">2013</a>)</cite>, or implement a fixed, sparse linear operator <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib15" title="" class="ltx_ref">2015</a>)</cite>.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Background - Product Quantization</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">To lay the groundwork for our own method, we begin by reviewing Product Quantization (PQ) <cite class="ltx_cite ltx_citemacro_citep">(Jegou et al., <a href="#bib.bib41" title="" class="ltx_ref">2011</a>)</cite>. PQ is a classic vector quantization algorithm for approximating inner products and Euclidean distances and serves as the basis for nearly all vector quantization methods similar to our own.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.10" class="ltx_p">The basic intuition behind PQ is that <math id="S3.p2.1.m1.1" class="ltx_Math" alttext="\bm{a}^{\top}\bm{b}\approx\hat{\bm{a}}^{\top}\bm{b}" display="inline"><semantics id="S3.p2.1.m1.1a"><mrow id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml"><mrow id="S3.p2.1.m1.1.1.2" xref="S3.p2.1.m1.1.1.2.cmml"><msup id="S3.p2.1.m1.1.1.2.2" xref="S3.p2.1.m1.1.1.2.2.cmml"><mi id="S3.p2.1.m1.1.1.2.2.2" xref="S3.p2.1.m1.1.1.2.2.2.cmml">𝒂</mi><mo id="S3.p2.1.m1.1.1.2.2.3" xref="S3.p2.1.m1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.2.1" xref="S3.p2.1.m1.1.1.2.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.2.3" xref="S3.p2.1.m1.1.1.2.3.cmml">𝒃</mi></mrow><mo id="S3.p2.1.m1.1.1.1" xref="S3.p2.1.m1.1.1.1.cmml">≈</mo><mrow id="S3.p2.1.m1.1.1.3" xref="S3.p2.1.m1.1.1.3.cmml"><msup id="S3.p2.1.m1.1.1.3.2" xref="S3.p2.1.m1.1.1.3.2.cmml"><mover accent="true" id="S3.p2.1.m1.1.1.3.2.2" xref="S3.p2.1.m1.1.1.3.2.2.cmml"><mi id="S3.p2.1.m1.1.1.3.2.2.2" xref="S3.p2.1.m1.1.1.3.2.2.2.cmml">𝒂</mi><mo id="S3.p2.1.m1.1.1.3.2.2.1" xref="S3.p2.1.m1.1.1.3.2.2.1.cmml">^</mo></mover><mo id="S3.p2.1.m1.1.1.3.2.3" xref="S3.p2.1.m1.1.1.3.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.3.1" xref="S3.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.3.3" xref="S3.p2.1.m1.1.1.3.3.cmml">𝒃</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.1b"><apply id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1"><approx id="S3.p2.1.m1.1.1.1.cmml" xref="S3.p2.1.m1.1.1.1"></approx><apply id="S3.p2.1.m1.1.1.2.cmml" xref="S3.p2.1.m1.1.1.2"><times id="S3.p2.1.m1.1.1.2.1.cmml" xref="S3.p2.1.m1.1.1.2.1"></times><apply id="S3.p2.1.m1.1.1.2.2.cmml" xref="S3.p2.1.m1.1.1.2.2"><csymbol cd="ambiguous" id="S3.p2.1.m1.1.1.2.2.1.cmml" xref="S3.p2.1.m1.1.1.2.2">superscript</csymbol><ci id="S3.p2.1.m1.1.1.2.2.2.cmml" xref="S3.p2.1.m1.1.1.2.2.2">𝒂</ci><csymbol cd="latexml" id="S3.p2.1.m1.1.1.2.2.3.cmml" xref="S3.p2.1.m1.1.1.2.2.3">top</csymbol></apply><ci id="S3.p2.1.m1.1.1.2.3.cmml" xref="S3.p2.1.m1.1.1.2.3">𝒃</ci></apply><apply id="S3.p2.1.m1.1.1.3.cmml" xref="S3.p2.1.m1.1.1.3"><times id="S3.p2.1.m1.1.1.3.1.cmml" xref="S3.p2.1.m1.1.1.3.1"></times><apply id="S3.p2.1.m1.1.1.3.2.cmml" xref="S3.p2.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.p2.1.m1.1.1.3.2.1.cmml" xref="S3.p2.1.m1.1.1.3.2">superscript</csymbol><apply id="S3.p2.1.m1.1.1.3.2.2.cmml" xref="S3.p2.1.m1.1.1.3.2.2"><ci id="S3.p2.1.m1.1.1.3.2.2.1.cmml" xref="S3.p2.1.m1.1.1.3.2.2.1">^</ci><ci id="S3.p2.1.m1.1.1.3.2.2.2.cmml" xref="S3.p2.1.m1.1.1.3.2.2.2">𝒂</ci></apply><csymbol cd="latexml" id="S3.p2.1.m1.1.1.3.2.3.cmml" xref="S3.p2.1.m1.1.1.3.2.3">top</csymbol></apply><ci id="S3.p2.1.m1.1.1.3.3.cmml" xref="S3.p2.1.m1.1.1.3.3">𝒃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.1c">\bm{a}^{\top}\bm{b}\approx\hat{\bm{a}}^{\top}\bm{b}</annotation></semantics></math>, where <math id="S3.p2.2.m2.1" class="ltx_Math" alttext="\lVert\hat{\bm{a}}-\bm{a}\rVert" display="inline"><semantics id="S3.p2.2.m2.1a"><mrow id="S3.p2.2.m2.1.1.1" xref="S3.p2.2.m2.1.1.2.cmml"><mo fence="true" rspace="0em" id="S3.p2.2.m2.1.1.1.2" xref="S3.p2.2.m2.1.1.2.1.cmml">∥</mo><mrow id="S3.p2.2.m2.1.1.1.1" xref="S3.p2.2.m2.1.1.1.1.cmml"><mover accent="true" id="S3.p2.2.m2.1.1.1.1.2" xref="S3.p2.2.m2.1.1.1.1.2.cmml"><mi id="S3.p2.2.m2.1.1.1.1.2.2" xref="S3.p2.2.m2.1.1.1.1.2.2.cmml">𝒂</mi><mo id="S3.p2.2.m2.1.1.1.1.2.1" xref="S3.p2.2.m2.1.1.1.1.2.1.cmml">^</mo></mover><mo id="S3.p2.2.m2.1.1.1.1.1" xref="S3.p2.2.m2.1.1.1.1.1.cmml">−</mo><mi id="S3.p2.2.m2.1.1.1.1.3" xref="S3.p2.2.m2.1.1.1.1.3.cmml">𝒂</mi></mrow><mo fence="true" lspace="0em" id="S3.p2.2.m2.1.1.1.3" xref="S3.p2.2.m2.1.1.2.1.cmml">∥</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.1b"><apply id="S3.p2.2.m2.1.1.2.cmml" xref="S3.p2.2.m2.1.1.1"><csymbol cd="latexml" id="S3.p2.2.m2.1.1.2.1.cmml" xref="S3.p2.2.m2.1.1.1.2">delimited-∥∥</csymbol><apply id="S3.p2.2.m2.1.1.1.1.cmml" xref="S3.p2.2.m2.1.1.1.1"><minus id="S3.p2.2.m2.1.1.1.1.1.cmml" xref="S3.p2.2.m2.1.1.1.1.1"></minus><apply id="S3.p2.2.m2.1.1.1.1.2.cmml" xref="S3.p2.2.m2.1.1.1.1.2"><ci id="S3.p2.2.m2.1.1.1.1.2.1.cmml" xref="S3.p2.2.m2.1.1.1.1.2.1">^</ci><ci id="S3.p2.2.m2.1.1.1.1.2.2.cmml" xref="S3.p2.2.m2.1.1.1.1.2.2">𝒂</ci></apply><ci id="S3.p2.2.m2.1.1.1.1.3.cmml" xref="S3.p2.2.m2.1.1.1.1.3">𝒂</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.1c">\lVert\hat{\bm{a}}-\bm{a}\rVert</annotation></semantics></math> is small but <math id="S3.p2.3.m3.1" class="ltx_Math" alttext="\hat{\bm{a}}" display="inline"><semantics id="S3.p2.3.m3.1a"><mover accent="true" id="S3.p2.3.m3.1.1" xref="S3.p2.3.m3.1.1.cmml"><mi id="S3.p2.3.m3.1.1.2" xref="S3.p2.3.m3.1.1.2.cmml">𝒂</mi><mo id="S3.p2.3.m3.1.1.1" xref="S3.p2.3.m3.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.p2.3.m3.1b"><apply id="S3.p2.3.m3.1.1.cmml" xref="S3.p2.3.m3.1.1"><ci id="S3.p2.3.m3.1.1.1.cmml" xref="S3.p2.3.m3.1.1.1">^</ci><ci id="S3.p2.3.m3.1.1.2.cmml" xref="S3.p2.3.m3.1.1.2">𝒂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.3.m3.1c">\hat{\bm{a}}</annotation></semantics></math> has special structure allowing the product to be computed quickly. This structure consists of <math id="S3.p2.4.m4.1" class="ltx_Math" alttext="\hat{\bm{a}}" display="inline"><semantics id="S3.p2.4.m4.1a"><mover accent="true" id="S3.p2.4.m4.1.1" xref="S3.p2.4.m4.1.1.cmml"><mi id="S3.p2.4.m4.1.1.2" xref="S3.p2.4.m4.1.1.2.cmml">𝒂</mi><mo id="S3.p2.4.m4.1.1.1" xref="S3.p2.4.m4.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.p2.4.m4.1b"><apply id="S3.p2.4.m4.1.1.cmml" xref="S3.p2.4.m4.1.1"><ci id="S3.p2.4.m4.1.1.1.cmml" xref="S3.p2.4.m4.1.1.1">^</ci><ci id="S3.p2.4.m4.1.1.2.cmml" xref="S3.p2.4.m4.1.1.2">𝒂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.4.m4.1c">\hat{\bm{a}}</annotation></semantics></math> being formed by concatenating learned prototypes in disjoint subspaces; one obtains a speedup by precomputing the dot products between <math id="S3.p2.5.m5.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="S3.p2.5.m5.1a"><mi id="S3.p2.5.m5.1.1" xref="S3.p2.5.m5.1.1.cmml">𝒃</mi><annotation-xml encoding="MathML-Content" id="S3.p2.5.m5.1b"><ci id="S3.p2.5.m5.1.1.cmml" xref="S3.p2.5.m5.1.1">𝒃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.5.m5.1c">\bm{b}</annotation></semantics></math> and the prototypes once, and then reusing these values across many <math id="S3.p2.6.m6.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.p2.6.m6.1a"><mi id="S3.p2.6.m6.1.1" xref="S3.p2.6.m6.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.p2.6.m6.1b"><ci id="S3.p2.6.m6.1.1.cmml" xref="S3.p2.6.m6.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.6.m6.1c">\bm{a}</annotation></semantics></math> vectors. The <math id="S3.p2.7.m7.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.p2.7.m7.1a"><mi id="S3.p2.7.m7.1.1" xref="S3.p2.7.m7.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.p2.7.m7.1b"><ci id="S3.p2.7.m7.1.1.cmml" xref="S3.p2.7.m7.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.7.m7.1c">\bm{a}</annotation></semantics></math> vectors here are the (transposed) rows of <math id="S3.p2.8.m8.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S3.p2.8.m8.1a"><mi id="S3.p2.8.m8.1.1" xref="S3.p2.8.m8.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S3.p2.8.m8.1b"><ci id="S3.p2.8.m8.1.1.cmml" xref="S3.p2.8.m8.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.8.m8.1c">\bm{A}</annotation></semantics></math> and the <math id="S3.p2.9.m9.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="S3.p2.9.m9.1a"><mi id="S3.p2.9.m9.1.1" xref="S3.p2.9.m9.1.1.cmml">𝒃</mi><annotation-xml encoding="MathML-Content" id="S3.p2.9.m9.1b"><ci id="S3.p2.9.m9.1.1.cmml" xref="S3.p2.9.m9.1.1">𝒃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.9.m9.1c">\bm{b}</annotation></semantics></math> vectors are the columns of <math id="S3.p2.10.m10.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S3.p2.10.m10.1a"><mi id="S3.p2.10.m10.1.1" xref="S3.p2.10.m10.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S3.p2.10.m10.1b"><ci id="S3.p2.10.m10.1.1.cmml" xref="S3.p2.10.m10.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.10.m10.1c">\bm{B}</annotation></semantics></math>.</p>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.3" class="ltx_p">In somewhat more detail, PQ consists of the following:</p>
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.5" class="ltx_p"><span id="S3.I1.i1.p1.5.1" class="ltx_text ltx_font_bold">Prototype Learning</span> - In an initial, offline training phase, cluster the rows of <math id="S3.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S3.I1.i1.p1.1.m1.1a"><mi id="S3.I1.i1.p1.1.m1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.1.m1.1b"><ci id="S3.I1.i1.p1.1.m1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.1.m1.1c">\bm{A}</annotation></semantics></math> (or a training set <math id="S3.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S3.I1.i1.p1.2.m2.1a"><mover accent="true" id="S3.I1.i1.p1.2.m2.1.1" xref="S3.I1.i1.p1.2.m2.1.1.cmml"><mi id="S3.I1.i1.p1.2.m2.1.1.2" xref="S3.I1.i1.p1.2.m2.1.1.2.cmml">𝑨</mi><mo id="S3.I1.i1.p1.2.m2.1.1.1" xref="S3.I1.i1.p1.2.m2.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.2.m2.1b"><apply id="S3.I1.i1.p1.2.m2.1.1.cmml" xref="S3.I1.i1.p1.2.m2.1.1"><ci id="S3.I1.i1.p1.2.m2.1.1.1.cmml" xref="S3.I1.i1.p1.2.m2.1.1.1">~</ci><ci id="S3.I1.i1.p1.2.m2.1.1.2.cmml" xref="S3.I1.i1.p1.2.m2.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.2.m2.1c">\tilde{\bm{A}}</annotation></semantics></math>) using K-means to create prototypes. A separate K-means is run in each of <math id="S3.I1.i1.p1.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.I1.i1.p1.3.m3.1a"><mi id="S3.I1.i1.p1.3.m3.1.1" xref="S3.I1.i1.p1.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.3.m3.1b"><ci id="S3.I1.i1.p1.3.m3.1.1.cmml" xref="S3.I1.i1.p1.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.3.m3.1c">C</annotation></semantics></math> disjoint subspaces to produce <math id="S3.I1.i1.p1.4.m4.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.I1.i1.p1.4.m4.1a"><mi id="S3.I1.i1.p1.4.m4.1.1" xref="S3.I1.i1.p1.4.m4.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.4.m4.1b"><ci id="S3.I1.i1.p1.4.m4.1.1.cmml" xref="S3.I1.i1.p1.4.m4.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.4.m4.1c">C</annotation></semantics></math> sets of <math id="S3.I1.i1.p1.5.m5.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.I1.i1.p1.5.m5.1a"><mi id="S3.I1.i1.p1.5.m5.1.1" xref="S3.I1.i1.p1.5.m5.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.5.m5.1b"><ci id="S3.I1.i1.p1.5.m5.1.1.cmml" xref="S3.I1.i1.p1.5.m5.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.5.m5.1c">K</annotation></semantics></math> prototypes. </p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;padding-top:4.3pt;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.3" class="ltx_p"><span id="S3.I1.i2.p1.3.1" class="ltx_text ltx_font_bold">Encoding Function</span>, <math id="S3.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="g(\bm{a})" display="inline"><semantics id="S3.I1.i2.p1.1.m1.1a"><mrow id="S3.I1.i2.p1.1.m1.1.2" xref="S3.I1.i2.p1.1.m1.1.2.cmml"><mi id="S3.I1.i2.p1.1.m1.1.2.2" xref="S3.I1.i2.p1.1.m1.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.I1.i2.p1.1.m1.1.2.1" xref="S3.I1.i2.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S3.I1.i2.p1.1.m1.1.2.3.2" xref="S3.I1.i2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.I1.i2.p1.1.m1.1.2.3.2.1" xref="S3.I1.i2.p1.1.m1.1.2.cmml">(</mo><mi id="S3.I1.i2.p1.1.m1.1.1" xref="S3.I1.i2.p1.1.m1.1.1.cmml">𝒂</mi><mo stretchy="false" id="S3.I1.i2.p1.1.m1.1.2.3.2.2" xref="S3.I1.i2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.1.m1.1b"><apply id="S3.I1.i2.p1.1.m1.1.2.cmml" xref="S3.I1.i2.p1.1.m1.1.2"><times id="S3.I1.i2.p1.1.m1.1.2.1.cmml" xref="S3.I1.i2.p1.1.m1.1.2.1"></times><ci id="S3.I1.i2.p1.1.m1.1.2.2.cmml" xref="S3.I1.i2.p1.1.m1.1.2.2">𝑔</ci><ci id="S3.I1.i2.p1.1.m1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1">𝒂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.1.m1.1c">g(\bm{a})</annotation></semantics></math> - Determine the most similar prototype to <math id="S3.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.I1.i2.p1.2.m2.1a"><mi id="S3.I1.i2.p1.2.m2.1.1" xref="S3.I1.i2.p1.2.m2.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.2.m2.1b"><ci id="S3.I1.i2.p1.2.m2.1.1.cmml" xref="S3.I1.i2.p1.2.m2.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.2.m2.1c">\bm{a}</annotation></semantics></math> in each subspace. Store these assignments as integer indices using <math id="S3.I1.i2.p1.3.m3.2" class="ltx_Math" alttext="C\log_{2}(K)" display="inline"><semantics id="S3.I1.i2.p1.3.m3.2a"><mrow id="S3.I1.i2.p1.3.m3.2.2" xref="S3.I1.i2.p1.3.m3.2.2.cmml"><mi id="S3.I1.i2.p1.3.m3.2.2.3" xref="S3.I1.i2.p1.3.m3.2.2.3.cmml">C</mi><mo lspace="0.167em" rspace="0em" id="S3.I1.i2.p1.3.m3.2.2.2" xref="S3.I1.i2.p1.3.m3.2.2.2.cmml">​</mo><mrow id="S3.I1.i2.p1.3.m3.2.2.1.1" xref="S3.I1.i2.p1.3.m3.2.2.1.2.cmml"><msub id="S3.I1.i2.p1.3.m3.2.2.1.1.1" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1.cmml"><mi id="S3.I1.i2.p1.3.m3.2.2.1.1.1.2" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1.2.cmml">log</mi><mn id="S3.I1.i2.p1.3.m3.2.2.1.1.1.3" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1.3.cmml">2</mn></msub><mo id="S3.I1.i2.p1.3.m3.2.2.1.1a" xref="S3.I1.i2.p1.3.m3.2.2.1.2.cmml">⁡</mo><mrow id="S3.I1.i2.p1.3.m3.2.2.1.1.2" xref="S3.I1.i2.p1.3.m3.2.2.1.2.cmml"><mo stretchy="false" id="S3.I1.i2.p1.3.m3.2.2.1.1.2.1" xref="S3.I1.i2.p1.3.m3.2.2.1.2.cmml">(</mo><mi id="S3.I1.i2.p1.3.m3.1.1" xref="S3.I1.i2.p1.3.m3.1.1.cmml">K</mi><mo stretchy="false" id="S3.I1.i2.p1.3.m3.2.2.1.1.2.2" xref="S3.I1.i2.p1.3.m3.2.2.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.3.m3.2b"><apply id="S3.I1.i2.p1.3.m3.2.2.cmml" xref="S3.I1.i2.p1.3.m3.2.2"><times id="S3.I1.i2.p1.3.m3.2.2.2.cmml" xref="S3.I1.i2.p1.3.m3.2.2.2"></times><ci id="S3.I1.i2.p1.3.m3.2.2.3.cmml" xref="S3.I1.i2.p1.3.m3.2.2.3">𝐶</ci><apply id="S3.I1.i2.p1.3.m3.2.2.1.2.cmml" xref="S3.I1.i2.p1.3.m3.2.2.1.1"><apply id="S3.I1.i2.p1.3.m3.2.2.1.1.1.cmml" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.I1.i2.p1.3.m3.2.2.1.1.1.1.cmml" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1">subscript</csymbol><log id="S3.I1.i2.p1.3.m3.2.2.1.1.1.2.cmml" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1.2"></log><cn type="integer" id="S3.I1.i2.p1.3.m3.2.2.1.1.1.3.cmml" xref="S3.I1.i2.p1.3.m3.2.2.1.1.1.3">2</cn></apply><ci id="S3.I1.i2.p1.3.m3.1.1.cmml" xref="S3.I1.i2.p1.3.m3.1.1">𝐾</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.3.m3.2c">C\log_{2}(K)</annotation></semantics></math> bits.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;padding-top:4.3pt;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.4" class="ltx_p"><span id="S3.I1.i3.p1.4.1" class="ltx_text ltx_font_bold">Table Construction</span>, <math id="S3.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="h(\bm{B})" display="inline"><semantics id="S3.I1.i3.p1.1.m1.1a"><mrow id="S3.I1.i3.p1.1.m1.1.2" xref="S3.I1.i3.p1.1.m1.1.2.cmml"><mi id="S3.I1.i3.p1.1.m1.1.2.2" xref="S3.I1.i3.p1.1.m1.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.I1.i3.p1.1.m1.1.2.1" xref="S3.I1.i3.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S3.I1.i3.p1.1.m1.1.2.3.2" xref="S3.I1.i3.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.I1.i3.p1.1.m1.1.2.3.2.1" xref="S3.I1.i3.p1.1.m1.1.2.cmml">(</mo><mi id="S3.I1.i3.p1.1.m1.1.1" xref="S3.I1.i3.p1.1.m1.1.1.cmml">𝑩</mi><mo stretchy="false" id="S3.I1.i3.p1.1.m1.1.2.3.2.2" xref="S3.I1.i3.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.1.m1.1b"><apply id="S3.I1.i3.p1.1.m1.1.2.cmml" xref="S3.I1.i3.p1.1.m1.1.2"><times id="S3.I1.i3.p1.1.m1.1.2.1.cmml" xref="S3.I1.i3.p1.1.m1.1.2.1"></times><ci id="S3.I1.i3.p1.1.m1.1.2.2.cmml" xref="S3.I1.i3.p1.1.m1.1.2.2">ℎ</ci><ci id="S3.I1.i3.p1.1.m1.1.1.cmml" xref="S3.I1.i3.p1.1.m1.1.1">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.1.m1.1c">h(\bm{B})</annotation></semantics></math> - Precompute the dot products between <math id="S3.I1.i3.p1.2.m2.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="S3.I1.i3.p1.2.m2.1a"><mi id="S3.I1.i3.p1.2.m2.1.1" xref="S3.I1.i3.p1.2.m2.1.1.cmml">𝒃</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.2.m2.1b"><ci id="S3.I1.i3.p1.2.m2.1.1.cmml" xref="S3.I1.i3.p1.2.m2.1.1">𝒃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.2.m2.1c">\bm{b}</annotation></semantics></math> and each prototype in each subspace. Store these partial dot products in <math id="S3.I1.i3.p1.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.I1.i3.p1.3.m3.1a"><mi id="S3.I1.i3.p1.3.m3.1.1" xref="S3.I1.i3.p1.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.3.m3.1b"><ci id="S3.I1.i3.p1.3.m3.1.1.cmml" xref="S3.I1.i3.p1.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.3.m3.1c">C</annotation></semantics></math> lookup tables of size <math id="S3.I1.i3.p1.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.I1.i3.p1.4.m4.1a"><mi id="S3.I1.i3.p1.4.m4.1.1" xref="S3.I1.i3.p1.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.4.m4.1b"><ci id="S3.I1.i3.p1.4.m4.1.1.cmml" xref="S3.I1.i3.p1.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.4.m4.1c">K</annotation></semantics></math>.</p>
</div>
</li>
<li id="S3.I1.i4" class="ltx_item" style="list-style-type:none;padding-top:4.3pt;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S3.I1.i4.p1" class="ltx_para">
<p id="S3.I1.i4.p1.3" class="ltx_p"><span id="S3.I1.i4.p1.3.1" class="ltx_text ltx_font_bold">Aggregation</span>, <math id="S3.I1.i4.p1.1.m1.2" class="ltx_Math" alttext="f(\cdot,\cdot)" display="inline"><semantics id="S3.I1.i4.p1.1.m1.2a"><mrow id="S3.I1.i4.p1.1.m1.2.3" xref="S3.I1.i4.p1.1.m1.2.3.cmml"><mi id="S3.I1.i4.p1.1.m1.2.3.2" xref="S3.I1.i4.p1.1.m1.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.I1.i4.p1.1.m1.2.3.1" xref="S3.I1.i4.p1.1.m1.2.3.1.cmml">​</mo><mrow id="S3.I1.i4.p1.1.m1.2.3.3.2" xref="S3.I1.i4.p1.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="S3.I1.i4.p1.1.m1.2.3.3.2.1" xref="S3.I1.i4.p1.1.m1.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.I1.i4.p1.1.m1.1.1" xref="S3.I1.i4.p1.1.m1.1.1.cmml">⋅</mo><mo rspace="0em" id="S3.I1.i4.p1.1.m1.2.3.3.2.2" xref="S3.I1.i4.p1.1.m1.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S3.I1.i4.p1.1.m1.2.2" xref="S3.I1.i4.p1.1.m1.2.2.cmml">⋅</mo><mo stretchy="false" id="S3.I1.i4.p1.1.m1.2.3.3.2.3" xref="S3.I1.i4.p1.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.1.m1.2b"><apply id="S3.I1.i4.p1.1.m1.2.3.cmml" xref="S3.I1.i4.p1.1.m1.2.3"><times id="S3.I1.i4.p1.1.m1.2.3.1.cmml" xref="S3.I1.i4.p1.1.m1.2.3.1"></times><ci id="S3.I1.i4.p1.1.m1.2.3.2.cmml" xref="S3.I1.i4.p1.1.m1.2.3.2">𝑓</ci><interval closure="open" id="S3.I1.i4.p1.1.m1.2.3.3.1.cmml" xref="S3.I1.i4.p1.1.m1.2.3.3.2"><ci id="S3.I1.i4.p1.1.m1.1.1.cmml" xref="S3.I1.i4.p1.1.m1.1.1">⋅</ci><ci id="S3.I1.i4.p1.1.m1.2.2.cmml" xref="S3.I1.i4.p1.1.m1.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.1.m1.2c">f(\cdot,\cdot)</annotation></semantics></math> - Use the indices and tables to <span id="S3.I1.i4.p1.3.2" class="ltx_text ltx_font_italic">lookup</span> the estimated partial <math id="S3.I1.i4.p1.2.m2.1" class="ltx_Math" alttext="\bm{a}^{\top}\bm{b}" display="inline"><semantics id="S3.I1.i4.p1.2.m2.1a"><mrow id="S3.I1.i4.p1.2.m2.1.1" xref="S3.I1.i4.p1.2.m2.1.1.cmml"><msup id="S3.I1.i4.p1.2.m2.1.1.2" xref="S3.I1.i4.p1.2.m2.1.1.2.cmml"><mi id="S3.I1.i4.p1.2.m2.1.1.2.2" xref="S3.I1.i4.p1.2.m2.1.1.2.2.cmml">𝒂</mi><mo id="S3.I1.i4.p1.2.m2.1.1.2.3" xref="S3.I1.i4.p1.2.m2.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.I1.i4.p1.2.m2.1.1.1" xref="S3.I1.i4.p1.2.m2.1.1.1.cmml">​</mo><mi id="S3.I1.i4.p1.2.m2.1.1.3" xref="S3.I1.i4.p1.2.m2.1.1.3.cmml">𝒃</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.2.m2.1b"><apply id="S3.I1.i4.p1.2.m2.1.1.cmml" xref="S3.I1.i4.p1.2.m2.1.1"><times id="S3.I1.i4.p1.2.m2.1.1.1.cmml" xref="S3.I1.i4.p1.2.m2.1.1.1"></times><apply id="S3.I1.i4.p1.2.m2.1.1.2.cmml" xref="S3.I1.i4.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.I1.i4.p1.2.m2.1.1.2.1.cmml" xref="S3.I1.i4.p1.2.m2.1.1.2">superscript</csymbol><ci id="S3.I1.i4.p1.2.m2.1.1.2.2.cmml" xref="S3.I1.i4.p1.2.m2.1.1.2.2">𝒂</ci><csymbol cd="latexml" id="S3.I1.i4.p1.2.m2.1.1.2.3.cmml" xref="S3.I1.i4.p1.2.m2.1.1.2.3">top</csymbol></apply><ci id="S3.I1.i4.p1.2.m2.1.1.3.cmml" xref="S3.I1.i4.p1.2.m2.1.1.3">𝒃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.2.m2.1c">\bm{a}^{\top}\bm{b}</annotation></semantics></math> in each subspace, then sum the results across all <math id="S3.I1.i4.p1.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.I1.i4.p1.3.m3.1a"><mi id="S3.I1.i4.p1.3.m3.1.1" xref="S3.I1.i4.p1.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.3.m3.1b"><ci id="S3.I1.i4.p1.3.m3.1.1.cmml" xref="S3.I1.i4.p1.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.3.m3.1c">C</annotation></semantics></math> subspaces.</p>
</div>
</li>
</ol>
<p id="S3.p3.2" class="ltx_p">PQ is depicted for a single pair of vectors <math id="S3.p3.1.m1.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.p3.1.m1.1a"><mi id="S3.p3.1.m1.1.1" xref="S3.p3.1.m1.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.p3.1.m1.1b"><ci id="S3.p3.1.m1.1.1.cmml" xref="S3.p3.1.m1.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p3.1.m1.1c">\bm{a}</annotation></semantics></math> and <math id="S3.p3.2.m2.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="S3.p3.2.m2.1a"><mi id="S3.p3.2.m2.1.1" xref="S3.p3.2.m2.1.1.cmml">𝒃</mi><annotation-xml encoding="MathML-Content" id="S3.p3.2.m2.1b"><ci id="S3.p3.2.m2.1.1.cmml" xref="S3.p3.2.m2.1.1">𝒃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p3.2.m2.1c">\bm{b}</annotation></semantics></math> in Figure <a href="#S3.F2" title="Figure 2 ‣ Prototype Learning: ‣ 3 Background - Product Quantization ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. We elaborate upon each of these steps below.</p>
</div>
<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Prototype Learning:</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px1.p1.7" class="ltx_p">Let <math id="S3.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\tilde{A}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S3.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><mover accent="true" id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">A</mi><mo id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.1.cmml">~</mo></mover><mo id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1"><in id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.1"></in><apply id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2"><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.1">~</ci><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.2.2">𝐴</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.2">ℝ</ci><apply id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3"><times id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.1"></times><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2">𝑁</ci><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.1.m1.1c">\tilde{A}\in\mathbb{R}^{N\times D}</annotation></semantics></math> be a training set, <math id="S3.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.2.m2.1a"><mi id="S3.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.2.m2.1c">K</annotation></semantics></math> be a number of prototypes per subspace, <math id="S3.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.3.m3.1a"><mi id="S3.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.3.m3.1b"><ci id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.3.m3.1c">C</annotation></semantics></math> be a number of subspaces, and <math id="S3.SS0.SSS0.Px1.p1.4.m4.2" class="ltx_Math" alttext="\{\mathcal{J}^{(c)}\}_{c=1}^{C}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.4.m4.2a"><msubsup id="S3.SS0.SSS0.Px1.p1.4.m4.2.2" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.cmml"><mrow id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2.cmml">{</mo><msup id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.2.cmml">𝒥</mi><mrow id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.cmml">)</mo></mrow></msup><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.2" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.2.cmml">c</mi><mo id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.1.cmml">=</mo><mn id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.3" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.3.cmml">1</mn></mrow><mi id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.3" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.3.cmml">C</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.4.m4.2b"><apply id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2">superscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2">subscript</csymbol><set id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1"><apply id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.1.2">𝒥</ci><ci id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.1">𝑐</ci></apply></set><apply id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3"><eq id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.1"></eq><ci id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.2">𝑐</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.1.3.3">1</cn></apply></apply><ci id="S3.SS0.SSS0.Px1.p1.4.m4.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.2.2.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.4.m4.2c">\{\mathcal{J}^{(c)}\}_{c=1}^{C}</annotation></semantics></math> be the mutually exclusive and collectively exhaustive sets of indices associated with each subspace. The training-time task of PQ is to learn <math id="S3.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.5.m5.1a"><mi id="S3.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.5.m5.1b"><ci id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.5.m5.1c">C</annotation></semantics></math> sets of prototypes <math id="S3.SS0.SSS0.Px1.p1.6.m6.3" class="ltx_Math" alttext="\bm{P}^{(c)}\in\mathbb{R}^{K\times|\mathcal{J}^{(c)}|}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.6.m6.3a"><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.3.4" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.cmml"><msup id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.2.cmml">𝑷</mi><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.cmml">)</mo></mrow></msup><mo id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.2.cmml">ℝ</mi><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.4" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.4.cmml">K</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.3.cmml">×</mo><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.2.1.cmml">|</mo><msup id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.2.cmml">𝒥</mi><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.cmml">)</mo></mrow></msup><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.2.1.cmml">|</mo></mrow></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.6.m6.3b"><apply id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4"><in id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.1"></in><apply id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.2.2">𝑷</ci><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.1">𝑐</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.4.3.2">ℝ</ci><apply id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2"><times id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.3"></times><ci id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.4.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.4">𝐾</ci><apply id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1"><abs id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.2"></abs><apply id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.3.3.2.2.1.1.2">𝒥</ci><ci id="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.2.2.1.1.1.1">𝑐</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.6.m6.3c">\bm{P}^{(c)}\in\mathbb{R}^{K\times|\mathcal{J}^{(c)}|}</annotation></semantics></math> and assignments <math id="S3.SS0.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="\bm{z}^{(c)}\in\mathbb{R}^{N}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.7.m7.1a"><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.cmml"><msup id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.2.cmml">𝒛</mi><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.cmml">)</mo></mrow></msup><mo id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.2.cmml">ℝ</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.3.cmml">N</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.7.m7.1b"><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2"><in id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.1"></in><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.2.2">𝒛</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1">𝑐</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.2">ℝ</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.2.3.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.7.m7.1c">\bm{z}^{(c)}\in\mathbb{R}^{N}</annotation></semantics></math> such that:</p>
<table id="A6.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.6" class="ltx_Math" alttext="\displaystyle\sum_{i=1}^{N}\sum_{c=1}^{C}\sum_{j\in\mathcal{J}^{(c)}}\left(\tilde{\bm{A}}_{ij}-\bm{P}^{(c)}_{z^{(c)},j}\right)^{2}\vspace*{-.5mm}" display="inline"><semantics id="S3.E3.m1.6a"><mrow id="S3.E3.m1.6.6" xref="S3.E3.m1.6.6.cmml"><mstyle displaystyle="true" id="S3.E3.m1.6.6.2" xref="S3.E3.m1.6.6.2.cmml"><munderover id="S3.E3.m1.6.6.2a" xref="S3.E3.m1.6.6.2.cmml"><mo movablelimits="false" id="S3.E3.m1.6.6.2.2.2" xref="S3.E3.m1.6.6.2.2.2.cmml">∑</mo><mrow id="S3.E3.m1.6.6.2.2.3" xref="S3.E3.m1.6.6.2.2.3.cmml"><mi id="S3.E3.m1.6.6.2.2.3.2" xref="S3.E3.m1.6.6.2.2.3.2.cmml">i</mi><mo id="S3.E3.m1.6.6.2.2.3.1" xref="S3.E3.m1.6.6.2.2.3.1.cmml">=</mo><mn id="S3.E3.m1.6.6.2.2.3.3" xref="S3.E3.m1.6.6.2.2.3.3.cmml">1</mn></mrow><mi id="S3.E3.m1.6.6.2.3" xref="S3.E3.m1.6.6.2.3.cmml">N</mi></munderover></mstyle><mrow id="S3.E3.m1.6.6.1" xref="S3.E3.m1.6.6.1.cmml"><mstyle displaystyle="true" id="S3.E3.m1.6.6.1.2" xref="S3.E3.m1.6.6.1.2.cmml"><munderover id="S3.E3.m1.6.6.1.2a" xref="S3.E3.m1.6.6.1.2.cmml"><mo movablelimits="false" id="S3.E3.m1.6.6.1.2.2.2" xref="S3.E3.m1.6.6.1.2.2.2.cmml">∑</mo><mrow id="S3.E3.m1.6.6.1.2.2.3" xref="S3.E3.m1.6.6.1.2.2.3.cmml"><mi id="S3.E3.m1.6.6.1.2.2.3.2" xref="S3.E3.m1.6.6.1.2.2.3.2.cmml">c</mi><mo id="S3.E3.m1.6.6.1.2.2.3.1" xref="S3.E3.m1.6.6.1.2.2.3.1.cmml">=</mo><mn id="S3.E3.m1.6.6.1.2.2.3.3" xref="S3.E3.m1.6.6.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.E3.m1.6.6.1.2.3" xref="S3.E3.m1.6.6.1.2.3.cmml">C</mi></munderover></mstyle><mrow id="S3.E3.m1.6.6.1.1" xref="S3.E3.m1.6.6.1.1.cmml"><mstyle displaystyle="true" id="S3.E3.m1.6.6.1.1.2" xref="S3.E3.m1.6.6.1.1.2.cmml"><munder id="S3.E3.m1.6.6.1.1.2a" xref="S3.E3.m1.6.6.1.1.2.cmml"><mo movablelimits="false" id="S3.E3.m1.6.6.1.1.2.2" xref="S3.E3.m1.6.6.1.1.2.2.cmml">∑</mo><mrow id="S3.E3.m1.1.1.1" xref="S3.E3.m1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.3" xref="S3.E3.m1.1.1.1.3.cmml">j</mi><mo id="S3.E3.m1.1.1.1.2" xref="S3.E3.m1.1.1.1.2.cmml">∈</mo><msup id="S3.E3.m1.1.1.1.4" xref="S3.E3.m1.1.1.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.1.1.1.4.2" xref="S3.E3.m1.1.1.1.4.2.cmml">𝒥</mi><mrow id="S3.E3.m1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.4.cmml"><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.3.1" xref="S3.E3.m1.1.1.1.4.cmml">(</mo><mi id="S3.E3.m1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.3.2" xref="S3.E3.m1.1.1.1.4.cmml">)</mo></mrow></msup></mrow></munder></mstyle><msup id="S3.E3.m1.6.6.1.1.1" xref="S3.E3.m1.6.6.1.1.1.cmml"><mrow id="S3.E3.m1.6.6.1.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.cmml"><mo id="S3.E3.m1.6.6.1.1.1.1.1.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.6.6.1.1.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.cmml"><msub id="S3.E3.m1.6.6.1.1.1.1.1.1.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.cmml"><mover accent="true" id="S3.E3.m1.6.6.1.1.1.1.1.1.2.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.cmml"><mi id="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.2.cmml">𝑨</mi><mo id="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.1.cmml">~</mo></mover><mrow id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.cmml"><mi id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.3" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.3.cmml">j</mi></mrow></msub><mo id="S3.E3.m1.6.6.1.1.1.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S3.E3.m1.6.6.1.1.1.1.1.1.3" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.2.cmml">𝑷</mi><mrow id="S3.E3.m1.5.5.3.3" xref="S3.E3.m1.5.5.3.4.cmml"><msup id="S3.E3.m1.5.5.3.3.1" xref="S3.E3.m1.5.5.3.3.1.cmml"><mi id="S3.E3.m1.5.5.3.3.1.2" xref="S3.E3.m1.5.5.3.3.1.2.cmml">z</mi><mrow id="S3.E3.m1.3.3.1.1.1.3" xref="S3.E3.m1.5.5.3.3.1.cmml"><mo stretchy="false" id="S3.E3.m1.3.3.1.1.1.3.1" xref="S3.E3.m1.5.5.3.3.1.cmml">(</mo><mi id="S3.E3.m1.3.3.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E3.m1.3.3.1.1.1.3.2" xref="S3.E3.m1.5.5.3.3.1.cmml">)</mo></mrow></msup><mo id="S3.E3.m1.5.5.3.3.2" xref="S3.E3.m1.5.5.3.4.cmml">,</mo><mi id="S3.E3.m1.4.4.2.2" xref="S3.E3.m1.4.4.2.2.cmml">j</mi></mrow><mrow id="S3.E3.m1.2.2.1.3" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S3.E3.m1.2.2.1.3.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.cmml">(</mo><mi id="S3.E3.m1.2.2.1.1" xref="S3.E3.m1.2.2.1.1.cmml">c</mi><mo stretchy="false" id="S3.E3.m1.2.2.1.3.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.cmml">)</mo></mrow></msubsup></mrow><mo id="S3.E3.m1.6.6.1.1.1.1.1.3" xref="S3.E3.m1.6.6.1.1.1.1.1.1.cmml">)</mo></mrow><mn id="S3.E3.m1.6.6.1.1.1.3" xref="S3.E3.m1.6.6.1.1.1.3.cmml">2</mn></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.6b"><apply id="S3.E3.m1.6.6.cmml" xref="S3.E3.m1.6.6"><apply id="S3.E3.m1.6.6.2.cmml" xref="S3.E3.m1.6.6.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.2.1.cmml" xref="S3.E3.m1.6.6.2">superscript</csymbol><apply id="S3.E3.m1.6.6.2.2.cmml" xref="S3.E3.m1.6.6.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.2.2.1.cmml" xref="S3.E3.m1.6.6.2">subscript</csymbol><sum id="S3.E3.m1.6.6.2.2.2.cmml" xref="S3.E3.m1.6.6.2.2.2"></sum><apply id="S3.E3.m1.6.6.2.2.3.cmml" xref="S3.E3.m1.6.6.2.2.3"><eq id="S3.E3.m1.6.6.2.2.3.1.cmml" xref="S3.E3.m1.6.6.2.2.3.1"></eq><ci id="S3.E3.m1.6.6.2.2.3.2.cmml" xref="S3.E3.m1.6.6.2.2.3.2">𝑖</ci><cn type="integer" id="S3.E3.m1.6.6.2.2.3.3.cmml" xref="S3.E3.m1.6.6.2.2.3.3">1</cn></apply></apply><ci id="S3.E3.m1.6.6.2.3.cmml" xref="S3.E3.m1.6.6.2.3">𝑁</ci></apply><apply id="S3.E3.m1.6.6.1.cmml" xref="S3.E3.m1.6.6.1"><apply id="S3.E3.m1.6.6.1.2.cmml" xref="S3.E3.m1.6.6.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.2.1.cmml" xref="S3.E3.m1.6.6.1.2">superscript</csymbol><apply id="S3.E3.m1.6.6.1.2.2.cmml" xref="S3.E3.m1.6.6.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.2.2.1.cmml" xref="S3.E3.m1.6.6.1.2">subscript</csymbol><sum id="S3.E3.m1.6.6.1.2.2.2.cmml" xref="S3.E3.m1.6.6.1.2.2.2"></sum><apply id="S3.E3.m1.6.6.1.2.2.3.cmml" xref="S3.E3.m1.6.6.1.2.2.3"><eq id="S3.E3.m1.6.6.1.2.2.3.1.cmml" xref="S3.E3.m1.6.6.1.2.2.3.1"></eq><ci id="S3.E3.m1.6.6.1.2.2.3.2.cmml" xref="S3.E3.m1.6.6.1.2.2.3.2">𝑐</ci><cn type="integer" id="S3.E3.m1.6.6.1.2.2.3.3.cmml" xref="S3.E3.m1.6.6.1.2.2.3.3">1</cn></apply></apply><ci id="S3.E3.m1.6.6.1.2.3.cmml" xref="S3.E3.m1.6.6.1.2.3">𝐶</ci></apply><apply id="S3.E3.m1.6.6.1.1.cmml" xref="S3.E3.m1.6.6.1.1"><apply id="S3.E3.m1.6.6.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.2.1.cmml" xref="S3.E3.m1.6.6.1.1.2">subscript</csymbol><sum id="S3.E3.m1.6.6.1.1.2.2.cmml" xref="S3.E3.m1.6.6.1.1.2.2"></sum><apply id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1.1"><in id="S3.E3.m1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.2"></in><ci id="S3.E3.m1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.3">𝑗</ci><apply id="S3.E3.m1.1.1.1.4.cmml" xref="S3.E3.m1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.4.1.cmml" xref="S3.E3.m1.1.1.1.4">superscript</csymbol><ci id="S3.E3.m1.1.1.1.4.2.cmml" xref="S3.E3.m1.1.1.1.4.2">𝒥</ci><ci id="S3.E3.m1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1">𝑐</ci></apply></apply></apply><apply id="S3.E3.m1.6.6.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.1">superscript</csymbol><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1"><minus id="S3.E3.m1.6.6.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.1"></minus><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2">subscript</csymbol><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.2"><ci id="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.1">~</ci><ci id="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.2.2">𝑨</ci></apply><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3"><times id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.1"></times><ci id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.2">𝑖</ci><ci id="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.3.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.2">𝑷</ci><ci id="S3.E3.m1.2.2.1.1.cmml" xref="S3.E3.m1.2.2.1.1">𝑐</ci></apply><list id="S3.E3.m1.5.5.3.4.cmml" xref="S3.E3.m1.5.5.3.3"><apply id="S3.E3.m1.5.5.3.3.1.cmml" xref="S3.E3.m1.5.5.3.3.1"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.3.3.1.1.cmml" xref="S3.E3.m1.5.5.3.3.1">superscript</csymbol><ci id="S3.E3.m1.5.5.3.3.1.2.cmml" xref="S3.E3.m1.5.5.3.3.1.2">𝑧</ci><ci id="S3.E3.m1.3.3.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1">𝑐</ci></apply><ci id="S3.E3.m1.4.4.2.2.cmml" xref="S3.E3.m1.4.4.2.2">𝑗</ci></list></apply></apply><cn type="integer" id="S3.E3.m1.6.6.1.1.1.3.cmml" xref="S3.E3.m1.6.6.1.1.1.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.6c">\displaystyle\sum_{i=1}^{N}\sum_{c=1}^{C}\sum_{j\in\mathcal{J}^{(c)}}\left(\tilde{\bm{A}}_{ij}-\bm{P}^{(c)}_{z^{(c)},j}\right)^{2}\vspace*{-.5mm}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px1.p1.10" class="ltx_p">is minimized. It does this by running K-means separately in each subspace <math id="S3.SS0.SSS0.Px1.p1.8.m1.1" class="ltx_Math" alttext="\mathcal{J}^{(c)}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.8.m1.1a"><msup id="S3.SS0.SSS0.Px1.p1.8.m1.1.2" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.8.m1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2.2.cmml">𝒥</mi><mrow id="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.8.m1.1b"><apply id="S3.SS0.SSS0.Px1.p1.8.m1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.8.m1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.8.m1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.2.2">𝒥</ci><ci id="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m1.1.1.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.8.m1.1c">\mathcal{J}^{(c)}</annotation></semantics></math> and using the resulting centroids and assignments to populate <math id="S3.SS0.SSS0.Px1.p1.9.m2.1" class="ltx_Math" alttext="\bm{P}^{(c)}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.9.m2.1a"><msup id="S3.SS0.SSS0.Px1.p1.9.m2.1.2" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.9.m2.1.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2.2.cmml">𝑷</mi><mrow id="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.9.m2.1b"><apply id="S3.SS0.SSS0.Px1.p1.9.m2.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m2.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.9.m2.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.2.2">𝑷</ci><ci id="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m2.1.1.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.9.m2.1c">\bm{P}^{(c)}</annotation></semantics></math> and <math id="S3.SS0.SSS0.Px1.p1.10.m3.1" class="ltx_Math" alttext="\bm{z}^{(c)}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.10.m3.1a"><msup id="S3.SS0.SSS0.Px1.p1.10.m3.1.2" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.10.m3.1.2.2" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2.2.cmml">𝒛</mi><mrow id="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.10.m3.1b"><apply id="S3.SS0.SSS0.Px1.p1.10.m3.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.10.m3.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.10.m3.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.2.2">𝒛</ci><ci id="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m3.1.1.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.10.m3.1c">\bm{z}^{(c)}</annotation></semantics></math>.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2106.10860/assets/x2.png" id="S3.F2.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="363" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Product Quantization. The <math id="S3.F2.5.4.m1.1" class="ltx_Math" alttext="g(\cdot)" display="inline"><semantics id="S3.F2.5.4.m1.1b"><mrow id="S3.F2.5.4.m1.1.2" xref="S3.F2.5.4.m1.1.2.cmml"><mi id="S3.F2.5.4.m1.1.2.2" xref="S3.F2.5.4.m1.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.F2.5.4.m1.1.2.1" xref="S3.F2.5.4.m1.1.2.1.cmml">​</mo><mrow id="S3.F2.5.4.m1.1.2.3.2" xref="S3.F2.5.4.m1.1.2.cmml"><mo stretchy="false" id="S3.F2.5.4.m1.1.2.3.2.1" xref="S3.F2.5.4.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.F2.5.4.m1.1.1" xref="S3.F2.5.4.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.F2.5.4.m1.1.2.3.2.2" xref="S3.F2.5.4.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.5.4.m1.1c"><apply id="S3.F2.5.4.m1.1.2.cmml" xref="S3.F2.5.4.m1.1.2"><times id="S3.F2.5.4.m1.1.2.1.cmml" xref="S3.F2.5.4.m1.1.2.1"></times><ci id="S3.F2.5.4.m1.1.2.2.cmml" xref="S3.F2.5.4.m1.1.2.2">𝑔</ci><ci id="S3.F2.5.4.m1.1.1.cmml" xref="S3.F2.5.4.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.5.4.m1.1d">g(\cdot)</annotation></semantics></math> function returns the index of the most similar prototype to the data vector a in each subspace. The <math id="S3.F2.6.5.m2.1" class="ltx_Math" alttext="h(\cdot)" display="inline"><semantics id="S3.F2.6.5.m2.1b"><mrow id="S3.F2.6.5.m2.1.2" xref="S3.F2.6.5.m2.1.2.cmml"><mi id="S3.F2.6.5.m2.1.2.2" xref="S3.F2.6.5.m2.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.F2.6.5.m2.1.2.1" xref="S3.F2.6.5.m2.1.2.1.cmml">​</mo><mrow id="S3.F2.6.5.m2.1.2.3.2" xref="S3.F2.6.5.m2.1.2.cmml"><mo stretchy="false" id="S3.F2.6.5.m2.1.2.3.2.1" xref="S3.F2.6.5.m2.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.F2.6.5.m2.1.1" xref="S3.F2.6.5.m2.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.F2.6.5.m2.1.2.3.2.2" xref="S3.F2.6.5.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.6.5.m2.1c"><apply id="S3.F2.6.5.m2.1.2.cmml" xref="S3.F2.6.5.m2.1.2"><times id="S3.F2.6.5.m2.1.2.1.cmml" xref="S3.F2.6.5.m2.1.2.1"></times><ci id="S3.F2.6.5.m2.1.2.2.cmml" xref="S3.F2.6.5.m2.1.2.2">ℎ</ci><ci id="S3.F2.6.5.m2.1.1.cmml" xref="S3.F2.6.5.m2.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.6.5.m2.1d">h(\cdot)</annotation></semantics></math> function computes a lookup table of dot products between the query vector b and each protoype in each subspace. The aggregation function <math id="S3.F2.7.6.m3.2" class="ltx_Math" alttext="f(\cdot,\cdot)" display="inline"><semantics id="S3.F2.7.6.m3.2b"><mrow id="S3.F2.7.6.m3.2.3" xref="S3.F2.7.6.m3.2.3.cmml"><mi id="S3.F2.7.6.m3.2.3.2" xref="S3.F2.7.6.m3.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.F2.7.6.m3.2.3.1" xref="S3.F2.7.6.m3.2.3.1.cmml">​</mo><mrow id="S3.F2.7.6.m3.2.3.3.2" xref="S3.F2.7.6.m3.2.3.3.1.cmml"><mo stretchy="false" id="S3.F2.7.6.m3.2.3.3.2.1" xref="S3.F2.7.6.m3.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.F2.7.6.m3.1.1" xref="S3.F2.7.6.m3.1.1.cmml">⋅</mo><mo rspace="0em" id="S3.F2.7.6.m3.2.3.3.2.2" xref="S3.F2.7.6.m3.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S3.F2.7.6.m3.2.2" xref="S3.F2.7.6.m3.2.2.cmml">⋅</mo><mo stretchy="false" id="S3.F2.7.6.m3.2.3.3.2.3" xref="S3.F2.7.6.m3.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F2.7.6.m3.2c"><apply id="S3.F2.7.6.m3.2.3.cmml" xref="S3.F2.7.6.m3.2.3"><times id="S3.F2.7.6.m3.2.3.1.cmml" xref="S3.F2.7.6.m3.2.3.1"></times><ci id="S3.F2.7.6.m3.2.3.2.cmml" xref="S3.F2.7.6.m3.2.3.2">𝑓</ci><interval closure="open" id="S3.F2.7.6.m3.2.3.3.1.cmml" xref="S3.F2.7.6.m3.2.3.3.2"><ci id="S3.F2.7.6.m3.1.1.cmml" xref="S3.F2.7.6.m3.1.1">⋅</ci><ci id="S3.F2.7.6.m3.2.2.cmml" xref="S3.F2.7.6.m3.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.7.6.m3.2d">f(\cdot,\cdot)</annotation></semantics></math> sums the table entries corresponding to each index.</figcaption>
</figure>
</section>
<section id="S3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Encoding Function, <math id="S3.SS0.SSS0.Px2.1.m1.1" class="ltx_Math" alttext="\bm{g(\bm{A})}" display="inline"><semantics id="S3.SS0.SSS0.Px2.1.m1.1b"><mrow id="S3.SS0.SSS0.Px2.1.m1.1.2" xref="S3.SS0.SSS0.Px2.1.m1.1.2.cmml"><mi id="S3.SS0.SSS0.Px2.1.m1.1.2.2" xref="S3.SS0.SSS0.Px2.1.m1.1.2.2.cmml">𝒈</mi><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px2.1.m1.1.2.1" xref="S3.SS0.SSS0.Px2.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS0.SSS0.Px2.1.m1.1.2.3.2" xref="S3.SS0.SSS0.Px2.1.m1.1.2.cmml"><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S3.SS0.SSS0.Px2.1.m1.1.2.3.2.1" xref="S3.SS0.SSS0.Px2.1.m1.1.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px2.1.m1.1.1" xref="S3.SS0.SSS0.Px2.1.m1.1.1.cmml">𝑨</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S3.SS0.SSS0.Px2.1.m1.1.2.3.2.2" xref="S3.SS0.SSS0.Px2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.1.m1.1c"><apply id="S3.SS0.SSS0.Px2.1.m1.1.2.cmml" xref="S3.SS0.SSS0.Px2.1.m1.1.2"><times id="S3.SS0.SSS0.Px2.1.m1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.1.m1.1.2.1"></times><ci id="S3.SS0.SSS0.Px2.1.m1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.1.m1.1.2.2">𝒈</ci><ci id="S3.SS0.SSS0.Px2.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.1.m1.1.1">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.1.m1.1d">\bm{g(\bm{A})}</annotation></semantics></math>:</h4>

<div id="S3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p1.4" class="ltx_p">Given the learned prototypes, PQ replaces each row <math id="S3.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.1.m1.1a"><mi id="S3.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.1.m1.1b"><ci id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.1.m1.1c">\bm{a}</annotation></semantics></math> of <math id="S3.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.2.m2.1a"><mi id="S3.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.2.m2.1c">\bm{A}</annotation></semantics></math> with the concatenation of its <math id="S3.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.3.m3.1a"><mi id="S3.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.3.m3.1b"><ci id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.3.m3.1c">C</annotation></semantics></math> K-means centroid assignments in each of the <math id="S3.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.4.m4.1a"><mi id="S3.SS0.SSS0.Px2.p1.4.m4.1.1" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.4.m4.1b"><ci id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.4.m4.1c">C</annotation></semantics></math> subspaces. Formally:</p>
<table id="A6.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E4.m1.7" class="ltx_Math" alttext="\displaystyle g^{(c)}(\bm{a})\triangleq\operatorname*{argmin}_{k}\sum_{j\in\mathcal{J}^{(c)}}\left(\bm{a}_{j}-\bm{P}^{(c)}_{k,j}\right)^{2}.\vspace*{-.5mm}" display="inline"><semantics id="S3.E4.m1.7a"><mrow id="S3.E4.m1.7.7.1" xref="S3.E4.m1.7.7.1.1.cmml"><mrow id="S3.E4.m1.7.7.1.1" xref="S3.E4.m1.7.7.1.1.cmml"><mrow id="S3.E4.m1.7.7.1.1.3" xref="S3.E4.m1.7.7.1.1.3.cmml"><msup id="S3.E4.m1.7.7.1.1.3.2" xref="S3.E4.m1.7.7.1.1.3.2.cmml"><mi id="S3.E4.m1.7.7.1.1.3.2.2" xref="S3.E4.m1.7.7.1.1.3.2.2.cmml">g</mi><mrow id="S3.E4.m1.1.1.1.3" xref="S3.E4.m1.7.7.1.1.3.2.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.3.1" xref="S3.E4.m1.7.7.1.1.3.2.cmml">(</mo><mi id="S3.E4.m1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E4.m1.1.1.1.3.2" xref="S3.E4.m1.7.7.1.1.3.2.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S3.E4.m1.7.7.1.1.3.1" xref="S3.E4.m1.7.7.1.1.3.1.cmml">​</mo><mrow id="S3.E4.m1.7.7.1.1.3.3.2" xref="S3.E4.m1.7.7.1.1.3.cmml"><mo stretchy="false" id="S3.E4.m1.7.7.1.1.3.3.2.1" xref="S3.E4.m1.7.7.1.1.3.cmml">(</mo><mi id="S3.E4.m1.6.6" xref="S3.E4.m1.6.6.cmml">𝒂</mi><mo stretchy="false" id="S3.E4.m1.7.7.1.1.3.3.2.2" xref="S3.E4.m1.7.7.1.1.3.cmml">)</mo></mrow></mrow><mo rspace="0.1389em" id="S3.E4.m1.7.7.1.1.2" xref="S3.E4.m1.7.7.1.1.2.cmml">≜</mo><mrow id="S3.E4.m1.7.7.1.1.1" xref="S3.E4.m1.7.7.1.1.1.cmml"><munder id="S3.E4.m1.7.7.1.1.1.3" xref="S3.E4.m1.7.7.1.1.1.3.cmml"><mo lspace="0.1389em" id="S3.E4.m1.7.7.1.1.1.3.2" xref="S3.E4.m1.7.7.1.1.1.3.2.cmml">argmin</mo><mi id="S3.E4.m1.7.7.1.1.1.3.3" xref="S3.E4.m1.7.7.1.1.1.3.3.cmml">k</mi></munder><mo lspace="0.167em" rspace="0em" id="S3.E4.m1.7.7.1.1.1.2" xref="S3.E4.m1.7.7.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.7.7.1.1.1.1" xref="S3.E4.m1.7.7.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E4.m1.7.7.1.1.1.1.2" xref="S3.E4.m1.7.7.1.1.1.1.2.cmml"><munder id="S3.E4.m1.7.7.1.1.1.1.2a" xref="S3.E4.m1.7.7.1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.E4.m1.7.7.1.1.1.1.2.2" xref="S3.E4.m1.7.7.1.1.1.1.2.2.cmml">∑</mo><mrow id="S3.E4.m1.2.2.1" xref="S3.E4.m1.2.2.1.cmml"><mi id="S3.E4.m1.2.2.1.3" xref="S3.E4.m1.2.2.1.3.cmml">j</mi><mo id="S3.E4.m1.2.2.1.2" xref="S3.E4.m1.2.2.1.2.cmml">∈</mo><msup id="S3.E4.m1.2.2.1.4" xref="S3.E4.m1.2.2.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.2.2.1.4.2" xref="S3.E4.m1.2.2.1.4.2.cmml">𝒥</mi><mrow id="S3.E4.m1.2.2.1.1.1.3" xref="S3.E4.m1.2.2.1.4.cmml"><mo stretchy="false" id="S3.E4.m1.2.2.1.1.1.3.1" xref="S3.E4.m1.2.2.1.4.cmml">(</mo><mi id="S3.E4.m1.2.2.1.1.1.1" xref="S3.E4.m1.2.2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E4.m1.2.2.1.1.1.3.2" xref="S3.E4.m1.2.2.1.4.cmml">)</mo></mrow></msup></mrow></munder></mstyle><msup id="S3.E4.m1.7.7.1.1.1.1.1" xref="S3.E4.m1.7.7.1.1.1.1.1.cmml"><mrow id="S3.E4.m1.7.7.1.1.1.1.1.1.1" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.cmml"><mo id="S3.E4.m1.7.7.1.1.1.1.1.1.1.2" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.cmml"><msub id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.2" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.2.cmml">𝒂</mi><mi id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.3" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.3.cmml">j</mi></msub><mo id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.2.2" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.2.2.cmml">𝑷</mi><mrow id="S3.E4.m1.5.5.2.4" xref="S3.E4.m1.5.5.2.3.cmml"><mi id="S3.E4.m1.4.4.1.1" xref="S3.E4.m1.4.4.1.1.cmml">k</mi><mo id="S3.E4.m1.5.5.2.4.1" xref="S3.E4.m1.5.5.2.3.cmml">,</mo><mi id="S3.E4.m1.5.5.2.2" xref="S3.E4.m1.5.5.2.2.cmml">j</mi></mrow><mrow id="S3.E4.m1.3.3.1.3" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S3.E4.m1.3.3.1.3.1" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.cmml">(</mo><mi id="S3.E4.m1.3.3.1.1" xref="S3.E4.m1.3.3.1.1.cmml">c</mi><mo stretchy="false" id="S3.E4.m1.3.3.1.3.2" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.cmml">)</mo></mrow></msubsup></mrow><mo id="S3.E4.m1.7.7.1.1.1.1.1.1.1.3" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mn id="S3.E4.m1.7.7.1.1.1.1.1.3" xref="S3.E4.m1.7.7.1.1.1.1.1.3.cmml">2</mn></msup></mrow></mrow></mrow><mo lspace="0em" id="S3.E4.m1.7.7.1.2" xref="S3.E4.m1.7.7.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.7b"><apply id="S3.E4.m1.7.7.1.1.cmml" xref="S3.E4.m1.7.7.1"><ci id="S3.E4.m1.7.7.1.1.2.cmml" xref="S3.E4.m1.7.7.1.1.2">≜</ci><apply id="S3.E4.m1.7.7.1.1.3.cmml" xref="S3.E4.m1.7.7.1.1.3"><times id="S3.E4.m1.7.7.1.1.3.1.cmml" xref="S3.E4.m1.7.7.1.1.3.1"></times><apply id="S3.E4.m1.7.7.1.1.3.2.cmml" xref="S3.E4.m1.7.7.1.1.3.2"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.3.2.1.cmml" xref="S3.E4.m1.7.7.1.1.3.2">superscript</csymbol><ci id="S3.E4.m1.7.7.1.1.3.2.2.cmml" xref="S3.E4.m1.7.7.1.1.3.2.2">𝑔</ci><ci id="S3.E4.m1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1">𝑐</ci></apply><ci id="S3.E4.m1.6.6.cmml" xref="S3.E4.m1.6.6">𝒂</ci></apply><apply id="S3.E4.m1.7.7.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1"><times id="S3.E4.m1.7.7.1.1.1.2.cmml" xref="S3.E4.m1.7.7.1.1.1.2"></times><apply id="S3.E4.m1.7.7.1.1.1.3.cmml" xref="S3.E4.m1.7.7.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.1.3.1.cmml" xref="S3.E4.m1.7.7.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.7.7.1.1.1.3.2.cmml" xref="S3.E4.m1.7.7.1.1.1.3.2">argmin</ci><ci id="S3.E4.m1.7.7.1.1.1.3.3.cmml" xref="S3.E4.m1.7.7.1.1.1.3.3">𝑘</ci></apply><apply id="S3.E4.m1.7.7.1.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1"><apply id="S3.E4.m1.7.7.1.1.1.1.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.1.1.2.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.2">subscript</csymbol><sum id="S3.E4.m1.7.7.1.1.1.1.2.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.2.2"></sum><apply id="S3.E4.m1.2.2.1.cmml" xref="S3.E4.m1.2.2.1"><in id="S3.E4.m1.2.2.1.2.cmml" xref="S3.E4.m1.2.2.1.2"></in><ci id="S3.E4.m1.2.2.1.3.cmml" xref="S3.E4.m1.2.2.1.3">𝑗</ci><apply id="S3.E4.m1.2.2.1.4.cmml" xref="S3.E4.m1.2.2.1.4"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.1.4.1.cmml" xref="S3.E4.m1.2.2.1.4">superscript</csymbol><ci id="S3.E4.m1.2.2.1.4.2.cmml" xref="S3.E4.m1.2.2.1.4.2">𝒥</ci><ci id="S3.E4.m1.2.2.1.1.1.1.cmml" xref="S3.E4.m1.2.2.1.1.1.1">𝑐</ci></apply></apply></apply><apply id="S3.E4.m1.7.7.1.1.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.1.1.1.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1">superscript</csymbol><apply id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1"><minus id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.2">𝒂</ci><ci id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.2.3">𝑗</ci></apply><apply id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.1.1.1.3.2.2">𝑷</ci><ci id="S3.E4.m1.3.3.1.1.cmml" xref="S3.E4.m1.3.3.1.1">𝑐</ci></apply><list id="S3.E4.m1.5.5.2.3.cmml" xref="S3.E4.m1.5.5.2.4"><ci id="S3.E4.m1.4.4.1.1.cmml" xref="S3.E4.m1.4.4.1.1">𝑘</ci><ci id="S3.E4.m1.5.5.2.2.cmml" xref="S3.E4.m1.5.5.2.2">𝑗</ci></list></apply></apply><cn type="integer" id="S3.E4.m1.7.7.1.1.1.1.1.3.cmml" xref="S3.E4.m1.7.7.1.1.1.1.1.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.7c">\displaystyle g^{(c)}(\bm{a})\triangleq\operatorname*{argmin}_{k}\sum_{j\in\mathcal{J}^{(c)}}\left(\bm{a}_{j}-\bm{P}^{(c)}_{k,j}\right)^{2}.\vspace*{-.5mm}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px2.p1.9" class="ltx_p">We will refer to the resulting sequence of indices as the <span id="S3.SS0.SSS0.Px2.p1.9.1" class="ltx_text ltx_font_italic">encoding</span> of <math id="S3.SS0.SSS0.Px2.p1.5.m1.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.5.m1.1a"><mi id="S3.SS0.SSS0.Px2.p1.5.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.5.m1.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.5.m1.1b"><ci id="S3.SS0.SSS0.Px2.p1.5.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.5.m1.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.5.m1.1c">\bm{a}</annotation></semantics></math> and the set of <math id="S3.SS0.SSS0.Px2.p1.6.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.6.m2.1a"><mi id="S3.SS0.SSS0.Px2.p1.6.m2.1.1" xref="S3.SS0.SSS0.Px2.p1.6.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.6.m2.1b"><ci id="S3.SS0.SSS0.Px2.p1.6.m2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.6.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.6.m2.1c">K</annotation></semantics></math> centroids as a <span id="S3.SS0.SSS0.Px2.p1.9.2" class="ltx_text ltx_font_italic">codebook</span>. For convenience, we will also refer to the vector <math id="S3.SS0.SSS0.Px2.p1.7.m3.4" class="ltx_Math" alttext="\bm{a}^{(c)}\triangleq\langle a_{j}\rangle,j\in\mathcal{J}^{(c)}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.7.m3.4a"><mrow id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.3.cmml"><mrow id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.cmml"><msup id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.2.cmml">𝒂</mi><mrow id="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.3.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.cmml">(</mo><mi id="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.3.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.cmml">)</mo></mrow></msup><mo id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.2.cmml">≜</mo><mrow id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.2.1.cmml">⟨</mo><msub id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.2.cmml">a</mi><mi id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.3.cmml">j</mi></msub><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.2.1.cmml">⟩</mo></mrow></mrow><mo id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.3a.cmml">,</mo><mrow id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.cmml"><mi id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.2.cmml">j</mi><mo id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.2.cmml">𝒥</mi><mrow id="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.3" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.3.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.cmml">(</mo><mi id="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.1" xref="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.3.2" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.cmml">)</mo></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.7.m3.4b"><apply id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.3.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.3a.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.3">formulae-sequence</csymbol><apply id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1"><ci id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.2">≜</ci><apply id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.3.2">𝒂</ci><ci id="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.1.1.1.1">𝑐</ci></apply><apply id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1"><csymbol cd="latexml" id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.2">delimited-⟨⟩</csymbol><apply id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.2">𝑎</ci><ci id="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.3.3.1.1.1.1.1.3">𝑗</ci></apply></apply></apply><apply id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2"><in id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.1"></in><ci id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.2">𝑗</ci><apply id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.4.4.2.2.3.2">𝒥</ci><ci id="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.m3.2.2.1.1">𝑐</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.7.m3.4c">\bm{a}^{(c)}\triangleq\langle a_{j}\rangle,j\in\mathcal{J}^{(c)}</annotation></semantics></math> as the <span id="S3.SS0.SSS0.Px2.p1.9.3" class="ltx_text ltx_font_italic">subvector</span> of <math id="S3.SS0.SSS0.Px2.p1.8.m4.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.8.m4.1a"><mi id="S3.SS0.SSS0.Px2.p1.8.m4.1.1" xref="S3.SS0.SSS0.Px2.p1.8.m4.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.8.m4.1b"><ci id="S3.SS0.SSS0.Px2.p1.8.m4.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.8.m4.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.8.m4.1c">\bm{a}</annotation></semantics></math> in subspace <math id="S3.SS0.SSS0.Px2.p1.9.m5.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.9.m5.1a"><mi id="S3.SS0.SSS0.Px2.p1.9.m5.1.1" xref="S3.SS0.SSS0.Px2.p1.9.m5.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.9.m5.1b"><ci id="S3.SS0.SSS0.Px2.p1.9.m5.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.9.m5.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.9.m5.1c">c</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Table Construction, <math id="S3.SS0.SSS0.Px3.1.m1.1" class="ltx_Math" alttext="\bm{h(\bm{B})}" display="inline"><semantics id="S3.SS0.SSS0.Px3.1.m1.1b"><mrow id="S3.SS0.SSS0.Px3.1.m1.1.2" xref="S3.SS0.SSS0.Px3.1.m1.1.2.cmml"><mi id="S3.SS0.SSS0.Px3.1.m1.1.2.2" xref="S3.SS0.SSS0.Px3.1.m1.1.2.2.cmml">𝒉</mi><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px3.1.m1.1.2.1" xref="S3.SS0.SSS0.Px3.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS0.SSS0.Px3.1.m1.1.2.3.2" xref="S3.SS0.SSS0.Px3.1.m1.1.2.cmml"><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S3.SS0.SSS0.Px3.1.m1.1.2.3.2.1" xref="S3.SS0.SSS0.Px3.1.m1.1.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px3.1.m1.1.1" xref="S3.SS0.SSS0.Px3.1.m1.1.1.cmml">𝑩</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S3.SS0.SSS0.Px3.1.m1.1.2.3.2.2" xref="S3.SS0.SSS0.Px3.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.1.m1.1c"><apply id="S3.SS0.SSS0.Px3.1.m1.1.2.cmml" xref="S3.SS0.SSS0.Px3.1.m1.1.2"><times id="S3.SS0.SSS0.Px3.1.m1.1.2.1.cmml" xref="S3.SS0.SSS0.Px3.1.m1.1.2.1"></times><ci id="S3.SS0.SSS0.Px3.1.m1.1.2.2.cmml" xref="S3.SS0.SSS0.Px3.1.m1.1.2.2">𝒉</ci><ci id="S3.SS0.SSS0.Px3.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.1.m1.1.1">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.1.m1.1d">\bm{h(\bm{B})}</annotation></semantics></math>:</h4>

<div id="S3.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px3.p1.4" class="ltx_p">Using these same prototypes, PQ constructs a lookup table <math id="S3.SS0.SSS0.Px3.p1.1.m1.2" class="ltx_Math" alttext="h^{(c)}(\bm{b})\in\mathbb{R}^{K}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.1.m1.2a"><mrow id="S3.SS0.SSS0.Px3.p1.1.m1.2.3" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.cmml"><mrow id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.cmml"><msup id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.cmml"><mi id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.2.cmml">h</mi><mrow id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.3" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.3.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.3.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.1.cmml">​</mo><mrow id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.3.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.3.2.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.cmml">(</mo><mi id="S3.SS0.SSS0.Px3.p1.1.m1.2.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.2.cmml">𝒃</mi><mo stretchy="false" id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.3.2.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.cmml">)</mo></mrow></mrow><mo id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.1.cmml">∈</mo><msup id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.cmml"><mi id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.2.cmml">ℝ</mi><mi id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.3" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.3.cmml">K</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.1.m1.2b"><apply id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3"><in id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.1"></in><apply id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2"><times id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.1"></times><apply id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.2.2.2">ℎ</ci><ci id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.1">𝑐</ci></apply><ci id="S3.SS0.SSS0.Px3.p1.1.m1.2.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.2">𝒃</ci></apply><apply id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3">superscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.2">ℝ</ci><ci id="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.3.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.2.3.3.3">𝐾</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.1.m1.2c">h^{(c)}(\bm{b})\in\mathbb{R}^{K}</annotation></semantics></math> in each of the <math id="S3.SS0.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.2.m2.1a"><mi id="S3.SS0.SSS0.Px3.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.2.m2.1c">C</annotation></semantics></math> subspaces for each column <math id="S3.SS0.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.3.m3.1a"><mi id="S3.SS0.SSS0.Px3.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.cmml">𝒃</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.3.m3.1b"><ci id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1">𝒃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.3.m3.1c">\bm{b}</annotation></semantics></math> of <math id="S3.SS0.SSS0.Px3.p1.4.m4.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.4.m4.1a"><mi id="S3.SS0.SSS0.Px3.p1.4.m4.1.1" xref="S3.SS0.SSS0.Px3.p1.4.m4.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.4.m4.1b"><ci id="S3.SS0.SSS0.Px3.p1.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.4.m4.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.4.m4.1c">\bm{B}</annotation></semantics></math>, where</p>
<table id="A6.EGx5" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E5.m1.7" class="ltx_Math" alttext="\displaystyle h^{(c)}(\bm{b})_{k}\triangleq\sum_{j\in\mathcal{J}^{(c)}}\bm{b}_{j}\bm{P}^{(c)}_{k,j}." display="inline"><semantics id="S3.E5.m1.7a"><mrow id="S3.E5.m1.7.7.1" xref="S3.E5.m1.7.7.1.1.cmml"><mrow id="S3.E5.m1.7.7.1.1" xref="S3.E5.m1.7.7.1.1.cmml"><mrow id="S3.E5.m1.7.7.1.1.2" xref="S3.E5.m1.7.7.1.1.2.cmml"><msup id="S3.E5.m1.7.7.1.1.2.2" xref="S3.E5.m1.7.7.1.1.2.2.cmml"><mi id="S3.E5.m1.7.7.1.1.2.2.2" xref="S3.E5.m1.7.7.1.1.2.2.2.cmml">h</mi><mrow id="S3.E5.m1.1.1.1.3" xref="S3.E5.m1.7.7.1.1.2.2.cmml"><mo stretchy="false" id="S3.E5.m1.1.1.1.3.1" xref="S3.E5.m1.7.7.1.1.2.2.cmml">(</mo><mi id="S3.E5.m1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E5.m1.1.1.1.3.2" xref="S3.E5.m1.7.7.1.1.2.2.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S3.E5.m1.7.7.1.1.2.1" xref="S3.E5.m1.7.7.1.1.2.1.cmml">​</mo><msub id="S3.E5.m1.7.7.1.1.2.3" xref="S3.E5.m1.7.7.1.1.2.3.cmml"><mrow id="S3.E5.m1.7.7.1.1.2.3.2.2" xref="S3.E5.m1.7.7.1.1.2.3.cmml"><mo stretchy="false" id="S3.E5.m1.7.7.1.1.2.3.2.2.1" xref="S3.E5.m1.7.7.1.1.2.3.cmml">(</mo><mi id="S3.E5.m1.6.6" xref="S3.E5.m1.6.6.cmml">𝒃</mi><mo stretchy="false" id="S3.E5.m1.7.7.1.1.2.3.2.2.2" xref="S3.E5.m1.7.7.1.1.2.3.cmml">)</mo></mrow><mi id="S3.E5.m1.7.7.1.1.2.3.3" xref="S3.E5.m1.7.7.1.1.2.3.3.cmml">k</mi></msub></mrow><mo id="S3.E5.m1.7.7.1.1.1" xref="S3.E5.m1.7.7.1.1.1.cmml">≜</mo><mrow id="S3.E5.m1.7.7.1.1.3" xref="S3.E5.m1.7.7.1.1.3.cmml"><mstyle displaystyle="true" id="S3.E5.m1.7.7.1.1.3.1" xref="S3.E5.m1.7.7.1.1.3.1.cmml"><munder id="S3.E5.m1.7.7.1.1.3.1a" xref="S3.E5.m1.7.7.1.1.3.1.cmml"><mo movablelimits="false" id="S3.E5.m1.7.7.1.1.3.1.2" xref="S3.E5.m1.7.7.1.1.3.1.2.cmml">∑</mo><mrow id="S3.E5.m1.2.2.1" xref="S3.E5.m1.2.2.1.cmml"><mi id="S3.E5.m1.2.2.1.3" xref="S3.E5.m1.2.2.1.3.cmml">j</mi><mo id="S3.E5.m1.2.2.1.2" xref="S3.E5.m1.2.2.1.2.cmml">∈</mo><msup id="S3.E5.m1.2.2.1.4" xref="S3.E5.m1.2.2.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5.m1.2.2.1.4.2" xref="S3.E5.m1.2.2.1.4.2.cmml">𝒥</mi><mrow id="S3.E5.m1.2.2.1.1.1.3" xref="S3.E5.m1.2.2.1.4.cmml"><mo stretchy="false" id="S3.E5.m1.2.2.1.1.1.3.1" xref="S3.E5.m1.2.2.1.4.cmml">(</mo><mi id="S3.E5.m1.2.2.1.1.1.1" xref="S3.E5.m1.2.2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E5.m1.2.2.1.1.1.3.2" xref="S3.E5.m1.2.2.1.4.cmml">)</mo></mrow></msup></mrow></munder></mstyle><mrow id="S3.E5.m1.7.7.1.1.3.2" xref="S3.E5.m1.7.7.1.1.3.2.cmml"><msub id="S3.E5.m1.7.7.1.1.3.2.2" xref="S3.E5.m1.7.7.1.1.3.2.2.cmml"><mi id="S3.E5.m1.7.7.1.1.3.2.2.2" xref="S3.E5.m1.7.7.1.1.3.2.2.2.cmml">𝒃</mi><mi id="S3.E5.m1.7.7.1.1.3.2.2.3" xref="S3.E5.m1.7.7.1.1.3.2.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.7.7.1.1.3.2.1" xref="S3.E5.m1.7.7.1.1.3.2.1.cmml">​</mo><msubsup id="S3.E5.m1.7.7.1.1.3.2.3" xref="S3.E5.m1.7.7.1.1.3.2.3.cmml"><mi id="S3.E5.m1.7.7.1.1.3.2.3.2.2" xref="S3.E5.m1.7.7.1.1.3.2.3.2.2.cmml">𝑷</mi><mrow id="S3.E5.m1.5.5.2.4" xref="S3.E5.m1.5.5.2.3.cmml"><mi id="S3.E5.m1.4.4.1.1" xref="S3.E5.m1.4.4.1.1.cmml">k</mi><mo id="S3.E5.m1.5.5.2.4.1" xref="S3.E5.m1.5.5.2.3.cmml">,</mo><mi id="S3.E5.m1.5.5.2.2" xref="S3.E5.m1.5.5.2.2.cmml">j</mi></mrow><mrow id="S3.E5.m1.3.3.1.3" xref="S3.E5.m1.7.7.1.1.3.2.3.cmml"><mo stretchy="false" id="S3.E5.m1.3.3.1.3.1" xref="S3.E5.m1.7.7.1.1.3.2.3.cmml">(</mo><mi id="S3.E5.m1.3.3.1.1" xref="S3.E5.m1.3.3.1.1.cmml">c</mi><mo stretchy="false" id="S3.E5.m1.3.3.1.3.2" xref="S3.E5.m1.7.7.1.1.3.2.3.cmml">)</mo></mrow></msubsup></mrow></mrow></mrow><mo lspace="0em" id="S3.E5.m1.7.7.1.2" xref="S3.E5.m1.7.7.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.7b"><apply id="S3.E5.m1.7.7.1.1.cmml" xref="S3.E5.m1.7.7.1"><ci id="S3.E5.m1.7.7.1.1.1.cmml" xref="S3.E5.m1.7.7.1.1.1">≜</ci><apply id="S3.E5.m1.7.7.1.1.2.cmml" xref="S3.E5.m1.7.7.1.1.2"><times id="S3.E5.m1.7.7.1.1.2.1.cmml" xref="S3.E5.m1.7.7.1.1.2.1"></times><apply id="S3.E5.m1.7.7.1.1.2.2.cmml" xref="S3.E5.m1.7.7.1.1.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.1.1.2.2.1.cmml" xref="S3.E5.m1.7.7.1.1.2.2">superscript</csymbol><ci id="S3.E5.m1.7.7.1.1.2.2.2.cmml" xref="S3.E5.m1.7.7.1.1.2.2.2">ℎ</ci><ci id="S3.E5.m1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1">𝑐</ci></apply><apply id="S3.E5.m1.7.7.1.1.2.3.cmml" xref="S3.E5.m1.7.7.1.1.2.3"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.1.1.2.3.1.cmml" xref="S3.E5.m1.7.7.1.1.2.3">subscript</csymbol><ci id="S3.E5.m1.6.6.cmml" xref="S3.E5.m1.6.6">𝒃</ci><ci id="S3.E5.m1.7.7.1.1.2.3.3.cmml" xref="S3.E5.m1.7.7.1.1.2.3.3">𝑘</ci></apply></apply><apply id="S3.E5.m1.7.7.1.1.3.cmml" xref="S3.E5.m1.7.7.1.1.3"><apply id="S3.E5.m1.7.7.1.1.3.1.cmml" xref="S3.E5.m1.7.7.1.1.3.1"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.1.1.3.1.1.cmml" xref="S3.E5.m1.7.7.1.1.3.1">subscript</csymbol><sum id="S3.E5.m1.7.7.1.1.3.1.2.cmml" xref="S3.E5.m1.7.7.1.1.3.1.2"></sum><apply id="S3.E5.m1.2.2.1.cmml" xref="S3.E5.m1.2.2.1"><in id="S3.E5.m1.2.2.1.2.cmml" xref="S3.E5.m1.2.2.1.2"></in><ci id="S3.E5.m1.2.2.1.3.cmml" xref="S3.E5.m1.2.2.1.3">𝑗</ci><apply id="S3.E5.m1.2.2.1.4.cmml" xref="S3.E5.m1.2.2.1.4"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.1.4.1.cmml" xref="S3.E5.m1.2.2.1.4">superscript</csymbol><ci id="S3.E5.m1.2.2.1.4.2.cmml" xref="S3.E5.m1.2.2.1.4.2">𝒥</ci><ci id="S3.E5.m1.2.2.1.1.1.1.cmml" xref="S3.E5.m1.2.2.1.1.1.1">𝑐</ci></apply></apply></apply><apply id="S3.E5.m1.7.7.1.1.3.2.cmml" xref="S3.E5.m1.7.7.1.1.3.2"><times id="S3.E5.m1.7.7.1.1.3.2.1.cmml" xref="S3.E5.m1.7.7.1.1.3.2.1"></times><apply id="S3.E5.m1.7.7.1.1.3.2.2.cmml" xref="S3.E5.m1.7.7.1.1.3.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.1.1.3.2.2.1.cmml" xref="S3.E5.m1.7.7.1.1.3.2.2">subscript</csymbol><ci id="S3.E5.m1.7.7.1.1.3.2.2.2.cmml" xref="S3.E5.m1.7.7.1.1.3.2.2.2">𝒃</ci><ci id="S3.E5.m1.7.7.1.1.3.2.2.3.cmml" xref="S3.E5.m1.7.7.1.1.3.2.2.3">𝑗</ci></apply><apply id="S3.E5.m1.7.7.1.1.3.2.3.cmml" xref="S3.E5.m1.7.7.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.1.1.3.2.3.1.cmml" xref="S3.E5.m1.7.7.1.1.3.2.3">subscript</csymbol><apply id="S3.E5.m1.7.7.1.1.3.2.3.2.cmml" xref="S3.E5.m1.7.7.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.1.1.3.2.3.2.1.cmml" xref="S3.E5.m1.7.7.1.1.3.2.3">superscript</csymbol><ci id="S3.E5.m1.7.7.1.1.3.2.3.2.2.cmml" xref="S3.E5.m1.7.7.1.1.3.2.3.2.2">𝑷</ci><ci id="S3.E5.m1.3.3.1.1.cmml" xref="S3.E5.m1.3.3.1.1">𝑐</ci></apply><list id="S3.E5.m1.5.5.2.3.cmml" xref="S3.E5.m1.5.5.2.4"><ci id="S3.E5.m1.4.4.1.1.cmml" xref="S3.E5.m1.4.4.1.1">𝑘</ci><ci id="S3.E5.m1.5.5.2.2.cmml" xref="S3.E5.m1.5.5.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.7c">\displaystyle h^{(c)}(\bm{b})_{k}\triangleq\sum_{j\in\mathcal{J}^{(c)}}\bm{b}_{j}\bm{P}^{(c)}_{k,j}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS0.SSS0.Px3.p1.8" class="ltx_p">Existing work has shown that setting <math id="S3.SS0.SSS0.Px3.p1.5.m1.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.5.m1.1a"><mrow id="S3.SS0.SSS0.Px3.p1.5.m1.1.1" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.2" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.2.cmml">K</mi><mo id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.1" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.1.cmml">=</mo><mn id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.3" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.5.m1.1b"><apply id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1"><eq id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.1"></eq><ci id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.2">𝐾</ci><cn type="integer" id="S3.SS0.SSS0.Px3.p1.5.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.5.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.5.m1.1c">K=16</annotation></semantics></math> and quantizing the lookup tables to 8 bits can offer enormous speedups compared to larger <math id="S3.SS0.SSS0.Px3.p1.6.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.6.m2.1a"><mi id="S3.SS0.SSS0.Px3.p1.6.m2.1.1" xref="S3.SS0.SSS0.Px3.p1.6.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.6.m2.1b"><ci id="S3.SS0.SSS0.Px3.p1.6.m2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.6.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.6.m2.1c">K</annotation></semantics></math> and/or floating-point tables <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>; André et al., <a href="#bib.bib4" title="" class="ltx_ref">2017</a>, <a href="#bib.bib5" title="" class="ltx_ref">2019</a>)</cite>. This is because 16 1-byte entries can be stored in a SIMD register, allowing 16 or more table lookups to be performed in parallel using a byte shuffle instruction. Since the table entries naturally occupy more than 8 bits even for 8-bit data, some means of quantizing these entries is necessary. This can easily be done by subtracting off the minimum entry in each table and linearly rescaling such that the maximum entry in any table is at most 255. Ignoring rounding error, this affine transform is invertible, and is reflected by the constants <math id="S3.SS0.SSS0.Px3.p1.7.m3.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.7.m3.1a"><mi id="S3.SS0.SSS0.Px3.p1.7.m3.1.1" xref="S3.SS0.SSS0.Px3.p1.7.m3.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.7.m3.1b"><ci id="S3.SS0.SSS0.Px3.p1.7.m3.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.7.m3.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.7.m3.1c">\alpha</annotation></semantics></math> and <math id="S3.SS0.SSS0.Px3.p1.8.m4.1" class="ltx_Math" alttext="\bm{\beta}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.8.m4.1a"><mi id="S3.SS0.SSS0.Px3.p1.8.m4.1.1" xref="S3.SS0.SSS0.Px3.p1.8.m4.1.1.cmml">𝜷</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.8.m4.1b"><ci id="S3.SS0.SSS0.Px3.p1.8.m4.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.8.m4.1.1">𝜷</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.8.m4.1c">\bm{\beta}</annotation></semantics></math> in equation <a href="#S1.E2" title="In 1.1 Problem Formulation ‣ 1 Introduction ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. See Appendix <a href="#A1" title="Appendix A Quantizing Lookup Tables ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a> for further details.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Aggregation, <math id="S3.SS0.SSS0.Px4.1.m1.2" class="ltx_Math" alttext="\bm{f(\cdot,\cdot)}" display="inline"><semantics id="S3.SS0.SSS0.Px4.1.m1.2b"><mrow id="S3.SS0.SSS0.Px4.1.m1.2.3" xref="S3.SS0.SSS0.Px4.1.m1.2.3.cmml"><mi id="S3.SS0.SSS0.Px4.1.m1.2.3.2" xref="S3.SS0.SSS0.Px4.1.m1.2.3.2.cmml">𝒇</mi><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px4.1.m1.2.3.1" xref="S3.SS0.SSS0.Px4.1.m1.2.3.1.cmml">​</mo><mrow id="S3.SS0.SSS0.Px4.1.m1.2.3.3.2" xref="S3.SS0.SSS0.Px4.1.m1.2.3.3.1.cmml"><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S3.SS0.SSS0.Px4.1.m1.2.3.3.2.1" xref="S3.SS0.SSS0.Px4.1.m1.2.3.3.1.cmml">(</mo><mo class="ltx_mathvariant_bold" lspace="0em" mathvariant="bold" rspace="0em" id="S3.SS0.SSS0.Px4.1.m1.1.1" xref="S3.SS0.SSS0.Px4.1.m1.1.1.cmml">⋅</mo><mo class="ltx_mathvariant_bold" mathvariant="bold" rspace="0em" id="S3.SS0.SSS0.Px4.1.m1.2.3.3.2.2" xref="S3.SS0.SSS0.Px4.1.m1.2.3.3.1.cmml">,</mo><mo class="ltx_mathvariant_bold" lspace="0em" mathvariant="bold" rspace="0em" id="S3.SS0.SSS0.Px4.1.m1.2.2" xref="S3.SS0.SSS0.Px4.1.m1.2.2.cmml">⋅</mo><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S3.SS0.SSS0.Px4.1.m1.2.3.3.2.3" xref="S3.SS0.SSS0.Px4.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.1.m1.2c"><apply id="S3.SS0.SSS0.Px4.1.m1.2.3.cmml" xref="S3.SS0.SSS0.Px4.1.m1.2.3"><times id="S3.SS0.SSS0.Px4.1.m1.2.3.1.cmml" xref="S3.SS0.SSS0.Px4.1.m1.2.3.1"></times><ci id="S3.SS0.SSS0.Px4.1.m1.2.3.2.cmml" xref="S3.SS0.SSS0.Px4.1.m1.2.3.2">𝒇</ci><interval closure="open" id="S3.SS0.SSS0.Px4.1.m1.2.3.3.1.cmml" xref="S3.SS0.SSS0.Px4.1.m1.2.3.3.2"><ci id="S3.SS0.SSS0.Px4.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px4.1.m1.1.1">bold-⋅</ci><ci id="S3.SS0.SSS0.Px4.1.m1.2.2.cmml" xref="S3.SS0.SSS0.Px4.1.m1.2.2">bold-⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.1.m1.2d">\bm{f(\cdot,\cdot)}</annotation></semantics></math>:</h4>

<div id="S3.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px4.p1.2" class="ltx_p">Given the encoding of <math id="S3.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.1.m1.1a"><mi id="S3.SS0.SSS0.Px4.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1.cmml">𝒂</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.1.m1.1b"><ci id="S3.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.1.m1.1.1">𝒂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.1.m1.1c">\bm{a}</annotation></semantics></math> and the lookup tables for <math id="S3.SS0.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="S3.SS0.SSS0.Px4.p1.2.m2.1a"><mi id="S3.SS0.SSS0.Px4.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px4.p1.2.m2.1.1.cmml">𝒃</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px4.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px4.p1.2.m2.1.1">𝒃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px4.p1.2.m2.1c">\bm{b}</annotation></semantics></math>, the product can be approximated as</p>
<table id="A6.EGx6" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E6.m1.7" class="ltx_Math" alttext="\displaystyle\bm{a}^{\top}\bm{b}=\sum_{c=1}^{C}\bm{a}^{(c)\top}\bm{b}^{(c)}\approx\sum_{c=1}^{C}h^{(c)}(\bm{b})_{k},\text{ }k=g^{(c)}(\bm{a})." display="inline"><semantics id="S3.E6.m1.7a"><mrow id="S3.E6.m1.7.7.1"><mrow id="S3.E6.m1.7.7.1.1.2" xref="S3.E6.m1.7.7.1.1.3.cmml"><mrow id="S3.E6.m1.7.7.1.1.1.1" xref="S3.E6.m1.7.7.1.1.1.1.cmml"><mrow id="S3.E6.m1.7.7.1.1.1.1.2" xref="S3.E6.m1.7.7.1.1.1.1.2.cmml"><msup id="S3.E6.m1.7.7.1.1.1.1.2.2" xref="S3.E6.m1.7.7.1.1.1.1.2.2.cmml"><mi id="S3.E6.m1.7.7.1.1.1.1.2.2.2" xref="S3.E6.m1.7.7.1.1.1.1.2.2.2.cmml">𝒂</mi><mo id="S3.E6.m1.7.7.1.1.1.1.2.2.3" xref="S3.E6.m1.7.7.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.7.7.1.1.1.1.2.1" xref="S3.E6.m1.7.7.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E6.m1.7.7.1.1.1.1.2.3" xref="S3.E6.m1.7.7.1.1.1.1.2.3.cmml">𝒃</mi></mrow><mo id="S3.E6.m1.7.7.1.1.1.1.3" xref="S3.E6.m1.7.7.1.1.1.1.3.cmml">=</mo><mrow id="S3.E6.m1.7.7.1.1.1.1.4" xref="S3.E6.m1.7.7.1.1.1.1.4.cmml"><mstyle displaystyle="true" id="S3.E6.m1.7.7.1.1.1.1.4.1" xref="S3.E6.m1.7.7.1.1.1.1.4.1.cmml"><munderover id="S3.E6.m1.7.7.1.1.1.1.4.1a" xref="S3.E6.m1.7.7.1.1.1.1.4.1.cmml"><mo movablelimits="false" id="S3.E6.m1.7.7.1.1.1.1.4.1.2.2" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.2.cmml">∑</mo><mrow id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.cmml"><mi id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.2" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.2.cmml">c</mi><mo id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.1" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.1.cmml">=</mo><mn id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.3" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.3.cmml">1</mn></mrow><mi id="S3.E6.m1.7.7.1.1.1.1.4.1.3" xref="S3.E6.m1.7.7.1.1.1.1.4.1.3.cmml">C</mi></munderover></mstyle><mrow id="S3.E6.m1.7.7.1.1.1.1.4.2" xref="S3.E6.m1.7.7.1.1.1.1.4.2.cmml"><msup id="S3.E6.m1.7.7.1.1.1.1.4.2.2" xref="S3.E6.m1.7.7.1.1.1.1.4.2.2.cmml"><mi id="S3.E6.m1.7.7.1.1.1.1.4.2.2.2" xref="S3.E6.m1.7.7.1.1.1.1.4.2.2.2.cmml">𝒂</mi><mrow id="S3.E6.m1.1.1.1" xref="S3.E6.m1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.3.2.1" xref="S3.E6.m1.1.1.1.cmml">(</mo><mi id="S3.E6.m1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S3.E6.m1.1.1.1.3.2.2" xref="S3.E6.m1.1.1.1.cmml">)</mo></mrow><mo lspace="0.222em" id="S3.E6.m1.1.1.1.4" xref="S3.E6.m1.1.1.1.4.cmml">⊤</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.7.7.1.1.1.1.4.2.1" xref="S3.E6.m1.7.7.1.1.1.1.4.2.1.cmml">​</mo><msup id="S3.E6.m1.7.7.1.1.1.1.4.2.3" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3.cmml"><mi id="S3.E6.m1.7.7.1.1.1.1.4.2.3.2" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3.2.cmml">𝒃</mi><mrow id="S3.E6.m1.2.2.1.3" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3.cmml"><mo stretchy="false" id="S3.E6.m1.2.2.1.3.1" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3.cmml">(</mo><mi id="S3.E6.m1.2.2.1.1" xref="S3.E6.m1.2.2.1.1.cmml">c</mi><mo stretchy="false" id="S3.E6.m1.2.2.1.3.2" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3.cmml">)</mo></mrow></msup></mrow></mrow><mo id="S3.E6.m1.7.7.1.1.1.1.5" xref="S3.E6.m1.7.7.1.1.1.1.5.cmml">≈</mo><mrow id="S3.E6.m1.7.7.1.1.1.1.6" xref="S3.E6.m1.7.7.1.1.1.1.6.cmml"><mstyle displaystyle="true" id="S3.E6.m1.7.7.1.1.1.1.6.1" xref="S3.E6.m1.7.7.1.1.1.1.6.1.cmml"><munderover id="S3.E6.m1.7.7.1.1.1.1.6.1a" xref="S3.E6.m1.7.7.1.1.1.1.6.1.cmml"><mo movablelimits="false" id="S3.E6.m1.7.7.1.1.1.1.6.1.2.2" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.2.cmml">∑</mo><mrow id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.cmml"><mi id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.2" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.2.cmml">c</mi><mo id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.1" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.1.cmml">=</mo><mn id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.3" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.3.cmml">1</mn></mrow><mi id="S3.E6.m1.7.7.1.1.1.1.6.1.3" xref="S3.E6.m1.7.7.1.1.1.1.6.1.3.cmml">C</mi></munderover></mstyle><mrow id="S3.E6.m1.7.7.1.1.1.1.6.2" xref="S3.E6.m1.7.7.1.1.1.1.6.2.cmml"><msup id="S3.E6.m1.7.7.1.1.1.1.6.2.2" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2.cmml"><mi id="S3.E6.m1.7.7.1.1.1.1.6.2.2.2" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2.2.cmml">h</mi><mrow id="S3.E6.m1.3.3.1.3" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2.cmml"><mo stretchy="false" id="S3.E6.m1.3.3.1.3.1" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2.cmml">(</mo><mi id="S3.E6.m1.3.3.1.1" xref="S3.E6.m1.3.3.1.1.cmml">c</mi><mo stretchy="false" id="S3.E6.m1.3.3.1.3.2" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.7.7.1.1.1.1.6.2.1" xref="S3.E6.m1.7.7.1.1.1.1.6.2.1.cmml">​</mo><msub id="S3.E6.m1.7.7.1.1.1.1.6.2.3" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3.cmml"><mrow id="S3.E6.m1.7.7.1.1.1.1.6.2.3.2.2" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3.cmml"><mo stretchy="false" id="S3.E6.m1.7.7.1.1.1.1.6.2.3.2.2.1" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3.cmml">(</mo><mi id="S3.E6.m1.5.5" xref="S3.E6.m1.5.5.cmml">𝒃</mi><mo stretchy="false" id="S3.E6.m1.7.7.1.1.1.1.6.2.3.2.2.2" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3.cmml">)</mo></mrow><mi id="S3.E6.m1.7.7.1.1.1.1.6.2.3.3" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3.3.cmml">k</mi></msub></mrow></mrow></mrow><mo id="S3.E6.m1.7.7.1.1.2.3" xref="S3.E6.m1.7.7.1.1.3a.cmml">,</mo><mrow id="S3.E6.m1.7.7.1.1.2.2" xref="S3.E6.m1.7.7.1.1.2.2.cmml"><mrow id="S3.E6.m1.7.7.1.1.2.2.2" xref="S3.E6.m1.7.7.1.1.2.2.2.cmml"><mtext id="S3.E6.m1.7.7.1.1.2.2.2.2" xref="S3.E6.m1.7.7.1.1.2.2.2.2a.cmml"> </mtext><mo lspace="0em" rspace="0em" id="S3.E6.m1.7.7.1.1.2.2.2.1" xref="S3.E6.m1.7.7.1.1.2.2.2.1.cmml">​</mo><mi id="S3.E6.m1.7.7.1.1.2.2.2.3" xref="S3.E6.m1.7.7.1.1.2.2.2.3.cmml">k</mi></mrow><mo id="S3.E6.m1.7.7.1.1.2.2.1" xref="S3.E6.m1.7.7.1.1.2.2.1.cmml">=</mo><mrow id="S3.E6.m1.7.7.1.1.2.2.3" xref="S3.E6.m1.7.7.1.1.2.2.3.cmml"><msup id="S3.E6.m1.7.7.1.1.2.2.3.2" xref="S3.E6.m1.7.7.1.1.2.2.3.2.cmml"><mi id="S3.E6.m1.7.7.1.1.2.2.3.2.2" xref="S3.E6.m1.7.7.1.1.2.2.3.2.2.cmml">g</mi><mrow id="S3.E6.m1.4.4.1.3" xref="S3.E6.m1.7.7.1.1.2.2.3.2.cmml"><mo stretchy="false" id="S3.E6.m1.4.4.1.3.1" xref="S3.E6.m1.7.7.1.1.2.2.3.2.cmml">(</mo><mi id="S3.E6.m1.4.4.1.1" xref="S3.E6.m1.4.4.1.1.cmml">c</mi><mo stretchy="false" id="S3.E6.m1.4.4.1.3.2" xref="S3.E6.m1.7.7.1.1.2.2.3.2.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.7.7.1.1.2.2.3.1" xref="S3.E6.m1.7.7.1.1.2.2.3.1.cmml">​</mo><mrow id="S3.E6.m1.7.7.1.1.2.2.3.3.2" xref="S3.E6.m1.7.7.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E6.m1.7.7.1.1.2.2.3.3.2.1" xref="S3.E6.m1.7.7.1.1.2.2.3.cmml">(</mo><mi id="S3.E6.m1.6.6" xref="S3.E6.m1.6.6.cmml">𝒂</mi><mo stretchy="false" id="S3.E6.m1.7.7.1.1.2.2.3.3.2.2" xref="S3.E6.m1.7.7.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.E6.m1.7.7.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.7b"><apply id="S3.E6.m1.7.7.1.1.3.cmml" xref="S3.E6.m1.7.7.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.3a.cmml" xref="S3.E6.m1.7.7.1.1.2.3">formulae-sequence</csymbol><apply id="S3.E6.m1.7.7.1.1.1.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1"><and id="S3.E6.m1.7.7.1.1.1.1a.cmml" xref="S3.E6.m1.7.7.1.1.1.1"></and><apply id="S3.E6.m1.7.7.1.1.1.1b.cmml" xref="S3.E6.m1.7.7.1.1.1.1"><eq id="S3.E6.m1.7.7.1.1.1.1.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.3"></eq><apply id="S3.E6.m1.7.7.1.1.1.1.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2"><times id="S3.E6.m1.7.7.1.1.1.1.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2.1"></times><apply id="S3.E6.m1.7.7.1.1.1.1.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.2.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2.2">superscript</csymbol><ci id="S3.E6.m1.7.7.1.1.1.1.2.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2.2.2">𝒂</ci><csymbol cd="latexml" id="S3.E6.m1.7.7.1.1.1.1.2.2.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2.2.3">top</csymbol></apply><ci id="S3.E6.m1.7.7.1.1.1.1.2.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.2.3">𝒃</ci></apply><apply id="S3.E6.m1.7.7.1.1.1.1.4.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4"><apply id="S3.E6.m1.7.7.1.1.1.1.4.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.4.1.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1">superscript</csymbol><apply id="S3.E6.m1.7.7.1.1.1.1.4.1.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.4.1.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1">subscript</csymbol><sum id="S3.E6.m1.7.7.1.1.1.1.4.1.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.2"></sum><apply id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3"><eq id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.1"></eq><ci id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.2">𝑐</ci><cn type="integer" id="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1.2.3.3">1</cn></apply></apply><ci id="S3.E6.m1.7.7.1.1.1.1.4.1.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.1.3">𝐶</ci></apply><apply id="S3.E6.m1.7.7.1.1.1.1.4.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2"><times id="S3.E6.m1.7.7.1.1.1.1.4.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.1"></times><apply id="S3.E6.m1.7.7.1.1.1.1.4.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.4.2.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.2">superscript</csymbol><ci id="S3.E6.m1.7.7.1.1.1.1.4.2.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.2.2">𝒂</ci><apply id="S3.E6.m1.1.1.1.cmml" xref="S3.E6.m1.1.1.1"><csymbol cd="latexml" id="S3.E6.m1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1">limit-from</csymbol><ci id="S3.E6.m1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1">𝑐</ci><csymbol cd="latexml" id="S3.E6.m1.1.1.1.4.cmml" xref="S3.E6.m1.1.1.1.4">top</csymbol></apply></apply><apply id="S3.E6.m1.7.7.1.1.1.1.4.2.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.4.2.3.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3">superscript</csymbol><ci id="S3.E6.m1.7.7.1.1.1.1.4.2.3.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.4.2.3.2">𝒃</ci><ci id="S3.E6.m1.2.2.1.1.cmml" xref="S3.E6.m1.2.2.1.1">𝑐</ci></apply></apply></apply></apply><apply id="S3.E6.m1.7.7.1.1.1.1c.cmml" xref="S3.E6.m1.7.7.1.1.1.1"><approx id="S3.E6.m1.7.7.1.1.1.1.5.cmml" xref="S3.E6.m1.7.7.1.1.1.1.5"></approx><share href="#S3.E6.m1.7.7.1.1.1.1.4.cmml" id="S3.E6.m1.7.7.1.1.1.1d.cmml" xref="S3.E6.m1.7.7.1.1.1.1"></share><apply id="S3.E6.m1.7.7.1.1.1.1.6.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6"><apply id="S3.E6.m1.7.7.1.1.1.1.6.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.6.1.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1">superscript</csymbol><apply id="S3.E6.m1.7.7.1.1.1.1.6.1.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.6.1.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1">subscript</csymbol><sum id="S3.E6.m1.7.7.1.1.1.1.6.1.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.2"></sum><apply id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3"><eq id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.1"></eq><ci id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.2">𝑐</ci><cn type="integer" id="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1.2.3.3">1</cn></apply></apply><ci id="S3.E6.m1.7.7.1.1.1.1.6.1.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.1.3">𝐶</ci></apply><apply id="S3.E6.m1.7.7.1.1.1.1.6.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2"><times id="S3.E6.m1.7.7.1.1.1.1.6.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.1"></times><apply id="S3.E6.m1.7.7.1.1.1.1.6.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.6.2.2.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2">superscript</csymbol><ci id="S3.E6.m1.7.7.1.1.1.1.6.2.2.2.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.2.2">ℎ</ci><ci id="S3.E6.m1.3.3.1.1.cmml" xref="S3.E6.m1.3.3.1.1">𝑐</ci></apply><apply id="S3.E6.m1.7.7.1.1.1.1.6.2.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.1.1.6.2.3.1.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3">subscript</csymbol><ci id="S3.E6.m1.5.5.cmml" xref="S3.E6.m1.5.5">𝒃</ci><ci id="S3.E6.m1.7.7.1.1.1.1.6.2.3.3.cmml" xref="S3.E6.m1.7.7.1.1.1.1.6.2.3.3">𝑘</ci></apply></apply></apply></apply></apply><apply id="S3.E6.m1.7.7.1.1.2.2.cmml" xref="S3.E6.m1.7.7.1.1.2.2"><eq id="S3.E6.m1.7.7.1.1.2.2.1.cmml" xref="S3.E6.m1.7.7.1.1.2.2.1"></eq><apply id="S3.E6.m1.7.7.1.1.2.2.2.cmml" xref="S3.E6.m1.7.7.1.1.2.2.2"><times id="S3.E6.m1.7.7.1.1.2.2.2.1.cmml" xref="S3.E6.m1.7.7.1.1.2.2.2.1"></times><ci id="S3.E6.m1.7.7.1.1.2.2.2.2a.cmml" xref="S3.E6.m1.7.7.1.1.2.2.2.2"><mtext id="S3.E6.m1.7.7.1.1.2.2.2.2.cmml" xref="S3.E6.m1.7.7.1.1.2.2.2.2"> </mtext></ci><ci id="S3.E6.m1.7.7.1.1.2.2.2.3.cmml" xref="S3.E6.m1.7.7.1.1.2.2.2.3">𝑘</ci></apply><apply id="S3.E6.m1.7.7.1.1.2.2.3.cmml" xref="S3.E6.m1.7.7.1.1.2.2.3"><times id="S3.E6.m1.7.7.1.1.2.2.3.1.cmml" xref="S3.E6.m1.7.7.1.1.2.2.3.1"></times><apply id="S3.E6.m1.7.7.1.1.2.2.3.2.cmml" xref="S3.E6.m1.7.7.1.1.2.2.3.2"><csymbol cd="ambiguous" id="S3.E6.m1.7.7.1.1.2.2.3.2.1.cmml" xref="S3.E6.m1.7.7.1.1.2.2.3.2">superscript</csymbol><ci id="S3.E6.m1.7.7.1.1.2.2.3.2.2.cmml" xref="S3.E6.m1.7.7.1.1.2.2.3.2.2">𝑔</ci><ci id="S3.E6.m1.4.4.1.1.cmml" xref="S3.E6.m1.4.4.1.1">𝑐</ci></apply><ci id="S3.E6.m1.6.6.cmml" xref="S3.E6.m1.6.6">𝒂</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.7c">\displaystyle\bm{a}^{\top}\bm{b}=\sum_{c=1}^{C}\bm{a}^{(c)\top}\bm{b}^{(c)}\approx\sum_{c=1}^{C}h^{(c)}(\bm{b})_{k},\text{ }k=g^{(c)}(\bm{a}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Our Method</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.5" class="ltx_p">Product Quantization and its variants yield a large speedup with <math id="S4.p1.1.m1.2" class="ltx_Math" alttext="N,M\gg D" display="inline"><semantics id="S4.p1.1.m1.2a"><mrow id="S4.p1.1.m1.2.3" xref="S4.p1.1.m1.2.3.cmml"><mrow id="S4.p1.1.m1.2.3.2.2" xref="S4.p1.1.m1.2.3.2.1.cmml"><mi id="S4.p1.1.m1.1.1" xref="S4.p1.1.m1.1.1.cmml">N</mi><mo id="S4.p1.1.m1.2.3.2.2.1" xref="S4.p1.1.m1.2.3.2.1.cmml">,</mo><mi id="S4.p1.1.m1.2.2" xref="S4.p1.1.m1.2.2.cmml">M</mi></mrow><mo id="S4.p1.1.m1.2.3.1" xref="S4.p1.1.m1.2.3.1.cmml">≫</mo><mi id="S4.p1.1.m1.2.3.3" xref="S4.p1.1.m1.2.3.3.cmml">D</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.1.m1.2b"><apply id="S4.p1.1.m1.2.3.cmml" xref="S4.p1.1.m1.2.3"><csymbol cd="latexml" id="S4.p1.1.m1.2.3.1.cmml" xref="S4.p1.1.m1.2.3.1">much-greater-than</csymbol><list id="S4.p1.1.m1.2.3.2.1.cmml" xref="S4.p1.1.m1.2.3.2.2"><ci id="S4.p1.1.m1.1.1.cmml" xref="S4.p1.1.m1.1.1">𝑁</ci><ci id="S4.p1.1.m1.2.2.cmml" xref="S4.p1.1.m1.2.2">𝑀</ci></list><ci id="S4.p1.1.m1.2.3.3.cmml" xref="S4.p1.1.m1.2.3.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.1.m1.2c">N,M\gg D</annotation></semantics></math>. However, we require an algorithm that only needs <math id="S4.p1.2.m2.2" class="ltx_Math" alttext="N\gg M,D" display="inline"><semantics id="S4.p1.2.m2.2a"><mrow id="S4.p1.2.m2.2.3" xref="S4.p1.2.m2.2.3.cmml"><mi id="S4.p1.2.m2.2.3.2" xref="S4.p1.2.m2.2.3.2.cmml">N</mi><mo id="S4.p1.2.m2.2.3.1" xref="S4.p1.2.m2.2.3.1.cmml">≫</mo><mrow id="S4.p1.2.m2.2.3.3.2" xref="S4.p1.2.m2.2.3.3.1.cmml"><mi id="S4.p1.2.m2.1.1" xref="S4.p1.2.m2.1.1.cmml">M</mi><mo id="S4.p1.2.m2.2.3.3.2.1" xref="S4.p1.2.m2.2.3.3.1.cmml">,</mo><mi id="S4.p1.2.m2.2.2" xref="S4.p1.2.m2.2.2.cmml">D</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.2.m2.2b"><apply id="S4.p1.2.m2.2.3.cmml" xref="S4.p1.2.m2.2.3"><csymbol cd="latexml" id="S4.p1.2.m2.2.3.1.cmml" xref="S4.p1.2.m2.2.3.1">much-greater-than</csymbol><ci id="S4.p1.2.m2.2.3.2.cmml" xref="S4.p1.2.m2.2.3.2">𝑁</ci><list id="S4.p1.2.m2.2.3.3.1.cmml" xref="S4.p1.2.m2.2.3.3.2"><ci id="S4.p1.2.m2.1.1.cmml" xref="S4.p1.2.m2.1.1">𝑀</ci><ci id="S4.p1.2.m2.2.2.cmml" xref="S4.p1.2.m2.2.2">𝐷</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.2.m2.2c">N\gg M,D</annotation></semantics></math>, a more relaxed scenario common when using linear models and transforms. In this setting, the preprocessing time <math id="S4.p1.3.m3.1" class="ltx_Math" alttext="g(\bm{A})" display="inline"><semantics id="S4.p1.3.m3.1a"><mrow id="S4.p1.3.m3.1.2" xref="S4.p1.3.m3.1.2.cmml"><mi id="S4.p1.3.m3.1.2.2" xref="S4.p1.3.m3.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S4.p1.3.m3.1.2.1" xref="S4.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S4.p1.3.m3.1.2.3.2" xref="S4.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S4.p1.3.m3.1.2.3.2.1" xref="S4.p1.3.m3.1.2.cmml">(</mo><mi id="S4.p1.3.m3.1.1" xref="S4.p1.3.m3.1.1.cmml">𝑨</mi><mo stretchy="false" id="S4.p1.3.m3.1.2.3.2.2" xref="S4.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.3.m3.1b"><apply id="S4.p1.3.m3.1.2.cmml" xref="S4.p1.3.m3.1.2"><times id="S4.p1.3.m3.1.2.1.cmml" xref="S4.p1.3.m3.1.2.1"></times><ci id="S4.p1.3.m3.1.2.2.cmml" xref="S4.p1.3.m3.1.2.2">𝑔</ci><ci id="S4.p1.3.m3.1.1.cmml" xref="S4.p1.3.m3.1.1">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.3.m3.1c">g(\bm{A})</annotation></semantics></math> can be significant, since <math id="S4.p1.4.m4.1" class="ltx_Math" alttext="ND" display="inline"><semantics id="S4.p1.4.m4.1a"><mrow id="S4.p1.4.m4.1.1" xref="S4.p1.4.m4.1.1.cmml"><mi id="S4.p1.4.m4.1.1.2" xref="S4.p1.4.m4.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S4.p1.4.m4.1.1.1" xref="S4.p1.4.m4.1.1.1.cmml">​</mo><mi id="S4.p1.4.m4.1.1.3" xref="S4.p1.4.m4.1.1.3.cmml">D</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.4.m4.1b"><apply id="S4.p1.4.m4.1.1.cmml" xref="S4.p1.4.m4.1.1"><times id="S4.p1.4.m4.1.1.1.cmml" xref="S4.p1.4.m4.1.1.1"></times><ci id="S4.p1.4.m4.1.1.2.cmml" xref="S4.p1.4.m4.1.1.2">𝑁</ci><ci id="S4.p1.4.m4.1.1.3.cmml" xref="S4.p1.4.m4.1.1.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.4.m4.1c">ND</annotation></semantics></math> may be similar to (or even larger than) <math id="S4.p1.5.m5.1" class="ltx_Math" alttext="NM" display="inline"><semantics id="S4.p1.5.m5.1a"><mrow id="S4.p1.5.m5.1.1" xref="S4.p1.5.m5.1.1.cmml"><mi id="S4.p1.5.m5.1.1.2" xref="S4.p1.5.m5.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S4.p1.5.m5.1.1.1" xref="S4.p1.5.m5.1.1.1.cmml">​</mo><mi id="S4.p1.5.m5.1.1.3" xref="S4.p1.5.m5.1.1.3.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.5.m5.1b"><apply id="S4.p1.5.m5.1.1.cmml" xref="S4.p1.5.m5.1.1"><times id="S4.p1.5.m5.1.1.1.cmml" xref="S4.p1.5.m5.1.1.1"></times><ci id="S4.p1.5.m5.1.1.2.cmml" xref="S4.p1.5.m5.1.1.2">𝑁</ci><ci id="S4.p1.5.m5.1.1.3.cmml" xref="S4.p1.5.m5.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.5.m5.1c">NM</annotation></semantics></math>.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.4" class="ltx_p">To address this case, we introduce a new <math id="S4.p2.1.m1.1" class="ltx_Math" alttext="g(\bm{A})" display="inline"><semantics id="S4.p2.1.m1.1a"><mrow id="S4.p2.1.m1.1.2" xref="S4.p2.1.m1.1.2.cmml"><mi id="S4.p2.1.m1.1.2.2" xref="S4.p2.1.m1.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S4.p2.1.m1.1.2.1" xref="S4.p2.1.m1.1.2.1.cmml">​</mo><mrow id="S4.p2.1.m1.1.2.3.2" xref="S4.p2.1.m1.1.2.cmml"><mo stretchy="false" id="S4.p2.1.m1.1.2.3.2.1" xref="S4.p2.1.m1.1.2.cmml">(</mo><mi id="S4.p2.1.m1.1.1" xref="S4.p2.1.m1.1.1.cmml">𝑨</mi><mo stretchy="false" id="S4.p2.1.m1.1.2.3.2.2" xref="S4.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.1.m1.1b"><apply id="S4.p2.1.m1.1.2.cmml" xref="S4.p2.1.m1.1.2"><times id="S4.p2.1.m1.1.2.1.cmml" xref="S4.p2.1.m1.1.2.1"></times><ci id="S4.p2.1.m1.1.2.2.cmml" xref="S4.p2.1.m1.1.2.2">𝑔</ci><ci id="S4.p2.1.m1.1.1.cmml" xref="S4.p2.1.m1.1.1">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.1.m1.1c">g(\bm{A})</annotation></semantics></math> function that yields large speedups even on much smaller matrices. The main idea behind our function is to determine the “most similar” prototype through locality-sensitive hashing <cite class="ltx_cite ltx_citemacro_citep">(Indyk &amp; Motwani, <a href="#bib.bib39" title="" class="ltx_ref">1998</a>)</cite>; i.e., rather than compute the Euclidean distance between a subvector <math id="S4.p2.2.m2.1" class="ltx_Math" alttext="\bm{a}^{(c)}" display="inline"><semantics id="S4.p2.2.m2.1a"><msup id="S4.p2.2.m2.1.2" xref="S4.p2.2.m2.1.2.cmml"><mi id="S4.p2.2.m2.1.2.2" xref="S4.p2.2.m2.1.2.2.cmml">𝒂</mi><mrow id="S4.p2.2.m2.1.1.1.3" xref="S4.p2.2.m2.1.2.cmml"><mo stretchy="false" id="S4.p2.2.m2.1.1.1.3.1" xref="S4.p2.2.m2.1.2.cmml">(</mo><mi id="S4.p2.2.m2.1.1.1.1" xref="S4.p2.2.m2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S4.p2.2.m2.1.1.1.3.2" xref="S4.p2.2.m2.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.p2.2.m2.1b"><apply id="S4.p2.2.m2.1.2.cmml" xref="S4.p2.2.m2.1.2"><csymbol cd="ambiguous" id="S4.p2.2.m2.1.2.1.cmml" xref="S4.p2.2.m2.1.2">superscript</csymbol><ci id="S4.p2.2.m2.1.2.2.cmml" xref="S4.p2.2.m2.1.2.2">𝒂</ci><ci id="S4.p2.2.m2.1.1.1.1.cmml" xref="S4.p2.2.m2.1.1.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.2.m2.1c">\bm{a}^{(c)}</annotation></semantics></math> and each prototype, we hash <math id="S4.p2.3.m3.1" class="ltx_Math" alttext="\bm{a}^{(c)}" display="inline"><semantics id="S4.p2.3.m3.1a"><msup id="S4.p2.3.m3.1.2" xref="S4.p2.3.m3.1.2.cmml"><mi id="S4.p2.3.m3.1.2.2" xref="S4.p2.3.m3.1.2.2.cmml">𝒂</mi><mrow id="S4.p2.3.m3.1.1.1.3" xref="S4.p2.3.m3.1.2.cmml"><mo stretchy="false" id="S4.p2.3.m3.1.1.1.3.1" xref="S4.p2.3.m3.1.2.cmml">(</mo><mi id="S4.p2.3.m3.1.1.1.1" xref="S4.p2.3.m3.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S4.p2.3.m3.1.1.1.3.2" xref="S4.p2.3.m3.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.p2.3.m3.1b"><apply id="S4.p2.3.m3.1.2.cmml" xref="S4.p2.3.m3.1.2"><csymbol cd="ambiguous" id="S4.p2.3.m3.1.2.1.cmml" xref="S4.p2.3.m3.1.2">superscript</csymbol><ci id="S4.p2.3.m3.1.2.2.cmml" xref="S4.p2.3.m3.1.2.2">𝒂</ci><ci id="S4.p2.3.m3.1.1.1.1.cmml" xref="S4.p2.3.m3.1.1.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.3.m3.1c">\bm{a}^{(c)}</annotation></semantics></math> to one of <math id="S4.p2.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.p2.4.m4.1a"><mi id="S4.p2.4.m4.1.1" xref="S4.p2.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.p2.4.m4.1b"><ci id="S4.p2.4.m4.1.1.cmml" xref="S4.p2.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.4.m4.1c">K</annotation></semantics></math> buckets where similar subvectors tend to hash to the same bucket. The prototypes are set to the means of the subvectors hashing to each bucket.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Hash Function Family, <math id="S4.SS1.1.m1.1" class="ltx_Math" alttext="\bm{g(\cdot)}" display="inline"><semantics id="S4.SS1.1.m1.1b"><mrow id="S4.SS1.1.m1.1.2" xref="S4.SS1.1.m1.1.2.cmml"><mi id="S4.SS1.1.m1.1.2.2" xref="S4.SS1.1.m1.1.2.2.cmml">𝒈</mi><mo lspace="0em" rspace="0em" id="S4.SS1.1.m1.1.2.1" xref="S4.SS1.1.m1.1.2.1.cmml">​</mo><mrow id="S4.SS1.1.m1.1.2.3.2" xref="S4.SS1.1.m1.1.2.cmml"><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S4.SS1.1.m1.1.2.3.2.1" xref="S4.SS1.1.m1.1.2.cmml">(</mo><mo class="ltx_mathvariant_bold" lspace="0em" mathvariant="bold" rspace="0em" id="S4.SS1.1.m1.1.1" xref="S4.SS1.1.m1.1.1.cmml">⋅</mo><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S4.SS1.1.m1.1.2.3.2.2" xref="S4.SS1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.1.m1.1c"><apply id="S4.SS1.1.m1.1.2.cmml" xref="S4.SS1.1.m1.1.2"><times id="S4.SS1.1.m1.1.2.1.cmml" xref="S4.SS1.1.m1.1.2.1"></times><ci id="S4.SS1.1.m1.1.2.2.cmml" xref="S4.SS1.1.m1.1.2.2">𝒈</ci><ci id="S4.SS1.1.m1.1.1.cmml" xref="S4.SS1.1.m1.1.1">bold-⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.1.m1.1d">\bm{g(\cdot)}</annotation></semantics></math>
</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.5" class="ltx_p">Because we seek to exploit a training set while also doing far less work than even a single linear transform, we found that existing hash functions did not meet our requirements. Consequently, we designed our own family of trainable hash functions.
The family of hash functions we choose is balanced binary regression trees, with each leaf of the tree acting as one hash bucket. The leaf for a vector <math id="S4.SS1.p1.1.m1.1" class="ltx_Math" alttext="\bm{x}" display="inline"><semantics id="S4.SS1.p1.1.m1.1a"><mi id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml">𝒙</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><ci id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1">𝒙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">\bm{x}</annotation></semantics></math> is chosen by traversing the tree from the root and moving to the left child if the value <math id="S4.SS1.p1.2.m2.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="S4.SS1.p1.2.m2.1a"><msub id="S4.SS1.p1.2.m2.1.1" xref="S4.SS1.p1.2.m2.1.1.cmml"><mi id="S4.SS1.p1.2.m2.1.1.2" xref="S4.SS1.p1.2.m2.1.1.2.cmml">x</mi><mi id="S4.SS1.p1.2.m2.1.1.3" xref="S4.SS1.p1.2.m2.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.2.m2.1b"><apply id="S4.SS1.p1.2.m2.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.2.m2.1.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS1.p1.2.m2.1.1.2.cmml" xref="S4.SS1.p1.2.m2.1.1.2">𝑥</ci><ci id="S4.SS1.p1.2.m2.1.1.3.cmml" xref="S4.SS1.p1.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.2.m2.1c">x_{j}</annotation></semantics></math> at some index <math id="S4.SS1.p1.3.m3.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S4.SS1.p1.3.m3.1a"><mi id="S4.SS1.p1.3.m3.1.1" xref="S4.SS1.p1.3.m3.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.3.m3.1b"><ci id="S4.SS1.p1.3.m3.1.1.cmml" xref="S4.SS1.p1.3.m3.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">j</annotation></semantics></math> is below a node-specific threshold <math id="S4.SS1.p1.4.m4.1" class="ltx_Math" alttext="v" display="inline"><semantics id="S4.SS1.p1.4.m4.1a"><mi id="S4.SS1.p1.4.m4.1.1" xref="S4.SS1.p1.4.m4.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.4.m4.1b"><ci id="S4.SS1.p1.4.m4.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.4.m4.1c">v</annotation></semantics></math>, and to the right child otherwise. To enable the use of SIMD instructions, the tree is limited to 16 leaves and all nodes at a given level of the tree are required to split on the same index <math id="S4.SS1.p1.5.m5.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S4.SS1.p1.5.m5.1a"><mi id="S4.SS1.p1.5.m5.1.1" xref="S4.SS1.p1.5.m5.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.5.m5.1b"><ci id="S4.SS1.p1.5.m5.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.5.m5.1c">j</annotation></semantics></math>. The number 16 holds across many processor architectures, and we refer the reader to Appendix <a href="#A2" title="Appendix B Quantization and MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> for further vectorization details.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.5" class="ltx_p">Formally, consider a set of four indices <math id="S4.SS1.p2.1.m1.3" class="ltx_Math" alttext="{j^{1},\ldots,j^{4}}" display="inline"><semantics id="S4.SS1.p2.1.m1.3a"><mrow id="S4.SS1.p2.1.m1.3.3.2" xref="S4.SS1.p2.1.m1.3.3.3.cmml"><msup id="S4.SS1.p2.1.m1.2.2.1.1" xref="S4.SS1.p2.1.m1.2.2.1.1.cmml"><mi id="S4.SS1.p2.1.m1.2.2.1.1.2" xref="S4.SS1.p2.1.m1.2.2.1.1.2.cmml">j</mi><mn id="S4.SS1.p2.1.m1.2.2.1.1.3" xref="S4.SS1.p2.1.m1.2.2.1.1.3.cmml">1</mn></msup><mo id="S4.SS1.p2.1.m1.3.3.2.3" xref="S4.SS1.p2.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml">…</mi><mo id="S4.SS1.p2.1.m1.3.3.2.4" xref="S4.SS1.p2.1.m1.3.3.3.cmml">,</mo><msup id="S4.SS1.p2.1.m1.3.3.2.2" xref="S4.SS1.p2.1.m1.3.3.2.2.cmml"><mi id="S4.SS1.p2.1.m1.3.3.2.2.2" xref="S4.SS1.p2.1.m1.3.3.2.2.2.cmml">j</mi><mn id="S4.SS1.p2.1.m1.3.3.2.2.3" xref="S4.SS1.p2.1.m1.3.3.2.2.3.cmml">4</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.3b"><list id="S4.SS1.p2.1.m1.3.3.3.cmml" xref="S4.SS1.p2.1.m1.3.3.2"><apply id="S4.SS1.p2.1.m1.2.2.1.1.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.1.m1.2.2.1.1.1.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1">superscript</csymbol><ci id="S4.SS1.p2.1.m1.2.2.1.1.2.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1.2">𝑗</ci><cn type="integer" id="S4.SS1.p2.1.m1.2.2.1.1.3.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1">…</ci><apply id="S4.SS1.p2.1.m1.3.3.2.2.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS1.p2.1.m1.3.3.2.2.1.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2">superscript</csymbol><ci id="S4.SS1.p2.1.m1.3.3.2.2.2.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2.2">𝑗</ci><cn type="integer" id="S4.SS1.p2.1.m1.3.3.2.2.3.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2.3">4</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.3c">{j^{1},\ldots,j^{4}}</annotation></semantics></math> and four arrays of split thresholds <math id="S4.SS1.p2.2.m2.3" class="ltx_Math" alttext="\bm{v}^{1},\ldots,\bm{v}^{4}" display="inline"><semantics id="S4.SS1.p2.2.m2.3a"><mrow id="S4.SS1.p2.2.m2.3.3.2" xref="S4.SS1.p2.2.m2.3.3.3.cmml"><msup id="S4.SS1.p2.2.m2.2.2.1.1" xref="S4.SS1.p2.2.m2.2.2.1.1.cmml"><mi id="S4.SS1.p2.2.m2.2.2.1.1.2" xref="S4.SS1.p2.2.m2.2.2.1.1.2.cmml">𝒗</mi><mn id="S4.SS1.p2.2.m2.2.2.1.1.3" xref="S4.SS1.p2.2.m2.2.2.1.1.3.cmml">1</mn></msup><mo id="S4.SS1.p2.2.m2.3.3.2.3" xref="S4.SS1.p2.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS1.p2.2.m2.1.1" xref="S4.SS1.p2.2.m2.1.1.cmml">…</mi><mo id="S4.SS1.p2.2.m2.3.3.2.4" xref="S4.SS1.p2.2.m2.3.3.3.cmml">,</mo><msup id="S4.SS1.p2.2.m2.3.3.2.2" xref="S4.SS1.p2.2.m2.3.3.2.2.cmml"><mi id="S4.SS1.p2.2.m2.3.3.2.2.2" xref="S4.SS1.p2.2.m2.3.3.2.2.2.cmml">𝒗</mi><mn id="S4.SS1.p2.2.m2.3.3.2.2.3" xref="S4.SS1.p2.2.m2.3.3.2.2.3.cmml">4</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.2.m2.3b"><list id="S4.SS1.p2.2.m2.3.3.3.cmml" xref="S4.SS1.p2.2.m2.3.3.2"><apply id="S4.SS1.p2.2.m2.2.2.1.1.cmml" xref="S4.SS1.p2.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.2.2.1.1.1.cmml" xref="S4.SS1.p2.2.m2.2.2.1.1">superscript</csymbol><ci id="S4.SS1.p2.2.m2.2.2.1.1.2.cmml" xref="S4.SS1.p2.2.m2.2.2.1.1.2">𝒗</ci><cn type="integer" id="S4.SS1.p2.2.m2.2.2.1.1.3.cmml" xref="S4.SS1.p2.2.m2.2.2.1.1.3">1</cn></apply><ci id="S4.SS1.p2.2.m2.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1">…</ci><apply id="S4.SS1.p2.2.m2.3.3.2.2.cmml" xref="S4.SS1.p2.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.3.3.2.2.1.cmml" xref="S4.SS1.p2.2.m2.3.3.2.2">superscript</csymbol><ci id="S4.SS1.p2.2.m2.3.3.2.2.2.cmml" xref="S4.SS1.p2.2.m2.3.3.2.2.2">𝒗</ci><cn type="integer" id="S4.SS1.p2.2.m2.3.3.2.2.3.cmml" xref="S4.SS1.p2.2.m2.3.3.2.2.3">4</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.2.m2.3c">\bm{v}^{1},\ldots,\bm{v}^{4}</annotation></semantics></math>, with <math id="S4.SS1.p2.3.m3.1" class="ltx_Math" alttext="v_{t}" display="inline"><semantics id="S4.SS1.p2.3.m3.1a"><msub id="S4.SS1.p2.3.m3.1.1" xref="S4.SS1.p2.3.m3.1.1.cmml"><mi id="S4.SS1.p2.3.m3.1.1.2" xref="S4.SS1.p2.3.m3.1.1.2.cmml">v</mi><mi id="S4.SS1.p2.3.m3.1.1.3" xref="S4.SS1.p2.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.3.m3.1b"><apply id="S4.SS1.p2.3.m3.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.3.m3.1.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.p2.3.m3.1.1.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2">𝑣</ci><ci id="S4.SS1.p2.3.m3.1.1.3.cmml" xref="S4.SS1.p2.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.3.m3.1c">v_{t}</annotation></semantics></math> having length <math id="S4.SS1.p2.4.m4.1" class="ltx_Math" alttext="2^{t-1}" display="inline"><semantics id="S4.SS1.p2.4.m4.1a"><msup id="S4.SS1.p2.4.m4.1.1" xref="S4.SS1.p2.4.m4.1.1.cmml"><mn id="S4.SS1.p2.4.m4.1.1.2" xref="S4.SS1.p2.4.m4.1.1.2.cmml">2</mn><mrow id="S4.SS1.p2.4.m4.1.1.3" xref="S4.SS1.p2.4.m4.1.1.3.cmml"><mi id="S4.SS1.p2.4.m4.1.1.3.2" xref="S4.SS1.p2.4.m4.1.1.3.2.cmml">t</mi><mo id="S4.SS1.p2.4.m4.1.1.3.1" xref="S4.SS1.p2.4.m4.1.1.3.1.cmml">−</mo><mn id="S4.SS1.p2.4.m4.1.1.3.3" xref="S4.SS1.p2.4.m4.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.4.m4.1b"><apply id="S4.SS1.p2.4.m4.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.4.m4.1.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1">superscript</csymbol><cn type="integer" id="S4.SS1.p2.4.m4.1.1.2.cmml" xref="S4.SS1.p2.4.m4.1.1.2">2</cn><apply id="S4.SS1.p2.4.m4.1.1.3.cmml" xref="S4.SS1.p2.4.m4.1.1.3"><minus id="S4.SS1.p2.4.m4.1.1.3.1.cmml" xref="S4.SS1.p2.4.m4.1.1.3.1"></minus><ci id="S4.SS1.p2.4.m4.1.1.3.2.cmml" xref="S4.SS1.p2.4.m4.1.1.3.2">𝑡</ci><cn type="integer" id="S4.SS1.p2.4.m4.1.1.3.3.cmml" xref="S4.SS1.p2.4.m4.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.4.m4.1c">2^{t-1}</annotation></semantics></math>. A vector <math id="S4.SS1.p2.5.m5.1" class="ltx_Math" alttext="\bm{x}" display="inline"><semantics id="S4.SS1.p2.5.m5.1a"><mi id="S4.SS1.p2.5.m5.1.1" xref="S4.SS1.p2.5.m5.1.1.cmml">𝒙</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.5.m5.1b"><ci id="S4.SS1.p2.5.m5.1.1.cmml" xref="S4.SS1.p2.5.m5.1.1">𝒙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.5.m5.1c">\bm{x}</annotation></semantics></math> is mapped to an index using Algorithm <a href="#alg1" title="Algorithm 1 ‣ 4.1 Hash Function Family, 𝒈⁢(⋅) ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.3.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> <span id="alg1.4.2" class="ltx_text ltx_font_smallcaps">MaddnessHash</span></figcaption>
<div id="alg1.5" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span>  <span id="alg1.l1.2" class="ltx_text ltx_font_bold">Input:</span> vector <math id="alg1.l1.m1.1" class="ltx_Math" alttext="\bm{x}" display="inline"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">𝒙</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝒙</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">\bm{x}</annotation></semantics></math>, split indices <math id="alg1.l1.m2.3" class="ltx_Math" alttext="{j^{1},\ldots,j^{4}}" display="inline"><semantics id="alg1.l1.m2.3a"><mrow id="alg1.l1.m2.3.3.2" xref="alg1.l1.m2.3.3.3.cmml"><msup id="alg1.l1.m2.2.2.1.1" xref="alg1.l1.m2.2.2.1.1.cmml"><mi id="alg1.l1.m2.2.2.1.1.2" xref="alg1.l1.m2.2.2.1.1.2.cmml">j</mi><mn id="alg1.l1.m2.2.2.1.1.3" xref="alg1.l1.m2.2.2.1.1.3.cmml">1</mn></msup><mo id="alg1.l1.m2.3.3.2.3" xref="alg1.l1.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml">…</mi><mo id="alg1.l1.m2.3.3.2.4" xref="alg1.l1.m2.3.3.3.cmml">,</mo><msup id="alg1.l1.m2.3.3.2.2" xref="alg1.l1.m2.3.3.2.2.cmml"><mi id="alg1.l1.m2.3.3.2.2.2" xref="alg1.l1.m2.3.3.2.2.2.cmml">j</mi><mn id="alg1.l1.m2.3.3.2.2.3" xref="alg1.l1.m2.3.3.2.2.3.cmml">4</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.3b"><list id="alg1.l1.m2.3.3.3.cmml" xref="alg1.l1.m2.3.3.2"><apply id="alg1.l1.m2.2.2.1.1.cmml" xref="alg1.l1.m2.2.2.1.1"><csymbol cd="ambiguous" id="alg1.l1.m2.2.2.1.1.1.cmml" xref="alg1.l1.m2.2.2.1.1">superscript</csymbol><ci id="alg1.l1.m2.2.2.1.1.2.cmml" xref="alg1.l1.m2.2.2.1.1.2">𝑗</ci><cn type="integer" id="alg1.l1.m2.2.2.1.1.3.cmml" xref="alg1.l1.m2.2.2.1.1.3">1</cn></apply><ci id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1">…</ci><apply id="alg1.l1.m2.3.3.2.2.cmml" xref="alg1.l1.m2.3.3.2.2"><csymbol cd="ambiguous" id="alg1.l1.m2.3.3.2.2.1.cmml" xref="alg1.l1.m2.3.3.2.2">superscript</csymbol><ci id="alg1.l1.m2.3.3.2.2.2.cmml" xref="alg1.l1.m2.3.3.2.2.2">𝑗</ci><cn type="integer" id="alg1.l1.m2.3.3.2.2.3.cmml" xref="alg1.l1.m2.3.3.2.2.3">4</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.3c">{j^{1},\ldots,j^{4}}</annotation></semantics></math>, split thresholds <math id="alg1.l1.m3.3" class="ltx_Math" alttext="\bm{v}^{1},\ldots,\bm{v}^{4}" display="inline"><semantics id="alg1.l1.m3.3a"><mrow id="alg1.l1.m3.3.3.2" xref="alg1.l1.m3.3.3.3.cmml"><msup id="alg1.l1.m3.2.2.1.1" xref="alg1.l1.m3.2.2.1.1.cmml"><mi id="alg1.l1.m3.2.2.1.1.2" xref="alg1.l1.m3.2.2.1.1.2.cmml">𝒗</mi><mn id="alg1.l1.m3.2.2.1.1.3" xref="alg1.l1.m3.2.2.1.1.3.cmml">1</mn></msup><mo id="alg1.l1.m3.3.3.2.3" xref="alg1.l1.m3.3.3.3.cmml">,</mo><mi mathvariant="normal" id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml">…</mi><mo id="alg1.l1.m3.3.3.2.4" xref="alg1.l1.m3.3.3.3.cmml">,</mo><msup id="alg1.l1.m3.3.3.2.2" xref="alg1.l1.m3.3.3.2.2.cmml"><mi id="alg1.l1.m3.3.3.2.2.2" xref="alg1.l1.m3.3.3.2.2.2.cmml">𝒗</mi><mn id="alg1.l1.m3.3.3.2.2.3" xref="alg1.l1.m3.3.3.2.2.3.cmml">4</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.3b"><list id="alg1.l1.m3.3.3.3.cmml" xref="alg1.l1.m3.3.3.2"><apply id="alg1.l1.m3.2.2.1.1.cmml" xref="alg1.l1.m3.2.2.1.1"><csymbol cd="ambiguous" id="alg1.l1.m3.2.2.1.1.1.cmml" xref="alg1.l1.m3.2.2.1.1">superscript</csymbol><ci id="alg1.l1.m3.2.2.1.1.2.cmml" xref="alg1.l1.m3.2.2.1.1.2">𝒗</ci><cn type="integer" id="alg1.l1.m3.2.2.1.1.3.cmml" xref="alg1.l1.m3.2.2.1.1.3">1</cn></apply><ci id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1">…</ci><apply id="alg1.l1.m3.3.3.2.2.cmml" xref="alg1.l1.m3.3.3.2.2"><csymbol cd="ambiguous" id="alg1.l1.m3.3.3.2.2.1.cmml" xref="alg1.l1.m3.3.3.2.2">superscript</csymbol><ci id="alg1.l1.m3.3.3.2.2.2.cmml" xref="alg1.l1.m3.3.3.2.2.2">𝒗</ci><cn type="integer" id="alg1.l1.m3.3.3.2.2.3.cmml" xref="alg1.l1.m3.3.3.2.2.3">4</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.3c">\bm{v}^{1},\ldots,\bm{v}^{4}</annotation></semantics></math>

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>  <math id="alg1.l2.m1.1" class="ltx_Math" alttext="i\leftarrow 1" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml">i</mi><mo stretchy="false" id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">←</mo><mn id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><ci id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1">←</ci><ci id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2">𝑖</ci><cn type="integer" id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">i\leftarrow 1</annotation></semantics></math> 
<span id="alg1.l2.2" class="ltx_text" style="color:#009900;">
<span id="alg1.l2.2.1" class="ltx_text" style="float:right;">// node index within level of tree</span></span>


</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>  <span id="alg1.l3.2" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l3.m1.1" class="ltx_Math" alttext="t\leftarrow 1\textbf{ to }4" display="inline"><semantics id="alg1.l3.m1.1a"><mrow id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml"><mi id="alg1.l3.m1.1.1.2" xref="alg1.l3.m1.1.1.2.cmml">t</mi><mo stretchy="false" id="alg1.l3.m1.1.1.1" xref="alg1.l3.m1.1.1.1.cmml">←</mo><mrow id="alg1.l3.m1.1.1.3" xref="alg1.l3.m1.1.1.3.cmml"><mn id="alg1.l3.m1.1.1.3.2" xref="alg1.l3.m1.1.1.3.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg1.l3.m1.1.1.3.1" xref="alg1.l3.m1.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg1.l3.m1.1.1.3.3" xref="alg1.l3.m1.1.1.3.3a.cmml"> to </mtext><mo lspace="0em" rspace="0em" id="alg1.l3.m1.1.1.3.1a" xref="alg1.l3.m1.1.1.3.1.cmml">​</mo><mn id="alg1.l3.m1.1.1.3.4" xref="alg1.l3.m1.1.1.3.4.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><apply id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1"><ci id="alg1.l3.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1">←</ci><ci id="alg1.l3.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.2">𝑡</ci><apply id="alg1.l3.m1.1.1.3.cmml" xref="alg1.l3.m1.1.1.3"><times id="alg1.l3.m1.1.1.3.1.cmml" xref="alg1.l3.m1.1.1.3.1"></times><cn type="integer" id="alg1.l3.m1.1.1.3.2.cmml" xref="alg1.l3.m1.1.1.3.2">1</cn><ci id="alg1.l3.m1.1.1.3.3a.cmml" xref="alg1.l3.m1.1.1.3.3"><mtext class="ltx_mathvariant_bold" id="alg1.l3.m1.1.1.3.3.cmml" xref="alg1.l3.m1.1.1.3.3"> to </mtext></ci><cn type="integer" id="alg1.l3.m1.1.1.3.4.cmml" xref="alg1.l3.m1.1.1.3.4">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">t\leftarrow 1\textbf{ to }4</annotation></semantics></math> <span id="alg1.l3.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l4.3.1.1" class="ltx_text" style="font-size:80%;">4:</span></span>     <math id="alg1.l4.m1.1" class="ltx_Math" alttext="v\leftarrow\bm{v}^{t}_{i}" display="inline"><semantics id="alg1.l4.m1.1a"><mrow id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mi id="alg1.l4.m1.1.1.2" xref="alg1.l4.m1.1.1.2.cmml">v</mi><mo stretchy="false" id="alg1.l4.m1.1.1.1" xref="alg1.l4.m1.1.1.1.cmml">←</mo><msubsup id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml"><mi id="alg1.l4.m1.1.1.3.2.2" xref="alg1.l4.m1.1.1.3.2.2.cmml">𝒗</mi><mi id="alg1.l4.m1.1.1.3.3" xref="alg1.l4.m1.1.1.3.3.cmml">i</mi><mi id="alg1.l4.m1.1.1.3.2.3" xref="alg1.l4.m1.1.1.3.2.3.cmml">t</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><ci id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1">←</ci><ci id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1.2">𝑣</ci><apply id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.3.1.cmml" xref="alg1.l4.m1.1.1.3">subscript</csymbol><apply id="alg1.l4.m1.1.1.3.2.cmml" xref="alg1.l4.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.3.2.1.cmml" xref="alg1.l4.m1.1.1.3">superscript</csymbol><ci id="alg1.l4.m1.1.1.3.2.2.cmml" xref="alg1.l4.m1.1.1.3.2.2">𝒗</ci><ci id="alg1.l4.m1.1.1.3.2.3.cmml" xref="alg1.l4.m1.1.1.3.2.3">𝑡</ci></apply><ci id="alg1.l4.m1.1.1.3.3.cmml" xref="alg1.l4.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">v\leftarrow\bm{v}^{t}_{i}</annotation></semantics></math> 
<span id="alg1.l4.2" class="ltx_text" style="color:#009900;">
<span id="alg1.l4.2.2" class="ltx_text" style="float:right;">// lookup split threshold for node <math id="alg1.l4.1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg1.l4.1.1.m1.1a"><mi mathcolor="#009900" id="alg1.l4.1.1.m1.1.1" xref="alg1.l4.1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg1.l4.1.1.m1.1b"><ci id="alg1.l4.1.1.m1.1.1.cmml" xref="alg1.l4.1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.1.1.m1.1c">i</annotation></semantics></math> at level <math id="alg1.l4.2.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="alg1.l4.2.2.m2.1a"><mi mathcolor="#009900" id="alg1.l4.2.2.m2.1.1" xref="alg1.l4.2.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="alg1.l4.2.2.m2.1b"><ci id="alg1.l4.2.2.m2.1.1.cmml" xref="alg1.l4.2.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.2.2.m2.1c">t</annotation></semantics></math></span></span>


</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>     <math id="alg1.l5.m1.1" class="ltx_Math" alttext="b\leftarrow x_{j^{t}}\geq v\text{ ? }1\text{ : }0" display="inline"><semantics id="alg1.l5.m1.1a"><mrow id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml"><mi id="alg1.l5.m1.1.1.2" xref="alg1.l5.m1.1.1.2.cmml">b</mi><mo stretchy="false" id="alg1.l5.m1.1.1.3" xref="alg1.l5.m1.1.1.3.cmml">←</mo><msub id="alg1.l5.m1.1.1.4" xref="alg1.l5.m1.1.1.4.cmml"><mi id="alg1.l5.m1.1.1.4.2" xref="alg1.l5.m1.1.1.4.2.cmml">x</mi><msup id="alg1.l5.m1.1.1.4.3" xref="alg1.l5.m1.1.1.4.3.cmml"><mi id="alg1.l5.m1.1.1.4.3.2" xref="alg1.l5.m1.1.1.4.3.2.cmml">j</mi><mi id="alg1.l5.m1.1.1.4.3.3" xref="alg1.l5.m1.1.1.4.3.3.cmml">t</mi></msup></msub><mo id="alg1.l5.m1.1.1.5" xref="alg1.l5.m1.1.1.5.cmml">≥</mo><mrow id="alg1.l5.m1.1.1.6" xref="alg1.l5.m1.1.1.6.cmml"><mi id="alg1.l5.m1.1.1.6.2" xref="alg1.l5.m1.1.1.6.2.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg1.l5.m1.1.1.6.1" xref="alg1.l5.m1.1.1.6.1.cmml">​</mo><mtext id="alg1.l5.m1.1.1.6.3" xref="alg1.l5.m1.1.1.6.3a.cmml"> ? </mtext><mo lspace="0em" rspace="0em" id="alg1.l5.m1.1.1.6.1a" xref="alg1.l5.m1.1.1.6.1.cmml">​</mo><mn id="alg1.l5.m1.1.1.6.4" xref="alg1.l5.m1.1.1.6.4.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg1.l5.m1.1.1.6.1b" xref="alg1.l5.m1.1.1.6.1.cmml">​</mo><mtext id="alg1.l5.m1.1.1.6.5" xref="alg1.l5.m1.1.1.6.5a.cmml"> : </mtext><mo lspace="0em" rspace="0em" id="alg1.l5.m1.1.1.6.1c" xref="alg1.l5.m1.1.1.6.1.cmml">​</mo><mn id="alg1.l5.m1.1.1.6.6" xref="alg1.l5.m1.1.1.6.6.cmml">0</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1"><and id="alg1.l5.m1.1.1a.cmml" xref="alg1.l5.m1.1.1"></and><apply id="alg1.l5.m1.1.1b.cmml" xref="alg1.l5.m1.1.1"><ci id="alg1.l5.m1.1.1.3.cmml" xref="alg1.l5.m1.1.1.3">←</ci><ci id="alg1.l5.m1.1.1.2.cmml" xref="alg1.l5.m1.1.1.2">𝑏</ci><apply id="alg1.l5.m1.1.1.4.cmml" xref="alg1.l5.m1.1.1.4"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.4.1.cmml" xref="alg1.l5.m1.1.1.4">subscript</csymbol><ci id="alg1.l5.m1.1.1.4.2.cmml" xref="alg1.l5.m1.1.1.4.2">𝑥</ci><apply id="alg1.l5.m1.1.1.4.3.cmml" xref="alg1.l5.m1.1.1.4.3"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.4.3.1.cmml" xref="alg1.l5.m1.1.1.4.3">superscript</csymbol><ci id="alg1.l5.m1.1.1.4.3.2.cmml" xref="alg1.l5.m1.1.1.4.3.2">𝑗</ci><ci id="alg1.l5.m1.1.1.4.3.3.cmml" xref="alg1.l5.m1.1.1.4.3.3">𝑡</ci></apply></apply></apply><apply id="alg1.l5.m1.1.1c.cmml" xref="alg1.l5.m1.1.1"><geq id="alg1.l5.m1.1.1.5.cmml" xref="alg1.l5.m1.1.1.5"></geq><share href="#alg1.l5.m1.1.1.4.cmml" id="alg1.l5.m1.1.1d.cmml" xref="alg1.l5.m1.1.1"></share><apply id="alg1.l5.m1.1.1.6.cmml" xref="alg1.l5.m1.1.1.6"><times id="alg1.l5.m1.1.1.6.1.cmml" xref="alg1.l5.m1.1.1.6.1"></times><ci id="alg1.l5.m1.1.1.6.2.cmml" xref="alg1.l5.m1.1.1.6.2">𝑣</ci><ci id="alg1.l5.m1.1.1.6.3a.cmml" xref="alg1.l5.m1.1.1.6.3"><mtext id="alg1.l5.m1.1.1.6.3.cmml" xref="alg1.l5.m1.1.1.6.3"> ? </mtext></ci><cn type="integer" id="alg1.l5.m1.1.1.6.4.cmml" xref="alg1.l5.m1.1.1.6.4">1</cn><ci id="alg1.l5.m1.1.1.6.5a.cmml" xref="alg1.l5.m1.1.1.6.5"><mtext id="alg1.l5.m1.1.1.6.5.cmml" xref="alg1.l5.m1.1.1.6.5"> : </mtext></ci><cn type="integer" id="alg1.l5.m1.1.1.6.6.cmml" xref="alg1.l5.m1.1.1.6.6">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">b\leftarrow x_{j^{t}}\geq v\text{ ? }1\text{ : }0</annotation></semantics></math> 
<span id="alg1.l5.2" class="ltx_text" style="color:#009900;">
<span id="alg1.l5.2.1" class="ltx_text" style="float:right;">// above split threshold?</span></span>


</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>     <math id="alg1.l6.m1.1" class="ltx_Math" alttext="i\leftarrow 2i-1+b" display="inline"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><mi id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml">i</mi><mo stretchy="false" id="alg1.l6.m1.1.1.1" xref="alg1.l6.m1.1.1.1.cmml">←</mo><mrow id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml"><mrow id="alg1.l6.m1.1.1.3.2" xref="alg1.l6.m1.1.1.3.2.cmml"><mrow id="alg1.l6.m1.1.1.3.2.2" xref="alg1.l6.m1.1.1.3.2.2.cmml"><mn id="alg1.l6.m1.1.1.3.2.2.2" xref="alg1.l6.m1.1.1.3.2.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.2.2.1" xref="alg1.l6.m1.1.1.3.2.2.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.2.2.3" xref="alg1.l6.m1.1.1.3.2.2.3.cmml">i</mi></mrow><mo id="alg1.l6.m1.1.1.3.2.1" xref="alg1.l6.m1.1.1.3.2.1.cmml">−</mo><mn id="alg1.l6.m1.1.1.3.2.3" xref="alg1.l6.m1.1.1.3.2.3.cmml">1</mn></mrow><mo id="alg1.l6.m1.1.1.3.1" xref="alg1.l6.m1.1.1.3.1.cmml">+</mo><mi id="alg1.l6.m1.1.1.3.3" xref="alg1.l6.m1.1.1.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><ci id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1">←</ci><ci id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2">𝑖</ci><apply id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3"><plus id="alg1.l6.m1.1.1.3.1.cmml" xref="alg1.l6.m1.1.1.3.1"></plus><apply id="alg1.l6.m1.1.1.3.2.cmml" xref="alg1.l6.m1.1.1.3.2"><minus id="alg1.l6.m1.1.1.3.2.1.cmml" xref="alg1.l6.m1.1.1.3.2.1"></minus><apply id="alg1.l6.m1.1.1.3.2.2.cmml" xref="alg1.l6.m1.1.1.3.2.2"><times id="alg1.l6.m1.1.1.3.2.2.1.cmml" xref="alg1.l6.m1.1.1.3.2.2.1"></times><cn type="integer" id="alg1.l6.m1.1.1.3.2.2.2.cmml" xref="alg1.l6.m1.1.1.3.2.2.2">2</cn><ci id="alg1.l6.m1.1.1.3.2.2.3.cmml" xref="alg1.l6.m1.1.1.3.2.2.3">𝑖</ci></apply><cn type="integer" id="alg1.l6.m1.1.1.3.2.3.cmml" xref="alg1.l6.m1.1.1.3.2.3">1</cn></apply><ci id="alg1.l6.m1.1.1.3.3.cmml" xref="alg1.l6.m1.1.1.3.3">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">i\leftarrow 2i-1+b</annotation></semantics></math> 
<span id="alg1.l6.2" class="ltx_text" style="color:#009900;">
<span id="alg1.l6.2.1" class="ltx_text" style="float:right;">// assign to left or right child</span></span>


</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>  <span id="alg1.l7.2" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l7.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>   <span id="alg1.l8.2" class="ltx_text ltx_font_bold">return</span> <math id="alg1.l8.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg1.l8.m1.1a"><mi id="alg1.l8.m1.1.1" xref="alg1.l8.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.1b"><ci id="alg1.l8.m1.1.1.cmml" xref="alg1.l8.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.1c">i</annotation></semantics></math>

</div>
</div>
<br class="ltx_break ltx_break">
</figure>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p">This function is simple, only depends on a constant number of indices in the input vector, and can easily be vectorized provided that the matrix whose rows are being encoded is stored in column-major order. </p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Learning the Hash Function Parameters</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.7" class="ltx_p">The split indices <math id="S4.SS2.p1.1.m1.3" class="ltx_Math" alttext="{j^{1},\ldots,j^{4}}" display="inline"><semantics id="S4.SS2.p1.1.m1.3a"><mrow id="S4.SS2.p1.1.m1.3.3.2" xref="S4.SS2.p1.1.m1.3.3.3.cmml"><msup id="S4.SS2.p1.1.m1.2.2.1.1" xref="S4.SS2.p1.1.m1.2.2.1.1.cmml"><mi id="S4.SS2.p1.1.m1.2.2.1.1.2" xref="S4.SS2.p1.1.m1.2.2.1.1.2.cmml">j</mi><mn id="S4.SS2.p1.1.m1.2.2.1.1.3" xref="S4.SS2.p1.1.m1.2.2.1.1.3.cmml">1</mn></msup><mo id="S4.SS2.p1.1.m1.3.3.2.3" xref="S4.SS2.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml">…</mi><mo id="S4.SS2.p1.1.m1.3.3.2.4" xref="S4.SS2.p1.1.m1.3.3.3.cmml">,</mo><msup id="S4.SS2.p1.1.m1.3.3.2.2" xref="S4.SS2.p1.1.m1.3.3.2.2.cmml"><mi id="S4.SS2.p1.1.m1.3.3.2.2.2" xref="S4.SS2.p1.1.m1.3.3.2.2.2.cmml">j</mi><mn id="S4.SS2.p1.1.m1.3.3.2.2.3" xref="S4.SS2.p1.1.m1.3.3.2.2.3.cmml">4</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.3b"><list id="S4.SS2.p1.1.m1.3.3.3.cmml" xref="S4.SS2.p1.1.m1.3.3.2"><apply id="S4.SS2.p1.1.m1.2.2.1.1.cmml" xref="S4.SS2.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.1.m1.2.2.1.1.1.cmml" xref="S4.SS2.p1.1.m1.2.2.1.1">superscript</csymbol><ci id="S4.SS2.p1.1.m1.2.2.1.1.2.cmml" xref="S4.SS2.p1.1.m1.2.2.1.1.2">𝑗</ci><cn type="integer" id="S4.SS2.p1.1.m1.2.2.1.1.3.cmml" xref="S4.SS2.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">…</ci><apply id="S4.SS2.p1.1.m1.3.3.2.2.cmml" xref="S4.SS2.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.1.m1.3.3.2.2.1.cmml" xref="S4.SS2.p1.1.m1.3.3.2.2">superscript</csymbol><ci id="S4.SS2.p1.1.m1.3.3.2.2.2.cmml" xref="S4.SS2.p1.1.m1.3.3.2.2.2">𝑗</ci><cn type="integer" id="S4.SS2.p1.1.m1.3.3.2.2.3.cmml" xref="S4.SS2.p1.1.m1.3.3.2.2.3">4</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.3c">{j^{1},\ldots,j^{4}}</annotation></semantics></math> and split thresholds <math id="S4.SS2.p1.2.m2.3" class="ltx_Math" alttext="\bm{v}^{1},\ldots,\bm{v}^{4}" display="inline"><semantics id="S4.SS2.p1.2.m2.3a"><mrow id="S4.SS2.p1.2.m2.3.3.2" xref="S4.SS2.p1.2.m2.3.3.3.cmml"><msup id="S4.SS2.p1.2.m2.2.2.1.1" xref="S4.SS2.p1.2.m2.2.2.1.1.cmml"><mi id="S4.SS2.p1.2.m2.2.2.1.1.2" xref="S4.SS2.p1.2.m2.2.2.1.1.2.cmml">𝒗</mi><mn id="S4.SS2.p1.2.m2.2.2.1.1.3" xref="S4.SS2.p1.2.m2.2.2.1.1.3.cmml">1</mn></msup><mo id="S4.SS2.p1.2.m2.3.3.2.3" xref="S4.SS2.p1.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.p1.2.m2.1.1" xref="S4.SS2.p1.2.m2.1.1.cmml">…</mi><mo id="S4.SS2.p1.2.m2.3.3.2.4" xref="S4.SS2.p1.2.m2.3.3.3.cmml">,</mo><msup id="S4.SS2.p1.2.m2.3.3.2.2" xref="S4.SS2.p1.2.m2.3.3.2.2.cmml"><mi id="S4.SS2.p1.2.m2.3.3.2.2.2" xref="S4.SS2.p1.2.m2.3.3.2.2.2.cmml">𝒗</mi><mn id="S4.SS2.p1.2.m2.3.3.2.2.3" xref="S4.SS2.p1.2.m2.3.3.2.2.3.cmml">4</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.2.m2.3b"><list id="S4.SS2.p1.2.m2.3.3.3.cmml" xref="S4.SS2.p1.2.m2.3.3.2"><apply id="S4.SS2.p1.2.m2.2.2.1.1.cmml" xref="S4.SS2.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.2.2.1.1.1.cmml" xref="S4.SS2.p1.2.m2.2.2.1.1">superscript</csymbol><ci id="S4.SS2.p1.2.m2.2.2.1.1.2.cmml" xref="S4.SS2.p1.2.m2.2.2.1.1.2">𝒗</ci><cn type="integer" id="S4.SS2.p1.2.m2.2.2.1.1.3.cmml" xref="S4.SS2.p1.2.m2.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.p1.2.m2.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1">…</ci><apply id="S4.SS2.p1.2.m2.3.3.2.2.cmml" xref="S4.SS2.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.3.3.2.2.1.cmml" xref="S4.SS2.p1.2.m2.3.3.2.2">superscript</csymbol><ci id="S4.SS2.p1.2.m2.3.3.2.2.2.cmml" xref="S4.SS2.p1.2.m2.3.3.2.2.2">𝒗</ci><cn type="integer" id="S4.SS2.p1.2.m2.3.3.2.2.3.cmml" xref="S4.SS2.p1.2.m2.3.3.2.2.3">4</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.2.m2.3c">\bm{v}^{1},\ldots,\bm{v}^{4}</annotation></semantics></math> are optimized on the training matrix <math id="S4.SS2.p1.3.m3.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S4.SS2.p1.3.m3.1a"><mover accent="true" id="S4.SS2.p1.3.m3.1.1" xref="S4.SS2.p1.3.m3.1.1.cmml"><mi id="S4.SS2.p1.3.m3.1.1.2" xref="S4.SS2.p1.3.m3.1.1.2.cmml">𝑨</mi><mo id="S4.SS2.p1.3.m3.1.1.1" xref="S4.SS2.p1.3.m3.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.3.m3.1b"><apply id="S4.SS2.p1.3.m3.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1"><ci id="S4.SS2.p1.3.m3.1.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1.1">~</ci><ci id="S4.SS2.p1.3.m3.1.1.2.cmml" xref="S4.SS2.p1.3.m3.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.3.m3.1c">\tilde{\bm{A}}</annotation></semantics></math> using a greedy tree construction algorithm.
To describe this algorithm, we introduce the notion of a <span id="S4.SS2.p1.7.1" class="ltx_text ltx_font_italic">bucket</span> <math id="S4.SS2.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{B}^{t}_{i}" display="inline"><semantics id="S4.SS2.p1.4.m4.1a"><msubsup id="S4.SS2.p1.4.m4.1.1" xref="S4.SS2.p1.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.4.m4.1.1.2.2" xref="S4.SS2.p1.4.m4.1.1.2.2.cmml">ℬ</mi><mi id="S4.SS2.p1.4.m4.1.1.3" xref="S4.SS2.p1.4.m4.1.1.3.cmml">i</mi><mi id="S4.SS2.p1.4.m4.1.1.2.3" xref="S4.SS2.p1.4.m4.1.1.2.3.cmml">t</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.4.m4.1b"><apply id="S4.SS2.p1.4.m4.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1">subscript</csymbol><apply id="S4.SS2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.4.m4.1.1.2.1.cmml" xref="S4.SS2.p1.4.m4.1.1">superscript</csymbol><ci id="S4.SS2.p1.4.m4.1.1.2.2.cmml" xref="S4.SS2.p1.4.m4.1.1.2.2">ℬ</ci><ci id="S4.SS2.p1.4.m4.1.1.2.3.cmml" xref="S4.SS2.p1.4.m4.1.1.2.3">𝑡</ci></apply><ci id="S4.SS2.p1.4.m4.1.1.3.cmml" xref="S4.SS2.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.4.m4.1c">\mathcal{B}^{t}_{i}</annotation></semantics></math>, which is the set of vectors mapped to node <math id="S4.SS2.p1.5.m5.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S4.SS2.p1.5.m5.1a"><mi id="S4.SS2.p1.5.m5.1.1" xref="S4.SS2.p1.5.m5.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.5.m5.1b"><ci id="S4.SS2.p1.5.m5.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.5.m5.1c">i</annotation></semantics></math> in level <math id="S4.SS2.p1.6.m6.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S4.SS2.p1.6.m6.1a"><mi id="S4.SS2.p1.6.m6.1.1" xref="S4.SS2.p1.6.m6.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.6.m6.1b"><ci id="S4.SS2.p1.6.m6.1.1.cmml" xref="S4.SS2.p1.6.m6.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.6.m6.1c">t</annotation></semantics></math> of the tree. The root of the tree is level 0 and <math id="S4.SS2.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{B}^{0}_{1}" display="inline"><semantics id="S4.SS2.p1.7.m7.1a"><msubsup id="S4.SS2.p1.7.m7.1.1" xref="S4.SS2.p1.7.m7.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.7.m7.1.1.2.2" xref="S4.SS2.p1.7.m7.1.1.2.2.cmml">ℬ</mi><mn id="S4.SS2.p1.7.m7.1.1.3" xref="S4.SS2.p1.7.m7.1.1.3.cmml">1</mn><mn id="S4.SS2.p1.7.m7.1.1.2.3" xref="S4.SS2.p1.7.m7.1.1.2.3.cmml">0</mn></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.7.m7.1b"><apply id="S4.SS2.p1.7.m7.1.1.cmml" xref="S4.SS2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.7.m7.1.1.1.cmml" xref="S4.SS2.p1.7.m7.1.1">subscript</csymbol><apply id="S4.SS2.p1.7.m7.1.1.2.cmml" xref="S4.SS2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.7.m7.1.1.2.1.cmml" xref="S4.SS2.p1.7.m7.1.1">superscript</csymbol><ci id="S4.SS2.p1.7.m7.1.1.2.2.cmml" xref="S4.SS2.p1.7.m7.1.1.2.2">ℬ</ci><cn type="integer" id="S4.SS2.p1.7.m7.1.1.2.3.cmml" xref="S4.SS2.p1.7.m7.1.1.2.3">0</cn></apply><cn type="integer" id="S4.SS2.p1.7.m7.1.1.3.cmml" xref="S4.SS2.p1.7.m7.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.7.m7.1c">\mathcal{B}^{0}_{1}</annotation></semantics></math> contains all the vectors. It will also be helpful to define the sum of squared errors (SSE) loss associated with a bucket, or a specific (index, bucket) pair:</p>
<table id="S4.E7" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S4.E7X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E7X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}(j\text{, }\mathcal{B})" display="inline"><semantics id="S4.E7X.2.1.1.m1.1a"><mrow id="S4.E7X.2.1.1.m1.1.1" xref="S4.E7X.2.1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E7X.2.1.1.m1.1.1.3" xref="S4.E7X.2.1.1.m1.1.1.3.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S4.E7X.2.1.1.m1.1.1.2" xref="S4.E7X.2.1.1.m1.1.1.2.cmml">​</mo><mrow id="S4.E7X.2.1.1.m1.1.1.1.1" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E7X.2.1.1.m1.1.1.1.1.2" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E7X.2.1.1.m1.1.1.1.1.1" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.cmml"><mi id="S4.E7X.2.1.1.m1.1.1.1.1.1.2" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.2.cmml">j</mi><mo lspace="0em" rspace="0em" id="S4.E7X.2.1.1.m1.1.1.1.1.1.1" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.1.cmml">​</mo><mtext id="S4.E7X.2.1.1.m1.1.1.1.1.1.3" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="S4.E7X.2.1.1.m1.1.1.1.1.1.1a" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S4.E7X.2.1.1.m1.1.1.1.1.1.4" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.4.cmml">ℬ</mi></mrow><mo stretchy="false" id="S4.E7X.2.1.1.m1.1.1.1.1.3" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E7X.2.1.1.m1.1b"><apply id="S4.E7X.2.1.1.m1.1.1.cmml" xref="S4.E7X.2.1.1.m1.1.1"><times id="S4.E7X.2.1.1.m1.1.1.2.cmml" xref="S4.E7X.2.1.1.m1.1.1.2"></times><ci id="S4.E7X.2.1.1.m1.1.1.3.cmml" xref="S4.E7X.2.1.1.m1.1.1.3">ℒ</ci><apply id="S4.E7X.2.1.1.m1.1.1.1.1.1.cmml" xref="S4.E7X.2.1.1.m1.1.1.1.1"><times id="S4.E7X.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.1"></times><ci id="S4.E7X.2.1.1.m1.1.1.1.1.1.2.cmml" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.2">𝑗</ci><ci id="S4.E7X.2.1.1.m1.1.1.1.1.1.3a.cmml" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.3"><mtext id="S4.E7X.2.1.1.m1.1.1.1.1.1.3.cmml" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.3">, </mtext></ci><ci id="S4.E7X.2.1.1.m1.1.1.1.1.1.4.cmml" xref="S4.E7X.2.1.1.m1.1.1.1.1.1.4">ℬ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E7X.2.1.1.m1.1c">\displaystyle\mathcal{L}(j\text{, }\mathcal{B})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S4.E7X.3.2.2.m1.2" class="ltx_Math" alttext="\displaystyle\triangleq\sum_{\bm{x}\in\mathcal{B}}\left(x_{j}-\frac{1}{|\mathcal{B}|}\sum_{\bm{x}^{\prime}\in\mathcal{B}}x^{\prime}_{j}\right)^{2}" display="inline"><semantics id="S4.E7X.3.2.2.m1.2a"><mrow id="S4.E7X.3.2.2.m1.2.2" xref="S4.E7X.3.2.2.m1.2.2.cmml"><mi id="S4.E7X.3.2.2.m1.2.2.3" xref="S4.E7X.3.2.2.m1.2.2.3.cmml"></mi><mo id="S4.E7X.3.2.2.m1.2.2.2" xref="S4.E7X.3.2.2.m1.2.2.2.cmml">≜</mo><mrow id="S4.E7X.3.2.2.m1.2.2.1" xref="S4.E7X.3.2.2.m1.2.2.1.cmml"><mstyle displaystyle="true" id="S4.E7X.3.2.2.m1.2.2.1.2" xref="S4.E7X.3.2.2.m1.2.2.1.2.cmml"><munder id="S4.E7X.3.2.2.m1.2.2.1.2a" xref="S4.E7X.3.2.2.m1.2.2.1.2.cmml"><mo movablelimits="false" id="S4.E7X.3.2.2.m1.2.2.1.2.2" xref="S4.E7X.3.2.2.m1.2.2.1.2.2.cmml">∑</mo><mrow id="S4.E7X.3.2.2.m1.2.2.1.2.3" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.cmml"><mi id="S4.E7X.3.2.2.m1.2.2.1.2.3.2" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.2.cmml">𝒙</mi><mo id="S4.E7X.3.2.2.m1.2.2.1.2.3.1" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S4.E7X.3.2.2.m1.2.2.1.2.3.3" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.3.cmml">ℬ</mi></mrow></munder></mstyle><msup id="S4.E7X.3.2.2.m1.2.2.1.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.cmml"><mrow id="S4.E7X.3.2.2.m1.2.2.1.1.1.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.cmml"><mo id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.cmml"><msub id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.cmml"><mi id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.2.cmml">x</mi><mi id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.3.cmml">j</mi></msub><mo id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.1.cmml">−</mo><mrow id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.cmml"><mstyle displaystyle="true" id="S4.E7X.3.2.2.m1.1.1" xref="S4.E7X.3.2.2.m1.1.1.cmml"><mfrac id="S4.E7X.3.2.2.m1.1.1a" xref="S4.E7X.3.2.2.m1.1.1.cmml"><mn id="S4.E7X.3.2.2.m1.1.1.3" xref="S4.E7X.3.2.2.m1.1.1.3.cmml">1</mn><mrow id="S4.E7X.3.2.2.m1.1.1.1.3" xref="S4.E7X.3.2.2.m1.1.1.1.2.cmml"><mo stretchy="false" id="S4.E7X.3.2.2.m1.1.1.1.3.1" xref="S4.E7X.3.2.2.m1.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S4.E7X.3.2.2.m1.1.1.1.1" xref="S4.E7X.3.2.2.m1.1.1.1.1.cmml">ℬ</mi><mo stretchy="false" id="S4.E7X.3.2.2.m1.1.1.1.3.2" xref="S4.E7X.3.2.2.m1.1.1.1.2.1.cmml">|</mo></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.1.cmml">​</mo><mrow id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.cmml"><mstyle displaystyle="true" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.cmml"><munder id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1a" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.cmml"><mo movablelimits="false" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.2.cmml">∑</mo><mrow id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.cmml"><msup id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.cmml"><mi id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.2.cmml">𝒙</mi><mo id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.3.cmml">′</mo></msup><mo id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.1" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.3.cmml">ℬ</mi></mrow></munder></mstyle><msubsup id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.cmml"><mi id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.2" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.2.cmml">x</mi><mi id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.3.cmml">j</mi><mo id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.3.cmml">′</mo></msubsup></mrow></mrow></mrow><mo id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow><mn id="S4.E7X.3.2.2.m1.2.2.1.1.3" xref="S4.E7X.3.2.2.m1.2.2.1.1.3.cmml">2</mn></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E7X.3.2.2.m1.2b"><apply id="S4.E7X.3.2.2.m1.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2"><ci id="S4.E7X.3.2.2.m1.2.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.2">≜</ci><csymbol cd="latexml" id="S4.E7X.3.2.2.m1.2.2.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.3">absent</csymbol><apply id="S4.E7X.3.2.2.m1.2.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1"><apply id="S4.E7X.3.2.2.m1.2.2.1.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2">subscript</csymbol><sum id="S4.E7X.3.2.2.m1.2.2.1.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2.2"></sum><apply id="S4.E7X.3.2.2.m1.2.2.1.2.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2.3"><in id="S4.E7X.3.2.2.m1.2.2.1.2.3.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.1"></in><ci id="S4.E7X.3.2.2.m1.2.2.1.2.3.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.2">𝒙</ci><ci id="S4.E7X.3.2.2.m1.2.2.1.2.3.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.2.3.3">ℬ</ci></apply></apply><apply id="S4.E7X.3.2.2.m1.2.2.1.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.1.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1">superscript</csymbol><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1"><minus id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.1"></minus><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.2">𝑥</ci><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.2.3">𝑗</ci></apply><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3"><times id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.1"></times><apply id="S4.E7X.3.2.2.m1.1.1.cmml" xref="S4.E7X.3.2.2.m1.1.1"><divide id="S4.E7X.3.2.2.m1.1.1.2.cmml" xref="S4.E7X.3.2.2.m1.1.1"></divide><cn type="integer" id="S4.E7X.3.2.2.m1.1.1.3.cmml" xref="S4.E7X.3.2.2.m1.1.1.3">1</cn><apply id="S4.E7X.3.2.2.m1.1.1.1.2.cmml" xref="S4.E7X.3.2.2.m1.1.1.1.3"><abs id="S4.E7X.3.2.2.m1.1.1.1.2.1.cmml" xref="S4.E7X.3.2.2.m1.1.1.1.3.1"></abs><ci id="S4.E7X.3.2.2.m1.1.1.1.1.cmml" xref="S4.E7X.3.2.2.m1.1.1.1.1">ℬ</ci></apply></apply><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2"><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1">subscript</csymbol><sum id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.2"></sum><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3"><in id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.1"></in><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2">superscript</csymbol><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.2">𝒙</ci><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.2.3">′</ci></apply><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.1.3.3">ℬ</ci></apply></apply><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2">subscript</csymbol><apply id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2"><csymbol cd="ambiguous" id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.1.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2">superscript</csymbol><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.2.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.2">𝑥</ci><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.2.3">′</ci></apply><ci id="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.1.1.1.3.2.2.3">𝑗</ci></apply></apply></apply></apply><cn type="integer" id="S4.E7X.3.2.2.m1.2.2.1.1.3.cmml" xref="S4.E7X.3.2.2.m1.2.2.1.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E7X.3.2.2.m1.2c">\displaystyle\triangleq\sum_{\bm{x}\in\mathcal{B}}\left(x_{j}-\frac{1}{|\mathcal{B}|}\sum_{\bm{x}^{\prime}\in\mathcal{B}}x^{\prime}_{j}\right)^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(7)</span></td>
</tr>
<tr id="S4.E7Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E7Xa.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}(\mathcal{B})" display="inline"><semantics id="S4.E7Xa.2.1.1.m1.1a"><mrow id="S4.E7Xa.2.1.1.m1.1.2" xref="S4.E7Xa.2.1.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E7Xa.2.1.1.m1.1.2.2" xref="S4.E7Xa.2.1.1.m1.1.2.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S4.E7Xa.2.1.1.m1.1.2.1" xref="S4.E7Xa.2.1.1.m1.1.2.1.cmml">​</mo><mrow id="S4.E7Xa.2.1.1.m1.1.2.3.2" xref="S4.E7Xa.2.1.1.m1.1.2.cmml"><mo stretchy="false" id="S4.E7Xa.2.1.1.m1.1.2.3.2.1" xref="S4.E7Xa.2.1.1.m1.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S4.E7Xa.2.1.1.m1.1.1" xref="S4.E7Xa.2.1.1.m1.1.1.cmml">ℬ</mi><mo stretchy="false" id="S4.E7Xa.2.1.1.m1.1.2.3.2.2" xref="S4.E7Xa.2.1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E7Xa.2.1.1.m1.1b"><apply id="S4.E7Xa.2.1.1.m1.1.2.cmml" xref="S4.E7Xa.2.1.1.m1.1.2"><times id="S4.E7Xa.2.1.1.m1.1.2.1.cmml" xref="S4.E7Xa.2.1.1.m1.1.2.1"></times><ci id="S4.E7Xa.2.1.1.m1.1.2.2.cmml" xref="S4.E7Xa.2.1.1.m1.1.2.2">ℒ</ci><ci id="S4.E7Xa.2.1.1.m1.1.1.cmml" xref="S4.E7Xa.2.1.1.m1.1.1">ℬ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E7Xa.2.1.1.m1.1c">\displaystyle\mathcal{L}(\mathcal{B})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S4.E7Xa.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle\triangleq\sum_{j}\mathcal{L}(j\text{, }\mathcal{B})." display="inline"><semantics id="S4.E7Xa.3.2.2.m1.1a"><mrow id="S4.E7Xa.3.2.2.m1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.cmml"><mrow id="S4.E7Xa.3.2.2.m1.1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.cmml"><mi id="S4.E7Xa.3.2.2.m1.1.1.1.1.3" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.3.cmml"></mi><mo id="S4.E7Xa.3.2.2.m1.1.1.1.1.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.2.cmml">≜</mo><mrow id="S4.E7Xa.3.2.2.m1.1.1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.cmml"><munder id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2a" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.2.cmml">∑</mo><mi id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.3" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.3.cmml">j</mi></munder></mstyle><mrow id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.3" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.3.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.2.cmml">j</mi><mo lspace="0em" rspace="0em" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mtext id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1a" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.4" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.4.cmml">ℬ</mi></mrow><mo stretchy="false" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.3" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S4.E7Xa.3.2.2.m1.1.1.1.2" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E7Xa.3.2.2.m1.1b"><apply id="S4.E7Xa.3.2.2.m1.1.1.1.1.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1"><ci id="S4.E7Xa.3.2.2.m1.1.1.1.1.2.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.2">≜</ci><csymbol cd="latexml" id="S4.E7Xa.3.2.2.m1.1.1.1.1.3.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.3">absent</csymbol><apply id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1"><apply id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.1.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2">subscript</csymbol><sum id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.2.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.2"></sum><ci id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.3.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.2.3">𝑗</ci></apply><apply id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1"><times id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.2.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.2"></times><ci id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.3.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.3">ℒ</ci><apply id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1"><times id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.1"></times><ci id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.2">𝑗</ci><ci id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3a.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3"><mtext id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.3">, </mtext></ci><ci id="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.4.cmml" xref="S4.E7Xa.3.2.2.m1.1.1.1.1.1.1.1.1.1.4">ℬ</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E7Xa.3.2.2.m1.1c">\displaystyle\triangleq\sum_{j}\mathcal{L}(j\text{, }\mathcal{B}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S4.SS2.p1.9" class="ltx_p">Using this notation, it suffices to characterize the learning algorithm by describing the construction of level <math id="S4.SS2.p1.8.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S4.SS2.p1.8.m1.1a"><mi id="S4.SS2.p1.8.m1.1.1" xref="S4.SS2.p1.8.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.8.m1.1b"><ci id="S4.SS2.p1.8.m1.1.1.cmml" xref="S4.SS2.p1.8.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.8.m1.1c">t</annotation></semantics></math> of the tree given the buckets <math id="S4.SS2.p1.9.m2.3" class="ltx_Math" alttext="\mathcal{B}^{t-1}_{1},\ldots,\mathcal{B}^{t-1}_{2^{t-1}}" display="inline"><semantics id="S4.SS2.p1.9.m2.3a"><mrow id="S4.SS2.p1.9.m2.3.3.2" xref="S4.SS2.p1.9.m2.3.3.3.cmml"><msubsup id="S4.SS2.p1.9.m2.2.2.1.1" xref="S4.SS2.p1.9.m2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.9.m2.2.2.1.1.2.2" xref="S4.SS2.p1.9.m2.2.2.1.1.2.2.cmml">ℬ</mi><mn id="S4.SS2.p1.9.m2.2.2.1.1.3" xref="S4.SS2.p1.9.m2.2.2.1.1.3.cmml">1</mn><mrow id="S4.SS2.p1.9.m2.2.2.1.1.2.3" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.cmml"><mi id="S4.SS2.p1.9.m2.2.2.1.1.2.3.2" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.2.cmml">t</mi><mo id="S4.SS2.p1.9.m2.2.2.1.1.2.3.1" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.1.cmml">−</mo><mn id="S4.SS2.p1.9.m2.2.2.1.1.2.3.3" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mo id="S4.SS2.p1.9.m2.3.3.2.3" xref="S4.SS2.p1.9.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.p1.9.m2.1.1" xref="S4.SS2.p1.9.m2.1.1.cmml">…</mi><mo id="S4.SS2.p1.9.m2.3.3.2.4" xref="S4.SS2.p1.9.m2.3.3.3.cmml">,</mo><msubsup id="S4.SS2.p1.9.m2.3.3.2.2" xref="S4.SS2.p1.9.m2.3.3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.9.m2.3.3.2.2.2.2" xref="S4.SS2.p1.9.m2.3.3.2.2.2.2.cmml">ℬ</mi><msup id="S4.SS2.p1.9.m2.3.3.2.2.3" xref="S4.SS2.p1.9.m2.3.3.2.2.3.cmml"><mn id="S4.SS2.p1.9.m2.3.3.2.2.3.2" xref="S4.SS2.p1.9.m2.3.3.2.2.3.2.cmml">2</mn><mrow id="S4.SS2.p1.9.m2.3.3.2.2.3.3" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.cmml"><mi id="S4.SS2.p1.9.m2.3.3.2.2.3.3.2" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.2.cmml">t</mi><mo id="S4.SS2.p1.9.m2.3.3.2.2.3.3.1" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.1.cmml">−</mo><mn id="S4.SS2.p1.9.m2.3.3.2.2.3.3.3" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.3.cmml">1</mn></mrow></msup><mrow id="S4.SS2.p1.9.m2.3.3.2.2.2.3" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.cmml"><mi id="S4.SS2.p1.9.m2.3.3.2.2.2.3.2" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.2.cmml">t</mi><mo id="S4.SS2.p1.9.m2.3.3.2.2.2.3.1" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.1.cmml">−</mo><mn id="S4.SS2.p1.9.m2.3.3.2.2.2.3.3" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.3.cmml">1</mn></mrow></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.9.m2.3b"><list id="S4.SS2.p1.9.m2.3.3.3.cmml" xref="S4.SS2.p1.9.m2.3.3.2"><apply id="S4.SS2.p1.9.m2.2.2.1.1.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.9.m2.2.2.1.1.1.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1">subscript</csymbol><apply id="S4.SS2.p1.9.m2.2.2.1.1.2.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.9.m2.2.2.1.1.2.1.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1">superscript</csymbol><ci id="S4.SS2.p1.9.m2.2.2.1.1.2.2.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1.2.2">ℬ</ci><apply id="S4.SS2.p1.9.m2.2.2.1.1.2.3.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3"><minus id="S4.SS2.p1.9.m2.2.2.1.1.2.3.1.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.1"></minus><ci id="S4.SS2.p1.9.m2.2.2.1.1.2.3.2.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.2">𝑡</ci><cn type="integer" id="S4.SS2.p1.9.m2.2.2.1.1.2.3.3.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1.2.3.3">1</cn></apply></apply><cn type="integer" id="S4.SS2.p1.9.m2.2.2.1.1.3.cmml" xref="S4.SS2.p1.9.m2.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.p1.9.m2.1.1.cmml" xref="S4.SS2.p1.9.m2.1.1">…</ci><apply id="S4.SS2.p1.9.m2.3.3.2.2.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.9.m2.3.3.2.2.1.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2">subscript</csymbol><apply id="S4.SS2.p1.9.m2.3.3.2.2.2.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.9.m2.3.3.2.2.2.1.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2">superscript</csymbol><ci id="S4.SS2.p1.9.m2.3.3.2.2.2.2.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.2.2">ℬ</ci><apply id="S4.SS2.p1.9.m2.3.3.2.2.2.3.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3"><minus id="S4.SS2.p1.9.m2.3.3.2.2.2.3.1.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.1"></minus><ci id="S4.SS2.p1.9.m2.3.3.2.2.2.3.2.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.2">𝑡</ci><cn type="integer" id="S4.SS2.p1.9.m2.3.3.2.2.2.3.3.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.2.3.3">1</cn></apply></apply><apply id="S4.SS2.p1.9.m2.3.3.2.2.3.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p1.9.m2.3.3.2.2.3.1.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3">superscript</csymbol><cn type="integer" id="S4.SS2.p1.9.m2.3.3.2.2.3.2.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3.2">2</cn><apply id="S4.SS2.p1.9.m2.3.3.2.2.3.3.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3"><minus id="S4.SS2.p1.9.m2.3.3.2.2.3.3.1.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.1"></minus><ci id="S4.SS2.p1.9.m2.3.3.2.2.3.3.2.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.2">𝑡</ci><cn type="integer" id="S4.SS2.p1.9.m2.3.3.2.2.3.3.3.cmml" xref="S4.SS2.p1.9.m2.3.3.2.2.3.3.3">1</cn></apply></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.9.m2.3c">\mathcal{B}^{t-1}_{1},\ldots,\mathcal{B}^{t-1}_{2^{t-1}}</annotation></semantics></math> from the previous level. This procedure is given in Algorithm <a href="#alg2" title="Algorithm 2 ‣ 4.2 Learning the Hash Function Parameters ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.4" class="ltx_p">In line 2, we select a fixed number of indices to evaluate. Several heuristics are possible, including evaluating all indices. We found that simply selecting the top <math id="S4.SS2.p2.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS2.p2.1.m1.1a"><mi id="S4.SS2.p2.1.m1.1.1" xref="S4.SS2.p2.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.1b"><ci id="S4.SS2.p2.1.m1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.1c">n</annotation></semantics></math> indices that contributed the most loss summed across all buckets was difficult to beat. In preliminary experiments, we found that using <math id="S4.SS2.p2.2.m2.1" class="ltx_Math" alttext="n&gt;4" display="inline"><semantics id="S4.SS2.p2.2.m2.1a"><mrow id="S4.SS2.p2.2.m2.1.1" xref="S4.SS2.p2.2.m2.1.1.cmml"><mi id="S4.SS2.p2.2.m2.1.1.2" xref="S4.SS2.p2.2.m2.1.1.2.cmml">n</mi><mo id="S4.SS2.p2.2.m2.1.1.1" xref="S4.SS2.p2.2.m2.1.1.1.cmml">&gt;</mo><mn id="S4.SS2.p2.2.m2.1.1.3" xref="S4.SS2.p2.2.m2.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.2.m2.1b"><apply id="S4.SS2.p2.2.m2.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1"><gt id="S4.SS2.p2.2.m2.1.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1.1"></gt><ci id="S4.SS2.p2.2.m2.1.1.2.cmml" xref="S4.SS2.p2.2.m2.1.1.2">𝑛</ci><cn type="integer" id="S4.SS2.p2.2.m2.1.1.3.cmml" xref="S4.SS2.p2.2.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.2.m2.1c">n&gt;4</annotation></semantics></math> indices offered no clear benefit (and even choosing <math id="S4.SS2.p2.3.m3.1" class="ltx_Math" alttext="n=1" display="inline"><semantics id="S4.SS2.p2.3.m3.1a"><mrow id="S4.SS2.p2.3.m3.1.1" xref="S4.SS2.p2.3.m3.1.1.cmml"><mi id="S4.SS2.p2.3.m3.1.1.2" xref="S4.SS2.p2.3.m3.1.1.2.cmml">n</mi><mo id="S4.SS2.p2.3.m3.1.1.1" xref="S4.SS2.p2.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS2.p2.3.m3.1.1.3" xref="S4.SS2.p2.3.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.3.m3.1b"><apply id="S4.SS2.p2.3.m3.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1"><eq id="S4.SS2.p2.3.m3.1.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1.1"></eq><ci id="S4.SS2.p2.3.m3.1.1.2.cmml" xref="S4.SS2.p2.3.m3.1.1.2">𝑛</ci><cn type="integer" id="S4.SS2.p2.3.m3.1.1.3.cmml" xref="S4.SS2.p2.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.3.m3.1c">n=1</annotation></semantics></math> was nearly as good), so we fix <math id="S4.SS2.p2.4.m4.1" class="ltx_Math" alttext="n=4" display="inline"><semantics id="S4.SS2.p2.4.m4.1a"><mrow id="S4.SS2.p2.4.m4.1.1" xref="S4.SS2.p2.4.m4.1.1.cmml"><mi id="S4.SS2.p2.4.m4.1.1.2" xref="S4.SS2.p2.4.m4.1.1.2.cmml">n</mi><mo id="S4.SS2.p2.4.m4.1.1.1" xref="S4.SS2.p2.4.m4.1.1.1.cmml">=</mo><mn id="S4.SS2.p2.4.m4.1.1.3" xref="S4.SS2.p2.4.m4.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.4.m4.1b"><apply id="S4.SS2.p2.4.m4.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1"><eq id="S4.SS2.p2.4.m4.1.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1.1"></eq><ci id="S4.SS2.p2.4.m4.1.1.2.cmml" xref="S4.SS2.p2.4.m4.1.1.2">𝑛</ci><cn type="integer" id="S4.SS2.p2.4.m4.1.1.3.cmml" xref="S4.SS2.p2.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.4.m4.1c">n=4</annotation></semantics></math>.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.4" class="ltx_p">In lines 4-15, we find the minimal loss obtainable by splitting all buckets along that index, but with bucket-specific cutoffs. This loss is minimal not in the sense that it leads to the globally optimal tree, but in that it minimizes the sum of the losses in the buckets produced in this iteration. To do this, we invoke the subroutine <span id="S4.SS2.p3.4.1" class="ltx_text ltx_font_typewriter">optimal_split_threshold</span>, which takes in a bucket <math id="S4.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="S4.SS2.p3.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p3.1.m1.1.1" xref="S4.SS2.p3.1.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.1.m1.1b"><ci id="S4.SS2.p3.1.m1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.1.m1.1c">\mathcal{B}</annotation></semantics></math> and an index <math id="S4.SS2.p3.2.m2.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S4.SS2.p3.2.m2.1a"><mi id="S4.SS2.p3.2.m2.1.1" xref="S4.SS2.p3.2.m2.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.2.m2.1b"><ci id="S4.SS2.p3.2.m2.1.1.cmml" xref="S4.SS2.p3.2.m2.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.2.m2.1c">j</annotation></semantics></math> and tests all possible thresholds to find one minimizing <math id="S4.SS2.p3.3.m3.1" class="ltx_Math" alttext="\mathcal{L}(j\text{, }\mathcal{B})" display="inline"><semantics id="S4.SS2.p3.3.m3.1a"><mrow id="S4.SS2.p3.3.m3.1.1" xref="S4.SS2.p3.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p3.3.m3.1.1.3" xref="S4.SS2.p3.3.m3.1.1.3.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p3.3.m3.1.1.2" xref="S4.SS2.p3.3.m3.1.1.2.cmml">​</mo><mrow id="S4.SS2.p3.3.m3.1.1.1.1" xref="S4.SS2.p3.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p3.3.m3.1.1.1.1.2" xref="S4.SS2.p3.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p3.3.m3.1.1.1.1.1" xref="S4.SS2.p3.3.m3.1.1.1.1.1.cmml"><mi id="S4.SS2.p3.3.m3.1.1.1.1.1.2" xref="S4.SS2.p3.3.m3.1.1.1.1.1.2.cmml">j</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p3.3.m3.1.1.1.1.1.1" xref="S4.SS2.p3.3.m3.1.1.1.1.1.1.cmml">​</mo><mtext id="S4.SS2.p3.3.m3.1.1.1.1.1.3" xref="S4.SS2.p3.3.m3.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="S4.SS2.p3.3.m3.1.1.1.1.1.1a" xref="S4.SS2.p3.3.m3.1.1.1.1.1.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p3.3.m3.1.1.1.1.1.4" xref="S4.SS2.p3.3.m3.1.1.1.1.1.4.cmml">ℬ</mi></mrow><mo stretchy="false" id="S4.SS2.p3.3.m3.1.1.1.1.3" xref="S4.SS2.p3.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.3.m3.1b"><apply id="S4.SS2.p3.3.m3.1.1.cmml" xref="S4.SS2.p3.3.m3.1.1"><times id="S4.SS2.p3.3.m3.1.1.2.cmml" xref="S4.SS2.p3.3.m3.1.1.2"></times><ci id="S4.SS2.p3.3.m3.1.1.3.cmml" xref="S4.SS2.p3.3.m3.1.1.3">ℒ</ci><apply id="S4.SS2.p3.3.m3.1.1.1.1.1.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1"><times id="S4.SS2.p3.3.m3.1.1.1.1.1.1.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1.1.1"></times><ci id="S4.SS2.p3.3.m3.1.1.1.1.1.2.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1.1.2">𝑗</ci><ci id="S4.SS2.p3.3.m3.1.1.1.1.1.3a.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1.1.3"><mtext id="S4.SS2.p3.3.m3.1.1.1.1.1.3.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1.1.3">, </mtext></ci><ci id="S4.SS2.p3.3.m3.1.1.1.1.1.4.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1.1.4">ℬ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.3.m3.1c">\mathcal{L}(j\text{, }\mathcal{B})</annotation></semantics></math>. This can be done in time <math id="S4.SS2.p3.4.m4.5" class="ltx_Math" alttext="O(|\mathcal{J}^{(c)}||\mathcal{B}|\log(|\mathcal{B}|))" display="inline"><semantics id="S4.SS2.p3.4.m4.5a"><mrow id="S4.SS2.p3.4.m4.5.5" xref="S4.SS2.p3.4.m4.5.5.cmml"><mi id="S4.SS2.p3.4.m4.5.5.3" xref="S4.SS2.p3.4.m4.5.5.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p3.4.m4.5.5.2" xref="S4.SS2.p3.4.m4.5.5.2.cmml">​</mo><mrow id="S4.SS2.p3.4.m4.5.5.1.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.cmml">(</mo><mrow id="S4.SS2.p3.4.m4.5.5.1.1.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.cmml"><mrow id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.2.1.cmml">|</mo><msup id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.2.cmml">𝒥</mi><mrow id="S4.SS2.p3.4.m4.1.1.1.3" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p3.4.m4.1.1.1.3.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.cmml">(</mo><mi id="S4.SS2.p3.4.m4.1.1.1.1" xref="S4.SS2.p3.4.m4.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S4.SS2.p3.4.m4.1.1.1.3.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.cmml">)</mo></mrow></msup><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.3" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.2.1.cmml">|</mo></mrow><mo lspace="0em" rspace="0em" id="S4.SS2.p3.4.m4.5.5.1.1.1.3" xref="S4.SS2.p3.4.m4.5.5.1.1.1.3.cmml">​</mo><mrow id="S4.SS2.p3.4.m4.5.5.1.1.1.4.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.4.1.cmml"><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.4.2.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.4.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p3.4.m4.2.2" xref="S4.SS2.p3.4.m4.2.2.cmml">ℬ</mi><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.4.2.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.4.1.1.cmml">|</mo></mrow><mo lspace="0.167em" rspace="0em" id="S4.SS2.p3.4.m4.5.5.1.1.1.3a" xref="S4.SS2.p3.4.m4.5.5.1.1.1.3.cmml">​</mo><mrow id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.2.cmml"><mi id="S4.SS2.p3.4.m4.4.4" xref="S4.SS2.p3.4.m4.4.4.cmml">log</mi><mo id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1a" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.2.cmml">⁡</mo><mrow id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.2.cmml">(</mo><mrow id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.2.1" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p3.4.m4.3.3" xref="S4.SS2.p3.4.m4.3.3.cmml">ℬ</mi><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.2.2" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.1.1.cmml">|</mo></mrow><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.3" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S4.SS2.p3.4.m4.5.5.1.1.3" xref="S4.SS2.p3.4.m4.5.5.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.4.m4.5b"><apply id="S4.SS2.p3.4.m4.5.5.cmml" xref="S4.SS2.p3.4.m4.5.5"><times id="S4.SS2.p3.4.m4.5.5.2.cmml" xref="S4.SS2.p3.4.m4.5.5.2"></times><ci id="S4.SS2.p3.4.m4.5.5.3.cmml" xref="S4.SS2.p3.4.m4.5.5.3">𝑂</ci><apply id="S4.SS2.p3.4.m4.5.5.1.1.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1"><times id="S4.SS2.p3.4.m4.5.5.1.1.1.3.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.3"></times><apply id="S4.SS2.p3.4.m4.5.5.1.1.1.1.2.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1"><abs id="S4.SS2.p3.4.m4.5.5.1.1.1.1.2.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.2"></abs><apply id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1">superscript</csymbol><ci id="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.1.1.1.2">𝒥</ci><ci id="S4.SS2.p3.4.m4.1.1.1.1.cmml" xref="S4.SS2.p3.4.m4.1.1.1.1">𝑐</ci></apply></apply><apply id="S4.SS2.p3.4.m4.5.5.1.1.1.4.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.4.2"><abs id="S4.SS2.p3.4.m4.5.5.1.1.1.4.1.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.4.2.1"></abs><ci id="S4.SS2.p3.4.m4.2.2.cmml" xref="S4.SS2.p3.4.m4.2.2">ℬ</ci></apply><apply id="S4.SS2.p3.4.m4.5.5.1.1.1.2.2.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.1"><log id="S4.SS2.p3.4.m4.4.4.cmml" xref="S4.SS2.p3.4.m4.4.4"></log><apply id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.2"><abs id="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.1.1.cmml" xref="S4.SS2.p3.4.m4.5.5.1.1.1.2.1.1.1.2.1"></abs><ci id="S4.SS2.p3.4.m4.3.3.cmml" xref="S4.SS2.p3.4.m4.3.3">ℬ</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.4.m4.5c">O(|\mathcal{J}^{(c)}||\mathcal{B}|\log(|\mathcal{B}|))</annotation></semantics></math>. The pseudocode for this subroutine is given in Algorithms <a href="#alg3" title="Algorithm 3 ‣ Appendix C Subroutines for Training MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and <a href="#alg4" title="Algorithm 4 ‣ Appendix C Subroutines for Training MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> in Appendix <a href="#A3" title="Appendix C Subroutines for Training MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.3" class="ltx_p">Once a split index <math id="S4.SS2.p4.1.m1.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S4.SS2.p4.1.m1.1a"><mi id="S4.SS2.p4.1.m1.1.1" xref="S4.SS2.p4.1.m1.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.1.m1.1b"><ci id="S4.SS2.p4.1.m1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.1.m1.1c">j</annotation></semantics></math> and an array of split thresholds <math id="S4.SS2.p4.2.m2.1" class="ltx_Math" alttext="\bm{v}" display="inline"><semantics id="S4.SS2.p4.2.m2.1a"><mi id="S4.SS2.p4.2.m2.1.1" xref="S4.SS2.p4.2.m2.1.1.cmml">𝒗</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.2.m2.1b"><ci id="S4.SS2.p4.2.m2.1.1.cmml" xref="S4.SS2.p4.2.m2.1.1">𝒗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.2.m2.1c">\bm{v}</annotation></semantics></math> are chosen, all that remains is to split the buckets to form the next level of the tree (lines 16-21). This entails forming two child buckets from each current bucket by grouping vectors whose <math id="S4.SS2.p4.3.m3.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S4.SS2.p4.3.m3.1a"><mi id="S4.SS2.p4.3.m3.1.1" xref="S4.SS2.p4.3.m3.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.3.m3.1b"><ci id="S4.SS2.p4.3.m3.1.1.cmml" xref="S4.SS2.p4.3.m3.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.3.m3.1c">j</annotation></semantics></math>th entries are above or below the bucket’s split threshold.</p>
</div>
<figure id="alg2" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg2.2.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> Adding The Next Level to the Hashing Tree</figcaption>
<div id="alg2.3" class="ltx_listing ltx_listing">
<div id="alg2.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span>  <span id="alg2.l1.2" class="ltx_text ltx_font_bold">Input:</span> buckets <math id="alg2.l1.m1.3" class="ltx_Math" alttext="\mathcal{B}^{t-1}_{1},\ldots,\mathcal{B}^{t-1}_{2^{t-1}}" display="inline"><semantics id="alg2.l1.m1.3a"><mrow id="alg2.l1.m1.3.3.2" xref="alg2.l1.m1.3.3.3.cmml"><msubsup id="alg2.l1.m1.2.2.1.1" xref="alg2.l1.m1.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l1.m1.2.2.1.1.2.2" xref="alg2.l1.m1.2.2.1.1.2.2.cmml">ℬ</mi><mn id="alg2.l1.m1.2.2.1.1.3" xref="alg2.l1.m1.2.2.1.1.3.cmml">1</mn><mrow id="alg2.l1.m1.2.2.1.1.2.3" xref="alg2.l1.m1.2.2.1.1.2.3.cmml"><mi id="alg2.l1.m1.2.2.1.1.2.3.2" xref="alg2.l1.m1.2.2.1.1.2.3.2.cmml">t</mi><mo id="alg2.l1.m1.2.2.1.1.2.3.1" xref="alg2.l1.m1.2.2.1.1.2.3.1.cmml">−</mo><mn id="alg2.l1.m1.2.2.1.1.2.3.3" xref="alg2.l1.m1.2.2.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mo id="alg2.l1.m1.3.3.2.3" xref="alg2.l1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="alg2.l1.m1.1.1" xref="alg2.l1.m1.1.1.cmml">…</mi><mo id="alg2.l1.m1.3.3.2.4" xref="alg2.l1.m1.3.3.3.cmml">,</mo><msubsup id="alg2.l1.m1.3.3.2.2" xref="alg2.l1.m1.3.3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l1.m1.3.3.2.2.2.2" xref="alg2.l1.m1.3.3.2.2.2.2.cmml">ℬ</mi><msup id="alg2.l1.m1.3.3.2.2.3" xref="alg2.l1.m1.3.3.2.2.3.cmml"><mn id="alg2.l1.m1.3.3.2.2.3.2" xref="alg2.l1.m1.3.3.2.2.3.2.cmml">2</mn><mrow id="alg2.l1.m1.3.3.2.2.3.3" xref="alg2.l1.m1.3.3.2.2.3.3.cmml"><mi id="alg2.l1.m1.3.3.2.2.3.3.2" xref="alg2.l1.m1.3.3.2.2.3.3.2.cmml">t</mi><mo id="alg2.l1.m1.3.3.2.2.3.3.1" xref="alg2.l1.m1.3.3.2.2.3.3.1.cmml">−</mo><mn id="alg2.l1.m1.3.3.2.2.3.3.3" xref="alg2.l1.m1.3.3.2.2.3.3.3.cmml">1</mn></mrow></msup><mrow id="alg2.l1.m1.3.3.2.2.2.3" xref="alg2.l1.m1.3.3.2.2.2.3.cmml"><mi id="alg2.l1.m1.3.3.2.2.2.3.2" xref="alg2.l1.m1.3.3.2.2.2.3.2.cmml">t</mi><mo id="alg2.l1.m1.3.3.2.2.2.3.1" xref="alg2.l1.m1.3.3.2.2.2.3.1.cmml">−</mo><mn id="alg2.l1.m1.3.3.2.2.2.3.3" xref="alg2.l1.m1.3.3.2.2.2.3.3.cmml">1</mn></mrow></msubsup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l1.m1.3b"><list id="alg2.l1.m1.3.3.3.cmml" xref="alg2.l1.m1.3.3.2"><apply id="alg2.l1.m1.2.2.1.1.cmml" xref="alg2.l1.m1.2.2.1.1"><csymbol cd="ambiguous" id="alg2.l1.m1.2.2.1.1.1.cmml" xref="alg2.l1.m1.2.2.1.1">subscript</csymbol><apply id="alg2.l1.m1.2.2.1.1.2.cmml" xref="alg2.l1.m1.2.2.1.1"><csymbol cd="ambiguous" id="alg2.l1.m1.2.2.1.1.2.1.cmml" xref="alg2.l1.m1.2.2.1.1">superscript</csymbol><ci id="alg2.l1.m1.2.2.1.1.2.2.cmml" xref="alg2.l1.m1.2.2.1.1.2.2">ℬ</ci><apply id="alg2.l1.m1.2.2.1.1.2.3.cmml" xref="alg2.l1.m1.2.2.1.1.2.3"><minus id="alg2.l1.m1.2.2.1.1.2.3.1.cmml" xref="alg2.l1.m1.2.2.1.1.2.3.1"></minus><ci id="alg2.l1.m1.2.2.1.1.2.3.2.cmml" xref="alg2.l1.m1.2.2.1.1.2.3.2">𝑡</ci><cn type="integer" id="alg2.l1.m1.2.2.1.1.2.3.3.cmml" xref="alg2.l1.m1.2.2.1.1.2.3.3">1</cn></apply></apply><cn type="integer" id="alg2.l1.m1.2.2.1.1.3.cmml" xref="alg2.l1.m1.2.2.1.1.3">1</cn></apply><ci id="alg2.l1.m1.1.1.cmml" xref="alg2.l1.m1.1.1">…</ci><apply id="alg2.l1.m1.3.3.2.2.cmml" xref="alg2.l1.m1.3.3.2.2"><csymbol cd="ambiguous" id="alg2.l1.m1.3.3.2.2.1.cmml" xref="alg2.l1.m1.3.3.2.2">subscript</csymbol><apply id="alg2.l1.m1.3.3.2.2.2.cmml" xref="alg2.l1.m1.3.3.2.2"><csymbol cd="ambiguous" id="alg2.l1.m1.3.3.2.2.2.1.cmml" xref="alg2.l1.m1.3.3.2.2">superscript</csymbol><ci id="alg2.l1.m1.3.3.2.2.2.2.cmml" xref="alg2.l1.m1.3.3.2.2.2.2">ℬ</ci><apply id="alg2.l1.m1.3.3.2.2.2.3.cmml" xref="alg2.l1.m1.3.3.2.2.2.3"><minus id="alg2.l1.m1.3.3.2.2.2.3.1.cmml" xref="alg2.l1.m1.3.3.2.2.2.3.1"></minus><ci id="alg2.l1.m1.3.3.2.2.2.3.2.cmml" xref="alg2.l1.m1.3.3.2.2.2.3.2">𝑡</ci><cn type="integer" id="alg2.l1.m1.3.3.2.2.2.3.3.cmml" xref="alg2.l1.m1.3.3.2.2.2.3.3">1</cn></apply></apply><apply id="alg2.l1.m1.3.3.2.2.3.cmml" xref="alg2.l1.m1.3.3.2.2.3"><csymbol cd="ambiguous" id="alg2.l1.m1.3.3.2.2.3.1.cmml" xref="alg2.l1.m1.3.3.2.2.3">superscript</csymbol><cn type="integer" id="alg2.l1.m1.3.3.2.2.3.2.cmml" xref="alg2.l1.m1.3.3.2.2.3.2">2</cn><apply id="alg2.l1.m1.3.3.2.2.3.3.cmml" xref="alg2.l1.m1.3.3.2.2.3.3"><minus id="alg2.l1.m1.3.3.2.2.3.3.1.cmml" xref="alg2.l1.m1.3.3.2.2.3.3.1"></minus><ci id="alg2.l1.m1.3.3.2.2.3.3.2.cmml" xref="alg2.l1.m1.3.3.2.2.3.3.2">𝑡</ci><cn type="integer" id="alg2.l1.m1.3.3.2.2.3.3.3.cmml" xref="alg2.l1.m1.3.3.2.2.3.3.3">1</cn></apply></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m1.3c">\mathcal{B}^{t-1}_{1},\ldots,\mathcal{B}^{t-1}_{2^{t-1}}</annotation></semantics></math>, training matrix <math id="alg2.l1.m2.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="alg2.l1.m2.1a"><mover accent="true" id="alg2.l1.m2.1.1" xref="alg2.l1.m2.1.1.cmml"><mi id="alg2.l1.m2.1.1.2" xref="alg2.l1.m2.1.1.2.cmml">𝑨</mi><mo id="alg2.l1.m2.1.1.1" xref="alg2.l1.m2.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="alg2.l1.m2.1b"><apply id="alg2.l1.m2.1.1.cmml" xref="alg2.l1.m2.1.1"><ci id="alg2.l1.m2.1.1.1.cmml" xref="alg2.l1.m2.1.1.1">~</ci><ci id="alg2.l1.m2.1.1.2.cmml" xref="alg2.l1.m2.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m2.1c">\tilde{\bm{A}}</annotation></semantics></math>

<span id="alg2.l1.3" class="ltx_text" style="color:#009900;">
// greedily choose next split index and thresholds
</span>


</div>
<div id="alg2.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>  <math id="alg2.l2.m1.3" class="ltx_Math" alttext="\mathcal{\hat{J}}\leftarrow\texttt{heuristic\_select\_idxs}(\mathcal{B}^{t-1}_{1},\ldots,\mathcal{B}^{t-1}_{2^{t-1}})" display="inline"><semantics id="alg2.l2.m1.3a"><mrow id="alg2.l2.m1.3.3" xref="alg2.l2.m1.3.3.cmml"><mover accent="true" id="alg2.l2.m1.3.3.4" xref="alg2.l2.m1.3.3.4.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l2.m1.3.3.4.2" xref="alg2.l2.m1.3.3.4.2.cmml">𝒥</mi><mo id="alg2.l2.m1.3.3.4.1" xref="alg2.l2.m1.3.3.4.1.cmml">^</mo></mover><mo stretchy="false" id="alg2.l2.m1.3.3.3" xref="alg2.l2.m1.3.3.3.cmml">←</mo><mrow id="alg2.l2.m1.3.3.2" xref="alg2.l2.m1.3.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg2.l2.m1.3.3.2.4" xref="alg2.l2.m1.3.3.2.4a.cmml">heuristic_select_idxs</mtext><mo lspace="0em" rspace="0em" id="alg2.l2.m1.3.3.2.3" xref="alg2.l2.m1.3.3.2.3.cmml">​</mo><mrow id="alg2.l2.m1.3.3.2.2.2" xref="alg2.l2.m1.3.3.2.2.3.cmml"><mo stretchy="false" id="alg2.l2.m1.3.3.2.2.2.3" xref="alg2.l2.m1.3.3.2.2.3.cmml">(</mo><msubsup id="alg2.l2.m1.2.2.1.1.1.1" xref="alg2.l2.m1.2.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l2.m1.2.2.1.1.1.1.2.2" xref="alg2.l2.m1.2.2.1.1.1.1.2.2.cmml">ℬ</mi><mn id="alg2.l2.m1.2.2.1.1.1.1.3" xref="alg2.l2.m1.2.2.1.1.1.1.3.cmml">1</mn><mrow id="alg2.l2.m1.2.2.1.1.1.1.2.3" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.cmml"><mi id="alg2.l2.m1.2.2.1.1.1.1.2.3.2" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.2.cmml">t</mi><mo id="alg2.l2.m1.2.2.1.1.1.1.2.3.1" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.1.cmml">−</mo><mn id="alg2.l2.m1.2.2.1.1.1.1.2.3.3" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mo id="alg2.l2.m1.3.3.2.2.2.4" xref="alg2.l2.m1.3.3.2.2.3.cmml">,</mo><mi mathvariant="normal" id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml">…</mi><mo id="alg2.l2.m1.3.3.2.2.2.5" xref="alg2.l2.m1.3.3.2.2.3.cmml">,</mo><msubsup id="alg2.l2.m1.3.3.2.2.2.2" xref="alg2.l2.m1.3.3.2.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l2.m1.3.3.2.2.2.2.2.2" xref="alg2.l2.m1.3.3.2.2.2.2.2.2.cmml">ℬ</mi><msup id="alg2.l2.m1.3.3.2.2.2.2.3" xref="alg2.l2.m1.3.3.2.2.2.2.3.cmml"><mn id="alg2.l2.m1.3.3.2.2.2.2.3.2" xref="alg2.l2.m1.3.3.2.2.2.2.3.2.cmml">2</mn><mrow id="alg2.l2.m1.3.3.2.2.2.2.3.3" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.cmml"><mi id="alg2.l2.m1.3.3.2.2.2.2.3.3.2" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.2.cmml">t</mi><mo id="alg2.l2.m1.3.3.2.2.2.2.3.3.1" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.1.cmml">−</mo><mn id="alg2.l2.m1.3.3.2.2.2.2.3.3.3" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.3.cmml">1</mn></mrow></msup><mrow id="alg2.l2.m1.3.3.2.2.2.2.2.3" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.cmml"><mi id="alg2.l2.m1.3.3.2.2.2.2.2.3.2" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.2.cmml">t</mi><mo id="alg2.l2.m1.3.3.2.2.2.2.2.3.1" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.1.cmml">−</mo><mn id="alg2.l2.m1.3.3.2.2.2.2.2.3.3" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="alg2.l2.m1.3.3.2.2.2.6" xref="alg2.l2.m1.3.3.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.3b"><apply id="alg2.l2.m1.3.3.cmml" xref="alg2.l2.m1.3.3"><ci id="alg2.l2.m1.3.3.3.cmml" xref="alg2.l2.m1.3.3.3">←</ci><apply id="alg2.l2.m1.3.3.4.cmml" xref="alg2.l2.m1.3.3.4"><ci id="alg2.l2.m1.3.3.4.1.cmml" xref="alg2.l2.m1.3.3.4.1">^</ci><ci id="alg2.l2.m1.3.3.4.2.cmml" xref="alg2.l2.m1.3.3.4.2">𝒥</ci></apply><apply id="alg2.l2.m1.3.3.2.cmml" xref="alg2.l2.m1.3.3.2"><times id="alg2.l2.m1.3.3.2.3.cmml" xref="alg2.l2.m1.3.3.2.3"></times><ci id="alg2.l2.m1.3.3.2.4a.cmml" xref="alg2.l2.m1.3.3.2.4"><mtext class="ltx_mathvariant_monospace" id="alg2.l2.m1.3.3.2.4.cmml" xref="alg2.l2.m1.3.3.2.4">heuristic_select_idxs</mtext></ci><vector id="alg2.l2.m1.3.3.2.2.3.cmml" xref="alg2.l2.m1.3.3.2.2.2"><apply id="alg2.l2.m1.2.2.1.1.1.1.cmml" xref="alg2.l2.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l2.m1.2.2.1.1.1.1.1.cmml" xref="alg2.l2.m1.2.2.1.1.1.1">subscript</csymbol><apply id="alg2.l2.m1.2.2.1.1.1.1.2.cmml" xref="alg2.l2.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l2.m1.2.2.1.1.1.1.2.1.cmml" xref="alg2.l2.m1.2.2.1.1.1.1">superscript</csymbol><ci id="alg2.l2.m1.2.2.1.1.1.1.2.2.cmml" xref="alg2.l2.m1.2.2.1.1.1.1.2.2">ℬ</ci><apply id="alg2.l2.m1.2.2.1.1.1.1.2.3.cmml" xref="alg2.l2.m1.2.2.1.1.1.1.2.3"><minus id="alg2.l2.m1.2.2.1.1.1.1.2.3.1.cmml" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.1"></minus><ci id="alg2.l2.m1.2.2.1.1.1.1.2.3.2.cmml" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.2">𝑡</ci><cn type="integer" id="alg2.l2.m1.2.2.1.1.1.1.2.3.3.cmml" xref="alg2.l2.m1.2.2.1.1.1.1.2.3.3">1</cn></apply></apply><cn type="integer" id="alg2.l2.m1.2.2.1.1.1.1.3.cmml" xref="alg2.l2.m1.2.2.1.1.1.1.3">1</cn></apply><ci id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1">…</ci><apply id="alg2.l2.m1.3.3.2.2.2.2.cmml" xref="alg2.l2.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l2.m1.3.3.2.2.2.2.1.cmml" xref="alg2.l2.m1.3.3.2.2.2.2">subscript</csymbol><apply id="alg2.l2.m1.3.3.2.2.2.2.2.cmml" xref="alg2.l2.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l2.m1.3.3.2.2.2.2.2.1.cmml" xref="alg2.l2.m1.3.3.2.2.2.2">superscript</csymbol><ci id="alg2.l2.m1.3.3.2.2.2.2.2.2.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.2.2">ℬ</ci><apply id="alg2.l2.m1.3.3.2.2.2.2.2.3.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.2.3"><minus id="alg2.l2.m1.3.3.2.2.2.2.2.3.1.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.1"></minus><ci id="alg2.l2.m1.3.3.2.2.2.2.2.3.2.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.2">𝑡</ci><cn type="integer" id="alg2.l2.m1.3.3.2.2.2.2.2.3.3.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.2.3.3">1</cn></apply></apply><apply id="alg2.l2.m1.3.3.2.2.2.2.3.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3"><csymbol cd="ambiguous" id="alg2.l2.m1.3.3.2.2.2.2.3.1.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3">superscript</csymbol><cn type="integer" id="alg2.l2.m1.3.3.2.2.2.2.3.2.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3.2">2</cn><apply id="alg2.l2.m1.3.3.2.2.2.2.3.3.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3.3"><minus id="alg2.l2.m1.3.3.2.2.2.2.3.3.1.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.1"></minus><ci id="alg2.l2.m1.3.3.2.2.2.2.3.3.2.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.2">𝑡</ci><cn type="integer" id="alg2.l2.m1.3.3.2.2.2.2.3.3.3.cmml" xref="alg2.l2.m1.3.3.2.2.2.2.3.3.3">1</cn></apply></apply></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.3c">\mathcal{\hat{J}}\leftarrow\texttt{heuristic\_select\_idxs}(\mathcal{B}^{t-1}_{1},\ldots,\mathcal{B}^{t-1}_{2^{t-1}})</annotation></semantics></math>

</div>
<div id="alg2.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>  <math id="alg2.l3.m1.2" class="ltx_Math" alttext="l^{min}\text{, }j^{min},\bm{v}^{min}\leftarrow\infty\text{, NaN, NaN}" display="inline"><semantics id="alg2.l3.m1.2a"><mrow id="alg2.l3.m1.2.2" xref="alg2.l3.m1.2.2.cmml"><mrow id="alg2.l3.m1.2.2.2.2" xref="alg2.l3.m1.2.2.2.3.cmml"><mrow id="alg2.l3.m1.1.1.1.1.1" xref="alg2.l3.m1.1.1.1.1.1.cmml"><msup id="alg2.l3.m1.1.1.1.1.1.2" xref="alg2.l3.m1.1.1.1.1.1.2.cmml"><mi id="alg2.l3.m1.1.1.1.1.1.2.2" xref="alg2.l3.m1.1.1.1.1.1.2.2.cmml">l</mi><mrow id="alg2.l3.m1.1.1.1.1.1.2.3" xref="alg2.l3.m1.1.1.1.1.1.2.3.cmml"><mi id="alg2.l3.m1.1.1.1.1.1.2.3.2" xref="alg2.l3.m1.1.1.1.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.1.1.1.2.3.1" xref="alg2.l3.m1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.1.1.1.2.3.3" xref="alg2.l3.m1.1.1.1.1.1.2.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.1.1.1.2.3.1a" xref="alg2.l3.m1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.1.1.1.2.3.4" xref="alg2.l3.m1.1.1.1.1.1.2.3.4.cmml">n</mi></mrow></msup><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.1.1.1.1" xref="alg2.l3.m1.1.1.1.1.1.1.cmml">​</mo><mtext id="alg2.l3.m1.1.1.1.1.1.3" xref="alg2.l3.m1.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.1.1.1.1a" xref="alg2.l3.m1.1.1.1.1.1.1.cmml">​</mo><msup id="alg2.l3.m1.1.1.1.1.1.4" xref="alg2.l3.m1.1.1.1.1.1.4.cmml"><mi id="alg2.l3.m1.1.1.1.1.1.4.2" xref="alg2.l3.m1.1.1.1.1.1.4.2.cmml">j</mi><mrow id="alg2.l3.m1.1.1.1.1.1.4.3" xref="alg2.l3.m1.1.1.1.1.1.4.3.cmml"><mi id="alg2.l3.m1.1.1.1.1.1.4.3.2" xref="alg2.l3.m1.1.1.1.1.1.4.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.1.1.1.4.3.1" xref="alg2.l3.m1.1.1.1.1.1.4.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.1.1.1.4.3.3" xref="alg2.l3.m1.1.1.1.1.1.4.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.1.1.1.4.3.1a" xref="alg2.l3.m1.1.1.1.1.1.4.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.1.1.1.4.3.4" xref="alg2.l3.m1.1.1.1.1.1.4.3.4.cmml">n</mi></mrow></msup></mrow><mo id="alg2.l3.m1.2.2.2.2.3" xref="alg2.l3.m1.2.2.2.3.cmml">,</mo><msup id="alg2.l3.m1.2.2.2.2.2" xref="alg2.l3.m1.2.2.2.2.2.cmml"><mi id="alg2.l3.m1.2.2.2.2.2.2" xref="alg2.l3.m1.2.2.2.2.2.2.cmml">𝒗</mi><mrow id="alg2.l3.m1.2.2.2.2.2.3" xref="alg2.l3.m1.2.2.2.2.2.3.cmml"><mi id="alg2.l3.m1.2.2.2.2.2.3.2" xref="alg2.l3.m1.2.2.2.2.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.2.2.2.2.2.3.1" xref="alg2.l3.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg2.l3.m1.2.2.2.2.2.3.3" xref="alg2.l3.m1.2.2.2.2.2.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.2.2.2.2.2.3.1a" xref="alg2.l3.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg2.l3.m1.2.2.2.2.2.3.4" xref="alg2.l3.m1.2.2.2.2.2.3.4.cmml">n</mi></mrow></msup></mrow><mo stretchy="false" id="alg2.l3.m1.2.2.3" xref="alg2.l3.m1.2.2.3.cmml">←</mo><mrow id="alg2.l3.m1.2.2.4" xref="alg2.l3.m1.2.2.4.cmml"><mi mathvariant="normal" id="alg2.l3.m1.2.2.4.2" xref="alg2.l3.m1.2.2.4.2.cmml">∞</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.2.2.4.1" xref="alg2.l3.m1.2.2.4.1.cmml">​</mo><mtext id="alg2.l3.m1.2.2.4.3" xref="alg2.l3.m1.2.2.4.3a.cmml">, NaN, NaN</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l3.m1.2b"><apply id="alg2.l3.m1.2.2.cmml" xref="alg2.l3.m1.2.2"><ci id="alg2.l3.m1.2.2.3.cmml" xref="alg2.l3.m1.2.2.3">←</ci><list id="alg2.l3.m1.2.2.2.3.cmml" xref="alg2.l3.m1.2.2.2.2"><apply id="alg2.l3.m1.1.1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1.1.1"><times id="alg2.l3.m1.1.1.1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1.1.1.1"></times><apply id="alg2.l3.m1.1.1.1.1.1.2.cmml" xref="alg2.l3.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.l3.m1.1.1.1.1.1.2.1.cmml" xref="alg2.l3.m1.1.1.1.1.1.2">superscript</csymbol><ci id="alg2.l3.m1.1.1.1.1.1.2.2.cmml" xref="alg2.l3.m1.1.1.1.1.1.2.2">𝑙</ci><apply id="alg2.l3.m1.1.1.1.1.1.2.3.cmml" xref="alg2.l3.m1.1.1.1.1.1.2.3"><times id="alg2.l3.m1.1.1.1.1.1.2.3.1.cmml" xref="alg2.l3.m1.1.1.1.1.1.2.3.1"></times><ci id="alg2.l3.m1.1.1.1.1.1.2.3.2.cmml" xref="alg2.l3.m1.1.1.1.1.1.2.3.2">𝑚</ci><ci id="alg2.l3.m1.1.1.1.1.1.2.3.3.cmml" xref="alg2.l3.m1.1.1.1.1.1.2.3.3">𝑖</ci><ci id="alg2.l3.m1.1.1.1.1.1.2.3.4.cmml" xref="alg2.l3.m1.1.1.1.1.1.2.3.4">𝑛</ci></apply></apply><ci id="alg2.l3.m1.1.1.1.1.1.3a.cmml" xref="alg2.l3.m1.1.1.1.1.1.3"><mtext id="alg2.l3.m1.1.1.1.1.1.3.cmml" xref="alg2.l3.m1.1.1.1.1.1.3">, </mtext></ci><apply id="alg2.l3.m1.1.1.1.1.1.4.cmml" xref="alg2.l3.m1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="alg2.l3.m1.1.1.1.1.1.4.1.cmml" xref="alg2.l3.m1.1.1.1.1.1.4">superscript</csymbol><ci id="alg2.l3.m1.1.1.1.1.1.4.2.cmml" xref="alg2.l3.m1.1.1.1.1.1.4.2">𝑗</ci><apply id="alg2.l3.m1.1.1.1.1.1.4.3.cmml" xref="alg2.l3.m1.1.1.1.1.1.4.3"><times id="alg2.l3.m1.1.1.1.1.1.4.3.1.cmml" xref="alg2.l3.m1.1.1.1.1.1.4.3.1"></times><ci id="alg2.l3.m1.1.1.1.1.1.4.3.2.cmml" xref="alg2.l3.m1.1.1.1.1.1.4.3.2">𝑚</ci><ci id="alg2.l3.m1.1.1.1.1.1.4.3.3.cmml" xref="alg2.l3.m1.1.1.1.1.1.4.3.3">𝑖</ci><ci id="alg2.l3.m1.1.1.1.1.1.4.3.4.cmml" xref="alg2.l3.m1.1.1.1.1.1.4.3.4">𝑛</ci></apply></apply></apply><apply id="alg2.l3.m1.2.2.2.2.2.cmml" xref="alg2.l3.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l3.m1.2.2.2.2.2.1.cmml" xref="alg2.l3.m1.2.2.2.2.2">superscript</csymbol><ci id="alg2.l3.m1.2.2.2.2.2.2.cmml" xref="alg2.l3.m1.2.2.2.2.2.2">𝒗</ci><apply id="alg2.l3.m1.2.2.2.2.2.3.cmml" xref="alg2.l3.m1.2.2.2.2.2.3"><times id="alg2.l3.m1.2.2.2.2.2.3.1.cmml" xref="alg2.l3.m1.2.2.2.2.2.3.1"></times><ci id="alg2.l3.m1.2.2.2.2.2.3.2.cmml" xref="alg2.l3.m1.2.2.2.2.2.3.2">𝑚</ci><ci id="alg2.l3.m1.2.2.2.2.2.3.3.cmml" xref="alg2.l3.m1.2.2.2.2.2.3.3">𝑖</ci><ci id="alg2.l3.m1.2.2.2.2.2.3.4.cmml" xref="alg2.l3.m1.2.2.2.2.2.3.4">𝑛</ci></apply></apply></list><apply id="alg2.l3.m1.2.2.4.cmml" xref="alg2.l3.m1.2.2.4"><times id="alg2.l3.m1.2.2.4.1.cmml" xref="alg2.l3.m1.2.2.4.1"></times><infinity id="alg2.l3.m1.2.2.4.2.cmml" xref="alg2.l3.m1.2.2.4.2"></infinity><ci id="alg2.l3.m1.2.2.4.3a.cmml" xref="alg2.l3.m1.2.2.4.3"><mtext id="alg2.l3.m1.2.2.4.3.cmml" xref="alg2.l3.m1.2.2.4.3">, NaN, NaN</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m1.2c">l^{min}\text{, }j^{min},\bm{v}^{min}\leftarrow\infty\text{, NaN, NaN}</annotation></semantics></math>

</div>
<div id="alg2.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span>  <span id="alg2.l4.2" class="ltx_text ltx_font_bold">for</span> <math id="alg2.l4.m1.1" class="ltx_Math" alttext="j\in\mathcal{\hat{J}}" display="inline"><semantics id="alg2.l4.m1.1a"><mrow id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml"><mi id="alg2.l4.m1.1.1.2" xref="alg2.l4.m1.1.1.2.cmml">j</mi><mo id="alg2.l4.m1.1.1.1" xref="alg2.l4.m1.1.1.1.cmml">∈</mo><mover accent="true" id="alg2.l4.m1.1.1.3" xref="alg2.l4.m1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l4.m1.1.1.3.2" xref="alg2.l4.m1.1.1.3.2.cmml">𝒥</mi><mo id="alg2.l4.m1.1.1.3.1" xref="alg2.l4.m1.1.1.3.1.cmml">^</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.1b"><apply id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1"><in id="alg2.l4.m1.1.1.1.cmml" xref="alg2.l4.m1.1.1.1"></in><ci id="alg2.l4.m1.1.1.2.cmml" xref="alg2.l4.m1.1.1.2">𝑗</ci><apply id="alg2.l4.m1.1.1.3.cmml" xref="alg2.l4.m1.1.1.3"><ci id="alg2.l4.m1.1.1.3.1.cmml" xref="alg2.l4.m1.1.1.3.1">^</ci><ci id="alg2.l4.m1.1.1.3.2.cmml" xref="alg2.l4.m1.1.1.3.2">𝒥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.1c">j\in\mathcal{\hat{J}}</annotation></semantics></math> <span id="alg2.l4.3" class="ltx_text ltx_font_bold">do</span>

 

</div>
<div id="alg2.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>     <math id="alg2.l5.m1.1" class="ltx_Math" alttext="l\leftarrow 0" display="inline"><semantics id="alg2.l5.m1.1a"><mrow id="alg2.l5.m1.1.1" xref="alg2.l5.m1.1.1.cmml"><mi id="alg2.l5.m1.1.1.2" xref="alg2.l5.m1.1.1.2.cmml">l</mi><mo stretchy="false" id="alg2.l5.m1.1.1.1" xref="alg2.l5.m1.1.1.1.cmml">←</mo><mn id="alg2.l5.m1.1.1.3" xref="alg2.l5.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m1.1b"><apply id="alg2.l5.m1.1.1.cmml" xref="alg2.l5.m1.1.1"><ci id="alg2.l5.m1.1.1.1.cmml" xref="alg2.l5.m1.1.1.1">←</ci><ci id="alg2.l5.m1.1.1.2.cmml" xref="alg2.l5.m1.1.1.2">𝑙</ci><cn type="integer" id="alg2.l5.m1.1.1.3.cmml" xref="alg2.l5.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m1.1c">l\leftarrow 0</annotation></semantics></math> 
<span id="alg2.l5.2" class="ltx_text" style="color:#009900;">
<span id="alg2.l5.2.1" class="ltx_text" style="float:right;">// initialize loss for this index to 0</span></span>


</div>
<div id="alg2.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>     <math id="alg2.l6.m1.1" class="ltx_Math" alttext="\bm{v}\leftarrow\text{[ ]}" display="inline"><semantics id="alg2.l6.m1.1a"><mrow id="alg2.l6.m1.1.1" xref="alg2.l6.m1.1.1.cmml"><mi id="alg2.l6.m1.1.1.2" xref="alg2.l6.m1.1.1.2.cmml">𝒗</mi><mo stretchy="false" id="alg2.l6.m1.1.1.1" xref="alg2.l6.m1.1.1.1.cmml">←</mo><mtext id="alg2.l6.m1.1.1.3" xref="alg2.l6.m1.1.1.3a.cmml">[ ]</mtext></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.1b"><apply id="alg2.l6.m1.1.1.cmml" xref="alg2.l6.m1.1.1"><ci id="alg2.l6.m1.1.1.1.cmml" xref="alg2.l6.m1.1.1.1">←</ci><ci id="alg2.l6.m1.1.1.2.cmml" xref="alg2.l6.m1.1.1.2">𝒗</ci><ci id="alg2.l6.m1.1.1.3a.cmml" xref="alg2.l6.m1.1.1.3"><mtext id="alg2.l6.m1.1.1.3.cmml" xref="alg2.l6.m1.1.1.3">[ ]</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.1c">\bm{v}\leftarrow\text{[ ]}</annotation></semantics></math> 
<span id="alg2.l6.2" class="ltx_text" style="color:#009900;">
<span id="alg2.l6.2.1" class="ltx_text" style="float:right;">// empty list of split thresholds</span></span>


</div>
<div id="alg2.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>     <span id="alg2.l7.2" class="ltx_text ltx_font_bold">for</span> <math id="alg2.l7.m1.1" class="ltx_Math" alttext="i\leftarrow 1\textbf{ to }2^{t-1}" display="inline"><semantics id="alg2.l7.m1.1a"><mrow id="alg2.l7.m1.1.1" xref="alg2.l7.m1.1.1.cmml"><mi id="alg2.l7.m1.1.1.2" xref="alg2.l7.m1.1.1.2.cmml">i</mi><mo stretchy="false" id="alg2.l7.m1.1.1.1" xref="alg2.l7.m1.1.1.1.cmml">←</mo><mrow id="alg2.l7.m1.1.1.3" xref="alg2.l7.m1.1.1.3.cmml"><mn id="alg2.l7.m1.1.1.3.2" xref="alg2.l7.m1.1.1.3.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg2.l7.m1.1.1.3.1" xref="alg2.l7.m1.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg2.l7.m1.1.1.3.3" xref="alg2.l7.m1.1.1.3.3a.cmml"> to </mtext><mo lspace="0em" rspace="0em" id="alg2.l7.m1.1.1.3.1a" xref="alg2.l7.m1.1.1.3.1.cmml">​</mo><msup id="alg2.l7.m1.1.1.3.4" xref="alg2.l7.m1.1.1.3.4.cmml"><mn id="alg2.l7.m1.1.1.3.4.2" xref="alg2.l7.m1.1.1.3.4.2.cmml">2</mn><mrow id="alg2.l7.m1.1.1.3.4.3" xref="alg2.l7.m1.1.1.3.4.3.cmml"><mi id="alg2.l7.m1.1.1.3.4.3.2" xref="alg2.l7.m1.1.1.3.4.3.2.cmml">t</mi><mo id="alg2.l7.m1.1.1.3.4.3.1" xref="alg2.l7.m1.1.1.3.4.3.1.cmml">−</mo><mn id="alg2.l7.m1.1.1.3.4.3.3" xref="alg2.l7.m1.1.1.3.4.3.3.cmml">1</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l7.m1.1b"><apply id="alg2.l7.m1.1.1.cmml" xref="alg2.l7.m1.1.1"><ci id="alg2.l7.m1.1.1.1.cmml" xref="alg2.l7.m1.1.1.1">←</ci><ci id="alg2.l7.m1.1.1.2.cmml" xref="alg2.l7.m1.1.1.2">𝑖</ci><apply id="alg2.l7.m1.1.1.3.cmml" xref="alg2.l7.m1.1.1.3"><times id="alg2.l7.m1.1.1.3.1.cmml" xref="alg2.l7.m1.1.1.3.1"></times><cn type="integer" id="alg2.l7.m1.1.1.3.2.cmml" xref="alg2.l7.m1.1.1.3.2">1</cn><ci id="alg2.l7.m1.1.1.3.3a.cmml" xref="alg2.l7.m1.1.1.3.3"><mtext class="ltx_mathvariant_bold" id="alg2.l7.m1.1.1.3.3.cmml" xref="alg2.l7.m1.1.1.3.3"> to </mtext></ci><apply id="alg2.l7.m1.1.1.3.4.cmml" xref="alg2.l7.m1.1.1.3.4"><csymbol cd="ambiguous" id="alg2.l7.m1.1.1.3.4.1.cmml" xref="alg2.l7.m1.1.1.3.4">superscript</csymbol><cn type="integer" id="alg2.l7.m1.1.1.3.4.2.cmml" xref="alg2.l7.m1.1.1.3.4.2">2</cn><apply id="alg2.l7.m1.1.1.3.4.3.cmml" xref="alg2.l7.m1.1.1.3.4.3"><minus id="alg2.l7.m1.1.1.3.4.3.1.cmml" xref="alg2.l7.m1.1.1.3.4.3.1"></minus><ci id="alg2.l7.m1.1.1.3.4.3.2.cmml" xref="alg2.l7.m1.1.1.3.4.3.2">𝑡</ci><cn type="integer" id="alg2.l7.m1.1.1.3.4.3.3.cmml" xref="alg2.l7.m1.1.1.3.4.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l7.m1.1c">i\leftarrow 1\textbf{ to }2^{t-1}</annotation></semantics></math> <span id="alg2.l7.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg2.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>        <math id="alg2.l8.m1.1" class="ltx_Math" alttext="v_{i}\text{, }l_{i}\leftarrow\texttt{optimal\_split\_threshold}(j\text{, }\mathcal{B}^{t-1}_{i})" display="inline"><semantics id="alg2.l8.m1.1a"><mrow id="alg2.l8.m1.1.1" xref="alg2.l8.m1.1.1.cmml"><mrow id="alg2.l8.m1.1.1.3" xref="alg2.l8.m1.1.1.3.cmml"><msub id="alg2.l8.m1.1.1.3.2" xref="alg2.l8.m1.1.1.3.2.cmml"><mi id="alg2.l8.m1.1.1.3.2.2" xref="alg2.l8.m1.1.1.3.2.2.cmml">v</mi><mi id="alg2.l8.m1.1.1.3.2.3" xref="alg2.l8.m1.1.1.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="alg2.l8.m1.1.1.3.1" xref="alg2.l8.m1.1.1.3.1.cmml">​</mo><mtext id="alg2.l8.m1.1.1.3.3" xref="alg2.l8.m1.1.1.3.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l8.m1.1.1.3.1a" xref="alg2.l8.m1.1.1.3.1.cmml">​</mo><msub id="alg2.l8.m1.1.1.3.4" xref="alg2.l8.m1.1.1.3.4.cmml"><mi id="alg2.l8.m1.1.1.3.4.2" xref="alg2.l8.m1.1.1.3.4.2.cmml">l</mi><mi id="alg2.l8.m1.1.1.3.4.3" xref="alg2.l8.m1.1.1.3.4.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="alg2.l8.m1.1.1.2" xref="alg2.l8.m1.1.1.2.cmml">←</mo><mrow id="alg2.l8.m1.1.1.1" xref="alg2.l8.m1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg2.l8.m1.1.1.1.3" xref="alg2.l8.m1.1.1.1.3a.cmml">optimal_split_threshold</mtext><mo lspace="0em" rspace="0em" id="alg2.l8.m1.1.1.1.2" xref="alg2.l8.m1.1.1.1.2.cmml">​</mo><mrow id="alg2.l8.m1.1.1.1.1.1" xref="alg2.l8.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.l8.m1.1.1.1.1.1.2" xref="alg2.l8.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.l8.m1.1.1.1.1.1.1" xref="alg2.l8.m1.1.1.1.1.1.1.cmml"><mi id="alg2.l8.m1.1.1.1.1.1.1.2" xref="alg2.l8.m1.1.1.1.1.1.1.2.cmml">j</mi><mo lspace="0em" rspace="0em" id="alg2.l8.m1.1.1.1.1.1.1.1" xref="alg2.l8.m1.1.1.1.1.1.1.1.cmml">​</mo><mtext id="alg2.l8.m1.1.1.1.1.1.1.3" xref="alg2.l8.m1.1.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l8.m1.1.1.1.1.1.1.1a" xref="alg2.l8.m1.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="alg2.l8.m1.1.1.1.1.1.1.4" xref="alg2.l8.m1.1.1.1.1.1.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l8.m1.1.1.1.1.1.1.4.2.2" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.2.cmml">ℬ</mi><mi id="alg2.l8.m1.1.1.1.1.1.1.4.3" xref="alg2.l8.m1.1.1.1.1.1.1.4.3.cmml">i</mi><mrow id="alg2.l8.m1.1.1.1.1.1.1.4.2.3" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.cmml"><mi id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.2" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.2.cmml">t</mi><mo id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.1" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.1.cmml">−</mo><mn id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.3" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.3.cmml">1</mn></mrow></msubsup></mrow><mo stretchy="false" id="alg2.l8.m1.1.1.1.1.1.3" xref="alg2.l8.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l8.m1.1b"><apply id="alg2.l8.m1.1.1.cmml" xref="alg2.l8.m1.1.1"><ci id="alg2.l8.m1.1.1.2.cmml" xref="alg2.l8.m1.1.1.2">←</ci><apply id="alg2.l8.m1.1.1.3.cmml" xref="alg2.l8.m1.1.1.3"><times id="alg2.l8.m1.1.1.3.1.cmml" xref="alg2.l8.m1.1.1.3.1"></times><apply id="alg2.l8.m1.1.1.3.2.cmml" xref="alg2.l8.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.3.2.1.cmml" xref="alg2.l8.m1.1.1.3.2">subscript</csymbol><ci id="alg2.l8.m1.1.1.3.2.2.cmml" xref="alg2.l8.m1.1.1.3.2.2">𝑣</ci><ci id="alg2.l8.m1.1.1.3.2.3.cmml" xref="alg2.l8.m1.1.1.3.2.3">𝑖</ci></apply><ci id="alg2.l8.m1.1.1.3.3a.cmml" xref="alg2.l8.m1.1.1.3.3"><mtext id="alg2.l8.m1.1.1.3.3.cmml" xref="alg2.l8.m1.1.1.3.3">, </mtext></ci><apply id="alg2.l8.m1.1.1.3.4.cmml" xref="alg2.l8.m1.1.1.3.4"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.3.4.1.cmml" xref="alg2.l8.m1.1.1.3.4">subscript</csymbol><ci id="alg2.l8.m1.1.1.3.4.2.cmml" xref="alg2.l8.m1.1.1.3.4.2">𝑙</ci><ci id="alg2.l8.m1.1.1.3.4.3.cmml" xref="alg2.l8.m1.1.1.3.4.3">𝑖</ci></apply></apply><apply id="alg2.l8.m1.1.1.1.cmml" xref="alg2.l8.m1.1.1.1"><times id="alg2.l8.m1.1.1.1.2.cmml" xref="alg2.l8.m1.1.1.1.2"></times><ci id="alg2.l8.m1.1.1.1.3a.cmml" xref="alg2.l8.m1.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="alg2.l8.m1.1.1.1.3.cmml" xref="alg2.l8.m1.1.1.1.3">optimal_split_threshold</mtext></ci><apply id="alg2.l8.m1.1.1.1.1.1.1.cmml" xref="alg2.l8.m1.1.1.1.1.1"><times id="alg2.l8.m1.1.1.1.1.1.1.1.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.1"></times><ci id="alg2.l8.m1.1.1.1.1.1.1.2.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.2">𝑗</ci><ci id="alg2.l8.m1.1.1.1.1.1.1.3a.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.3"><mtext id="alg2.l8.m1.1.1.1.1.1.1.3.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.3">, </mtext></ci><apply id="alg2.l8.m1.1.1.1.1.1.1.4.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.1.1.1.1.4.1.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4">subscript</csymbol><apply id="alg2.l8.m1.1.1.1.1.1.1.4.2.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="alg2.l8.m1.1.1.1.1.1.1.4.2.1.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4">superscript</csymbol><ci id="alg2.l8.m1.1.1.1.1.1.1.4.2.2.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.2">ℬ</ci><apply id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3"><minus id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.1.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.1"></minus><ci id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.2.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.2">𝑡</ci><cn type="integer" id="alg2.l8.m1.1.1.1.1.1.1.4.2.3.3.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4.2.3.3">1</cn></apply></apply><ci id="alg2.l8.m1.1.1.1.1.1.1.4.3.cmml" xref="alg2.l8.m1.1.1.1.1.1.1.4.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m1.1c">v_{i}\text{, }l_{i}\leftarrow\texttt{optimal\_split\_threshold}(j\text{, }\mathcal{B}^{t-1}_{i})</annotation></semantics></math>

</div>
<div id="alg2.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l9.2.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>        <math id="alg2.l9.m1.2" class="ltx_Math" alttext="\texttt{append}(\bm{v},v_{i})" display="inline"><semantics id="alg2.l9.m1.2a"><mrow id="alg2.l9.m1.2.2" xref="alg2.l9.m1.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg2.l9.m1.2.2.3" xref="alg2.l9.m1.2.2.3a.cmml">append</mtext><mo lspace="0em" rspace="0em" id="alg2.l9.m1.2.2.2" xref="alg2.l9.m1.2.2.2.cmml">​</mo><mrow id="alg2.l9.m1.2.2.1.1" xref="alg2.l9.m1.2.2.1.2.cmml"><mo stretchy="false" id="alg2.l9.m1.2.2.1.1.2" xref="alg2.l9.m1.2.2.1.2.cmml">(</mo><mi id="alg2.l9.m1.1.1" xref="alg2.l9.m1.1.1.cmml">𝒗</mi><mo id="alg2.l9.m1.2.2.1.1.3" xref="alg2.l9.m1.2.2.1.2.cmml">,</mo><msub id="alg2.l9.m1.2.2.1.1.1" xref="alg2.l9.m1.2.2.1.1.1.cmml"><mi id="alg2.l9.m1.2.2.1.1.1.2" xref="alg2.l9.m1.2.2.1.1.1.2.cmml">v</mi><mi id="alg2.l9.m1.2.2.1.1.1.3" xref="alg2.l9.m1.2.2.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="alg2.l9.m1.2.2.1.1.4" xref="alg2.l9.m1.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l9.m1.2b"><apply id="alg2.l9.m1.2.2.cmml" xref="alg2.l9.m1.2.2"><times id="alg2.l9.m1.2.2.2.cmml" xref="alg2.l9.m1.2.2.2"></times><ci id="alg2.l9.m1.2.2.3a.cmml" xref="alg2.l9.m1.2.2.3"><mtext class="ltx_mathvariant_monospace" id="alg2.l9.m1.2.2.3.cmml" xref="alg2.l9.m1.2.2.3">append</mtext></ci><interval closure="open" id="alg2.l9.m1.2.2.1.2.cmml" xref="alg2.l9.m1.2.2.1.1"><ci id="alg2.l9.m1.1.1.cmml" xref="alg2.l9.m1.1.1">𝒗</ci><apply id="alg2.l9.m1.2.2.1.1.1.cmml" xref="alg2.l9.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="alg2.l9.m1.2.2.1.1.1.1.cmml" xref="alg2.l9.m1.2.2.1.1.1">subscript</csymbol><ci id="alg2.l9.m1.2.2.1.1.1.2.cmml" xref="alg2.l9.m1.2.2.1.1.1.2">𝑣</ci><ci id="alg2.l9.m1.2.2.1.1.1.3.cmml" xref="alg2.l9.m1.2.2.1.1.1.3">𝑖</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l9.m1.2c">\texttt{append}(\bm{v},v_{i})</annotation></semantics></math> 
<span id="alg2.l9.1" class="ltx_text" style="color:#009900;">
<span id="alg2.l9.1.1" class="ltx_text" style="float:right;">// append threshold for bucket <math id="alg2.l9.1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg2.l9.1.1.m1.1a"><mi mathcolor="#009900" id="alg2.l9.1.1.m1.1.1" xref="alg2.l9.1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg2.l9.1.1.m1.1b"><ci id="alg2.l9.1.1.m1.1.1.cmml" xref="alg2.l9.1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l9.1.1.m1.1c">i</annotation></semantics></math></span></span>


</div>
<div id="alg2.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l10.2.1.1" class="ltx_text" style="font-size:80%;">10:</span></span>        <math id="alg2.l10.m1.1" class="ltx_Math" alttext="l\leftarrow l+l_{i}" display="inline"><semantics id="alg2.l10.m1.1a"><mrow id="alg2.l10.m1.1.1" xref="alg2.l10.m1.1.1.cmml"><mi id="alg2.l10.m1.1.1.2" xref="alg2.l10.m1.1.1.2.cmml">l</mi><mo stretchy="false" id="alg2.l10.m1.1.1.1" xref="alg2.l10.m1.1.1.1.cmml">←</mo><mrow id="alg2.l10.m1.1.1.3" xref="alg2.l10.m1.1.1.3.cmml"><mi id="alg2.l10.m1.1.1.3.2" xref="alg2.l10.m1.1.1.3.2.cmml">l</mi><mo id="alg2.l10.m1.1.1.3.1" xref="alg2.l10.m1.1.1.3.1.cmml">+</mo><msub id="alg2.l10.m1.1.1.3.3" xref="alg2.l10.m1.1.1.3.3.cmml"><mi id="alg2.l10.m1.1.1.3.3.2" xref="alg2.l10.m1.1.1.3.3.2.cmml">l</mi><mi id="alg2.l10.m1.1.1.3.3.3" xref="alg2.l10.m1.1.1.3.3.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l10.m1.1b"><apply id="alg2.l10.m1.1.1.cmml" xref="alg2.l10.m1.1.1"><ci id="alg2.l10.m1.1.1.1.cmml" xref="alg2.l10.m1.1.1.1">←</ci><ci id="alg2.l10.m1.1.1.2.cmml" xref="alg2.l10.m1.1.1.2">𝑙</ci><apply id="alg2.l10.m1.1.1.3.cmml" xref="alg2.l10.m1.1.1.3"><plus id="alg2.l10.m1.1.1.3.1.cmml" xref="alg2.l10.m1.1.1.3.1"></plus><ci id="alg2.l10.m1.1.1.3.2.cmml" xref="alg2.l10.m1.1.1.3.2">𝑙</ci><apply id="alg2.l10.m1.1.1.3.3.cmml" xref="alg2.l10.m1.1.1.3.3"><csymbol cd="ambiguous" id="alg2.l10.m1.1.1.3.3.1.cmml" xref="alg2.l10.m1.1.1.3.3">subscript</csymbol><ci id="alg2.l10.m1.1.1.3.3.2.cmml" xref="alg2.l10.m1.1.1.3.3.2">𝑙</ci><ci id="alg2.l10.m1.1.1.3.3.3.cmml" xref="alg2.l10.m1.1.1.3.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.m1.1c">l\leftarrow l+l_{i}</annotation></semantics></math> 
<span id="alg2.l10.1" class="ltx_text" style="color:#009900;">
<span id="alg2.l10.1.1" class="ltx_text" style="float:right;">// accumulate loss from bucket <math id="alg2.l10.1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg2.l10.1.1.m1.1a"><mi mathcolor="#009900" id="alg2.l10.1.1.m1.1.1" xref="alg2.l10.1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg2.l10.1.1.m1.1b"><ci id="alg2.l10.1.1.m1.1.1.cmml" xref="alg2.l10.1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.1.1.m1.1c">i</annotation></semantics></math></span></span>


</div>
<div id="alg2.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span>     <span id="alg2.l11.2" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l11.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg2.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span>     <span id="alg2.l12.2" class="ltx_text ltx_font_bold">if</span> <math id="alg2.l12.m1.1" class="ltx_Math" alttext="l&lt;l^{min}" display="inline"><semantics id="alg2.l12.m1.1a"><mrow id="alg2.l12.m1.1.1" xref="alg2.l12.m1.1.1.cmml"><mi id="alg2.l12.m1.1.1.2" xref="alg2.l12.m1.1.1.2.cmml">l</mi><mo id="alg2.l12.m1.1.1.1" xref="alg2.l12.m1.1.1.1.cmml">&lt;</mo><msup id="alg2.l12.m1.1.1.3" xref="alg2.l12.m1.1.1.3.cmml"><mi id="alg2.l12.m1.1.1.3.2" xref="alg2.l12.m1.1.1.3.2.cmml">l</mi><mrow id="alg2.l12.m1.1.1.3.3" xref="alg2.l12.m1.1.1.3.3.cmml"><mi id="alg2.l12.m1.1.1.3.3.2" xref="alg2.l12.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l12.m1.1.1.3.3.1" xref="alg2.l12.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l12.m1.1.1.3.3.3" xref="alg2.l12.m1.1.1.3.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l12.m1.1.1.3.3.1a" xref="alg2.l12.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l12.m1.1.1.3.3.4" xref="alg2.l12.m1.1.1.3.3.4.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l12.m1.1b"><apply id="alg2.l12.m1.1.1.cmml" xref="alg2.l12.m1.1.1"><lt id="alg2.l12.m1.1.1.1.cmml" xref="alg2.l12.m1.1.1.1"></lt><ci id="alg2.l12.m1.1.1.2.cmml" xref="alg2.l12.m1.1.1.2">𝑙</ci><apply id="alg2.l12.m1.1.1.3.cmml" xref="alg2.l12.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l12.m1.1.1.3.1.cmml" xref="alg2.l12.m1.1.1.3">superscript</csymbol><ci id="alg2.l12.m1.1.1.3.2.cmml" xref="alg2.l12.m1.1.1.3.2">𝑙</ci><apply id="alg2.l12.m1.1.1.3.3.cmml" xref="alg2.l12.m1.1.1.3.3"><times id="alg2.l12.m1.1.1.3.3.1.cmml" xref="alg2.l12.m1.1.1.3.3.1"></times><ci id="alg2.l12.m1.1.1.3.3.2.cmml" xref="alg2.l12.m1.1.1.3.3.2">𝑚</ci><ci id="alg2.l12.m1.1.1.3.3.3.cmml" xref="alg2.l12.m1.1.1.3.3.3">𝑖</ci><ci id="alg2.l12.m1.1.1.3.3.4.cmml" xref="alg2.l12.m1.1.1.3.3.4">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l12.m1.1c">l&lt;l^{min}</annotation></semantics></math> <span id="alg2.l12.3" class="ltx_text ltx_font_bold">then</span>



</div>
<div id="alg2.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span>        <math id="alg2.l13.m1.1" class="ltx_Math" alttext="l^{min}\leftarrow l\text{, }j^{min}\leftarrow j\text{, }\bm{v}^{min}\leftarrow\bm{v}" display="inline"><semantics id="alg2.l13.m1.1a"><mrow id="alg2.l13.m1.1.1" xref="alg2.l13.m1.1.1.cmml"><msup id="alg2.l13.m1.1.1.2" xref="alg2.l13.m1.1.1.2.cmml"><mi id="alg2.l13.m1.1.1.2.2" xref="alg2.l13.m1.1.1.2.2.cmml">l</mi><mrow id="alg2.l13.m1.1.1.2.3" xref="alg2.l13.m1.1.1.2.3.cmml"><mi id="alg2.l13.m1.1.1.2.3.2" xref="alg2.l13.m1.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.2.3.1" xref="alg2.l13.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l13.m1.1.1.2.3.3" xref="alg2.l13.m1.1.1.2.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.2.3.1a" xref="alg2.l13.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l13.m1.1.1.2.3.4" xref="alg2.l13.m1.1.1.2.3.4.cmml">n</mi></mrow></msup><mo stretchy="false" id="alg2.l13.m1.1.1.3" xref="alg2.l13.m1.1.1.3.cmml">←</mo><mrow id="alg2.l13.m1.1.1.4" xref="alg2.l13.m1.1.1.4.cmml"><mi id="alg2.l13.m1.1.1.4.2" xref="alg2.l13.m1.1.1.4.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.4.1" xref="alg2.l13.m1.1.1.4.1.cmml">​</mo><mtext id="alg2.l13.m1.1.1.4.3" xref="alg2.l13.m1.1.1.4.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.4.1a" xref="alg2.l13.m1.1.1.4.1.cmml">​</mo><msup id="alg2.l13.m1.1.1.4.4" xref="alg2.l13.m1.1.1.4.4.cmml"><mi id="alg2.l13.m1.1.1.4.4.2" xref="alg2.l13.m1.1.1.4.4.2.cmml">j</mi><mrow id="alg2.l13.m1.1.1.4.4.3" xref="alg2.l13.m1.1.1.4.4.3.cmml"><mi id="alg2.l13.m1.1.1.4.4.3.2" xref="alg2.l13.m1.1.1.4.4.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.4.4.3.1" xref="alg2.l13.m1.1.1.4.4.3.1.cmml">​</mo><mi id="alg2.l13.m1.1.1.4.4.3.3" xref="alg2.l13.m1.1.1.4.4.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.4.4.3.1a" xref="alg2.l13.m1.1.1.4.4.3.1.cmml">​</mo><mi id="alg2.l13.m1.1.1.4.4.3.4" xref="alg2.l13.m1.1.1.4.4.3.4.cmml">n</mi></mrow></msup></mrow><mo stretchy="false" id="alg2.l13.m1.1.1.5" xref="alg2.l13.m1.1.1.5.cmml">←</mo><mrow id="alg2.l13.m1.1.1.6" xref="alg2.l13.m1.1.1.6.cmml"><mi id="alg2.l13.m1.1.1.6.2" xref="alg2.l13.m1.1.1.6.2.cmml">j</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.6.1" xref="alg2.l13.m1.1.1.6.1.cmml">​</mo><mtext id="alg2.l13.m1.1.1.6.3" xref="alg2.l13.m1.1.1.6.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.6.1a" xref="alg2.l13.m1.1.1.6.1.cmml">​</mo><msup id="alg2.l13.m1.1.1.6.4" xref="alg2.l13.m1.1.1.6.4.cmml"><mi id="alg2.l13.m1.1.1.6.4.2" xref="alg2.l13.m1.1.1.6.4.2.cmml">𝒗</mi><mrow id="alg2.l13.m1.1.1.6.4.3" xref="alg2.l13.m1.1.1.6.4.3.cmml"><mi id="alg2.l13.m1.1.1.6.4.3.2" xref="alg2.l13.m1.1.1.6.4.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.6.4.3.1" xref="alg2.l13.m1.1.1.6.4.3.1.cmml">​</mo><mi id="alg2.l13.m1.1.1.6.4.3.3" xref="alg2.l13.m1.1.1.6.4.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l13.m1.1.1.6.4.3.1a" xref="alg2.l13.m1.1.1.6.4.3.1.cmml">​</mo><mi id="alg2.l13.m1.1.1.6.4.3.4" xref="alg2.l13.m1.1.1.6.4.3.4.cmml">n</mi></mrow></msup></mrow><mo stretchy="false" id="alg2.l13.m1.1.1.7" xref="alg2.l13.m1.1.1.7.cmml">←</mo><mi id="alg2.l13.m1.1.1.8" xref="alg2.l13.m1.1.1.8.cmml">𝒗</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.l13.m1.1b"><apply id="alg2.l13.m1.1.1.cmml" xref="alg2.l13.m1.1.1"><and id="alg2.l13.m1.1.1a.cmml" xref="alg2.l13.m1.1.1"></and><apply id="alg2.l13.m1.1.1b.cmml" xref="alg2.l13.m1.1.1"><ci id="alg2.l13.m1.1.1.3.cmml" xref="alg2.l13.m1.1.1.3">←</ci><apply id="alg2.l13.m1.1.1.2.cmml" xref="alg2.l13.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l13.m1.1.1.2.1.cmml" xref="alg2.l13.m1.1.1.2">superscript</csymbol><ci id="alg2.l13.m1.1.1.2.2.cmml" xref="alg2.l13.m1.1.1.2.2">𝑙</ci><apply id="alg2.l13.m1.1.1.2.3.cmml" xref="alg2.l13.m1.1.1.2.3"><times id="alg2.l13.m1.1.1.2.3.1.cmml" xref="alg2.l13.m1.1.1.2.3.1"></times><ci id="alg2.l13.m1.1.1.2.3.2.cmml" xref="alg2.l13.m1.1.1.2.3.2">𝑚</ci><ci id="alg2.l13.m1.1.1.2.3.3.cmml" xref="alg2.l13.m1.1.1.2.3.3">𝑖</ci><ci id="alg2.l13.m1.1.1.2.3.4.cmml" xref="alg2.l13.m1.1.1.2.3.4">𝑛</ci></apply></apply><apply id="alg2.l13.m1.1.1.4.cmml" xref="alg2.l13.m1.1.1.4"><times id="alg2.l13.m1.1.1.4.1.cmml" xref="alg2.l13.m1.1.1.4.1"></times><ci id="alg2.l13.m1.1.1.4.2.cmml" xref="alg2.l13.m1.1.1.4.2">𝑙</ci><ci id="alg2.l13.m1.1.1.4.3a.cmml" xref="alg2.l13.m1.1.1.4.3"><mtext id="alg2.l13.m1.1.1.4.3.cmml" xref="alg2.l13.m1.1.1.4.3">, </mtext></ci><apply id="alg2.l13.m1.1.1.4.4.cmml" xref="alg2.l13.m1.1.1.4.4"><csymbol cd="ambiguous" id="alg2.l13.m1.1.1.4.4.1.cmml" xref="alg2.l13.m1.1.1.4.4">superscript</csymbol><ci id="alg2.l13.m1.1.1.4.4.2.cmml" xref="alg2.l13.m1.1.1.4.4.2">𝑗</ci><apply id="alg2.l13.m1.1.1.4.4.3.cmml" xref="alg2.l13.m1.1.1.4.4.3"><times id="alg2.l13.m1.1.1.4.4.3.1.cmml" xref="alg2.l13.m1.1.1.4.4.3.1"></times><ci id="alg2.l13.m1.1.1.4.4.3.2.cmml" xref="alg2.l13.m1.1.1.4.4.3.2">𝑚</ci><ci id="alg2.l13.m1.1.1.4.4.3.3.cmml" xref="alg2.l13.m1.1.1.4.4.3.3">𝑖</ci><ci id="alg2.l13.m1.1.1.4.4.3.4.cmml" xref="alg2.l13.m1.1.1.4.4.3.4">𝑛</ci></apply></apply></apply></apply><apply id="alg2.l13.m1.1.1c.cmml" xref="alg2.l13.m1.1.1"><ci id="alg2.l13.m1.1.1.5.cmml" xref="alg2.l13.m1.1.1.5">←</ci><share href="#alg2.l13.m1.1.1.4.cmml" id="alg2.l13.m1.1.1d.cmml" xref="alg2.l13.m1.1.1"></share><apply id="alg2.l13.m1.1.1.6.cmml" xref="alg2.l13.m1.1.1.6"><times id="alg2.l13.m1.1.1.6.1.cmml" xref="alg2.l13.m1.1.1.6.1"></times><ci id="alg2.l13.m1.1.1.6.2.cmml" xref="alg2.l13.m1.1.1.6.2">𝑗</ci><ci id="alg2.l13.m1.1.1.6.3a.cmml" xref="alg2.l13.m1.1.1.6.3"><mtext id="alg2.l13.m1.1.1.6.3.cmml" xref="alg2.l13.m1.1.1.6.3">, </mtext></ci><apply id="alg2.l13.m1.1.1.6.4.cmml" xref="alg2.l13.m1.1.1.6.4"><csymbol cd="ambiguous" id="alg2.l13.m1.1.1.6.4.1.cmml" xref="alg2.l13.m1.1.1.6.4">superscript</csymbol><ci id="alg2.l13.m1.1.1.6.4.2.cmml" xref="alg2.l13.m1.1.1.6.4.2">𝒗</ci><apply id="alg2.l13.m1.1.1.6.4.3.cmml" xref="alg2.l13.m1.1.1.6.4.3"><times id="alg2.l13.m1.1.1.6.4.3.1.cmml" xref="alg2.l13.m1.1.1.6.4.3.1"></times><ci id="alg2.l13.m1.1.1.6.4.3.2.cmml" xref="alg2.l13.m1.1.1.6.4.3.2">𝑚</ci><ci id="alg2.l13.m1.1.1.6.4.3.3.cmml" xref="alg2.l13.m1.1.1.6.4.3.3">𝑖</ci><ci id="alg2.l13.m1.1.1.6.4.3.4.cmml" xref="alg2.l13.m1.1.1.6.4.3.4">𝑛</ci></apply></apply></apply></apply><apply id="alg2.l13.m1.1.1e.cmml" xref="alg2.l13.m1.1.1"><ci id="alg2.l13.m1.1.1.7.cmml" xref="alg2.l13.m1.1.1.7">←</ci><share href="#alg2.l13.m1.1.1.6.cmml" id="alg2.l13.m1.1.1f.cmml" xref="alg2.l13.m1.1.1"></share><ci id="alg2.l13.m1.1.1.8.cmml" xref="alg2.l13.m1.1.1.8">𝒗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l13.m1.1c">l^{min}\leftarrow l\text{, }j^{min}\leftarrow j\text{, }\bm{v}^{min}\leftarrow\bm{v}</annotation></semantics></math>

<span id="alg2.l13.2" class="ltx_text" style="color:#009900;">
<span id="alg2.l13.2.1" class="ltx_text" style="float:right;">// new best split</span></span>


</div>
<div id="alg2.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span>     <span id="alg2.l14.2" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l14.3" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg2.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span>  <span id="alg2.l15.2" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l15.3" class="ltx_text ltx_font_bold">for</span>

<span id="alg2.l15.4" class="ltx_text" style="color:#009900;">
// create new buckets using chosen split
</span>


</div>
<div id="alg2.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span>  <math id="alg2.l16.m1.1" class="ltx_Math" alttext="\bm{\mathcal{B}}\leftarrow" display="inline"><semantics id="alg2.l16.m1.1a"><mrow id="alg2.l16.m1.1.1" xref="alg2.l16.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l16.m1.1.1.2" xref="alg2.l16.m1.1.1.2.cmml">𝓑</mi><mo stretchy="false" id="alg2.l16.m1.1.1.1" xref="alg2.l16.m1.1.1.1.cmml">←</mo><mi id="alg2.l16.m1.1.1.3" xref="alg2.l16.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.l16.m1.1b"><apply id="alg2.l16.m1.1.1.cmml" xref="alg2.l16.m1.1.1"><ci id="alg2.l16.m1.1.1.1.cmml" xref="alg2.l16.m1.1.1.1">←</ci><ci id="alg2.l16.m1.1.1.2.cmml" xref="alg2.l16.m1.1.1.2">𝓑</ci><csymbol cd="latexml" id="alg2.l16.m1.1.1.3.cmml" xref="alg2.l16.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l16.m1.1c">\bm{\mathcal{B}}\leftarrow</annotation></semantics></math> [ ] 

</div>
<div id="alg2.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span>  <span id="alg2.l17.2" class="ltx_text ltx_font_bold">for</span> <math id="alg2.l17.m1.1" class="ltx_Math" alttext="i\leftarrow 1\textbf{ to }2^{t-1}" display="inline"><semantics id="alg2.l17.m1.1a"><mrow id="alg2.l17.m1.1.1" xref="alg2.l17.m1.1.1.cmml"><mi id="alg2.l17.m1.1.1.2" xref="alg2.l17.m1.1.1.2.cmml">i</mi><mo stretchy="false" id="alg2.l17.m1.1.1.1" xref="alg2.l17.m1.1.1.1.cmml">←</mo><mrow id="alg2.l17.m1.1.1.3" xref="alg2.l17.m1.1.1.3.cmml"><mn id="alg2.l17.m1.1.1.3.2" xref="alg2.l17.m1.1.1.3.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg2.l17.m1.1.1.3.1" xref="alg2.l17.m1.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg2.l17.m1.1.1.3.3" xref="alg2.l17.m1.1.1.3.3a.cmml"> to </mtext><mo lspace="0em" rspace="0em" id="alg2.l17.m1.1.1.3.1a" xref="alg2.l17.m1.1.1.3.1.cmml">​</mo><msup id="alg2.l17.m1.1.1.3.4" xref="alg2.l17.m1.1.1.3.4.cmml"><mn id="alg2.l17.m1.1.1.3.4.2" xref="alg2.l17.m1.1.1.3.4.2.cmml">2</mn><mrow id="alg2.l17.m1.1.1.3.4.3" xref="alg2.l17.m1.1.1.3.4.3.cmml"><mi id="alg2.l17.m1.1.1.3.4.3.2" xref="alg2.l17.m1.1.1.3.4.3.2.cmml">t</mi><mo id="alg2.l17.m1.1.1.3.4.3.1" xref="alg2.l17.m1.1.1.3.4.3.1.cmml">−</mo><mn id="alg2.l17.m1.1.1.3.4.3.3" xref="alg2.l17.m1.1.1.3.4.3.3.cmml">1</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l17.m1.1b"><apply id="alg2.l17.m1.1.1.cmml" xref="alg2.l17.m1.1.1"><ci id="alg2.l17.m1.1.1.1.cmml" xref="alg2.l17.m1.1.1.1">←</ci><ci id="alg2.l17.m1.1.1.2.cmml" xref="alg2.l17.m1.1.1.2">𝑖</ci><apply id="alg2.l17.m1.1.1.3.cmml" xref="alg2.l17.m1.1.1.3"><times id="alg2.l17.m1.1.1.3.1.cmml" xref="alg2.l17.m1.1.1.3.1"></times><cn type="integer" id="alg2.l17.m1.1.1.3.2.cmml" xref="alg2.l17.m1.1.1.3.2">1</cn><ci id="alg2.l17.m1.1.1.3.3a.cmml" xref="alg2.l17.m1.1.1.3.3"><mtext class="ltx_mathvariant_bold" id="alg2.l17.m1.1.1.3.3.cmml" xref="alg2.l17.m1.1.1.3.3"> to </mtext></ci><apply id="alg2.l17.m1.1.1.3.4.cmml" xref="alg2.l17.m1.1.1.3.4"><csymbol cd="ambiguous" id="alg2.l17.m1.1.1.3.4.1.cmml" xref="alg2.l17.m1.1.1.3.4">superscript</csymbol><cn type="integer" id="alg2.l17.m1.1.1.3.4.2.cmml" xref="alg2.l17.m1.1.1.3.4.2">2</cn><apply id="alg2.l17.m1.1.1.3.4.3.cmml" xref="alg2.l17.m1.1.1.3.4.3"><minus id="alg2.l17.m1.1.1.3.4.3.1.cmml" xref="alg2.l17.m1.1.1.3.4.3.1"></minus><ci id="alg2.l17.m1.1.1.3.4.3.2.cmml" xref="alg2.l17.m1.1.1.3.4.3.2">𝑡</ci><cn type="integer" id="alg2.l17.m1.1.1.3.4.3.3.cmml" xref="alg2.l17.m1.1.1.3.4.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l17.m1.1c">i\leftarrow 1\textbf{ to }2^{t-1}</annotation></semantics></math> <span id="alg2.l17.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg2.l18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l18.1.1.1" class="ltx_text" style="font-size:80%;">18:</span></span>      <math id="alg2.l18.m1.1" class="ltx_Math" alttext="\mathcal{B}_{below}\text{, }\mathcal{B}_{above}\leftarrow\texttt{apply\_split}(v^{min}_{i}\text{, }\mathcal{B}^{t-1}_{i})" display="inline"><semantics id="alg2.l18.m1.1a"><mrow id="alg2.l18.m1.1.1" xref="alg2.l18.m1.1.1.cmml"><mrow id="alg2.l18.m1.1.1.3" xref="alg2.l18.m1.1.1.3.cmml"><msub id="alg2.l18.m1.1.1.3.2" xref="alg2.l18.m1.1.1.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l18.m1.1.1.3.2.2" xref="alg2.l18.m1.1.1.3.2.2.cmml">ℬ</mi><mrow id="alg2.l18.m1.1.1.3.2.3" xref="alg2.l18.m1.1.1.3.2.3.cmml"><mi id="alg2.l18.m1.1.1.3.2.3.2" xref="alg2.l18.m1.1.1.3.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.2.3.1" xref="alg2.l18.m1.1.1.3.2.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.2.3.3" xref="alg2.l18.m1.1.1.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.2.3.1a" xref="alg2.l18.m1.1.1.3.2.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.2.3.4" xref="alg2.l18.m1.1.1.3.2.3.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.2.3.1b" xref="alg2.l18.m1.1.1.3.2.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.2.3.5" xref="alg2.l18.m1.1.1.3.2.3.5.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.2.3.1c" xref="alg2.l18.m1.1.1.3.2.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.2.3.6" xref="alg2.l18.m1.1.1.3.2.3.6.cmml">w</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.1" xref="alg2.l18.m1.1.1.3.1.cmml">​</mo><mtext id="alg2.l18.m1.1.1.3.3" xref="alg2.l18.m1.1.1.3.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.1a" xref="alg2.l18.m1.1.1.3.1.cmml">​</mo><msub id="alg2.l18.m1.1.1.3.4" xref="alg2.l18.m1.1.1.3.4.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l18.m1.1.1.3.4.2" xref="alg2.l18.m1.1.1.3.4.2.cmml">ℬ</mi><mrow id="alg2.l18.m1.1.1.3.4.3" xref="alg2.l18.m1.1.1.3.4.3.cmml"><mi id="alg2.l18.m1.1.1.3.4.3.2" xref="alg2.l18.m1.1.1.3.4.3.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.4.3.1" xref="alg2.l18.m1.1.1.3.4.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.4.3.3" xref="alg2.l18.m1.1.1.3.4.3.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.4.3.1a" xref="alg2.l18.m1.1.1.3.4.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.4.3.4" xref="alg2.l18.m1.1.1.3.4.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.4.3.1b" xref="alg2.l18.m1.1.1.3.4.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.4.3.5" xref="alg2.l18.m1.1.1.3.4.3.5.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.3.4.3.1c" xref="alg2.l18.m1.1.1.3.4.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.3.4.3.6" xref="alg2.l18.m1.1.1.3.4.3.6.cmml">e</mi></mrow></msub></mrow><mo stretchy="false" id="alg2.l18.m1.1.1.2" xref="alg2.l18.m1.1.1.2.cmml">←</mo><mrow id="alg2.l18.m1.1.1.1" xref="alg2.l18.m1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg2.l18.m1.1.1.1.3" xref="alg2.l18.m1.1.1.1.3a.cmml">apply_split</mtext><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.1.2" xref="alg2.l18.m1.1.1.1.2.cmml">​</mo><mrow id="alg2.l18.m1.1.1.1.1.1" xref="alg2.l18.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.l18.m1.1.1.1.1.1.2" xref="alg2.l18.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.l18.m1.1.1.1.1.1.1" xref="alg2.l18.m1.1.1.1.1.1.1.cmml"><msubsup id="alg2.l18.m1.1.1.1.1.1.1.2" xref="alg2.l18.m1.1.1.1.1.1.1.2.cmml"><mi id="alg2.l18.m1.1.1.1.1.1.1.2.2.2" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.2.cmml">v</mi><mi id="alg2.l18.m1.1.1.1.1.1.1.2.3" xref="alg2.l18.m1.1.1.1.1.1.1.2.3.cmml">i</mi><mrow id="alg2.l18.m1.1.1.1.1.1.1.2.2.3" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.cmml"><mi id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.2" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.1" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.3" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.1a" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.4" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.4.cmml">n</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.1.1.1.1.1" xref="alg2.l18.m1.1.1.1.1.1.1.1.cmml">​</mo><mtext id="alg2.l18.m1.1.1.1.1.1.1.3" xref="alg2.l18.m1.1.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l18.m1.1.1.1.1.1.1.1a" xref="alg2.l18.m1.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="alg2.l18.m1.1.1.1.1.1.1.4" xref="alg2.l18.m1.1.1.1.1.1.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l18.m1.1.1.1.1.1.1.4.2.2" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.2.cmml">ℬ</mi><mi id="alg2.l18.m1.1.1.1.1.1.1.4.3" xref="alg2.l18.m1.1.1.1.1.1.1.4.3.cmml">i</mi><mrow id="alg2.l18.m1.1.1.1.1.1.1.4.2.3" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.cmml"><mi id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.2" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.2.cmml">t</mi><mo id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.1" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.1.cmml">−</mo><mn id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.3" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.3.cmml">1</mn></mrow></msubsup></mrow><mo stretchy="false" id="alg2.l18.m1.1.1.1.1.1.3" xref="alg2.l18.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l18.m1.1b"><apply id="alg2.l18.m1.1.1.cmml" xref="alg2.l18.m1.1.1"><ci id="alg2.l18.m1.1.1.2.cmml" xref="alg2.l18.m1.1.1.2">←</ci><apply id="alg2.l18.m1.1.1.3.cmml" xref="alg2.l18.m1.1.1.3"><times id="alg2.l18.m1.1.1.3.1.cmml" xref="alg2.l18.m1.1.1.3.1"></times><apply id="alg2.l18.m1.1.1.3.2.cmml" xref="alg2.l18.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg2.l18.m1.1.1.3.2.1.cmml" xref="alg2.l18.m1.1.1.3.2">subscript</csymbol><ci id="alg2.l18.m1.1.1.3.2.2.cmml" xref="alg2.l18.m1.1.1.3.2.2">ℬ</ci><apply id="alg2.l18.m1.1.1.3.2.3.cmml" xref="alg2.l18.m1.1.1.3.2.3"><times id="alg2.l18.m1.1.1.3.2.3.1.cmml" xref="alg2.l18.m1.1.1.3.2.3.1"></times><ci id="alg2.l18.m1.1.1.3.2.3.2.cmml" xref="alg2.l18.m1.1.1.3.2.3.2">𝑏</ci><ci id="alg2.l18.m1.1.1.3.2.3.3.cmml" xref="alg2.l18.m1.1.1.3.2.3.3">𝑒</ci><ci id="alg2.l18.m1.1.1.3.2.3.4.cmml" xref="alg2.l18.m1.1.1.3.2.3.4">𝑙</ci><ci id="alg2.l18.m1.1.1.3.2.3.5.cmml" xref="alg2.l18.m1.1.1.3.2.3.5">𝑜</ci><ci id="alg2.l18.m1.1.1.3.2.3.6.cmml" xref="alg2.l18.m1.1.1.3.2.3.6">𝑤</ci></apply></apply><ci id="alg2.l18.m1.1.1.3.3a.cmml" xref="alg2.l18.m1.1.1.3.3"><mtext id="alg2.l18.m1.1.1.3.3.cmml" xref="alg2.l18.m1.1.1.3.3">, </mtext></ci><apply id="alg2.l18.m1.1.1.3.4.cmml" xref="alg2.l18.m1.1.1.3.4"><csymbol cd="ambiguous" id="alg2.l18.m1.1.1.3.4.1.cmml" xref="alg2.l18.m1.1.1.3.4">subscript</csymbol><ci id="alg2.l18.m1.1.1.3.4.2.cmml" xref="alg2.l18.m1.1.1.3.4.2">ℬ</ci><apply id="alg2.l18.m1.1.1.3.4.3.cmml" xref="alg2.l18.m1.1.1.3.4.3"><times id="alg2.l18.m1.1.1.3.4.3.1.cmml" xref="alg2.l18.m1.1.1.3.4.3.1"></times><ci id="alg2.l18.m1.1.1.3.4.3.2.cmml" xref="alg2.l18.m1.1.1.3.4.3.2">𝑎</ci><ci id="alg2.l18.m1.1.1.3.4.3.3.cmml" xref="alg2.l18.m1.1.1.3.4.3.3">𝑏</ci><ci id="alg2.l18.m1.1.1.3.4.3.4.cmml" xref="alg2.l18.m1.1.1.3.4.3.4">𝑜</ci><ci id="alg2.l18.m1.1.1.3.4.3.5.cmml" xref="alg2.l18.m1.1.1.3.4.3.5">𝑣</ci><ci id="alg2.l18.m1.1.1.3.4.3.6.cmml" xref="alg2.l18.m1.1.1.3.4.3.6">𝑒</ci></apply></apply></apply><apply id="alg2.l18.m1.1.1.1.cmml" xref="alg2.l18.m1.1.1.1"><times id="alg2.l18.m1.1.1.1.2.cmml" xref="alg2.l18.m1.1.1.1.2"></times><ci id="alg2.l18.m1.1.1.1.3a.cmml" xref="alg2.l18.m1.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="alg2.l18.m1.1.1.1.3.cmml" xref="alg2.l18.m1.1.1.1.3">apply_split</mtext></ci><apply id="alg2.l18.m1.1.1.1.1.1.1.cmml" xref="alg2.l18.m1.1.1.1.1.1"><times id="alg2.l18.m1.1.1.1.1.1.1.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.1"></times><apply id="alg2.l18.m1.1.1.1.1.1.1.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.l18.m1.1.1.1.1.1.1.2.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2">subscript</csymbol><apply id="alg2.l18.m1.1.1.1.1.1.1.2.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.l18.m1.1.1.1.1.1.1.2.2.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2">superscript</csymbol><ci id="alg2.l18.m1.1.1.1.1.1.1.2.2.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.2">𝑣</ci><apply id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3"><times id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.1"></times><ci id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.2">𝑚</ci><ci id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.3">𝑖</ci><ci id="alg2.l18.m1.1.1.1.1.1.1.2.2.3.4.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.2.3.4">𝑛</ci></apply></apply><ci id="alg2.l18.m1.1.1.1.1.1.1.2.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><ci id="alg2.l18.m1.1.1.1.1.1.1.3a.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.3"><mtext id="alg2.l18.m1.1.1.1.1.1.1.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.3">, </mtext></ci><apply id="alg2.l18.m1.1.1.1.1.1.1.4.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="alg2.l18.m1.1.1.1.1.1.1.4.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4">subscript</csymbol><apply id="alg2.l18.m1.1.1.1.1.1.1.4.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="alg2.l18.m1.1.1.1.1.1.1.4.2.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4">superscript</csymbol><ci id="alg2.l18.m1.1.1.1.1.1.1.4.2.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.2">ℬ</ci><apply id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3"><minus id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.1.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.1"></minus><ci id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.2.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.2">𝑡</ci><cn type="integer" id="alg2.l18.m1.1.1.1.1.1.1.4.2.3.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4.2.3.3">1</cn></apply></apply><ci id="alg2.l18.m1.1.1.1.1.1.1.4.3.cmml" xref="alg2.l18.m1.1.1.1.1.1.1.4.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l18.m1.1c">\mathcal{B}_{below}\text{, }\mathcal{B}_{above}\leftarrow\texttt{apply\_split}(v^{min}_{i}\text{, }\mathcal{B}^{t-1}_{i})</annotation></semantics></math>

</div>
<div id="alg2.l19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l19.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span>     <math id="alg2.l19.m1.2" class="ltx_Math" alttext="\texttt{append}(\bm{\mathcal{B}},\mathcal{B}_{below})" display="inline"><semantics id="alg2.l19.m1.2a"><mrow id="alg2.l19.m1.2.2" xref="alg2.l19.m1.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg2.l19.m1.2.2.3" xref="alg2.l19.m1.2.2.3a.cmml">append</mtext><mo lspace="0em" rspace="0em" id="alg2.l19.m1.2.2.2" xref="alg2.l19.m1.2.2.2.cmml">​</mo><mrow id="alg2.l19.m1.2.2.1.1" xref="alg2.l19.m1.2.2.1.2.cmml"><mo stretchy="false" id="alg2.l19.m1.2.2.1.1.2" xref="alg2.l19.m1.2.2.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="alg2.l19.m1.1.1" xref="alg2.l19.m1.1.1.cmml">𝓑</mi><mo id="alg2.l19.m1.2.2.1.1.3" xref="alg2.l19.m1.2.2.1.2.cmml">,</mo><msub id="alg2.l19.m1.2.2.1.1.1" xref="alg2.l19.m1.2.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l19.m1.2.2.1.1.1.2" xref="alg2.l19.m1.2.2.1.1.1.2.cmml">ℬ</mi><mrow id="alg2.l19.m1.2.2.1.1.1.3" xref="alg2.l19.m1.2.2.1.1.1.3.cmml"><mi id="alg2.l19.m1.2.2.1.1.1.3.2" xref="alg2.l19.m1.2.2.1.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l19.m1.2.2.1.1.1.3.1" xref="alg2.l19.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l19.m1.2.2.1.1.1.3.3" xref="alg2.l19.m1.2.2.1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l19.m1.2.2.1.1.1.3.1a" xref="alg2.l19.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l19.m1.2.2.1.1.1.3.4" xref="alg2.l19.m1.2.2.1.1.1.3.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l19.m1.2.2.1.1.1.3.1b" xref="alg2.l19.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l19.m1.2.2.1.1.1.3.5" xref="alg2.l19.m1.2.2.1.1.1.3.5.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg2.l19.m1.2.2.1.1.1.3.1c" xref="alg2.l19.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l19.m1.2.2.1.1.1.3.6" xref="alg2.l19.m1.2.2.1.1.1.3.6.cmml">w</mi></mrow></msub><mo stretchy="false" id="alg2.l19.m1.2.2.1.1.4" xref="alg2.l19.m1.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l19.m1.2b"><apply id="alg2.l19.m1.2.2.cmml" xref="alg2.l19.m1.2.2"><times id="alg2.l19.m1.2.2.2.cmml" xref="alg2.l19.m1.2.2.2"></times><ci id="alg2.l19.m1.2.2.3a.cmml" xref="alg2.l19.m1.2.2.3"><mtext class="ltx_mathvariant_monospace" id="alg2.l19.m1.2.2.3.cmml" xref="alg2.l19.m1.2.2.3">append</mtext></ci><interval closure="open" id="alg2.l19.m1.2.2.1.2.cmml" xref="alg2.l19.m1.2.2.1.1"><ci id="alg2.l19.m1.1.1.cmml" xref="alg2.l19.m1.1.1">𝓑</ci><apply id="alg2.l19.m1.2.2.1.1.1.cmml" xref="alg2.l19.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="alg2.l19.m1.2.2.1.1.1.1.cmml" xref="alg2.l19.m1.2.2.1.1.1">subscript</csymbol><ci id="alg2.l19.m1.2.2.1.1.1.2.cmml" xref="alg2.l19.m1.2.2.1.1.1.2">ℬ</ci><apply id="alg2.l19.m1.2.2.1.1.1.3.cmml" xref="alg2.l19.m1.2.2.1.1.1.3"><times id="alg2.l19.m1.2.2.1.1.1.3.1.cmml" xref="alg2.l19.m1.2.2.1.1.1.3.1"></times><ci id="alg2.l19.m1.2.2.1.1.1.3.2.cmml" xref="alg2.l19.m1.2.2.1.1.1.3.2">𝑏</ci><ci id="alg2.l19.m1.2.2.1.1.1.3.3.cmml" xref="alg2.l19.m1.2.2.1.1.1.3.3">𝑒</ci><ci id="alg2.l19.m1.2.2.1.1.1.3.4.cmml" xref="alg2.l19.m1.2.2.1.1.1.3.4">𝑙</ci><ci id="alg2.l19.m1.2.2.1.1.1.3.5.cmml" xref="alg2.l19.m1.2.2.1.1.1.3.5">𝑜</ci><ci id="alg2.l19.m1.2.2.1.1.1.3.6.cmml" xref="alg2.l19.m1.2.2.1.1.1.3.6">𝑤</ci></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l19.m1.2c">\texttt{append}(\bm{\mathcal{B}},\mathcal{B}_{below})</annotation></semantics></math>

</div>
<div id="alg2.l20" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l20.1.1.1" class="ltx_text" style="font-size:80%;">20:</span></span>     <math id="alg2.l20.m1.2" class="ltx_Math" alttext="\texttt{append}(\bm{\mathcal{B}},\mathcal{B}_{above})" display="inline"><semantics id="alg2.l20.m1.2a"><mrow id="alg2.l20.m1.2.2" xref="alg2.l20.m1.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg2.l20.m1.2.2.3" xref="alg2.l20.m1.2.2.3a.cmml">append</mtext><mo lspace="0em" rspace="0em" id="alg2.l20.m1.2.2.2" xref="alg2.l20.m1.2.2.2.cmml">​</mo><mrow id="alg2.l20.m1.2.2.1.1" xref="alg2.l20.m1.2.2.1.2.cmml"><mo stretchy="false" id="alg2.l20.m1.2.2.1.1.2" xref="alg2.l20.m1.2.2.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="alg2.l20.m1.1.1" xref="alg2.l20.m1.1.1.cmml">𝓑</mi><mo id="alg2.l20.m1.2.2.1.1.3" xref="alg2.l20.m1.2.2.1.2.cmml">,</mo><msub id="alg2.l20.m1.2.2.1.1.1" xref="alg2.l20.m1.2.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l20.m1.2.2.1.1.1.2" xref="alg2.l20.m1.2.2.1.1.1.2.cmml">ℬ</mi><mrow id="alg2.l20.m1.2.2.1.1.1.3" xref="alg2.l20.m1.2.2.1.1.1.3.cmml"><mi id="alg2.l20.m1.2.2.1.1.1.3.2" xref="alg2.l20.m1.2.2.1.1.1.3.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg2.l20.m1.2.2.1.1.1.3.1" xref="alg2.l20.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l20.m1.2.2.1.1.1.3.3" xref="alg2.l20.m1.2.2.1.1.1.3.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l20.m1.2.2.1.1.1.3.1a" xref="alg2.l20.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l20.m1.2.2.1.1.1.3.4" xref="alg2.l20.m1.2.2.1.1.1.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg2.l20.m1.2.2.1.1.1.3.1b" xref="alg2.l20.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l20.m1.2.2.1.1.1.3.5" xref="alg2.l20.m1.2.2.1.1.1.3.5.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg2.l20.m1.2.2.1.1.1.3.1c" xref="alg2.l20.m1.2.2.1.1.1.3.1.cmml">​</mo><mi id="alg2.l20.m1.2.2.1.1.1.3.6" xref="alg2.l20.m1.2.2.1.1.1.3.6.cmml">e</mi></mrow></msub><mo stretchy="false" id="alg2.l20.m1.2.2.1.1.4" xref="alg2.l20.m1.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l20.m1.2b"><apply id="alg2.l20.m1.2.2.cmml" xref="alg2.l20.m1.2.2"><times id="alg2.l20.m1.2.2.2.cmml" xref="alg2.l20.m1.2.2.2"></times><ci id="alg2.l20.m1.2.2.3a.cmml" xref="alg2.l20.m1.2.2.3"><mtext class="ltx_mathvariant_monospace" id="alg2.l20.m1.2.2.3.cmml" xref="alg2.l20.m1.2.2.3">append</mtext></ci><interval closure="open" id="alg2.l20.m1.2.2.1.2.cmml" xref="alg2.l20.m1.2.2.1.1"><ci id="alg2.l20.m1.1.1.cmml" xref="alg2.l20.m1.1.1">𝓑</ci><apply id="alg2.l20.m1.2.2.1.1.1.cmml" xref="alg2.l20.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="alg2.l20.m1.2.2.1.1.1.1.cmml" xref="alg2.l20.m1.2.2.1.1.1">subscript</csymbol><ci id="alg2.l20.m1.2.2.1.1.1.2.cmml" xref="alg2.l20.m1.2.2.1.1.1.2">ℬ</ci><apply id="alg2.l20.m1.2.2.1.1.1.3.cmml" xref="alg2.l20.m1.2.2.1.1.1.3"><times id="alg2.l20.m1.2.2.1.1.1.3.1.cmml" xref="alg2.l20.m1.2.2.1.1.1.3.1"></times><ci id="alg2.l20.m1.2.2.1.1.1.3.2.cmml" xref="alg2.l20.m1.2.2.1.1.1.3.2">𝑎</ci><ci id="alg2.l20.m1.2.2.1.1.1.3.3.cmml" xref="alg2.l20.m1.2.2.1.1.1.3.3">𝑏</ci><ci id="alg2.l20.m1.2.2.1.1.1.3.4.cmml" xref="alg2.l20.m1.2.2.1.1.1.3.4">𝑜</ci><ci id="alg2.l20.m1.2.2.1.1.1.3.5.cmml" xref="alg2.l20.m1.2.2.1.1.1.3.5">𝑣</ci><ci id="alg2.l20.m1.2.2.1.1.1.3.6.cmml" xref="alg2.l20.m1.2.2.1.1.1.3.6">𝑒</ci></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l20.m1.2c">\texttt{append}(\bm{\mathcal{B}},\mathcal{B}_{above})</annotation></semantics></math>

</div>
<div id="alg2.l21" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l21.1.1.1" class="ltx_text" style="font-size:80%;">21:</span></span>  <span id="alg2.l21.2" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l21.3" class="ltx_text ltx_font_bold">for</span>

</div>
<div id="alg2.l22" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l22.1.1.1" class="ltx_text" style="font-size:80%;">22:</span></span>  <span id="alg2.l22.2" class="ltx_text ltx_font_bold">return </span> <math id="alg2.l22.m1.1" class="ltx_Math" alttext="\bm{\mathcal{B}}\text{, }l^{min}\text{, }j^{min}\text{, }v^{min}" display="inline"><semantics id="alg2.l22.m1.1a"><mrow id="alg2.l22.m1.1.1" xref="alg2.l22.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l22.m1.1.1.2" xref="alg2.l22.m1.1.1.2.cmml">𝓑</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.1" xref="alg2.l22.m1.1.1.1.cmml">​</mo><mtext id="alg2.l22.m1.1.1.3" xref="alg2.l22.m1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.1a" xref="alg2.l22.m1.1.1.1.cmml">​</mo><msup id="alg2.l22.m1.1.1.4" xref="alg2.l22.m1.1.1.4.cmml"><mi id="alg2.l22.m1.1.1.4.2" xref="alg2.l22.m1.1.1.4.2.cmml">l</mi><mrow id="alg2.l22.m1.1.1.4.3" xref="alg2.l22.m1.1.1.4.3.cmml"><mi id="alg2.l22.m1.1.1.4.3.2" xref="alg2.l22.m1.1.1.4.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.4.3.1" xref="alg2.l22.m1.1.1.4.3.1.cmml">​</mo><mi id="alg2.l22.m1.1.1.4.3.3" xref="alg2.l22.m1.1.1.4.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.4.3.1a" xref="alg2.l22.m1.1.1.4.3.1.cmml">​</mo><mi id="alg2.l22.m1.1.1.4.3.4" xref="alg2.l22.m1.1.1.4.3.4.cmml">n</mi></mrow></msup><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.1b" xref="alg2.l22.m1.1.1.1.cmml">​</mo><mtext id="alg2.l22.m1.1.1.5" xref="alg2.l22.m1.1.1.5a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.1c" xref="alg2.l22.m1.1.1.1.cmml">​</mo><msup id="alg2.l22.m1.1.1.6" xref="alg2.l22.m1.1.1.6.cmml"><mi id="alg2.l22.m1.1.1.6.2" xref="alg2.l22.m1.1.1.6.2.cmml">j</mi><mrow id="alg2.l22.m1.1.1.6.3" xref="alg2.l22.m1.1.1.6.3.cmml"><mi id="alg2.l22.m1.1.1.6.3.2" xref="alg2.l22.m1.1.1.6.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.6.3.1" xref="alg2.l22.m1.1.1.6.3.1.cmml">​</mo><mi id="alg2.l22.m1.1.1.6.3.3" xref="alg2.l22.m1.1.1.6.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.6.3.1a" xref="alg2.l22.m1.1.1.6.3.1.cmml">​</mo><mi id="alg2.l22.m1.1.1.6.3.4" xref="alg2.l22.m1.1.1.6.3.4.cmml">n</mi></mrow></msup><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.1d" xref="alg2.l22.m1.1.1.1.cmml">​</mo><mtext id="alg2.l22.m1.1.1.7" xref="alg2.l22.m1.1.1.7a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.1e" xref="alg2.l22.m1.1.1.1.cmml">​</mo><msup id="alg2.l22.m1.1.1.8" xref="alg2.l22.m1.1.1.8.cmml"><mi id="alg2.l22.m1.1.1.8.2" xref="alg2.l22.m1.1.1.8.2.cmml">v</mi><mrow id="alg2.l22.m1.1.1.8.3" xref="alg2.l22.m1.1.1.8.3.cmml"><mi id="alg2.l22.m1.1.1.8.3.2" xref="alg2.l22.m1.1.1.8.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.8.3.1" xref="alg2.l22.m1.1.1.8.3.1.cmml">​</mo><mi id="alg2.l22.m1.1.1.8.3.3" xref="alg2.l22.m1.1.1.8.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l22.m1.1.1.8.3.1a" xref="alg2.l22.m1.1.1.8.3.1.cmml">​</mo><mi id="alg2.l22.m1.1.1.8.3.4" xref="alg2.l22.m1.1.1.8.3.4.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l22.m1.1b"><apply id="alg2.l22.m1.1.1.cmml" xref="alg2.l22.m1.1.1"><times id="alg2.l22.m1.1.1.1.cmml" xref="alg2.l22.m1.1.1.1"></times><ci id="alg2.l22.m1.1.1.2.cmml" xref="alg2.l22.m1.1.1.2">𝓑</ci><ci id="alg2.l22.m1.1.1.3a.cmml" xref="alg2.l22.m1.1.1.3"><mtext id="alg2.l22.m1.1.1.3.cmml" xref="alg2.l22.m1.1.1.3">, </mtext></ci><apply id="alg2.l22.m1.1.1.4.cmml" xref="alg2.l22.m1.1.1.4"><csymbol cd="ambiguous" id="alg2.l22.m1.1.1.4.1.cmml" xref="alg2.l22.m1.1.1.4">superscript</csymbol><ci id="alg2.l22.m1.1.1.4.2.cmml" xref="alg2.l22.m1.1.1.4.2">𝑙</ci><apply id="alg2.l22.m1.1.1.4.3.cmml" xref="alg2.l22.m1.1.1.4.3"><times id="alg2.l22.m1.1.1.4.3.1.cmml" xref="alg2.l22.m1.1.1.4.3.1"></times><ci id="alg2.l22.m1.1.1.4.3.2.cmml" xref="alg2.l22.m1.1.1.4.3.2">𝑚</ci><ci id="alg2.l22.m1.1.1.4.3.3.cmml" xref="alg2.l22.m1.1.1.4.3.3">𝑖</ci><ci id="alg2.l22.m1.1.1.4.3.4.cmml" xref="alg2.l22.m1.1.1.4.3.4">𝑛</ci></apply></apply><ci id="alg2.l22.m1.1.1.5a.cmml" xref="alg2.l22.m1.1.1.5"><mtext id="alg2.l22.m1.1.1.5.cmml" xref="alg2.l22.m1.1.1.5">, </mtext></ci><apply id="alg2.l22.m1.1.1.6.cmml" xref="alg2.l22.m1.1.1.6"><csymbol cd="ambiguous" id="alg2.l22.m1.1.1.6.1.cmml" xref="alg2.l22.m1.1.1.6">superscript</csymbol><ci id="alg2.l22.m1.1.1.6.2.cmml" xref="alg2.l22.m1.1.1.6.2">𝑗</ci><apply id="alg2.l22.m1.1.1.6.3.cmml" xref="alg2.l22.m1.1.1.6.3"><times id="alg2.l22.m1.1.1.6.3.1.cmml" xref="alg2.l22.m1.1.1.6.3.1"></times><ci id="alg2.l22.m1.1.1.6.3.2.cmml" xref="alg2.l22.m1.1.1.6.3.2">𝑚</ci><ci id="alg2.l22.m1.1.1.6.3.3.cmml" xref="alg2.l22.m1.1.1.6.3.3">𝑖</ci><ci id="alg2.l22.m1.1.1.6.3.4.cmml" xref="alg2.l22.m1.1.1.6.3.4">𝑛</ci></apply></apply><ci id="alg2.l22.m1.1.1.7a.cmml" xref="alg2.l22.m1.1.1.7"><mtext id="alg2.l22.m1.1.1.7.cmml" xref="alg2.l22.m1.1.1.7">, </mtext></ci><apply id="alg2.l22.m1.1.1.8.cmml" xref="alg2.l22.m1.1.1.8"><csymbol cd="ambiguous" id="alg2.l22.m1.1.1.8.1.cmml" xref="alg2.l22.m1.1.1.8">superscript</csymbol><ci id="alg2.l22.m1.1.1.8.2.cmml" xref="alg2.l22.m1.1.1.8.2">𝑣</ci><apply id="alg2.l22.m1.1.1.8.3.cmml" xref="alg2.l22.m1.1.1.8.3"><times id="alg2.l22.m1.1.1.8.3.1.cmml" xref="alg2.l22.m1.1.1.8.3.1"></times><ci id="alg2.l22.m1.1.1.8.3.2.cmml" xref="alg2.l22.m1.1.1.8.3.2">𝑚</ci><ci id="alg2.l22.m1.1.1.8.3.3.cmml" xref="alg2.l22.m1.1.1.8.3.3">𝑖</ci><ci id="alg2.l22.m1.1.1.8.3.4.cmml" xref="alg2.l22.m1.1.1.8.3.4">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l22.m1.1c">\bm{\mathcal{B}}\text{, }l^{min}\text{, }j^{min}\text{, }v^{min}</annotation></semantics></math>

</div>
</div>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Optimizing the Prototypes</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">At this point, we have a complete algorithm. We could simply drop our hash-based encoding function into PQ and approximate matrix products. However, we contribute two additional enhancements: a means of optimizing the prototypes with no runtime overhead, and a means of quickly summing low-bitwidth integers.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.4" class="ltx_p">Several works propose prototype or table optimizations based on knowledge of <math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><mi id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><ci id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">\bm{B}</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Babenko et al., <a href="#bib.bib8" title="" class="ltx_ref">2016</a>; Wang et al., <a href="#bib.bib65" title="" class="ltx_ref">2014b</a>)</cite>, and others optimize them at the expense of slowing down the function <math id="S4.SS3.p2.2.m2.1" class="ltx_Math" alttext="g(\cdot)" display="inline"><semantics id="S4.SS3.p2.2.m2.1a"><mrow id="S4.SS3.p2.2.m2.1.2" xref="S4.SS3.p2.2.m2.1.2.cmml"><mi id="S4.SS3.p2.2.m2.1.2.2" xref="S4.SS3.p2.2.m2.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p2.2.m2.1.2.1" xref="S4.SS3.p2.2.m2.1.2.1.cmml">​</mo><mrow id="S4.SS3.p2.2.m2.1.2.3.2" xref="S4.SS3.p2.2.m2.1.2.cmml"><mo stretchy="false" id="S4.SS3.p2.2.m2.1.2.3.2.1" xref="S4.SS3.p2.2.m2.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml">⋅</mo><mo stretchy="false" id="S4.SS3.p2.2.m2.1.2.3.2.2" xref="S4.SS3.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><apply id="S4.SS3.p2.2.m2.1.2.cmml" xref="S4.SS3.p2.2.m2.1.2"><times id="S4.SS3.p2.2.m2.1.2.1.cmml" xref="S4.SS3.p2.2.m2.1.2.1"></times><ci id="S4.SS3.p2.2.m2.1.2.2.cmml" xref="S4.SS3.p2.2.m2.1.2.2">𝑔</ci><ci id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">g(\cdot)</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib72" title="" class="ltx_ref">2014</a>, <a href="#bib.bib73" title="" class="ltx_ref">2015</a>)</cite>. In contrast, we introduce a method that does not do either of these. The idea is to choose prototypes such that <math id="S4.SS3.p2.3.m3.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S4.SS3.p2.3.m3.1a"><mover accent="true" id="S4.SS3.p2.3.m3.1.1" xref="S4.SS3.p2.3.m3.1.1.cmml"><mi id="S4.SS3.p2.3.m3.1.1.2" xref="S4.SS3.p2.3.m3.1.1.2.cmml">𝑨</mi><mo id="S4.SS3.p2.3.m3.1.1.1" xref="S4.SS3.p2.3.m3.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.3.m3.1b"><apply id="S4.SS3.p2.3.m3.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1"><ci id="S4.SS3.p2.3.m3.1.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1.1">~</ci><ci id="S4.SS3.p2.3.m3.1.1.2.cmml" xref="S4.SS3.p2.3.m3.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.3.m3.1c">\tilde{\bm{A}}</annotation></semantics></math> can be reconstructed from its prototypes with as little squared error as possible—this improves results since less error means that less information about <math id="S4.SS3.p2.4.m4.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S4.SS3.p2.4.m4.1a"><mover accent="true" id="S4.SS3.p2.4.m4.1.1" xref="S4.SS3.p2.4.m4.1.1.cmml"><mi id="S4.SS3.p2.4.m4.1.1.2" xref="S4.SS3.p2.4.m4.1.1.2.cmml">𝑨</mi><mo id="S4.SS3.p2.4.m4.1.1.1" xref="S4.SS3.p2.4.m4.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.4.m4.1b"><apply id="S4.SS3.p2.4.m4.1.1.cmml" xref="S4.SS3.p2.4.m4.1.1"><ci id="S4.SS3.p2.4.m4.1.1.1.cmml" xref="S4.SS3.p2.4.m4.1.1.1">~</ci><ci id="S4.SS3.p2.4.m4.1.1.2.cmml" xref="S4.SS3.p2.4.m4.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.4.m4.1c">\tilde{\bm{A}}</annotation></semantics></math> is being lost.</p>
</div>
<div id="S4.SS3.p3" class="ltx_para">
<p id="S4.SS3.p3.19" class="ltx_p">Let <math id="S4.SS3.p3.1.m1.1" class="ltx_Math" alttext="\bm{P}\in\mathbb{R}^{KC\times D}" display="inline"><semantics id="S4.SS3.p3.1.m1.1a"><mrow id="S4.SS3.p3.1.m1.1.1" xref="S4.SS3.p3.1.m1.1.1.cmml"><mi id="S4.SS3.p3.1.m1.1.1.2" xref="S4.SS3.p3.1.m1.1.1.2.cmml">𝑷</mi><mo id="S4.SS3.p3.1.m1.1.1.1" xref="S4.SS3.p3.1.m1.1.1.1.cmml">∈</mo><msup id="S4.SS3.p3.1.m1.1.1.3" xref="S4.SS3.p3.1.m1.1.1.3.cmml"><mi id="S4.SS3.p3.1.m1.1.1.3.2" xref="S4.SS3.p3.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S4.SS3.p3.1.m1.1.1.3.3" xref="S4.SS3.p3.1.m1.1.1.3.3.cmml"><mrow id="S4.SS3.p3.1.m1.1.1.3.3.2" xref="S4.SS3.p3.1.m1.1.1.3.3.2.cmml"><mi id="S4.SS3.p3.1.m1.1.1.3.3.2.2" xref="S4.SS3.p3.1.m1.1.1.3.3.2.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p3.1.m1.1.1.3.3.2.1" xref="S4.SS3.p3.1.m1.1.1.3.3.2.1.cmml">​</mo><mi id="S4.SS3.p3.1.m1.1.1.3.3.2.3" xref="S4.SS3.p3.1.m1.1.1.3.3.2.3.cmml">C</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S4.SS3.p3.1.m1.1.1.3.3.1" xref="S4.SS3.p3.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.SS3.p3.1.m1.1.1.3.3.3" xref="S4.SS3.p3.1.m1.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.1.m1.1b"><apply id="S4.SS3.p3.1.m1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1"><in id="S4.SS3.p3.1.m1.1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1.1"></in><ci id="S4.SS3.p3.1.m1.1.1.2.cmml" xref="S4.SS3.p3.1.m1.1.1.2">𝑷</ci><apply id="S4.SS3.p3.1.m1.1.1.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.SS3.p3.1.m1.1.1.3.1.cmml" xref="S4.SS3.p3.1.m1.1.1.3">superscript</csymbol><ci id="S4.SS3.p3.1.m1.1.1.3.2.cmml" xref="S4.SS3.p3.1.m1.1.1.3.2">ℝ</ci><apply id="S4.SS3.p3.1.m1.1.1.3.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3"><times id="S4.SS3.p3.1.m1.1.1.3.3.1.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3.1"></times><apply id="S4.SS3.p3.1.m1.1.1.3.3.2.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3.2"><times id="S4.SS3.p3.1.m1.1.1.3.3.2.1.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3.2.1"></times><ci id="S4.SS3.p3.1.m1.1.1.3.3.2.2.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3.2.2">𝐾</ci><ci id="S4.SS3.p3.1.m1.1.1.3.3.2.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3.2.3">𝐶</ci></apply><ci id="S4.SS3.p3.1.m1.1.1.3.3.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.1.m1.1c">\bm{P}\in\mathbb{R}^{KC\times D}</annotation></semantics></math> be a matrix whose diagonal blocks of size <math id="S4.SS3.p3.2.m2.2" class="ltx_Math" alttext="K\times|\mathcal{J}^{(c)}|" display="inline"><semantics id="S4.SS3.p3.2.m2.2a"><mrow id="S4.SS3.p3.2.m2.2.2" xref="S4.SS3.p3.2.m2.2.2.cmml"><mi id="S4.SS3.p3.2.m2.2.2.3" xref="S4.SS3.p3.2.m2.2.2.3.cmml">K</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS3.p3.2.m2.2.2.2" xref="S4.SS3.p3.2.m2.2.2.2.cmml">×</mo><mrow id="S4.SS3.p3.2.m2.2.2.1.1" xref="S4.SS3.p3.2.m2.2.2.1.2.cmml"><mo stretchy="false" id="S4.SS3.p3.2.m2.2.2.1.1.2" xref="S4.SS3.p3.2.m2.2.2.1.2.1.cmml">|</mo><msup id="S4.SS3.p3.2.m2.2.2.1.1.1" xref="S4.SS3.p3.2.m2.2.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p3.2.m2.2.2.1.1.1.2" xref="S4.SS3.p3.2.m2.2.2.1.1.1.2.cmml">𝒥</mi><mrow id="S4.SS3.p3.2.m2.1.1.1.3" xref="S4.SS3.p3.2.m2.2.2.1.1.1.cmml"><mo stretchy="false" id="S4.SS3.p3.2.m2.1.1.1.3.1" xref="S4.SS3.p3.2.m2.2.2.1.1.1.cmml">(</mo><mi id="S4.SS3.p3.2.m2.1.1.1.1" xref="S4.SS3.p3.2.m2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S4.SS3.p3.2.m2.1.1.1.3.2" xref="S4.SS3.p3.2.m2.2.2.1.1.1.cmml">)</mo></mrow></msup><mo stretchy="false" id="S4.SS3.p3.2.m2.2.2.1.1.3" xref="S4.SS3.p3.2.m2.2.2.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.2.m2.2b"><apply id="S4.SS3.p3.2.m2.2.2.cmml" xref="S4.SS3.p3.2.m2.2.2"><times id="S4.SS3.p3.2.m2.2.2.2.cmml" xref="S4.SS3.p3.2.m2.2.2.2"></times><ci id="S4.SS3.p3.2.m2.2.2.3.cmml" xref="S4.SS3.p3.2.m2.2.2.3">𝐾</ci><apply id="S4.SS3.p3.2.m2.2.2.1.2.cmml" xref="S4.SS3.p3.2.m2.2.2.1.1"><abs id="S4.SS3.p3.2.m2.2.2.1.2.1.cmml" xref="S4.SS3.p3.2.m2.2.2.1.1.2"></abs><apply id="S4.SS3.p3.2.m2.2.2.1.1.1.cmml" xref="S4.SS3.p3.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.SS3.p3.2.m2.2.2.1.1.1.1.cmml" xref="S4.SS3.p3.2.m2.2.2.1.1.1">superscript</csymbol><ci id="S4.SS3.p3.2.m2.2.2.1.1.1.2.cmml" xref="S4.SS3.p3.2.m2.2.2.1.1.1.2">𝒥</ci><ci id="S4.SS3.p3.2.m2.1.1.1.1.cmml" xref="S4.SS3.p3.2.m2.1.1.1.1">𝑐</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.2.m2.2c">K\times|\mathcal{J}^{(c)}|</annotation></semantics></math> consist of the <math id="S4.SS3.p3.3.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS3.p3.3.m3.1a"><mi id="S4.SS3.p3.3.m3.1.1" xref="S4.SS3.p3.3.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.3.m3.1b"><ci id="S4.SS3.p3.3.m3.1.1.cmml" xref="S4.SS3.p3.3.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.3.m3.1c">K</annotation></semantics></math> learned prototypes in each subspace <math id="S4.SS3.p3.4.m4.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S4.SS3.p3.4.m4.1a"><mi id="S4.SS3.p3.4.m4.1.1" xref="S4.SS3.p3.4.m4.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.4.m4.1b"><ci id="S4.SS3.p3.4.m4.1.1.cmml" xref="S4.SS3.p3.4.m4.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.4.m4.1c">c</annotation></semantics></math>. The training matrix <math id="S4.SS3.p3.5.m5.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S4.SS3.p3.5.m5.1a"><mover accent="true" id="S4.SS3.p3.5.m5.1.1" xref="S4.SS3.p3.5.m5.1.1.cmml"><mi id="S4.SS3.p3.5.m5.1.1.2" xref="S4.SS3.p3.5.m5.1.1.2.cmml">𝑨</mi><mo id="S4.SS3.p3.5.m5.1.1.1" xref="S4.SS3.p3.5.m5.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.5.m5.1b"><apply id="S4.SS3.p3.5.m5.1.1.cmml" xref="S4.SS3.p3.5.m5.1.1"><ci id="S4.SS3.p3.5.m5.1.1.1.cmml" xref="S4.SS3.p3.5.m5.1.1.1">~</ci><ci id="S4.SS3.p3.5.m5.1.1.2.cmml" xref="S4.SS3.p3.5.m5.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.5.m5.1c">\tilde{\bm{A}}</annotation></semantics></math> can be approximately reconstructed as <math id="S4.SS3.p3.6.m6.1" class="ltx_Math" alttext="\tilde{\bm{A}}\approx\bm{G}\bm{P}" display="inline"><semantics id="S4.SS3.p3.6.m6.1a"><mrow id="S4.SS3.p3.6.m6.1.1" xref="S4.SS3.p3.6.m6.1.1.cmml"><mover accent="true" id="S4.SS3.p3.6.m6.1.1.2" xref="S4.SS3.p3.6.m6.1.1.2.cmml"><mi id="S4.SS3.p3.6.m6.1.1.2.2" xref="S4.SS3.p3.6.m6.1.1.2.2.cmml">𝑨</mi><mo id="S4.SS3.p3.6.m6.1.1.2.1" xref="S4.SS3.p3.6.m6.1.1.2.1.cmml">~</mo></mover><mo id="S4.SS3.p3.6.m6.1.1.1" xref="S4.SS3.p3.6.m6.1.1.1.cmml">≈</mo><mrow id="S4.SS3.p3.6.m6.1.1.3" xref="S4.SS3.p3.6.m6.1.1.3.cmml"><mi id="S4.SS3.p3.6.m6.1.1.3.2" xref="S4.SS3.p3.6.m6.1.1.3.2.cmml">𝑮</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p3.6.m6.1.1.3.1" xref="S4.SS3.p3.6.m6.1.1.3.1.cmml">​</mo><mi id="S4.SS3.p3.6.m6.1.1.3.3" xref="S4.SS3.p3.6.m6.1.1.3.3.cmml">𝑷</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.6.m6.1b"><apply id="S4.SS3.p3.6.m6.1.1.cmml" xref="S4.SS3.p3.6.m6.1.1"><approx id="S4.SS3.p3.6.m6.1.1.1.cmml" xref="S4.SS3.p3.6.m6.1.1.1"></approx><apply id="S4.SS3.p3.6.m6.1.1.2.cmml" xref="S4.SS3.p3.6.m6.1.1.2"><ci id="S4.SS3.p3.6.m6.1.1.2.1.cmml" xref="S4.SS3.p3.6.m6.1.1.2.1">~</ci><ci id="S4.SS3.p3.6.m6.1.1.2.2.cmml" xref="S4.SS3.p3.6.m6.1.1.2.2">𝑨</ci></apply><apply id="S4.SS3.p3.6.m6.1.1.3.cmml" xref="S4.SS3.p3.6.m6.1.1.3"><times id="S4.SS3.p3.6.m6.1.1.3.1.cmml" xref="S4.SS3.p3.6.m6.1.1.3.1"></times><ci id="S4.SS3.p3.6.m6.1.1.3.2.cmml" xref="S4.SS3.p3.6.m6.1.1.3.2">𝑮</ci><ci id="S4.SS3.p3.6.m6.1.1.3.3.cmml" xref="S4.SS3.p3.6.m6.1.1.3.3">𝑷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.6.m6.1c">\tilde{\bm{A}}\approx\bm{G}\bm{P}</annotation></semantics></math>,
where <math id="S4.SS3.p3.7.m7.1" class="ltx_Math" alttext="\bm{G}" display="inline"><semantics id="S4.SS3.p3.7.m7.1a"><mi id="S4.SS3.p3.7.m7.1.1" xref="S4.SS3.p3.7.m7.1.1.cmml">𝑮</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.7.m7.1b"><ci id="S4.SS3.p3.7.m7.1.1.cmml" xref="S4.SS3.p3.7.m7.1.1">𝑮</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.7.m7.1c">\bm{G}</annotation></semantics></math> serves to select the appropriate prototype in each subspace. Rows of <math id="S4.SS3.p3.8.m8.1" class="ltx_Math" alttext="\bm{G}" display="inline"><semantics id="S4.SS3.p3.8.m8.1a"><mi id="S4.SS3.p3.8.m8.1.1" xref="S4.SS3.p3.8.m8.1.1.cmml">𝑮</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.8.m8.1b"><ci id="S4.SS3.p3.8.m8.1.1.cmml" xref="S4.SS3.p3.8.m8.1.1">𝑮</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.8.m8.1c">\bm{G}</annotation></semantics></math> are formed by concatenating the one-hot encoded representations of each assignment for the corresponding row of <math id="S4.SS3.p3.9.m9.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S4.SS3.p3.9.m9.1a"><mover accent="true" id="S4.SS3.p3.9.m9.1.1" xref="S4.SS3.p3.9.m9.1.1.cmml"><mi id="S4.SS3.p3.9.m9.1.1.2" xref="S4.SS3.p3.9.m9.1.1.2.cmml">𝑨</mi><mo id="S4.SS3.p3.9.m9.1.1.1" xref="S4.SS3.p3.9.m9.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.9.m9.1b"><apply id="S4.SS3.p3.9.m9.1.1.cmml" xref="S4.SS3.p3.9.m9.1.1"><ci id="S4.SS3.p3.9.m9.1.1.1.cmml" xref="S4.SS3.p3.9.m9.1.1.1">~</ci><ci id="S4.SS3.p3.9.m9.1.1.2.cmml" xref="S4.SS3.p3.9.m9.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.9.m9.1c">\tilde{\bm{A}}</annotation></semantics></math>. For example, if a row were assigned prototypes <math id="S4.SS3.p3.10.m10.1" class="ltx_Math" alttext="\langle" display="inline"><semantics id="S4.SS3.p3.10.m10.1a"><mo stretchy="false" id="S4.SS3.p3.10.m10.1.1" xref="S4.SS3.p3.10.m10.1.1.cmml">⟨</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.10.m10.1b"><ci id="S4.SS3.p3.10.m10.1.1.cmml" xref="S4.SS3.p3.10.m10.1.1">⟨</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.10.m10.1c">\langle</annotation></semantics></math><span id="S4.SS3.p3.11.1" class="ltx_text ltx_font_typewriter">3 1 2<math id="S4.SS3.p3.11.1.m1.1" class="ltx_Math" alttext="\rangle" display="inline"><semantics id="S4.SS3.p3.11.1.m1.1a"><mo stretchy="false" id="S4.SS3.p3.11.1.m1.1.1" xref="S4.SS3.p3.11.1.m1.1.1.cmml">⟩</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.11.1.m1.1b"><ci id="S4.SS3.p3.11.1.m1.1.1.cmml" xref="S4.SS3.p3.11.1.m1.1.1">⟩</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.11.1.m1.1c">\rangle</annotation></semantics></math></span> with <math id="S4.SS3.p3.12.m11.1" class="ltx_Math" alttext="K=4" display="inline"><semantics id="S4.SS3.p3.12.m11.1a"><mrow id="S4.SS3.p3.12.m11.1.1" xref="S4.SS3.p3.12.m11.1.1.cmml"><mi id="S4.SS3.p3.12.m11.1.1.2" xref="S4.SS3.p3.12.m11.1.1.2.cmml">K</mi><mo id="S4.SS3.p3.12.m11.1.1.1" xref="S4.SS3.p3.12.m11.1.1.1.cmml">=</mo><mn id="S4.SS3.p3.12.m11.1.1.3" xref="S4.SS3.p3.12.m11.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.12.m11.1b"><apply id="S4.SS3.p3.12.m11.1.1.cmml" xref="S4.SS3.p3.12.m11.1.1"><eq id="S4.SS3.p3.12.m11.1.1.1.cmml" xref="S4.SS3.p3.12.m11.1.1.1"></eq><ci id="S4.SS3.p3.12.m11.1.1.2.cmml" xref="S4.SS3.p3.12.m11.1.1.2">𝐾</ci><cn type="integer" id="S4.SS3.p3.12.m11.1.1.3.cmml" xref="S4.SS3.p3.12.m11.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.12.m11.1c">K=4</annotation></semantics></math>, <math id="S4.SS3.p3.13.m12.1" class="ltx_Math" alttext="C=3" display="inline"><semantics id="S4.SS3.p3.13.m12.1a"><mrow id="S4.SS3.p3.13.m12.1.1" xref="S4.SS3.p3.13.m12.1.1.cmml"><mi id="S4.SS3.p3.13.m12.1.1.2" xref="S4.SS3.p3.13.m12.1.1.2.cmml">C</mi><mo id="S4.SS3.p3.13.m12.1.1.1" xref="S4.SS3.p3.13.m12.1.1.1.cmml">=</mo><mn id="S4.SS3.p3.13.m12.1.1.3" xref="S4.SS3.p3.13.m12.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.13.m12.1b"><apply id="S4.SS3.p3.13.m12.1.1.cmml" xref="S4.SS3.p3.13.m12.1.1"><eq id="S4.SS3.p3.13.m12.1.1.1.cmml" xref="S4.SS3.p3.13.m12.1.1.1"></eq><ci id="S4.SS3.p3.13.m12.1.1.2.cmml" xref="S4.SS3.p3.13.m12.1.1.2">𝐶</ci><cn type="integer" id="S4.SS3.p3.13.m12.1.1.3.cmml" xref="S4.SS3.p3.13.m12.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.13.m12.1c">C=3</annotation></semantics></math>, its row in <math id="S4.SS3.p3.14.m13.1" class="ltx_Math" alttext="\bm{G}" display="inline"><semantics id="S4.SS3.p3.14.m13.1a"><mi id="S4.SS3.p3.14.m13.1.1" xref="S4.SS3.p3.14.m13.1.1.cmml">𝑮</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.14.m13.1b"><ci id="S4.SS3.p3.14.m13.1.1.cmml" xref="S4.SS3.p3.14.m13.1.1">𝑮</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.14.m13.1c">\bm{G}</annotation></semantics></math> would be <math id="S4.SS3.p3.15.m14.1" class="ltx_Math" alttext="\langle" display="inline"><semantics id="S4.SS3.p3.15.m14.1a"><mo stretchy="false" id="S4.SS3.p3.15.m14.1.1" xref="S4.SS3.p3.15.m14.1.1.cmml">⟨</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.15.m14.1b"><ci id="S4.SS3.p3.15.m14.1.1.cmml" xref="S4.SS3.p3.15.m14.1.1">⟨</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.15.m14.1c">\langle</annotation></semantics></math><span id="S4.SS3.p3.16.2" class="ltx_text ltx_font_typewriter">0010 1000 0100<math id="S4.SS3.p3.16.2.m1.1" class="ltx_math_unparsed" alttext="\rangle\in\mathbb{R}^{12}" display="inline"><semantics id="S4.SS3.p3.16.2.m1.1a"><mrow id="S4.SS3.p3.16.2.m1.1b"><mo stretchy="false" id="S4.SS3.p3.16.2.m1.1.1">⟩</mo><mo id="S4.SS3.p3.16.2.m1.1.2">∈</mo><mi id="S4.SS3.p3.16.2.m1.1.3">ℝ</mi><msup id="S4.SS3.p3.16.2.m1.1.4"><mi id="S4.SS3.p3.16.2.m1.1.4a"></mi><mn id="S4.SS3.p3.16.2.m1.1.4.1">12</mn></msup></mrow><annotation encoding="application/x-tex" id="S4.SS3.p3.16.2.m1.1c">\rangle\in\mathbb{R}^{12}</annotation></semantics></math></span>. Our idea is to optimize <math id="S4.SS3.p3.17.m15.1" class="ltx_Math" alttext="\bm{P}" display="inline"><semantics id="S4.SS3.p3.17.m15.1a"><mi id="S4.SS3.p3.17.m15.1.1" xref="S4.SS3.p3.17.m15.1.1.cmml">𝑷</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.17.m15.1b"><ci id="S4.SS3.p3.17.m15.1.1.cmml" xref="S4.SS3.p3.17.m15.1.1">𝑷</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.17.m15.1c">\bm{P}</annotation></semantics></math> conditioned on <math id="S4.SS3.p3.18.m16.1" class="ltx_Math" alttext="\bm{G}" display="inline"><semantics id="S4.SS3.p3.18.m16.1a"><mi id="S4.SS3.p3.18.m16.1.1" xref="S4.SS3.p3.18.m16.1.1.cmml">𝑮</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.18.m16.1b"><ci id="S4.SS3.p3.18.m16.1.1.cmml" xref="S4.SS3.p3.18.m16.1.1">𝑮</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.18.m16.1c">\bm{G}</annotation></semantics></math> and <math id="S4.SS3.p3.19.m17.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S4.SS3.p3.19.m17.1a"><mover accent="true" id="S4.SS3.p3.19.m17.1.1" xref="S4.SS3.p3.19.m17.1.1.cmml"><mi id="S4.SS3.p3.19.m17.1.1.2" xref="S4.SS3.p3.19.m17.1.1.2.cmml">𝑨</mi><mo id="S4.SS3.p3.19.m17.1.1.1" xref="S4.SS3.p3.19.m17.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.19.m17.1b"><apply id="S4.SS3.p3.19.m17.1.1.cmml" xref="S4.SS3.p3.19.m17.1.1"><ci id="S4.SS3.p3.19.m17.1.1.1.cmml" xref="S4.SS3.p3.19.m17.1.1.1">~</ci><ci id="S4.SS3.p3.19.m17.1.1.2.cmml" xref="S4.SS3.p3.19.m17.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.19.m17.1c">\tilde{\bm{A}}</annotation></semantics></math>. This is an ordinary least squares problem, and we solve it with ridge regression:</p>
<table id="A6.EGx7" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E8.m1.1" class="ltx_Math" alttext="\displaystyle\bm{P}\triangleq(\bm{G}^{\top}\bm{G}+\lambda\bm{I})^{-1}\bm{G}^{\top}\tilde{\bm{A}}." display="inline"><semantics id="S4.E8.m1.1a"><mrow id="S4.E8.m1.1.1.1" xref="S4.E8.m1.1.1.1.1.cmml"><mrow id="S4.E8.m1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.cmml"><mi id="S4.E8.m1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.3.cmml">𝑷</mi><mo id="S4.E8.m1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.2.cmml">≜</mo><mrow id="S4.E8.m1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.cmml"><msup id="S4.E8.m1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.cmml"><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E8.m1.1.1.1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.cmml"><msup id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.2.cmml">𝑮</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">𝑮</mi></mrow><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.3.cmml">𝑰</mi></mrow></mrow><mo stretchy="false" id="S4.E8.m1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S4.E8.m1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.3.cmml"><mo id="S4.E8.m1.1.1.1.1.1.1.3a" xref="S4.E8.m1.1.1.1.1.1.1.3.cmml">−</mo><mn id="S4.E8.m1.1.1.1.1.1.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.2.cmml">​</mo><msup id="S4.E8.m1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.3.2" xref="S4.E8.m1.1.1.1.1.1.3.2.cmml">𝑮</mi><mo id="S4.E8.m1.1.1.1.1.1.3.3" xref="S4.E8.m1.1.1.1.1.1.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.2a" xref="S4.E8.m1.1.1.1.1.1.2.cmml">​</mo><mover accent="true" id="S4.E8.m1.1.1.1.1.1.4" xref="S4.E8.m1.1.1.1.1.1.4.cmml"><mi id="S4.E8.m1.1.1.1.1.1.4.2" xref="S4.E8.m1.1.1.1.1.1.4.2.cmml">𝑨</mi><mo id="S4.E8.m1.1.1.1.1.1.4.1" xref="S4.E8.m1.1.1.1.1.1.4.1.cmml">~</mo></mover></mrow></mrow><mo lspace="0em" id="S4.E8.m1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E8.m1.1b"><apply id="S4.E8.m1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1"><ci id="S4.E8.m1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.2">≜</ci><ci id="S4.E8.m1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.3">𝑷</ci><apply id="S4.E8.m1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1"><times id="S4.E8.m1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.2"></times><apply id="S4.E8.m1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E8.m1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1">superscript</csymbol><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1"><plus id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2"><times id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.1"></times><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.2">𝑮</ci><csymbol cd="latexml" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.2.3">top</csymbol></apply><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.2.3">𝑮</ci></apply><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3"><times id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.1"></times><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.3.3">𝑰</ci></apply></apply><apply id="S4.E8.m1.1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.3"><minus id="S4.E8.m1.1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.3"></minus><cn type="integer" id="S4.E8.m1.1.1.1.1.1.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.3.2">1</cn></apply></apply><apply id="S4.E8.m1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E8.m1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.3">superscript</csymbol><ci id="S4.E8.m1.1.1.1.1.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.3.2">𝑮</ci><csymbol cd="latexml" id="S4.E8.m1.1.1.1.1.1.3.3.cmml" xref="S4.E8.m1.1.1.1.1.1.3.3">top</csymbol></apply><apply id="S4.E8.m1.1.1.1.1.1.4.cmml" xref="S4.E8.m1.1.1.1.1.1.4"><ci id="S4.E8.m1.1.1.1.1.1.4.1.cmml" xref="S4.E8.m1.1.1.1.1.1.4.1">~</ci><ci id="S4.E8.m1.1.1.1.1.1.4.2.cmml" xref="S4.E8.m1.1.1.1.1.1.4.2">𝑨</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E8.m1.1c">\displaystyle\bm{P}\triangleq(\bm{G}^{\top}\bm{G}+\lambda\bm{I})^{-1}\bm{G}^{\top}\tilde{\bm{A}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="S4.SS3.p3.21" class="ltx_p">One could obtain better performance by cross-validating to find <math id="S4.SS3.p3.20.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S4.SS3.p3.20.m1.1a"><mi id="S4.SS3.p3.20.m1.1.1" xref="S4.SS3.p3.20.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.20.m1.1b"><ci id="S4.SS3.p3.20.m1.1.1.cmml" xref="S4.SS3.p3.20.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.20.m1.1c">\lambda</annotation></semantics></math>, but for simplicity, we fix <math id="S4.SS3.p3.21.m2.1" class="ltx_Math" alttext="\lambda=1" display="inline"><semantics id="S4.SS3.p3.21.m2.1a"><mrow id="S4.SS3.p3.21.m2.1.1" xref="S4.SS3.p3.21.m2.1.1.cmml"><mi id="S4.SS3.p3.21.m2.1.1.2" xref="S4.SS3.p3.21.m2.1.1.2.cmml">λ</mi><mo id="S4.SS3.p3.21.m2.1.1.1" xref="S4.SS3.p3.21.m2.1.1.1.cmml">=</mo><mn id="S4.SS3.p3.21.m2.1.1.3" xref="S4.SS3.p3.21.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.21.m2.1b"><apply id="S4.SS3.p3.21.m2.1.1.cmml" xref="S4.SS3.p3.21.m2.1.1"><eq id="S4.SS3.p3.21.m2.1.1.1.cmml" xref="S4.SS3.p3.21.m2.1.1.1"></eq><ci id="S4.SS3.p3.21.m2.1.1.2.cmml" xref="S4.SS3.p3.21.m2.1.1.2">𝜆</ci><cn type="integer" id="S4.SS3.p3.21.m2.1.1.3.cmml" xref="S4.SS3.p3.21.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.21.m2.1c">\lambda=1</annotation></semantics></math>.</p>
</div>
<div id="S4.SS3.p4" class="ltx_para">
<p id="S4.SS3.p4.1" class="ltx_p">This procedure allows the prototypes to be nonzero outside of their original subspaces. Because of our hashing procedure, we avoid the dramatically increased overhead faced by other methods with non-orthogonal prototypes (c.f. <cite class="ltx_cite ltx_citemacro_citep">(Babenko &amp; Lempitsky, <a href="#bib.bib7" title="" class="ltx_ref">2015</a>, <a href="#bib.bib6" title="" class="ltx_ref">2014</a>; Zhang et al., <a href="#bib.bib72" title="" class="ltx_ref">2014</a>; Liu et al., <a href="#bib.bib50" title="" class="ltx_ref">2016</a>; Martinez et al., <a href="#bib.bib55" title="" class="ltx_ref">2016</a>, <a href="#bib.bib54" title="" class="ltx_ref">2014</a>)</cite>).</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Fast 8-bit Aggregation, <math id="S4.SS4.1.m1.2" class="ltx_Math" alttext="\bm{f(\cdot,\cdot)}" display="inline"><semantics id="S4.SS4.1.m1.2b"><mrow id="S4.SS4.1.m1.2.3" xref="S4.SS4.1.m1.2.3.cmml"><mi id="S4.SS4.1.m1.2.3.2" xref="S4.SS4.1.m1.2.3.2.cmml">𝒇</mi><mo lspace="0em" rspace="0em" id="S4.SS4.1.m1.2.3.1" xref="S4.SS4.1.m1.2.3.1.cmml">​</mo><mrow id="S4.SS4.1.m1.2.3.3.2" xref="S4.SS4.1.m1.2.3.3.1.cmml"><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S4.SS4.1.m1.2.3.3.2.1" xref="S4.SS4.1.m1.2.3.3.1.cmml">(</mo><mo class="ltx_mathvariant_bold" lspace="0em" mathvariant="bold" rspace="0em" id="S4.SS4.1.m1.1.1" xref="S4.SS4.1.m1.1.1.cmml">⋅</mo><mo class="ltx_mathvariant_bold" mathvariant="bold" rspace="0em" id="S4.SS4.1.m1.2.3.3.2.2" xref="S4.SS4.1.m1.2.3.3.1.cmml">,</mo><mo class="ltx_mathvariant_bold" lspace="0em" mathvariant="bold" rspace="0em" id="S4.SS4.1.m1.2.2" xref="S4.SS4.1.m1.2.2.cmml">⋅</mo><mo class="ltx_mathvariant_bold" mathvariant="bold" stretchy="false" id="S4.SS4.1.m1.2.3.3.2.3" xref="S4.SS4.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.1.m1.2c"><apply id="S4.SS4.1.m1.2.3.cmml" xref="S4.SS4.1.m1.2.3"><times id="S4.SS4.1.m1.2.3.1.cmml" xref="S4.SS4.1.m1.2.3.1"></times><ci id="S4.SS4.1.m1.2.3.2.cmml" xref="S4.SS4.1.m1.2.3.2">𝒇</ci><interval closure="open" id="S4.SS4.1.m1.2.3.3.1.cmml" xref="S4.SS4.1.m1.2.3.3.2"><ci id="S4.SS4.1.m1.1.1.cmml" xref="S4.SS4.1.m1.1.1">bold-⋅</ci><ci id="S4.SS4.1.m1.2.2.cmml" xref="S4.SS4.1.m1.2.2">bold-⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.1.m1.2d">\bm{f(\cdot,\cdot)}</annotation></semantics></math>
</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.5" class="ltx_p">Let <math id="S4.SS4.p1.1.m1.1" class="ltx_Math" alttext="\bm{T}\in\mathbb{R}^{M\times C\times K}" display="inline"><semantics id="S4.SS4.p1.1.m1.1a"><mrow id="S4.SS4.p1.1.m1.1.1" xref="S4.SS4.p1.1.m1.1.1.cmml"><mi id="S4.SS4.p1.1.m1.1.1.2" xref="S4.SS4.p1.1.m1.1.1.2.cmml">𝑻</mi><mo id="S4.SS4.p1.1.m1.1.1.1" xref="S4.SS4.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S4.SS4.p1.1.m1.1.1.3" xref="S4.SS4.p1.1.m1.1.1.3.cmml"><mi id="S4.SS4.p1.1.m1.1.1.3.2" xref="S4.SS4.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S4.SS4.p1.1.m1.1.1.3.3" xref="S4.SS4.p1.1.m1.1.1.3.3.cmml"><mi id="S4.SS4.p1.1.m1.1.1.3.3.2" xref="S4.SS4.p1.1.m1.1.1.3.3.2.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS4.p1.1.m1.1.1.3.3.1" xref="S4.SS4.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.SS4.p1.1.m1.1.1.3.3.3" xref="S4.SS4.p1.1.m1.1.1.3.3.3.cmml">C</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS4.p1.1.m1.1.1.3.3.1a" xref="S4.SS4.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.SS4.p1.1.m1.1.1.3.3.4" xref="S4.SS4.p1.1.m1.1.1.3.3.4.cmml">K</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.1.m1.1b"><apply id="S4.SS4.p1.1.m1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.1"><in id="S4.SS4.p1.1.m1.1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.1.1"></in><ci id="S4.SS4.p1.1.m1.1.1.2.cmml" xref="S4.SS4.p1.1.m1.1.1.2">𝑻</ci><apply id="S4.SS4.p1.1.m1.1.1.3.cmml" xref="S4.SS4.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.SS4.p1.1.m1.1.1.3.1.cmml" xref="S4.SS4.p1.1.m1.1.1.3">superscript</csymbol><ci id="S4.SS4.p1.1.m1.1.1.3.2.cmml" xref="S4.SS4.p1.1.m1.1.1.3.2">ℝ</ci><apply id="S4.SS4.p1.1.m1.1.1.3.3.cmml" xref="S4.SS4.p1.1.m1.1.1.3.3"><times id="S4.SS4.p1.1.m1.1.1.3.3.1.cmml" xref="S4.SS4.p1.1.m1.1.1.3.3.1"></times><ci id="S4.SS4.p1.1.m1.1.1.3.3.2.cmml" xref="S4.SS4.p1.1.m1.1.1.3.3.2">𝑀</ci><ci id="S4.SS4.p1.1.m1.1.1.3.3.3.cmml" xref="S4.SS4.p1.1.m1.1.1.3.3.3">𝐶</ci><ci id="S4.SS4.p1.1.m1.1.1.3.3.4.cmml" xref="S4.SS4.p1.1.m1.1.1.3.3.4">𝐾</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.1.m1.1c">\bm{T}\in\mathbb{R}^{M\times C\times K}</annotation></semantics></math> be the tensor of lookup tables for all <math id="S4.SS4.p1.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.SS4.p1.2.m2.1a"><mi id="S4.SS4.p1.2.m2.1.1" xref="S4.SS4.p1.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.2.m2.1b"><ci id="S4.SS4.p1.2.m2.1.1.cmml" xref="S4.SS4.p1.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.2.m2.1c">M</annotation></semantics></math> columns of <math id="S4.SS4.p1.3.m3.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S4.SS4.p1.3.m3.1a"><mi id="S4.SS4.p1.3.m3.1.1" xref="S4.SS4.p1.3.m3.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.3.m3.1b"><ci id="S4.SS4.p1.3.m3.1.1.cmml" xref="S4.SS4.p1.3.m3.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.3.m3.1c">\bm{B}</annotation></semantics></math>.
Given the encodings <math id="S4.SS4.p1.4.m4.1" class="ltx_Math" alttext="\bm{G}" display="inline"><semantics id="S4.SS4.p1.4.m4.1a"><mi id="S4.SS4.p1.4.m4.1.1" xref="S4.SS4.p1.4.m4.1.1.cmml">𝑮</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.4.m4.1b"><ci id="S4.SS4.p1.4.m4.1.1.cmml" xref="S4.SS4.p1.4.m4.1.1">𝑮</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.4.m4.1c">\bm{G}</annotation></semantics></math>, the function <math id="S4.SS4.p1.5.m5.2" class="ltx_Math" alttext="f(\cdot,\cdot)" display="inline"><semantics id="S4.SS4.p1.5.m5.2a"><mrow id="S4.SS4.p1.5.m5.2.3" xref="S4.SS4.p1.5.m5.2.3.cmml"><mi id="S4.SS4.p1.5.m5.2.3.2" xref="S4.SS4.p1.5.m5.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p1.5.m5.2.3.1" xref="S4.SS4.p1.5.m5.2.3.1.cmml">​</mo><mrow id="S4.SS4.p1.5.m5.2.3.3.2" xref="S4.SS4.p1.5.m5.2.3.3.1.cmml"><mo stretchy="false" id="S4.SS4.p1.5.m5.2.3.3.2.1" xref="S4.SS4.p1.5.m5.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S4.SS4.p1.5.m5.1.1" xref="S4.SS4.p1.5.m5.1.1.cmml">⋅</mo><mo rspace="0em" id="S4.SS4.p1.5.m5.2.3.3.2.2" xref="S4.SS4.p1.5.m5.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S4.SS4.p1.5.m5.2.2" xref="S4.SS4.p1.5.m5.2.2.cmml">⋅</mo><mo stretchy="false" id="S4.SS4.p1.5.m5.2.3.3.2.3" xref="S4.SS4.p1.5.m5.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.5.m5.2b"><apply id="S4.SS4.p1.5.m5.2.3.cmml" xref="S4.SS4.p1.5.m5.2.3"><times id="S4.SS4.p1.5.m5.2.3.1.cmml" xref="S4.SS4.p1.5.m5.2.3.1"></times><ci id="S4.SS4.p1.5.m5.2.3.2.cmml" xref="S4.SS4.p1.5.m5.2.3.2">𝑓</ci><interval closure="open" id="S4.SS4.p1.5.m5.2.3.3.1.cmml" xref="S4.SS4.p1.5.m5.2.3.3.2"><ci id="S4.SS4.p1.5.m5.1.1.cmml" xref="S4.SS4.p1.5.m5.1.1">⋅</ci><ci id="S4.SS4.p1.5.m5.2.2.cmml" xref="S4.SS4.p1.5.m5.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.5.m5.2c">f(\cdot,\cdot)</annotation></semantics></math> is defined as</p>
<table id="A6.EGx8" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E9"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E9.m1.9" class="ltx_Math" alttext="\displaystyle\vspace*{2mm}f(g(\bm{A}),h(\bm{B}))_{n,m}\triangleq\sum_{c=1}^{C}\bm{T}_{m,c,k}\text{, }\text{ }k=g^{(c)}(\bm{a}_{n})." display="inline"><semantics id="S4.E9.m1.9a"><mrow id="S4.E9.m1.9.9.1" xref="S4.E9.m1.9.9.1.1.cmml"><mrow id="S4.E9.m1.9.9.1.1" xref="S4.E9.m1.9.9.1.1.cmml"><mrow id="S4.E9.m1.9.9.1.1.2" xref="S4.E9.m1.9.9.1.1.2.cmml"><mi id="S4.E9.m1.9.9.1.1.2.4" xref="S4.E9.m1.9.9.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.E9.m1.9.9.1.1.2.3" xref="S4.E9.m1.9.9.1.1.2.3.cmml">​</mo><msub id="S4.E9.m1.9.9.1.1.2.2" xref="S4.E9.m1.9.9.1.1.2.2.cmml"><mrow id="S4.E9.m1.9.9.1.1.2.2.2.2" xref="S4.E9.m1.9.9.1.1.2.2.2.3.cmml"><mo stretchy="false" id="S4.E9.m1.9.9.1.1.2.2.2.2.3" xref="S4.E9.m1.9.9.1.1.2.2.2.3.cmml">(</mo><mrow id="S4.E9.m1.9.9.1.1.1.1.1.1.1" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.cmml"><mi id="S4.E9.m1.9.9.1.1.1.1.1.1.1.2" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S4.E9.m1.9.9.1.1.1.1.1.1.1.1" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S4.E9.m1.9.9.1.1.1.1.1.1.1.3.2" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E9.m1.9.9.1.1.1.1.1.1.1.3.2.1" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.cmml">(</mo><mi id="S4.E9.m1.7.7" xref="S4.E9.m1.7.7.cmml">𝑨</mi><mo stretchy="false" id="S4.E9.m1.9.9.1.1.1.1.1.1.1.3.2.2" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E9.m1.9.9.1.1.2.2.2.2.4" xref="S4.E9.m1.9.9.1.1.2.2.2.3.cmml">,</mo><mrow id="S4.E9.m1.9.9.1.1.2.2.2.2.2" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.cmml"><mi id="S4.E9.m1.9.9.1.1.2.2.2.2.2.2" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S4.E9.m1.9.9.1.1.2.2.2.2.2.1" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.1.cmml">​</mo><mrow id="S4.E9.m1.9.9.1.1.2.2.2.2.2.3.2" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.cmml"><mo stretchy="false" id="S4.E9.m1.9.9.1.1.2.2.2.2.2.3.2.1" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.cmml">(</mo><mi id="S4.E9.m1.8.8" xref="S4.E9.m1.8.8.cmml">𝑩</mi><mo stretchy="false" id="S4.E9.m1.9.9.1.1.2.2.2.2.2.3.2.2" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S4.E9.m1.9.9.1.1.2.2.2.2.5" xref="S4.E9.m1.9.9.1.1.2.2.2.3.cmml">)</mo></mrow><mrow id="S4.E9.m1.2.2.2.4" xref="S4.E9.m1.2.2.2.3.cmml"><mi id="S4.E9.m1.1.1.1.1" xref="S4.E9.m1.1.1.1.1.cmml">n</mi><mo id="S4.E9.m1.2.2.2.4.1" xref="S4.E9.m1.2.2.2.3.cmml">,</mo><mi id="S4.E9.m1.2.2.2.2" xref="S4.E9.m1.2.2.2.2.cmml">m</mi></mrow></msub></mrow><mo id="S4.E9.m1.9.9.1.1.5" xref="S4.E9.m1.9.9.1.1.5.cmml">≜</mo><mrow id="S4.E9.m1.9.9.1.1.6" xref="S4.E9.m1.9.9.1.1.6.cmml"><mstyle displaystyle="true" id="S4.E9.m1.9.9.1.1.6.1" xref="S4.E9.m1.9.9.1.1.6.1.cmml"><munderover id="S4.E9.m1.9.9.1.1.6.1a" xref="S4.E9.m1.9.9.1.1.6.1.cmml"><mo movablelimits="false" id="S4.E9.m1.9.9.1.1.6.1.2.2" xref="S4.E9.m1.9.9.1.1.6.1.2.2.cmml">∑</mo><mrow id="S4.E9.m1.9.9.1.1.6.1.2.3" xref="S4.E9.m1.9.9.1.1.6.1.2.3.cmml"><mi id="S4.E9.m1.9.9.1.1.6.1.2.3.2" xref="S4.E9.m1.9.9.1.1.6.1.2.3.2.cmml">c</mi><mo id="S4.E9.m1.9.9.1.1.6.1.2.3.1" xref="S4.E9.m1.9.9.1.1.6.1.2.3.1.cmml">=</mo><mn id="S4.E9.m1.9.9.1.1.6.1.2.3.3" xref="S4.E9.m1.9.9.1.1.6.1.2.3.3.cmml">1</mn></mrow><mi id="S4.E9.m1.9.9.1.1.6.1.3" xref="S4.E9.m1.9.9.1.1.6.1.3.cmml">C</mi></munderover></mstyle><mrow id="S4.E9.m1.9.9.1.1.6.2" xref="S4.E9.m1.9.9.1.1.6.2.cmml"><msub id="S4.E9.m1.9.9.1.1.6.2.2" xref="S4.E9.m1.9.9.1.1.6.2.2.cmml"><mi id="S4.E9.m1.9.9.1.1.6.2.2.2" xref="S4.E9.m1.9.9.1.1.6.2.2.2.cmml">𝑻</mi><mrow id="S4.E9.m1.5.5.3.5" xref="S4.E9.m1.5.5.3.4.cmml"><mi id="S4.E9.m1.3.3.1.1" xref="S4.E9.m1.3.3.1.1.cmml">m</mi><mo id="S4.E9.m1.5.5.3.5.1" xref="S4.E9.m1.5.5.3.4.cmml">,</mo><mi id="S4.E9.m1.4.4.2.2" xref="S4.E9.m1.4.4.2.2.cmml">c</mi><mo id="S4.E9.m1.5.5.3.5.2" xref="S4.E9.m1.5.5.3.4.cmml">,</mo><mi id="S4.E9.m1.5.5.3.3" xref="S4.E9.m1.5.5.3.3.cmml">k</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.E9.m1.9.9.1.1.6.2.1" xref="S4.E9.m1.9.9.1.1.6.2.1.cmml">​</mo><mrow id="S4.E9.m1.9.9.1.1.6.2.3" xref="S4.E9.m1.9.9.1.1.6.2.3c.cmml"><mtext id="S4.E9.m1.9.9.1.1.6.2.3a" xref="S4.E9.m1.9.9.1.1.6.2.3c.cmml">, </mtext><mtext id="S4.E9.m1.9.9.1.1.6.2.3b" xref="S4.E9.m1.9.9.1.1.6.2.3c.cmml"> </mtext></mrow><mo lspace="0em" rspace="0em" id="S4.E9.m1.9.9.1.1.6.2.1a" xref="S4.E9.m1.9.9.1.1.6.2.1.cmml">​</mo><mi id="S4.E9.m1.9.9.1.1.6.2.4" xref="S4.E9.m1.9.9.1.1.6.2.4.cmml">k</mi></mrow></mrow><mo id="S4.E9.m1.9.9.1.1.7" xref="S4.E9.m1.9.9.1.1.7.cmml">=</mo><mrow id="S4.E9.m1.9.9.1.1.3" xref="S4.E9.m1.9.9.1.1.3.cmml"><msup id="S4.E9.m1.9.9.1.1.3.3" xref="S4.E9.m1.9.9.1.1.3.3.cmml"><mi id="S4.E9.m1.9.9.1.1.3.3.2" xref="S4.E9.m1.9.9.1.1.3.3.2.cmml">g</mi><mrow id="S4.E9.m1.6.6.1.3" xref="S4.E9.m1.9.9.1.1.3.3.cmml"><mo stretchy="false" id="S4.E9.m1.6.6.1.3.1" xref="S4.E9.m1.9.9.1.1.3.3.cmml">(</mo><mi id="S4.E9.m1.6.6.1.1" xref="S4.E9.m1.6.6.1.1.cmml">c</mi><mo stretchy="false" id="S4.E9.m1.6.6.1.3.2" xref="S4.E9.m1.9.9.1.1.3.3.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S4.E9.m1.9.9.1.1.3.2" xref="S4.E9.m1.9.9.1.1.3.2.cmml">​</mo><mrow id="S4.E9.m1.9.9.1.1.3.1.1" xref="S4.E9.m1.9.9.1.1.3.1.1.1.cmml"><mo stretchy="false" id="S4.E9.m1.9.9.1.1.3.1.1.2" xref="S4.E9.m1.9.9.1.1.3.1.1.1.cmml">(</mo><msub id="S4.E9.m1.9.9.1.1.3.1.1.1" xref="S4.E9.m1.9.9.1.1.3.1.1.1.cmml"><mi id="S4.E9.m1.9.9.1.1.3.1.1.1.2" xref="S4.E9.m1.9.9.1.1.3.1.1.1.2.cmml">𝒂</mi><mi id="S4.E9.m1.9.9.1.1.3.1.1.1.3" xref="S4.E9.m1.9.9.1.1.3.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="S4.E9.m1.9.9.1.1.3.1.1.3" xref="S4.E9.m1.9.9.1.1.3.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S4.E9.m1.9.9.1.2" xref="S4.E9.m1.9.9.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E9.m1.9b"><apply id="S4.E9.m1.9.9.1.1.cmml" xref="S4.E9.m1.9.9.1"><and id="S4.E9.m1.9.9.1.1a.cmml" xref="S4.E9.m1.9.9.1"></and><apply id="S4.E9.m1.9.9.1.1b.cmml" xref="S4.E9.m1.9.9.1"><ci id="S4.E9.m1.9.9.1.1.5.cmml" xref="S4.E9.m1.9.9.1.1.5">≜</ci><apply id="S4.E9.m1.9.9.1.1.2.cmml" xref="S4.E9.m1.9.9.1.1.2"><times id="S4.E9.m1.9.9.1.1.2.3.cmml" xref="S4.E9.m1.9.9.1.1.2.3"></times><ci id="S4.E9.m1.9.9.1.1.2.4.cmml" xref="S4.E9.m1.9.9.1.1.2.4">𝑓</ci><apply id="S4.E9.m1.9.9.1.1.2.2.cmml" xref="S4.E9.m1.9.9.1.1.2.2"><csymbol cd="ambiguous" id="S4.E9.m1.9.9.1.1.2.2.3.cmml" xref="S4.E9.m1.9.9.1.1.2.2">subscript</csymbol><interval closure="open" id="S4.E9.m1.9.9.1.1.2.2.2.3.cmml" xref="S4.E9.m1.9.9.1.1.2.2.2.2"><apply id="S4.E9.m1.9.9.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1"><times id="S4.E9.m1.9.9.1.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.1"></times><ci id="S4.E9.m1.9.9.1.1.1.1.1.1.1.2.cmml" xref="S4.E9.m1.9.9.1.1.1.1.1.1.1.2">𝑔</ci><ci id="S4.E9.m1.7.7.cmml" xref="S4.E9.m1.7.7">𝑨</ci></apply><apply id="S4.E9.m1.9.9.1.1.2.2.2.2.2.cmml" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2"><times id="S4.E9.m1.9.9.1.1.2.2.2.2.2.1.cmml" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.1"></times><ci id="S4.E9.m1.9.9.1.1.2.2.2.2.2.2.cmml" xref="S4.E9.m1.9.9.1.1.2.2.2.2.2.2">ℎ</ci><ci id="S4.E9.m1.8.8.cmml" xref="S4.E9.m1.8.8">𝑩</ci></apply></interval><list id="S4.E9.m1.2.2.2.3.cmml" xref="S4.E9.m1.2.2.2.4"><ci id="S4.E9.m1.1.1.1.1.cmml" xref="S4.E9.m1.1.1.1.1">𝑛</ci><ci id="S4.E9.m1.2.2.2.2.cmml" xref="S4.E9.m1.2.2.2.2">𝑚</ci></list></apply></apply><apply id="S4.E9.m1.9.9.1.1.6.cmml" xref="S4.E9.m1.9.9.1.1.6"><apply id="S4.E9.m1.9.9.1.1.6.1.cmml" xref="S4.E9.m1.9.9.1.1.6.1"><csymbol cd="ambiguous" id="S4.E9.m1.9.9.1.1.6.1.1.cmml" xref="S4.E9.m1.9.9.1.1.6.1">superscript</csymbol><apply id="S4.E9.m1.9.9.1.1.6.1.2.cmml" xref="S4.E9.m1.9.9.1.1.6.1"><csymbol cd="ambiguous" id="S4.E9.m1.9.9.1.1.6.1.2.1.cmml" xref="S4.E9.m1.9.9.1.1.6.1">subscript</csymbol><sum id="S4.E9.m1.9.9.1.1.6.1.2.2.cmml" xref="S4.E9.m1.9.9.1.1.6.1.2.2"></sum><apply id="S4.E9.m1.9.9.1.1.6.1.2.3.cmml" xref="S4.E9.m1.9.9.1.1.6.1.2.3"><eq id="S4.E9.m1.9.9.1.1.6.1.2.3.1.cmml" xref="S4.E9.m1.9.9.1.1.6.1.2.3.1"></eq><ci id="S4.E9.m1.9.9.1.1.6.1.2.3.2.cmml" xref="S4.E9.m1.9.9.1.1.6.1.2.3.2">𝑐</ci><cn type="integer" id="S4.E9.m1.9.9.1.1.6.1.2.3.3.cmml" xref="S4.E9.m1.9.9.1.1.6.1.2.3.3">1</cn></apply></apply><ci id="S4.E9.m1.9.9.1.1.6.1.3.cmml" xref="S4.E9.m1.9.9.1.1.6.1.3">𝐶</ci></apply><apply id="S4.E9.m1.9.9.1.1.6.2.cmml" xref="S4.E9.m1.9.9.1.1.6.2"><times id="S4.E9.m1.9.9.1.1.6.2.1.cmml" xref="S4.E9.m1.9.9.1.1.6.2.1"></times><apply id="S4.E9.m1.9.9.1.1.6.2.2.cmml" xref="S4.E9.m1.9.9.1.1.6.2.2"><csymbol cd="ambiguous" id="S4.E9.m1.9.9.1.1.6.2.2.1.cmml" xref="S4.E9.m1.9.9.1.1.6.2.2">subscript</csymbol><ci id="S4.E9.m1.9.9.1.1.6.2.2.2.cmml" xref="S4.E9.m1.9.9.1.1.6.2.2.2">𝑻</ci><list id="S4.E9.m1.5.5.3.4.cmml" xref="S4.E9.m1.5.5.3.5"><ci id="S4.E9.m1.3.3.1.1.cmml" xref="S4.E9.m1.3.3.1.1">𝑚</ci><ci id="S4.E9.m1.4.4.2.2.cmml" xref="S4.E9.m1.4.4.2.2">𝑐</ci><ci id="S4.E9.m1.5.5.3.3.cmml" xref="S4.E9.m1.5.5.3.3">𝑘</ci></list></apply><ci id="S4.E9.m1.9.9.1.1.6.2.3c.cmml" xref="S4.E9.m1.9.9.1.1.6.2.3"><mrow id="S4.E9.m1.9.9.1.1.6.2.3.cmml" xref="S4.E9.m1.9.9.1.1.6.2.3"><mtext id="S4.E9.m1.9.9.1.1.6.2.3a.cmml" xref="S4.E9.m1.9.9.1.1.6.2.3">, </mtext><mtext id="S4.E9.m1.9.9.1.1.6.2.3b.cmml" xref="S4.E9.m1.9.9.1.1.6.2.3"> </mtext></mrow></ci><ci id="S4.E9.m1.9.9.1.1.6.2.4.cmml" xref="S4.E9.m1.9.9.1.1.6.2.4">𝑘</ci></apply></apply></apply><apply id="S4.E9.m1.9.9.1.1c.cmml" xref="S4.E9.m1.9.9.1"><eq id="S4.E9.m1.9.9.1.1.7.cmml" xref="S4.E9.m1.9.9.1.1.7"></eq><share href="#S4.E9.m1.9.9.1.1.6.cmml" id="S4.E9.m1.9.9.1.1d.cmml" xref="S4.E9.m1.9.9.1"></share><apply id="S4.E9.m1.9.9.1.1.3.cmml" xref="S4.E9.m1.9.9.1.1.3"><times id="S4.E9.m1.9.9.1.1.3.2.cmml" xref="S4.E9.m1.9.9.1.1.3.2"></times><apply id="S4.E9.m1.9.9.1.1.3.3.cmml" xref="S4.E9.m1.9.9.1.1.3.3"><csymbol cd="ambiguous" id="S4.E9.m1.9.9.1.1.3.3.1.cmml" xref="S4.E9.m1.9.9.1.1.3.3">superscript</csymbol><ci id="S4.E9.m1.9.9.1.1.3.3.2.cmml" xref="S4.E9.m1.9.9.1.1.3.3.2">𝑔</ci><ci id="S4.E9.m1.6.6.1.1.cmml" xref="S4.E9.m1.6.6.1.1">𝑐</ci></apply><apply id="S4.E9.m1.9.9.1.1.3.1.1.1.cmml" xref="S4.E9.m1.9.9.1.1.3.1.1"><csymbol cd="ambiguous" id="S4.E9.m1.9.9.1.1.3.1.1.1.1.cmml" xref="S4.E9.m1.9.9.1.1.3.1.1">subscript</csymbol><ci id="S4.E9.m1.9.9.1.1.3.1.1.1.2.cmml" xref="S4.E9.m1.9.9.1.1.3.1.1.1.2">𝒂</ci><ci id="S4.E9.m1.9.9.1.1.3.1.1.1.3.cmml" xref="S4.E9.m1.9.9.1.1.3.1.1.1.3">𝑛</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E9.m1.9c">\displaystyle\vspace*{2mm}f(g(\bm{A}),h(\bm{B}))_{n,m}\triangleq\sum_{c=1}^{C}\bm{T}_{m,c,k}\text{, }\text{ }k=g^{(c)}(\bm{a}_{n}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="S4.SS4.p1.6" class="ltx_p">Because the entries of <math id="S4.SS4.p1.6.m1.1" class="ltx_Math" alttext="\bm{T}" display="inline"><semantics id="S4.SS4.p1.6.m1.1a"><mi id="S4.SS4.p1.6.m1.1.1" xref="S4.SS4.p1.6.m1.1.1.cmml">𝑻</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.6.m1.1b"><ci id="S4.SS4.p1.6.m1.1.1.cmml" xref="S4.SS4.p1.6.m1.1.1">𝑻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.6.m1.1c">\bm{T}</annotation></semantics></math> are stored as 8-bit values, exact summation requires immediately upcasting each looked-up entry to 16 bits before performing any addition instructions <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>. This not only imposes overhead directly, but also means that one must perform 16-bit additions, which have half the throughput of 8-bit additions.</p>
</div>
<div id="S4.SS4.p2" class="ltx_para">
<p id="S4.SS4.p2.2" class="ltx_p">We propose an alternative that sacrifices a small amount of accuracy for a significant increase in speed. Instead of using <span id="S4.SS4.p2.2.1" class="ltx_text ltx_font_italic">addition</span> instructions, we use <span id="S4.SS4.p2.2.2" class="ltx_text ltx_font_italic">averaging</span> instructions, such as <span id="S4.SS4.p2.2.3" class="ltx_text ltx_font_typewriter">vpavgb</span> on x86 or <span id="S4.SS4.p2.2.4" class="ltx_text ltx_font_typewriter">vrhadd</span> on ARM. While non-saturating additions compute <math id="S4.SS4.p2.1.m1.1" class="ltx_Math" alttext="(a+b)\text{ \% }256" display="inline"><semantics id="S4.SS4.p2.1.m1.1a"><mrow id="S4.SS4.p2.1.m1.1.1" xref="S4.SS4.p2.1.m1.1.1.cmml"><mrow id="S4.SS4.p2.1.m1.1.1.1.1" xref="S4.SS4.p2.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS4.p2.1.m1.1.1.1.1.2" xref="S4.SS4.p2.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS4.p2.1.m1.1.1.1.1.1" xref="S4.SS4.p2.1.m1.1.1.1.1.1.cmml"><mi id="S4.SS4.p2.1.m1.1.1.1.1.1.2" xref="S4.SS4.p2.1.m1.1.1.1.1.1.2.cmml">a</mi><mo id="S4.SS4.p2.1.m1.1.1.1.1.1.1" xref="S4.SS4.p2.1.m1.1.1.1.1.1.1.cmml">+</mo><mi id="S4.SS4.p2.1.m1.1.1.1.1.1.3" xref="S4.SS4.p2.1.m1.1.1.1.1.1.3.cmml">b</mi></mrow><mo stretchy="false" id="S4.SS4.p2.1.m1.1.1.1.1.3" xref="S4.SS4.p2.1.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.SS4.p2.1.m1.1.1.2" xref="S4.SS4.p2.1.m1.1.1.2.cmml">​</mo><mtext id="S4.SS4.p2.1.m1.1.1.3" xref="S4.SS4.p2.1.m1.1.1.3a.cmml"> % </mtext><mo lspace="0em" rspace="0em" id="S4.SS4.p2.1.m1.1.1.2a" xref="S4.SS4.p2.1.m1.1.1.2.cmml">​</mo><mn id="S4.SS4.p2.1.m1.1.1.4" xref="S4.SS4.p2.1.m1.1.1.4.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.1.m1.1b"><apply id="S4.SS4.p2.1.m1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1"><times id="S4.SS4.p2.1.m1.1.1.2.cmml" xref="S4.SS4.p2.1.m1.1.1.2"></times><apply id="S4.SS4.p2.1.m1.1.1.1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1.1.1"><plus id="S4.SS4.p2.1.m1.1.1.1.1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1.1.1.1.1"></plus><ci id="S4.SS4.p2.1.m1.1.1.1.1.1.2.cmml" xref="S4.SS4.p2.1.m1.1.1.1.1.1.2">𝑎</ci><ci id="S4.SS4.p2.1.m1.1.1.1.1.1.3.cmml" xref="S4.SS4.p2.1.m1.1.1.1.1.1.3">𝑏</ci></apply><ci id="S4.SS4.p2.1.m1.1.1.3a.cmml" xref="S4.SS4.p2.1.m1.1.1.3"><mtext id="S4.SS4.p2.1.m1.1.1.3.cmml" xref="S4.SS4.p2.1.m1.1.1.3"> % </mtext></ci><cn type="integer" id="S4.SS4.p2.1.m1.1.1.4.cmml" xref="S4.SS4.p2.1.m1.1.1.4">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.1.m1.1c">(a+b)\text{ \% }256</annotation></semantics></math>, these instructions compute <math id="S4.SS4.p2.2.m2.1" class="ltx_Math" alttext="(a+b+1)/2" display="inline"><semantics id="S4.SS4.p2.2.m2.1a"><mrow id="S4.SS4.p2.2.m2.1.1" xref="S4.SS4.p2.2.m2.1.1.cmml"><mrow id="S4.SS4.p2.2.m2.1.1.1.1" xref="S4.SS4.p2.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS4.p2.2.m2.1.1.1.1.2" xref="S4.SS4.p2.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS4.p2.2.m2.1.1.1.1.1" xref="S4.SS4.p2.2.m2.1.1.1.1.1.cmml"><mi id="S4.SS4.p2.2.m2.1.1.1.1.1.2" xref="S4.SS4.p2.2.m2.1.1.1.1.1.2.cmml">a</mi><mo id="S4.SS4.p2.2.m2.1.1.1.1.1.1" xref="S4.SS4.p2.2.m2.1.1.1.1.1.1.cmml">+</mo><mi id="S4.SS4.p2.2.m2.1.1.1.1.1.3" xref="S4.SS4.p2.2.m2.1.1.1.1.1.3.cmml">b</mi><mo id="S4.SS4.p2.2.m2.1.1.1.1.1.1a" xref="S4.SS4.p2.2.m2.1.1.1.1.1.1.cmml">+</mo><mn id="S4.SS4.p2.2.m2.1.1.1.1.1.4" xref="S4.SS4.p2.2.m2.1.1.1.1.1.4.cmml">1</mn></mrow><mo stretchy="false" id="S4.SS4.p2.2.m2.1.1.1.1.3" xref="S4.SS4.p2.2.m2.1.1.1.1.1.cmml">)</mo></mrow><mo id="S4.SS4.p2.2.m2.1.1.2" xref="S4.SS4.p2.2.m2.1.1.2.cmml">/</mo><mn id="S4.SS4.p2.2.m2.1.1.3" xref="S4.SS4.p2.2.m2.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.2.m2.1b"><apply id="S4.SS4.p2.2.m2.1.1.cmml" xref="S4.SS4.p2.2.m2.1.1"><divide id="S4.SS4.p2.2.m2.1.1.2.cmml" xref="S4.SS4.p2.2.m2.1.1.2"></divide><apply id="S4.SS4.p2.2.m2.1.1.1.1.1.cmml" xref="S4.SS4.p2.2.m2.1.1.1.1"><plus id="S4.SS4.p2.2.m2.1.1.1.1.1.1.cmml" xref="S4.SS4.p2.2.m2.1.1.1.1.1.1"></plus><ci id="S4.SS4.p2.2.m2.1.1.1.1.1.2.cmml" xref="S4.SS4.p2.2.m2.1.1.1.1.1.2">𝑎</ci><ci id="S4.SS4.p2.2.m2.1.1.1.1.1.3.cmml" xref="S4.SS4.p2.2.m2.1.1.1.1.1.3">𝑏</ci><cn type="integer" id="S4.SS4.p2.2.m2.1.1.1.1.1.4.cmml" xref="S4.SS4.p2.2.m2.1.1.1.1.1.4">1</cn></apply><cn type="integer" id="S4.SS4.p2.2.m2.1.1.3.cmml" xref="S4.SS4.p2.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.2.m2.1c">(a+b+1)/2</annotation></semantics></math>. This means that they lose information about the low bit instead of the high bit of the sum. We estimate the overall mean by averaging pairs of values, then pairs of pairs, and so on. We refer the reader to Appendix <a href="#A4" title="Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a> for details.</p>
</div>
<div id="S4.SS4.p3" class="ltx_para">
<p id="S4.SS4.p3.1" class="ltx_p">The challenging part of this approach is computing the bias in the estimated sum in order to correct for it. We prove in Appendix <a href="#A4" title="Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a> that this bias has a closed-form solution under the realistic assumption that the low bits are equally likely to be 0 or 1.</p>
</div>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Theoretical Guarantees</h3>

<div id="S4.SS5.p1" class="ltx_para">
<p id="S4.SS5.p1.1" class="ltx_p">Our central theoretical result is a generalization guarantee for the overall approximation error of <span id="S4.SS5.p1.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>, stated below. See Appendix <a href="#A6" title="Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F</span></a> for a proof and additional analysis, including a discussion of algorithmic complexity. Besides this main guarantee, we also inherit all of the guarantees for Bolt <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>, modulo a small amount of additional error from averaging integers rather than summing exactly. This follows from Bolt’s guarantees depending only on the quantization errors, rather than the method used to obtain them.</p>
</div>
<div id="S4.Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem 4.1</span></span><span id="S4.Thmtheorem1.2.2" class="ltx_text ltx_font_bold"> </span>(Generalization Error of <span id="S4.Thmtheorem1.3.3" class="ltx_text ltx_font_smallcaps">Maddness</span>)<span id="S4.Thmtheorem1.4.4" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmtheorem1.p1" class="ltx_para">
<p id="S4.Thmtheorem1.p1.11" class="ltx_p"><span id="S4.Thmtheorem1.p1.11.11" class="ltx_text ltx_font_italic">Let <math id="S4.Thmtheorem1.p1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S4.Thmtheorem1.p1.1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.Thmtheorem1.p1.1.1.m1.1.1" xref="S4.Thmtheorem1.p1.1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.1.1.m1.1b"><ci id="S4.Thmtheorem1.p1.1.1.m1.1.1.cmml" xref="S4.Thmtheorem1.p1.1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.1.1.m1.1c">\mathcal{D}</annotation></semantics></math> be a probability distribution over <math id="S4.Thmtheorem1.p1.2.2.m2.1" class="ltx_Math" alttext="\mathbb{R}^{D}" display="inline"><semantics id="S4.Thmtheorem1.p1.2.2.m2.1a"><msup id="S4.Thmtheorem1.p1.2.2.m2.1.1" xref="S4.Thmtheorem1.p1.2.2.m2.1.1.cmml"><mi id="S4.Thmtheorem1.p1.2.2.m2.1.1.2" xref="S4.Thmtheorem1.p1.2.2.m2.1.1.2.cmml">ℝ</mi><mi id="S4.Thmtheorem1.p1.2.2.m2.1.1.3" xref="S4.Thmtheorem1.p1.2.2.m2.1.1.3.cmml">D</mi></msup><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.2.2.m2.1b"><apply id="S4.Thmtheorem1.p1.2.2.m2.1.1.cmml" xref="S4.Thmtheorem1.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.2.2.m2.1.1.1.cmml" xref="S4.Thmtheorem1.p1.2.2.m2.1.1">superscript</csymbol><ci id="S4.Thmtheorem1.p1.2.2.m2.1.1.2.cmml" xref="S4.Thmtheorem1.p1.2.2.m2.1.1.2">ℝ</ci><ci id="S4.Thmtheorem1.p1.2.2.m2.1.1.3.cmml" xref="S4.Thmtheorem1.p1.2.2.m2.1.1.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.2.2.m2.1c">\mathbb{R}^{D}</annotation></semantics></math> and suppose that <span id="S4.Thmtheorem1.p1.11.11.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S4.Thmtheorem1.p1.11.11.1.1" class="ltx_text ltx_font_italic"> </span></span>is trained on a matrix <math id="S4.Thmtheorem1.p1.3.3.m3.1" class="ltx_Math" alttext="\tilde{\bm{A}}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="S4.Thmtheorem1.p1.3.3.m3.1a"><mrow id="S4.Thmtheorem1.p1.3.3.m3.1.1" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.cmml"><mover accent="true" id="S4.Thmtheorem1.p1.3.3.m3.1.1.2" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.2.cmml"><mi id="S4.Thmtheorem1.p1.3.3.m3.1.1.2.2" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.2.2.cmml">𝐀</mi><mo id="S4.Thmtheorem1.p1.3.3.m3.1.1.2.1" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.2.1.cmml">~</mo></mover><mo id="S4.Thmtheorem1.p1.3.3.m3.1.1.1" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.1.cmml">∈</mo><msup id="S4.Thmtheorem1.p1.3.3.m3.1.1.3" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.cmml"><mi id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.2" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.cmml"><mi id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.2" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.1" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.3" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.3.3.m3.1b"><apply id="S4.Thmtheorem1.p1.3.3.m3.1.1.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1"><in id="S4.Thmtheorem1.p1.3.3.m3.1.1.1.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.1"></in><apply id="S4.Thmtheorem1.p1.3.3.m3.1.1.2.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.2"><ci id="S4.Thmtheorem1.p1.3.3.m3.1.1.2.1.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.2.1">~</ci><ci id="S4.Thmtheorem1.p1.3.3.m3.1.1.2.2.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.2.2">𝐀</ci></apply><apply id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.1.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3">superscript</csymbol><ci id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.2.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.2">ℝ</ci><apply id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3"><times id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.1.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.1"></times><ci id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.2.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.2">𝑁</ci><ci id="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.3.cmml" xref="S4.Thmtheorem1.p1.3.3.m3.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.3.3.m3.1c">\tilde{\bm{A}}\in\mathbb{R}^{N\times D}</annotation></semantics></math> whose rows are drawn independently from <math id="S4.Thmtheorem1.p1.4.4.m4.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S4.Thmtheorem1.p1.4.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S4.Thmtheorem1.p1.4.4.m4.1.1" xref="S4.Thmtheorem1.p1.4.4.m4.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.4.4.m4.1b"><ci id="S4.Thmtheorem1.p1.4.4.m4.1.1.cmml" xref="S4.Thmtheorem1.p1.4.4.m4.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.4.4.m4.1c">\mathcal{D}</annotation></semantics></math> and with maximum singular value bounded by <math id="S4.Thmtheorem1.p1.5.5.m5.1" class="ltx_Math" alttext="\sigma_{A}" display="inline"><semantics id="S4.Thmtheorem1.p1.5.5.m5.1a"><msub id="S4.Thmtheorem1.p1.5.5.m5.1.1" xref="S4.Thmtheorem1.p1.5.5.m5.1.1.cmml"><mi id="S4.Thmtheorem1.p1.5.5.m5.1.1.2" xref="S4.Thmtheorem1.p1.5.5.m5.1.1.2.cmml">σ</mi><mi id="S4.Thmtheorem1.p1.5.5.m5.1.1.3" xref="S4.Thmtheorem1.p1.5.5.m5.1.1.3.cmml">A</mi></msub><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.5.5.m5.1b"><apply id="S4.Thmtheorem1.p1.5.5.m5.1.1.cmml" xref="S4.Thmtheorem1.p1.5.5.m5.1.1"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.5.5.m5.1.1.1.cmml" xref="S4.Thmtheorem1.p1.5.5.m5.1.1">subscript</csymbol><ci id="S4.Thmtheorem1.p1.5.5.m5.1.1.2.cmml" xref="S4.Thmtheorem1.p1.5.5.m5.1.1.2">𝜎</ci><ci id="S4.Thmtheorem1.p1.5.5.m5.1.1.3.cmml" xref="S4.Thmtheorem1.p1.5.5.m5.1.1.3">𝐴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.5.5.m5.1c">\sigma_{A}</annotation></semantics></math>. Let <math id="S4.Thmtheorem1.p1.6.6.m6.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S4.Thmtheorem1.p1.6.6.m6.1a"><mi id="S4.Thmtheorem1.p1.6.6.m6.1.1" xref="S4.Thmtheorem1.p1.6.6.m6.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.6.6.m6.1b"><ci id="S4.Thmtheorem1.p1.6.6.m6.1.1.cmml" xref="S4.Thmtheorem1.p1.6.6.m6.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.6.6.m6.1c">C</annotation></semantics></math> be the number of codebooks used by <span id="S4.Thmtheorem1.p1.11.11.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="S4.Thmtheorem1.p1.11.11.2.1" class="ltx_text ltx_font_italic"> </span></span>and <math id="S4.Thmtheorem1.p1.7.7.m7.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="S4.Thmtheorem1.p1.7.7.m7.1a"><mrow id="S4.Thmtheorem1.p1.7.7.m7.1.1" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.cmml"><mi id="S4.Thmtheorem1.p1.7.7.m7.1.1.2" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.2.cmml">λ</mi><mo id="S4.Thmtheorem1.p1.7.7.m7.1.1.1" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.1.cmml">&gt;</mo><mn id="S4.Thmtheorem1.p1.7.7.m7.1.1.3" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.7.7.m7.1b"><apply id="S4.Thmtheorem1.p1.7.7.m7.1.1.cmml" xref="S4.Thmtheorem1.p1.7.7.m7.1.1"><gt id="S4.Thmtheorem1.p1.7.7.m7.1.1.1.cmml" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.1"></gt><ci id="S4.Thmtheorem1.p1.7.7.m7.1.1.2.cmml" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.2">𝜆</ci><cn type="integer" id="S4.Thmtheorem1.p1.7.7.m7.1.1.3.cmml" xref="S4.Thmtheorem1.p1.7.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.7.7.m7.1c">\lambda&gt;0</annotation></semantics></math> be the regularization parameter used in the ridge regression step. Then for any <math id="S4.Thmtheorem1.p1.8.8.m8.1" class="ltx_Math" alttext="\bm{b}\in\mathbb{R}^{D}" display="inline"><semantics id="S4.Thmtheorem1.p1.8.8.m8.1a"><mrow id="S4.Thmtheorem1.p1.8.8.m8.1.1" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.cmml"><mi id="S4.Thmtheorem1.p1.8.8.m8.1.1.2" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.2.cmml">𝐛</mi><mo id="S4.Thmtheorem1.p1.8.8.m8.1.1.1" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.1.cmml">∈</mo><msup id="S4.Thmtheorem1.p1.8.8.m8.1.1.3" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3.cmml"><mi id="S4.Thmtheorem1.p1.8.8.m8.1.1.3.2" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3.2.cmml">ℝ</mi><mi id="S4.Thmtheorem1.p1.8.8.m8.1.1.3.3" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.8.8.m8.1b"><apply id="S4.Thmtheorem1.p1.8.8.m8.1.1.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1"><in id="S4.Thmtheorem1.p1.8.8.m8.1.1.1.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.1"></in><ci id="S4.Thmtheorem1.p1.8.8.m8.1.1.2.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.2">𝐛</ci><apply id="S4.Thmtheorem1.p1.8.8.m8.1.1.3.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.8.8.m8.1.1.3.1.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3">superscript</csymbol><ci id="S4.Thmtheorem1.p1.8.8.m8.1.1.3.2.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3.2">ℝ</ci><ci id="S4.Thmtheorem1.p1.8.8.m8.1.1.3.3.cmml" xref="S4.Thmtheorem1.p1.8.8.m8.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.8.8.m8.1c">\bm{b}\in\mathbb{R}^{D}</annotation></semantics></math>, any <math id="S4.Thmtheorem1.p1.9.9.m9.1" class="ltx_Math" alttext="\bm{a}\sim\mathcal{D}" display="inline"><semantics id="S4.Thmtheorem1.p1.9.9.m9.1a"><mrow id="S4.Thmtheorem1.p1.9.9.m9.1.1" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.cmml"><mi id="S4.Thmtheorem1.p1.9.9.m9.1.1.2" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.2.cmml">𝐚</mi><mo id="S4.Thmtheorem1.p1.9.9.m9.1.1.1" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.1.cmml">∼</mo><mi class="ltx_font_mathcaligraphic" id="S4.Thmtheorem1.p1.9.9.m9.1.1.3" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.3.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.9.9.m9.1b"><apply id="S4.Thmtheorem1.p1.9.9.m9.1.1.cmml" xref="S4.Thmtheorem1.p1.9.9.m9.1.1"><csymbol cd="latexml" id="S4.Thmtheorem1.p1.9.9.m9.1.1.1.cmml" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.1">similar-to</csymbol><ci id="S4.Thmtheorem1.p1.9.9.m9.1.1.2.cmml" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.2">𝐚</ci><ci id="S4.Thmtheorem1.p1.9.9.m9.1.1.3.cmml" xref="S4.Thmtheorem1.p1.9.9.m9.1.1.3">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.9.9.m9.1c">\bm{a}\sim\mathcal{D}</annotation></semantics></math>, and any <math id="S4.Thmtheorem1.p1.10.10.m10.1" class="ltx_Math" alttext="0&lt;\delta&lt;1" display="inline"><semantics id="S4.Thmtheorem1.p1.10.10.m10.1a"><mrow id="S4.Thmtheorem1.p1.10.10.m10.1.1" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.cmml"><mn id="S4.Thmtheorem1.p1.10.10.m10.1.1.2" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.2.cmml">0</mn><mo id="S4.Thmtheorem1.p1.10.10.m10.1.1.3" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.3.cmml">&lt;</mo><mi id="S4.Thmtheorem1.p1.10.10.m10.1.1.4" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.4.cmml">δ</mi><mo id="S4.Thmtheorem1.p1.10.10.m10.1.1.5" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.5.cmml">&lt;</mo><mn id="S4.Thmtheorem1.p1.10.10.m10.1.1.6" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.6.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.10.10.m10.1b"><apply id="S4.Thmtheorem1.p1.10.10.m10.1.1.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1"><and id="S4.Thmtheorem1.p1.10.10.m10.1.1a.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1"></and><apply id="S4.Thmtheorem1.p1.10.10.m10.1.1b.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1"><lt id="S4.Thmtheorem1.p1.10.10.m10.1.1.3.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.3"></lt><cn type="integer" id="S4.Thmtheorem1.p1.10.10.m10.1.1.2.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.2">0</cn><ci id="S4.Thmtheorem1.p1.10.10.m10.1.1.4.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.4">𝛿</ci></apply><apply id="S4.Thmtheorem1.p1.10.10.m10.1.1c.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1"><lt id="S4.Thmtheorem1.p1.10.10.m10.1.1.5.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.5"></lt><share href="#S4.Thmtheorem1.p1.10.10.m10.1.1.4.cmml" id="S4.Thmtheorem1.p1.10.10.m10.1.1d.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1"></share><cn type="integer" id="S4.Thmtheorem1.p1.10.10.m10.1.1.6.cmml" xref="S4.Thmtheorem1.p1.10.10.m10.1.1.6">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.10.10.m10.1c">0&lt;\delta&lt;1</annotation></semantics></math>, we have with probability at least <math id="S4.Thmtheorem1.p1.11.11.m11.1" class="ltx_Math" alttext="1-\delta" display="inline"><semantics id="S4.Thmtheorem1.p1.11.11.m11.1a"><mrow id="S4.Thmtheorem1.p1.11.11.m11.1.1" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.cmml"><mn id="S4.Thmtheorem1.p1.11.11.m11.1.1.2" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.2.cmml">1</mn><mo id="S4.Thmtheorem1.p1.11.11.m11.1.1.1" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.1.cmml">−</mo><mi id="S4.Thmtheorem1.p1.11.11.m11.1.1.3" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.3.cmml">δ</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.11.11.m11.1b"><apply id="S4.Thmtheorem1.p1.11.11.m11.1.1.cmml" xref="S4.Thmtheorem1.p1.11.11.m11.1.1"><minus id="S4.Thmtheorem1.p1.11.11.m11.1.1.1.cmml" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.1"></minus><cn type="integer" id="S4.Thmtheorem1.p1.11.11.m11.1.1.2.cmml" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.2">1</cn><ci id="S4.Thmtheorem1.p1.11.11.m11.1.1.3.cmml" xref="S4.Thmtheorem1.p1.11.11.m11.1.1.3">𝛿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.11.11.m11.1c">1-\delta</annotation></semantics></math> that</span></p>
<table id="A6.EGx9" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E10"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td colspan="2" class="ltx_td ltx_align_center ltx_eqn_cell"><math id="S4.E10.m1.32" class="ltx_Math" alttext="\displaystyle\begin{split}\mathbb{E}_{\mathcal{D}}[&amp;\mathcal{L}(\bm{a},\bm{b})]\leq\mathbb{E}_{\tilde{\bm{A}}}[\mathcal{L}(\bm{a},\bm{b})]+\\
&amp;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}\left(\frac{1}{256}+\frac{8+\sqrt{\nu(C,D,\delta)}}{\sqrt{2n}}\right)\end{split}" display="inline"><semantics id="S4.E10.m1.32a"><mtable columnspacing="0pt" rowspacing="0pt" id="S4.E10.m1.32.32.4" xref="S4.E10.m1.31.31.3.cmml"><mtr id="S4.E10.m1.32.32.4a" xref="S4.E10.m1.31.31.3.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E10.m1.32.32.4b" xref="S4.E10.m1.31.31.3.cmml"><mrow id="S4.E10.m1.3.3.3.3.3a" xref="S4.E10.m1.31.31.3.cmml"><msub id="S4.E10.m1.3.3.3.3.3a.4" xref="S4.E10.m1.31.31.3.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1" xref="S4.E10.m1.1.1.1.1.1.1.cmml">𝔼</mi><mi class="ltx_font_mathcaligraphic" id="S4.E10.m1.2.2.2.2.2.2.1" xref="S4.E10.m1.2.2.2.2.2.2.1.cmml">𝒟</mi></msub><mo stretchy="false" id="S4.E10.m1.3.3.3.3.3.3" xref="S4.E10.m1.31.31.3.cmml">[</mo></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E10.m1.32.32.4c" xref="S4.E10.m1.31.31.3.cmml"><mrow id="S4.E10.m1.22.22.22.22.19a" xref="S4.E10.m1.31.31.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E10.m1.4.4.4.4.1.1" xref="S4.E10.m1.4.4.4.4.1.1.cmml">ℒ</mi><mrow id="S4.E10.m1.22.22.22.22.19a.20" xref="S4.E10.m1.31.31.3.cmml"><mo stretchy="false" id="S4.E10.m1.5.5.5.5.2.2" xref="S4.E10.m1.31.31.3.cmml">(</mo><mi id="S4.E10.m1.6.6.6.6.3.3" xref="S4.E10.m1.6.6.6.6.3.3.cmml">𝒂</mi><mo id="S4.E10.m1.7.7.7.7.4.4" xref="S4.E10.m1.31.31.3.cmml">,</mo><mi id="S4.E10.m1.8.8.8.8.5.5" xref="S4.E10.m1.8.8.8.8.5.5.cmml">𝒃</mi><mo stretchy="false" id="S4.E10.m1.9.9.9.9.6.6" xref="S4.E10.m1.31.31.3.cmml">)</mo></mrow><mo stretchy="false" id="S4.E10.m1.10.10.10.10.7.7" xref="S4.E10.m1.31.31.3.cmml">]</mo><mo id="S4.E10.m1.11.11.11.11.8.8" xref="S4.E10.m1.11.11.11.11.8.8.cmml">≤</mo><mi id="S4.E10.m1.12.12.12.12.9.9" xref="S4.E10.m1.12.12.12.12.9.9.cmml">𝔼</mi><msub id="S4.E10.m1.22.22.22.22.19a.21" xref="S4.E10.m1.31.31.3.cmml"><mi id="S4.E10.m1.22.22.22.22.19a.21a" xref="S4.E10.m1.31.31.3.cmml"></mi><mover accent="true" id="S4.E10.m1.13.13.13.13.10.10.1" xref="S4.E10.m1.13.13.13.13.10.10.1.cmml"><mi id="S4.E10.m1.13.13.13.13.10.10.1.2" xref="S4.E10.m1.13.13.13.13.10.10.1.2.cmml">𝑨</mi><mo id="S4.E10.m1.13.13.13.13.10.10.1.1" xref="S4.E10.m1.13.13.13.13.10.10.1.1.cmml">~</mo></mover></msub><mo stretchy="false" id="S4.E10.m1.14.14.14.14.11.11" xref="S4.E10.m1.31.31.3.cmml">[</mo><mi class="ltx_font_mathcaligraphic" id="S4.E10.m1.15.15.15.15.12.12" xref="S4.E10.m1.15.15.15.15.12.12.cmml">ℒ</mi><mrow id="S4.E10.m1.22.22.22.22.19a.22" xref="S4.E10.m1.31.31.3.cmml"><mo stretchy="false" id="S4.E10.m1.16.16.16.16.13.13" xref="S4.E10.m1.31.31.3.cmml">(</mo><mi id="S4.E10.m1.17.17.17.17.14.14" xref="S4.E10.m1.17.17.17.17.14.14.cmml">𝒂</mi><mo id="S4.E10.m1.18.18.18.18.15.15" xref="S4.E10.m1.31.31.3.cmml">,</mo><mi id="S4.E10.m1.19.19.19.19.16.16" xref="S4.E10.m1.19.19.19.19.16.16.cmml">𝒃</mi><mo stretchy="false" id="S4.E10.m1.20.20.20.20.17.17" xref="S4.E10.m1.31.31.3.cmml">)</mo></mrow><mo stretchy="false" id="S4.E10.m1.21.21.21.21.18.18" xref="S4.E10.m1.31.31.3.cmml">]</mo><mo id="S4.E10.m1.22.22.22.22.19.19" xref="S4.E10.m1.22.22.22.22.19.19.cmml">+</mo></mrow></mtd></mtr><mtr id="S4.E10.m1.32.32.4d" xref="S4.E10.m1.31.31.3.cmml"><mtd id="S4.E10.m1.32.32.4e" xref="S4.E10.m1.31.31.3.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E10.m1.32.32.4f" xref="S4.E10.m1.31.31.3.cmml"><mrow id="S4.E10.m1.32.32.4.29.7.7" xref="S4.E10.m1.31.31.3.cmml"><mstyle displaystyle="true" id="S4.E10.m1.23.23.23.1.1.1" xref="S4.E10.m1.23.23.23.1.1.1.cmml"><mfrac id="S4.E10.m1.23.23.23.1.1.1a" xref="S4.E10.m1.23.23.23.1.1.1.cmml"><mrow id="S4.E10.m1.23.23.23.1.1.1.1" xref="S4.E10.m1.23.23.23.1.1.1.1.cmml"><mi id="S4.E10.m1.23.23.23.1.1.1.1.3" xref="S4.E10.m1.23.23.23.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="S4.E10.m1.23.23.23.1.1.1.1.2" xref="S4.E10.m1.23.23.23.1.1.1.1.2.cmml">​</mo><msub id="S4.E10.m1.23.23.23.1.1.1.1.4" xref="S4.E10.m1.23.23.23.1.1.1.1.4.cmml"><mi id="S4.E10.m1.23.23.23.1.1.1.1.4.2" xref="S4.E10.m1.23.23.23.1.1.1.1.4.2.cmml">σ</mi><mi id="S4.E10.m1.23.23.23.1.1.1.1.4.3" xref="S4.E10.m1.23.23.23.1.1.1.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="S4.E10.m1.23.23.23.1.1.1.1.2a" xref="S4.E10.m1.23.23.23.1.1.1.1.2.cmml">​</mo><msub id="S4.E10.m1.23.23.23.1.1.1.1.5" xref="S4.E10.m1.23.23.23.1.1.1.1.5.cmml"><mrow id="S4.E10.m1.23.23.23.1.1.1.1.5.2.2" xref="S4.E10.m1.23.23.23.1.1.1.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="S4.E10.m1.23.23.23.1.1.1.1.5.2.2.1" xref="S4.E10.m1.23.23.23.1.1.1.1.5.2.1.1.cmml">∥</mo><mi id="S4.E10.m1.23.23.23.1.1.1.1.1" xref="S4.E10.m1.23.23.23.1.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="S4.E10.m1.23.23.23.1.1.1.1.5.2.2.2" xref="S4.E10.m1.23.23.23.1.1.1.1.5.2.1.1.cmml">∥</mo></mrow><mn id="S4.E10.m1.23.23.23.1.1.1.1.5.3" xref="S4.E10.m1.23.23.23.1.1.1.1.5.3.cmml">2</mn></msub></mrow><mrow id="S4.E10.m1.23.23.23.1.1.1.3" xref="S4.E10.m1.23.23.23.1.1.1.3.cmml"><mn id="S4.E10.m1.23.23.23.1.1.1.3.2" xref="S4.E10.m1.23.23.23.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.E10.m1.23.23.23.1.1.1.3.1" xref="S4.E10.m1.23.23.23.1.1.1.3.1.cmml">​</mo><msqrt id="S4.E10.m1.23.23.23.1.1.1.3.3" xref="S4.E10.m1.23.23.23.1.1.1.3.3.cmml"><mi id="S4.E10.m1.23.23.23.1.1.1.3.3.2" xref="S4.E10.m1.23.23.23.1.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="S4.E10.m1.32.32.4.29.7.7.8" xref="S4.E10.m1.31.31.3.cmml">​</mo><mrow id="S4.E10.m1.32.32.4.29.7.7.7.1" xref="S4.E10.m1.31.31.3.cmml"><mo id="S4.E10.m1.24.24.24.2.2.2" xref="S4.E10.m1.31.31.3.cmml">(</mo><mrow id="S4.E10.m1.32.32.4.29.7.7.7.1.1" xref="S4.E10.m1.31.31.3.cmml"><mstyle displaystyle="true" id="S4.E10.m1.25.25.25.3.3.3" xref="S4.E10.m1.25.25.25.3.3.3.cmml"><mfrac id="S4.E10.m1.25.25.25.3.3.3a" xref="S4.E10.m1.25.25.25.3.3.3.cmml"><mn id="S4.E10.m1.25.25.25.3.3.3.2" xref="S4.E10.m1.25.25.25.3.3.3.2.cmml">1</mn><mn id="S4.E10.m1.25.25.25.3.3.3.3" xref="S4.E10.m1.25.25.25.3.3.3.3.cmml">256</mn></mfrac></mstyle><mo id="S4.E10.m1.26.26.26.4.4.4" xref="S4.E10.m1.26.26.26.4.4.4.cmml">+</mo><mstyle displaystyle="true" id="S4.E10.m1.27.27.27.5.5.5" xref="S4.E10.m1.27.27.27.5.5.5.cmml"><mfrac id="S4.E10.m1.27.27.27.5.5.5a" xref="S4.E10.m1.27.27.27.5.5.5.cmml"><mrow id="S4.E10.m1.27.27.27.5.5.5.3" xref="S4.E10.m1.27.27.27.5.5.5.3.cmml"><mn id="S4.E10.m1.27.27.27.5.5.5.3.5" xref="S4.E10.m1.27.27.27.5.5.5.3.5.cmml">8</mn><mo id="S4.E10.m1.27.27.27.5.5.5.3.4" xref="S4.E10.m1.27.27.27.5.5.5.3.4.cmml">+</mo><msqrt id="S4.E10.m1.27.27.27.5.5.5.3.3" xref="S4.E10.m1.27.27.27.5.5.5.3.3.cmml"><mrow id="S4.E10.m1.27.27.27.5.5.5.3.3.3" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.cmml"><mi id="S4.E10.m1.27.27.27.5.5.5.3.3.3.5" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.5.cmml">ν</mi><mo lspace="0em" rspace="0em" id="S4.E10.m1.27.27.27.5.5.5.3.3.3.4" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.4.cmml">​</mo><mrow id="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.2" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.1.cmml"><mo stretchy="false" id="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.2.1" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">(</mo><mi id="S4.E10.m1.27.27.27.5.5.5.1.1.1.1" xref="S4.E10.m1.27.27.27.5.5.5.1.1.1.1.cmml">C</mi><mo id="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.2.2" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">,</mo><mi id="S4.E10.m1.27.27.27.5.5.5.2.2.2.2" xref="S4.E10.m1.27.27.27.5.5.5.2.2.2.2.cmml">D</mi><mo id="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.2.3" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">,</mo><mi id="S4.E10.m1.27.27.27.5.5.5.3.3.3.3" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.3.cmml">δ</mi><mo stretchy="false" id="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.2.4" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">)</mo></mrow></mrow></msqrt></mrow><msqrt id="S4.E10.m1.27.27.27.5.5.5.5" xref="S4.E10.m1.27.27.27.5.5.5.5.cmml"><mrow id="S4.E10.m1.27.27.27.5.5.5.5.2" xref="S4.E10.m1.27.27.27.5.5.5.5.2.cmml"><mn id="S4.E10.m1.27.27.27.5.5.5.5.2.2" xref="S4.E10.m1.27.27.27.5.5.5.5.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.E10.m1.27.27.27.5.5.5.5.2.1" xref="S4.E10.m1.27.27.27.5.5.5.5.2.1.cmml">​</mo><mi id="S4.E10.m1.27.27.27.5.5.5.5.2.3" xref="S4.E10.m1.27.27.27.5.5.5.5.2.3.cmml">n</mi></mrow></msqrt></mfrac></mstyle></mrow><mo id="S4.E10.m1.28.28.28.6.6.6" xref="S4.E10.m1.31.31.3.cmml">)</mo></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E10.m1.32b"><apply id="S4.E10.m1.31.31.3.cmml" xref="S4.E10.m1.32.32.4"><leq id="S4.E10.m1.11.11.11.11.8.8.cmml" xref="S4.E10.m1.11.11.11.11.8.8"></leq><apply id="S4.E10.m1.29.29.1.1.cmml" xref="S4.E10.m1.32.32.4"><times id="S4.E10.m1.29.29.1.1.2.cmml" xref="S4.E10.m1.32.32.4"></times><apply id="S4.E10.m1.29.29.1.1.3.cmml" xref="S4.E10.m1.32.32.4"><csymbol cd="ambiguous" id="S4.E10.m1.29.29.1.1.3.1.cmml" xref="S4.E10.m1.32.32.4">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1">𝔼</ci><ci id="S4.E10.m1.2.2.2.2.2.2.1.cmml" xref="S4.E10.m1.2.2.2.2.2.2.1">𝒟</ci></apply><apply id="S4.E10.m1.29.29.1.1.1.2.cmml" xref="S4.E10.m1.32.32.4"><csymbol cd="latexml" id="S4.E10.m1.29.29.1.1.1.2.1.cmml" xref="S4.E10.m1.32.32.4">delimited-[]</csymbol><apply id="S4.E10.m1.29.29.1.1.1.1.1.cmml" xref="S4.E10.m1.32.32.4"><times id="S4.E10.m1.29.29.1.1.1.1.1.1.cmml" xref="S4.E10.m1.32.32.4"></times><ci id="S4.E10.m1.4.4.4.4.1.1.cmml" xref="S4.E10.m1.4.4.4.4.1.1">ℒ</ci><interval closure="open" id="S4.E10.m1.29.29.1.1.1.1.1.3.cmml" xref="S4.E10.m1.32.32.4"><ci id="S4.E10.m1.6.6.6.6.3.3.cmml" xref="S4.E10.m1.6.6.6.6.3.3">𝒂</ci><ci id="S4.E10.m1.8.8.8.8.5.5.cmml" xref="S4.E10.m1.8.8.8.8.5.5">𝒃</ci></interval></apply></apply></apply><apply id="S4.E10.m1.31.31.3.3.cmml" xref="S4.E10.m1.32.32.4"><plus id="S4.E10.m1.22.22.22.22.19.19.cmml" xref="S4.E10.m1.22.22.22.22.19.19"></plus><apply id="S4.E10.m1.30.30.2.2.1.cmml" xref="S4.E10.m1.32.32.4"><times id="S4.E10.m1.30.30.2.2.1.2.cmml" xref="S4.E10.m1.32.32.4"></times><apply id="S4.E10.m1.30.30.2.2.1.3.cmml" xref="S4.E10.m1.32.32.4"><csymbol cd="ambiguous" id="S4.E10.m1.30.30.2.2.1.3.1.cmml" xref="S4.E10.m1.32.32.4">subscript</csymbol><ci id="S4.E10.m1.12.12.12.12.9.9.cmml" xref="S4.E10.m1.12.12.12.12.9.9">𝔼</ci><apply id="S4.E10.m1.13.13.13.13.10.10.1.cmml" xref="S4.E10.m1.13.13.13.13.10.10.1"><ci id="S4.E10.m1.13.13.13.13.10.10.1.1.cmml" xref="S4.E10.m1.13.13.13.13.10.10.1.1">~</ci><ci id="S4.E10.m1.13.13.13.13.10.10.1.2.cmml" xref="S4.E10.m1.13.13.13.13.10.10.1.2">𝑨</ci></apply></apply><apply id="S4.E10.m1.30.30.2.2.1.1.2.cmml" xref="S4.E10.m1.32.32.4"><csymbol cd="latexml" id="S4.E10.m1.30.30.2.2.1.1.2.1.cmml" xref="S4.E10.m1.32.32.4">delimited-[]</csymbol><apply id="S4.E10.m1.30.30.2.2.1.1.1.1.cmml" xref="S4.E10.m1.32.32.4"><times id="S4.E10.m1.30.30.2.2.1.1.1.1.1.cmml" xref="S4.E10.m1.32.32.4"></times><ci id="S4.E10.m1.15.15.15.15.12.12.cmml" xref="S4.E10.m1.15.15.15.15.12.12">ℒ</ci><interval closure="open" id="S4.E10.m1.30.30.2.2.1.1.1.1.3.cmml" xref="S4.E10.m1.32.32.4"><ci id="S4.E10.m1.17.17.17.17.14.14.cmml" xref="S4.E10.m1.17.17.17.17.14.14">𝒂</ci><ci id="S4.E10.m1.19.19.19.19.16.16.cmml" xref="S4.E10.m1.19.19.19.19.16.16">𝒃</ci></interval></apply></apply></apply><apply id="S4.E10.m1.31.31.3.3.2.cmml" xref="S4.E10.m1.32.32.4"><times id="S4.E10.m1.31.31.3.3.2.2.cmml" xref="S4.E10.m1.32.32.4"></times><apply id="S4.E10.m1.23.23.23.1.1.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1"><divide id="S4.E10.m1.23.23.23.1.1.1.2.cmml" xref="S4.E10.m1.23.23.23.1.1.1"></divide><apply id="S4.E10.m1.23.23.23.1.1.1.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1"><times id="S4.E10.m1.23.23.23.1.1.1.1.2.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.2"></times><ci id="S4.E10.m1.23.23.23.1.1.1.1.3.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.3">𝐶</ci><apply id="S4.E10.m1.23.23.23.1.1.1.1.4.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.4"><csymbol cd="ambiguous" id="S4.E10.m1.23.23.23.1.1.1.1.4.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.4">subscript</csymbol><ci id="S4.E10.m1.23.23.23.1.1.1.1.4.2.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.4.2">𝜎</ci><ci id="S4.E10.m1.23.23.23.1.1.1.1.4.3.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.4.3">𝐴</ci></apply><apply id="S4.E10.m1.23.23.23.1.1.1.1.5.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.5"><csymbol cd="ambiguous" id="S4.E10.m1.23.23.23.1.1.1.1.5.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.5">subscript</csymbol><apply id="S4.E10.m1.23.23.23.1.1.1.1.5.2.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.5.2.2"><csymbol cd="latexml" id="S4.E10.m1.23.23.23.1.1.1.1.5.2.1.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.5.2.2.1">delimited-∥∥</csymbol><ci id="S4.E10.m1.23.23.23.1.1.1.1.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.1">𝒃</ci></apply><cn type="integer" id="S4.E10.m1.23.23.23.1.1.1.1.5.3.cmml" xref="S4.E10.m1.23.23.23.1.1.1.1.5.3">2</cn></apply></apply><apply id="S4.E10.m1.23.23.23.1.1.1.3.cmml" xref="S4.E10.m1.23.23.23.1.1.1.3"><times id="S4.E10.m1.23.23.23.1.1.1.3.1.cmml" xref="S4.E10.m1.23.23.23.1.1.1.3.1"></times><cn type="integer" id="S4.E10.m1.23.23.23.1.1.1.3.2.cmml" xref="S4.E10.m1.23.23.23.1.1.1.3.2">2</cn><apply id="S4.E10.m1.23.23.23.1.1.1.3.3.cmml" xref="S4.E10.m1.23.23.23.1.1.1.3.3"><root id="S4.E10.m1.23.23.23.1.1.1.3.3a.cmml" xref="S4.E10.m1.23.23.23.1.1.1.3.3"></root><ci id="S4.E10.m1.23.23.23.1.1.1.3.3.2.cmml" xref="S4.E10.m1.23.23.23.1.1.1.3.3.2">𝜆</ci></apply></apply></apply><apply id="S4.E10.m1.31.31.3.3.2.1.1.1.cmml" xref="S4.E10.m1.32.32.4"><plus id="S4.E10.m1.26.26.26.4.4.4.cmml" xref="S4.E10.m1.26.26.26.4.4.4"></plus><apply id="S4.E10.m1.25.25.25.3.3.3.cmml" xref="S4.E10.m1.25.25.25.3.3.3"><divide id="S4.E10.m1.25.25.25.3.3.3.1.cmml" xref="S4.E10.m1.25.25.25.3.3.3"></divide><cn type="integer" id="S4.E10.m1.25.25.25.3.3.3.2.cmml" xref="S4.E10.m1.25.25.25.3.3.3.2">1</cn><cn type="integer" id="S4.E10.m1.25.25.25.3.3.3.3.cmml" xref="S4.E10.m1.25.25.25.3.3.3.3">256</cn></apply><apply id="S4.E10.m1.27.27.27.5.5.5.cmml" xref="S4.E10.m1.27.27.27.5.5.5"><divide id="S4.E10.m1.27.27.27.5.5.5.4.cmml" xref="S4.E10.m1.27.27.27.5.5.5"></divide><apply id="S4.E10.m1.27.27.27.5.5.5.3.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3"><plus id="S4.E10.m1.27.27.27.5.5.5.3.4.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.4"></plus><cn type="integer" id="S4.E10.m1.27.27.27.5.5.5.3.5.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.5">8</cn><apply id="S4.E10.m1.27.27.27.5.5.5.3.3.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3"><root id="S4.E10.m1.27.27.27.5.5.5.3.3a.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3"></root><apply id="S4.E10.m1.27.27.27.5.5.5.3.3.3.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3"><times id="S4.E10.m1.27.27.27.5.5.5.3.3.3.4.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.4"></times><ci id="S4.E10.m1.27.27.27.5.5.5.3.3.3.5.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.5">𝜈</ci><vector id="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.1.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.6.2"><ci id="S4.E10.m1.27.27.27.5.5.5.1.1.1.1.cmml" xref="S4.E10.m1.27.27.27.5.5.5.1.1.1.1">𝐶</ci><ci id="S4.E10.m1.27.27.27.5.5.5.2.2.2.2.cmml" xref="S4.E10.m1.27.27.27.5.5.5.2.2.2.2">𝐷</ci><ci id="S4.E10.m1.27.27.27.5.5.5.3.3.3.3.cmml" xref="S4.E10.m1.27.27.27.5.5.5.3.3.3.3">𝛿</ci></vector></apply></apply></apply><apply id="S4.E10.m1.27.27.27.5.5.5.5.cmml" xref="S4.E10.m1.27.27.27.5.5.5.5"><root id="S4.E10.m1.27.27.27.5.5.5.5a.cmml" xref="S4.E10.m1.27.27.27.5.5.5.5"></root><apply id="S4.E10.m1.27.27.27.5.5.5.5.2.cmml" xref="S4.E10.m1.27.27.27.5.5.5.5.2"><times id="S4.E10.m1.27.27.27.5.5.5.5.2.1.cmml" xref="S4.E10.m1.27.27.27.5.5.5.5.2.1"></times><cn type="integer" id="S4.E10.m1.27.27.27.5.5.5.5.2.2.cmml" xref="S4.E10.m1.27.27.27.5.5.5.5.2.2">2</cn><ci id="S4.E10.m1.27.27.27.5.5.5.5.2.3.cmml" xref="S4.E10.m1.27.27.27.5.5.5.5.2.3">𝑛</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E10.m1.32c">\displaystyle\begin{split}\mathbb{E}_{\mathcal{D}}[&amp;\mathcal{L}(\bm{a},\bm{b})]\leq\mathbb{E}_{\tilde{\bm{A}}}[\mathcal{L}(\bm{a},\bm{b})]+\\
&amp;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}\left(\frac{1}{256}+\frac{8+\sqrt{\nu(C,D,\delta)}}{\sqrt{2n}}\right)\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="S4.Thmtheorem1.p1.14" class="ltx_p"><span id="S4.Thmtheorem1.p1.14.3" class="ltx_text ltx_font_italic">where <math id="S4.Thmtheorem1.p1.12.1.m1.5" class="ltx_Math" alttext="\mathcal{L}(\bm{a},\bm{b})\triangleq|\bm{a}^{\top}\bm{b}-\alpha f(g(\bm{a}),h(\bm{b}))-\bm{\beta}|" display="inline"><semantics id="S4.Thmtheorem1.p1.12.1.m1.5a"><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.cmml"><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.1.cmml">​</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.1.cmml"><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.2.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.1.cmml">(</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.1.1" xref="S4.Thmtheorem1.p1.12.1.m1.1.1.cmml">𝐚</mi><mo id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.1.cmml">,</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.2.2.cmml">𝐛</mi><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.2.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.1.cmml">)</mo></mrow></mrow><mo id="S4.Thmtheorem1.p1.12.1.m1.5.5.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.2.cmml">≜</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.2.cmml"><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.2.1.cmml">|</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.cmml"><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.cmml"><msup id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.cmml"><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.2.cmml">𝐚</mi><mo id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.1.cmml">​</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.3.cmml">𝐛</mi></mrow><mo id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.3.cmml">−</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.cmml"><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.4" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.4.cmml">α</mi><mo lspace="0em" rspace="0em" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.3.cmml">​</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.5" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.3a" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.3.cmml">​</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml">(</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml"><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.3.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.3.2.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.3.3" xref="S4.Thmtheorem1.p1.12.1.m1.3.3.cmml">𝐚</mi><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.3.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.4" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml">,</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml"><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1.cmml">​</mo><mrow id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.3.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml"><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.3.2.1" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml">(</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.4.4" xref="S4.Thmtheorem1.p1.12.1.m1.4.4.cmml">𝐛</mi><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.3.2.2" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.5" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.3a" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.3.cmml">−</mo><mi id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.5" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.5.cmml">𝛃</mi></mrow><mo stretchy="false" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.3" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.12.1.m1.5b"><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5"><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.2">≜</ci><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3"><times id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.1"></times><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.2">ℒ</ci><interval closure="open" id="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.3.3.2"><ci id="S4.Thmtheorem1.p1.12.1.m1.1.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.1.1">𝐚</ci><ci id="S4.Thmtheorem1.p1.12.1.m1.2.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.2.2">𝐛</ci></interval></apply><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1"><abs id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.2.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.2"></abs><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1"><minus id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.3"></minus><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4"><times id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.1"></times><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2"><csymbol cd="ambiguous" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2">superscript</csymbol><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.2">𝐚</ci><csymbol cd="latexml" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.2.3">top</csymbol></apply><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.4.3">𝐛</ci></apply><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2"><times id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.3"></times><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.4.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.4">𝛼</ci><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.5.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.5">𝑓</ci><interval closure="open" id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2"><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1"><times id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1"></times><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2">𝑔</ci><ci id="S4.Thmtheorem1.p1.12.1.m1.3.3.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.3.3">𝐚</ci></apply><apply id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2"><times id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1"></times><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2">ℎ</ci><ci id="S4.Thmtheorem1.p1.12.1.m1.4.4.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.4.4">𝐛</ci></apply></interval></apply><ci id="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.5.cmml" xref="S4.Thmtheorem1.p1.12.1.m1.5.5.1.1.1.5">𝛃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.12.1.m1.5c">\mathcal{L}(\bm{a},\bm{b})\triangleq|\bm{a}^{\top}\bm{b}-\alpha f(g(\bm{a}),h(\bm{b}))-\bm{\beta}|</annotation></semantics></math>, <math id="S4.Thmtheorem1.p1.13.2.m2.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="S4.Thmtheorem1.p1.13.2.m2.1a"><mi id="S4.Thmtheorem1.p1.13.2.m2.1.1" xref="S4.Thmtheorem1.p1.13.2.m2.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.13.2.m2.1b"><ci id="S4.Thmtheorem1.p1.13.2.m2.1.1.cmml" xref="S4.Thmtheorem1.p1.13.2.m2.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.13.2.m2.1c">\alpha</annotation></semantics></math> is the scale used for quantizing the lookup tables, <math id="S4.Thmtheorem1.p1.14.3.m3.1" class="ltx_Math" alttext="\bm{\beta}" display="inline"><semantics id="S4.Thmtheorem1.p1.14.3.m3.1a"><mi id="S4.Thmtheorem1.p1.14.3.m3.1.1" xref="S4.Thmtheorem1.p1.14.3.m3.1.1.cmml">𝛃</mi><annotation-xml encoding="MathML-Content" id="S4.Thmtheorem1.p1.14.3.m3.1b"><ci id="S4.Thmtheorem1.p1.14.3.m3.1.1.cmml" xref="S4.Thmtheorem1.p1.14.3.m3.1.1">𝛃</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmtheorem1.p1.14.3.m3.1c">\bm{\beta}</annotation></semantics></math> is the constants used in quantizing the lookup tables plus the debiasing constant of Section <a href="#S4.SS4" title="4.4 Fast 8-bit Aggregation, 𝒇⁢(⋅,⋅) ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>, and</span></p>
<table id="A6.EGx10" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E11"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E11.m1.5" class="ltx_Math" alttext="\displaystyle\vspace*{-2mm}\nu(C,D,\delta)\triangleq C(4\left\lceil\log_{2}(D)\right\rceil+256)\log{2}-\log{\delta}.\vspace*{-2mm}" display="inline"><semantics id="S4.E11.m1.5a"><mrow id="S4.E11.m1.5.5.1" xref="S4.E11.m1.5.5.1.1.cmml"><mrow id="S4.E11.m1.5.5.1.1" xref="S4.E11.m1.5.5.1.1.cmml"><mrow id="S4.E11.m1.5.5.1.1.3" xref="S4.E11.m1.5.5.1.1.3.cmml"><mi id="S4.E11.m1.5.5.1.1.3.2" xref="S4.E11.m1.5.5.1.1.3.2.cmml">ν</mi><mo lspace="0em" rspace="0em" id="S4.E11.m1.5.5.1.1.3.1" xref="S4.E11.m1.5.5.1.1.3.1.cmml">​</mo><mrow id="S4.E11.m1.5.5.1.1.3.3.2" xref="S4.E11.m1.5.5.1.1.3.3.1.cmml"><mo stretchy="false" id="S4.E11.m1.5.5.1.1.3.3.2.1" xref="S4.E11.m1.5.5.1.1.3.3.1.cmml">(</mo><mi id="S4.E11.m1.1.1" xref="S4.E11.m1.1.1.cmml">C</mi><mo id="S4.E11.m1.5.5.1.1.3.3.2.2" xref="S4.E11.m1.5.5.1.1.3.3.1.cmml">,</mo><mi id="S4.E11.m1.2.2" xref="S4.E11.m1.2.2.cmml">D</mi><mo id="S4.E11.m1.5.5.1.1.3.3.2.3" xref="S4.E11.m1.5.5.1.1.3.3.1.cmml">,</mo><mi id="S4.E11.m1.3.3" xref="S4.E11.m1.3.3.cmml">δ</mi><mo stretchy="false" id="S4.E11.m1.5.5.1.1.3.3.2.4" xref="S4.E11.m1.5.5.1.1.3.3.1.cmml">)</mo></mrow></mrow><mo id="S4.E11.m1.5.5.1.1.2" xref="S4.E11.m1.5.5.1.1.2.cmml">≜</mo><mrow id="S4.E11.m1.5.5.1.1.1" xref="S4.E11.m1.5.5.1.1.1.cmml"><mrow id="S4.E11.m1.5.5.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.cmml"><mi id="S4.E11.m1.5.5.1.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="S4.E11.m1.5.5.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="S4.E11.m1.5.5.1.1.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E11.m1.5.5.1.1.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E11.m1.5.5.1.1.1.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.cmml"><mrow id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml"><mo id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml">⌈</mo><mrow id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><msub id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">log</mi><mn id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1a" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml">(</mo><mi id="S4.E11.m1.4.4" xref="S4.E11.m1.4.4.cmml">D</mi><mo stretchy="false" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml">⌉</mo></mrow></mrow><mo id="S4.E11.m1.5.5.1.1.1.1.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.2.cmml">+</mo><mn id="S4.E11.m1.5.5.1.1.1.1.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.3.cmml">256</mn></mrow><mo stretchy="false" id="S4.E11.m1.5.5.1.1.1.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S4.E11.m1.5.5.1.1.1.1.2a" xref="S4.E11.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="S4.E11.m1.5.5.1.1.1.1.4" xref="S4.E11.m1.5.5.1.1.1.1.4.cmml"><mi id="S4.E11.m1.5.5.1.1.1.1.4.1" xref="S4.E11.m1.5.5.1.1.1.1.4.1.cmml">log</mi><mo lspace="0.167em" id="S4.E11.m1.5.5.1.1.1.1.4a" xref="S4.E11.m1.5.5.1.1.1.1.4.cmml">⁡</mo><mn id="S4.E11.m1.5.5.1.1.1.1.4.2" xref="S4.E11.m1.5.5.1.1.1.1.4.2.cmml">2</mn></mrow></mrow><mo id="S4.E11.m1.5.5.1.1.1.2" xref="S4.E11.m1.5.5.1.1.1.2.cmml">−</mo><mrow id="S4.E11.m1.5.5.1.1.1.3" xref="S4.E11.m1.5.5.1.1.1.3.cmml"><mi id="S4.E11.m1.5.5.1.1.1.3.1" xref="S4.E11.m1.5.5.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.E11.m1.5.5.1.1.1.3a" xref="S4.E11.m1.5.5.1.1.1.3.cmml">⁡</mo><mi id="S4.E11.m1.5.5.1.1.1.3.2" xref="S4.E11.m1.5.5.1.1.1.3.2.cmml">δ</mi></mrow></mrow></mrow><mo lspace="0em" id="S4.E11.m1.5.5.1.2" xref="S4.E11.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E11.m1.5b"><apply id="S4.E11.m1.5.5.1.1.cmml" xref="S4.E11.m1.5.5.1"><ci id="S4.E11.m1.5.5.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.2">≜</ci><apply id="S4.E11.m1.5.5.1.1.3.cmml" xref="S4.E11.m1.5.5.1.1.3"><times id="S4.E11.m1.5.5.1.1.3.1.cmml" xref="S4.E11.m1.5.5.1.1.3.1"></times><ci id="S4.E11.m1.5.5.1.1.3.2.cmml" xref="S4.E11.m1.5.5.1.1.3.2">𝜈</ci><vector id="S4.E11.m1.5.5.1.1.3.3.1.cmml" xref="S4.E11.m1.5.5.1.1.3.3.2"><ci id="S4.E11.m1.1.1.cmml" xref="S4.E11.m1.1.1">𝐶</ci><ci id="S4.E11.m1.2.2.cmml" xref="S4.E11.m1.2.2">𝐷</ci><ci id="S4.E11.m1.3.3.cmml" xref="S4.E11.m1.3.3">𝛿</ci></vector></apply><apply id="S4.E11.m1.5.5.1.1.1.cmml" xref="S4.E11.m1.5.5.1.1.1"><minus id="S4.E11.m1.5.5.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.2"></minus><apply id="S4.E11.m1.5.5.1.1.1.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1"><times id="S4.E11.m1.5.5.1.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.2"></times><ci id="S4.E11.m1.5.5.1.1.1.1.3.cmml" xref="S4.E11.m1.5.5.1.1.1.1.3">𝐶</ci><apply id="S4.E11.m1.5.5.1.1.1.1.1.1.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1"><plus id="S4.E11.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.2"></plus><apply id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1"><times id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.2"></times><cn type="integer" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.3">4</cn><apply id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1"><ceiling id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.2"></ceiling><apply id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1"><apply id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><log id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2"></log><cn type="integer" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3">2</cn></apply><ci id="S4.E11.m1.4.4.cmml" xref="S4.E11.m1.4.4">𝐷</ci></apply></apply></apply><cn type="integer" id="S4.E11.m1.5.5.1.1.1.1.1.1.1.3.cmml" xref="S4.E11.m1.5.5.1.1.1.1.1.1.1.3">256</cn></apply><apply id="S4.E11.m1.5.5.1.1.1.1.4.cmml" xref="S4.E11.m1.5.5.1.1.1.1.4"><log id="S4.E11.m1.5.5.1.1.1.1.4.1.cmml" xref="S4.E11.m1.5.5.1.1.1.1.4.1"></log><cn type="integer" id="S4.E11.m1.5.5.1.1.1.1.4.2.cmml" xref="S4.E11.m1.5.5.1.1.1.1.4.2">2</cn></apply></apply><apply id="S4.E11.m1.5.5.1.1.1.3.cmml" xref="S4.E11.m1.5.5.1.1.1.3"><log id="S4.E11.m1.5.5.1.1.1.3.1.cmml" xref="S4.E11.m1.5.5.1.1.1.3.1"></log><ci id="S4.E11.m1.5.5.1.1.1.3.2.cmml" xref="S4.E11.m1.5.5.1.1.1.3.2">𝛿</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E11.m1.5c">\displaystyle\vspace*{-2mm}\nu(C,D,\delta)\triangleq C(4\left\lceil\log_{2}(D)\right\rceil+256)\log{2}-\log{\delta}.\vspace*{-2mm}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
</div>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experiments</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.2" class="ltx_p">To assess <span id="S5.p1.2.1" class="ltx_text ltx_font_smallcaps">Maddness</span>’s effectiveness, we implemented both it and existing algorithms in C++ and Python. All of our code and raw numerical results are publicly available at <span id="S5.p1.2.2" class="ltx_text ltx_font_typewriter">https://smarturl.it/Maddness</span>. All experiments use a single thread on a Macbook Pro with a 2.6GHz Intel Core i7-4960HQ processor. Unless stated otherwise, all timing results use five trials, with each trial reporting the fastest among 20 executions. We use the best, rather than the average, since this is standard practice in performance benchmarking and is robust to the purely additive noise introduced by competing CPU tasks. Standard deviations are shown for all curves as shaded areas.
Since training can be performed offline and all methods except SparsePCA <cite class="ltx_cite ltx_citemacro_citep">(Mairal et al., <a href="#bib.bib52" title="" class="ltx_ref">2009</a>)</cite> train in at most a few minutes, we omit profiling of training times. We also do not profile the time to preprocess <math id="S5.p1.1.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S5.p1.1.m1.1a"><mi id="S5.p1.1.m1.1.1" xref="S5.p1.1.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S5.p1.1.m1.1b"><ci id="S5.p1.1.m1.1.1.cmml" xref="S5.p1.1.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.1.m1.1c">\bm{B}</annotation></semantics></math>, since 1) this time is inconsequential in most cases, and 2) <math id="S5.p1.2.m2.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S5.p1.2.m2.1a"><mi id="S5.p1.2.m2.1.1" xref="S5.p1.2.m2.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S5.p1.2.m2.1b"><ci id="S5.p1.2.m2.1.1.cmml" xref="S5.p1.2.m2.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.2.m2.1c">\bm{B}</annotation></semantics></math> is fixed and could be processed offline in all the problems we consider.
In order to avoid implementation bias, we build upon the source code provided by <cite class="ltx_cite ltx_citemacro_citet">Blalock &amp; Guttag (<a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>https://github.com/dblalock/bolt</span></span></span>, which includes highly tuned implementations of many algorithms to which we compare.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">We do not need to tune any hyperparameters for <span id="S5.p2.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>, but we do take steps to ensure that other methods are not hindered by insufficient hyperparameter tuning. Concretely, we sweep a wide range of hyperparameter values and allow them to cherry-pick their best hyperparameters on each test matrix.
Further details regarding our experimental setup and choices (e.g., use of a single thread) can be found in Appendix <a href="#A5" title="Appendix E Additional Experimental Details ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Methods Tested</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.4" class="ltx_p">Recall that most baselines take the form of selecting a matrix <math id="S5.SS1.p1.1.m1.2" class="ltx_Math" alttext="\bm{V}\in\mathbb{R}^{D\times d},d&lt;D" display="inline"><semantics id="S5.SS1.p1.1.m1.2a"><mrow id="S5.SS1.p1.1.m1.2.2.2" xref="S5.SS1.p1.1.m1.2.2.3.cmml"><mrow id="S5.SS1.p1.1.m1.1.1.1.1" xref="S5.SS1.p1.1.m1.1.1.1.1.cmml"><mi id="S5.SS1.p1.1.m1.1.1.1.1.2" xref="S5.SS1.p1.1.m1.1.1.1.1.2.cmml">𝑽</mi><mo id="S5.SS1.p1.1.m1.1.1.1.1.1" xref="S5.SS1.p1.1.m1.1.1.1.1.1.cmml">∈</mo><msup id="S5.SS1.p1.1.m1.1.1.1.1.3" xref="S5.SS1.p1.1.m1.1.1.1.1.3.cmml"><mi id="S5.SS1.p1.1.m1.1.1.1.1.3.2" xref="S5.SS1.p1.1.m1.1.1.1.1.3.2.cmml">ℝ</mi><mrow id="S5.SS1.p1.1.m1.1.1.1.1.3.3" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.cmml"><mi id="S5.SS1.p1.1.m1.1.1.1.1.3.3.2" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS1.p1.1.m1.1.1.1.1.3.3.1" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.1.cmml">×</mo><mi id="S5.SS1.p1.1.m1.1.1.1.1.3.3.3" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><mo id="S5.SS1.p1.1.m1.2.2.2.3" xref="S5.SS1.p1.1.m1.2.2.3a.cmml">,</mo><mrow id="S5.SS1.p1.1.m1.2.2.2.2" xref="S5.SS1.p1.1.m1.2.2.2.2.cmml"><mi id="S5.SS1.p1.1.m1.2.2.2.2.2" xref="S5.SS1.p1.1.m1.2.2.2.2.2.cmml">d</mi><mo id="S5.SS1.p1.1.m1.2.2.2.2.1" xref="S5.SS1.p1.1.m1.2.2.2.2.1.cmml">&lt;</mo><mi id="S5.SS1.p1.1.m1.2.2.2.2.3" xref="S5.SS1.p1.1.m1.2.2.2.2.3.cmml">D</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.1.m1.2b"><apply id="S5.SS1.p1.1.m1.2.2.3.cmml" xref="S5.SS1.p1.1.m1.2.2.2"><csymbol cd="ambiguous" id="S5.SS1.p1.1.m1.2.2.3a.cmml" xref="S5.SS1.p1.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S5.SS1.p1.1.m1.1.1.1.1.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1"><in id="S5.SS1.p1.1.m1.1.1.1.1.1.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.1"></in><ci id="S5.SS1.p1.1.m1.1.1.1.1.2.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.2">𝑽</ci><apply id="S5.SS1.p1.1.m1.1.1.1.1.3.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S5.SS1.p1.1.m1.1.1.1.1.3.1.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3">superscript</csymbol><ci id="S5.SS1.p1.1.m1.1.1.1.1.3.2.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3.2">ℝ</ci><apply id="S5.SS1.p1.1.m1.1.1.1.1.3.3.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3"><times id="S5.SS1.p1.1.m1.1.1.1.1.3.3.1.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.1"></times><ci id="S5.SS1.p1.1.m1.1.1.1.1.3.3.2.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.2">𝐷</ci><ci id="S5.SS1.p1.1.m1.1.1.1.1.3.3.3.cmml" xref="S5.SS1.p1.1.m1.1.1.1.1.3.3.3">𝑑</ci></apply></apply></apply><apply id="S5.SS1.p1.1.m1.2.2.2.2.cmml" xref="S5.SS1.p1.1.m1.2.2.2.2"><lt id="S5.SS1.p1.1.m1.2.2.2.2.1.cmml" xref="S5.SS1.p1.1.m1.2.2.2.2.1"></lt><ci id="S5.SS1.p1.1.m1.2.2.2.2.2.cmml" xref="S5.SS1.p1.1.m1.2.2.2.2.2">𝑑</ci><ci id="S5.SS1.p1.1.m1.2.2.2.2.3.cmml" xref="S5.SS1.p1.1.m1.2.2.2.2.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.1.m1.2c">\bm{V}\in\mathbb{R}^{D\times d},d&lt;D</annotation></semantics></math> such that <math id="S5.SS1.p1.2.m2.2" class="ltx_Math" alttext="\bm{A}\bm{B}\approx(\bm{A}\bm{V})(\bm{V}^{\top}\bm{B})" display="inline"><semantics id="S5.SS1.p1.2.m2.2a"><mrow id="S5.SS1.p1.2.m2.2.2" xref="S5.SS1.p1.2.m2.2.2.cmml"><mrow id="S5.SS1.p1.2.m2.2.2.4" xref="S5.SS1.p1.2.m2.2.2.4.cmml"><mi id="S5.SS1.p1.2.m2.2.2.4.2" xref="S5.SS1.p1.2.m2.2.2.4.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.2.m2.2.2.4.1" xref="S5.SS1.p1.2.m2.2.2.4.1.cmml">​</mo><mi id="S5.SS1.p1.2.m2.2.2.4.3" xref="S5.SS1.p1.2.m2.2.2.4.3.cmml">𝑩</mi></mrow><mo id="S5.SS1.p1.2.m2.2.2.3" xref="S5.SS1.p1.2.m2.2.2.3.cmml">≈</mo><mrow id="S5.SS1.p1.2.m2.2.2.2" xref="S5.SS1.p1.2.m2.2.2.2.cmml"><mrow id="S5.SS1.p1.2.m2.1.1.1.1.1" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS1.p1.2.m2.1.1.1.1.1.2" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS1.p1.2.m2.1.1.1.1.1.1" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml"><mi id="S5.SS1.p1.2.m2.1.1.1.1.1.1.2" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.2.m2.1.1.1.1.1.1.1" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.2.m2.1.1.1.1.1.1.3" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.3.cmml">𝑽</mi></mrow><mo stretchy="false" id="S5.SS1.p1.2.m2.1.1.1.1.1.3" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S5.SS1.p1.2.m2.2.2.2.3" xref="S5.SS1.p1.2.m2.2.2.2.3.cmml">​</mo><mrow id="S5.SS1.p1.2.m2.2.2.2.2.1" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.cmml"><mo stretchy="false" id="S5.SS1.p1.2.m2.2.2.2.2.1.2" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.cmml">(</mo><mrow id="S5.SS1.p1.2.m2.2.2.2.2.1.1" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.cmml"><msup id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.cmml"><mi id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.2" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.2.cmml">𝑽</mi><mo id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.3" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S5.SS1.p1.2.m2.2.2.2.2.1.1.1" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.2.m2.2.2.2.2.1.1.3" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.3.cmml">𝑩</mi></mrow><mo stretchy="false" id="S5.SS1.p1.2.m2.2.2.2.2.1.3" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.2.m2.2b"><apply id="S5.SS1.p1.2.m2.2.2.cmml" xref="S5.SS1.p1.2.m2.2.2"><approx id="S5.SS1.p1.2.m2.2.2.3.cmml" xref="S5.SS1.p1.2.m2.2.2.3"></approx><apply id="S5.SS1.p1.2.m2.2.2.4.cmml" xref="S5.SS1.p1.2.m2.2.2.4"><times id="S5.SS1.p1.2.m2.2.2.4.1.cmml" xref="S5.SS1.p1.2.m2.2.2.4.1"></times><ci id="S5.SS1.p1.2.m2.2.2.4.2.cmml" xref="S5.SS1.p1.2.m2.2.2.4.2">𝑨</ci><ci id="S5.SS1.p1.2.m2.2.2.4.3.cmml" xref="S5.SS1.p1.2.m2.2.2.4.3">𝑩</ci></apply><apply id="S5.SS1.p1.2.m2.2.2.2.cmml" xref="S5.SS1.p1.2.m2.2.2.2"><times id="S5.SS1.p1.2.m2.2.2.2.3.cmml" xref="S5.SS1.p1.2.m2.2.2.2.3"></times><apply id="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1"><times id="S5.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.1"></times><ci id="S5.SS1.p1.2.m2.1.1.1.1.1.1.2.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.2">𝑨</ci><ci id="S5.SS1.p1.2.m2.1.1.1.1.1.1.3.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.3">𝑽</ci></apply><apply id="S5.SS1.p1.2.m2.2.2.2.2.1.1.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1"><times id="S5.SS1.p1.2.m2.2.2.2.2.1.1.1.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.1"></times><apply id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2"><csymbol cd="ambiguous" id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.1.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2">superscript</csymbol><ci id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.2.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.2">𝑽</ci><csymbol cd="latexml" id="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.3.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.2.3">top</csymbol></apply><ci id="S5.SS1.p1.2.m2.2.2.2.2.1.1.3.cmml" xref="S5.SS1.p1.2.m2.2.2.2.2.1.1.3">𝑩</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.2.m2.2c">\bm{A}\bm{B}\approx(\bm{A}\bm{V})(\bm{V}^{\top}\bm{B})</annotation></semantics></math>. Here <math id="S5.SS1.p1.3.m3.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S5.SS1.p1.3.m3.1a"><mi id="S5.SS1.p1.3.m3.1.1" xref="S5.SS1.p1.3.m3.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.3.m3.1b"><ci id="S5.SS1.p1.3.m3.1.1.cmml" xref="S5.SS1.p1.3.m3.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.3.m3.1c">d</annotation></semantics></math> is a free parameter that adjusts the quality versus speed tradeoff. We therefore characterize most of these methods by how they set <math id="S5.SS1.p1.4.m4.1" class="ltx_Math" alttext="\bm{V}" display="inline"><semantics id="S5.SS1.p1.4.m4.1a"><mi id="S5.SS1.p1.4.m4.1.1" xref="S5.SS1.p1.4.m4.1.1.cmml">𝑽</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.4.m4.1b"><ci id="S5.SS1.p1.4.m4.1.1.cmml" xref="S5.SS1.p1.4.m4.1.1">𝑽</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.4.m4.1c">\bm{V}</annotation></semantics></math>.</p>
<ul id="S5.I1" class="ltx_itemize">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.2" class="ltx_p"><span id="S5.I1.i1.p1.2.1" class="ltx_text ltx_font_bold">PCA</span>. Set <math id="S5.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\bm{V}" display="inline"><semantics id="S5.I1.i1.p1.1.m1.1a"><mi id="S5.I1.i1.p1.1.m1.1.1" xref="S5.I1.i1.p1.1.m1.1.1.cmml">𝑽</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i1.p1.1.m1.1b"><ci id="S5.I1.i1.p1.1.m1.1.1.cmml" xref="S5.I1.i1.p1.1.m1.1.1">𝑽</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i1.p1.1.m1.1c">\bm{V}</annotation></semantics></math> equal to the top principal components of <math id="S5.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S5.I1.i1.p1.2.m2.1a"><mover accent="true" id="S5.I1.i1.p1.2.m2.1.1" xref="S5.I1.i1.p1.2.m2.1.1.cmml"><mi id="S5.I1.i1.p1.2.m2.1.1.2" xref="S5.I1.i1.p1.2.m2.1.1.2.cmml">𝑨</mi><mo id="S5.I1.i1.p1.2.m2.1.1.1" xref="S5.I1.i1.p1.2.m2.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S5.I1.i1.p1.2.m2.1b"><apply id="S5.I1.i1.p1.2.m2.1.1.cmml" xref="S5.I1.i1.p1.2.m2.1.1"><ci id="S5.I1.i1.p1.2.m2.1.1.1.cmml" xref="S5.I1.i1.p1.2.m2.1.1.1">~</ci><ci id="S5.I1.i1.p1.2.m2.1.1.2.cmml" xref="S5.I1.i1.p1.2.m2.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i1.p1.2.m2.1c">\tilde{\bm{A}}</annotation></semantics></math>.</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.2" class="ltx_p"><span id="S5.I1.i2.p1.2.1" class="ltx_text ltx_font_bold">SparsePCA</span> <cite class="ltx_cite ltx_citemacro_citep">(Mairal et al., <a href="#bib.bib52" title="" class="ltx_ref">2009</a>)</cite>. Set <math id="S5.I1.i2.p1.1.m1.2" class="ltx_Math" alttext="\bm{V}=\operatorname*{argmin}_{\bm{V}}\min_{\bm{U}}\frac{1}{2N_{train}}\lVert\tilde{\bm{A}}-\bm{U}\bm{V}^{\top}\rVert^{2}_{F}+\lambda\lVert\bm{V}\rVert_{1}" display="inline"><semantics id="S5.I1.i2.p1.1.m1.2a"><mrow id="S5.I1.i2.p1.1.m1.2.2" xref="S5.I1.i2.p1.1.m1.2.2.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.3" xref="S5.I1.i2.p1.1.m1.2.2.3.cmml">𝑽</mi><mo rspace="0.1389em" id="S5.I1.i2.p1.1.m1.2.2.2" xref="S5.I1.i2.p1.1.m1.2.2.2.cmml">=</mo><mrow id="S5.I1.i2.p1.1.m1.2.2.1" xref="S5.I1.i2.p1.1.m1.2.2.1.cmml"><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.cmml"><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.cmml"><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.cmml"><msub id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.cmml"><mo lspace="0.1389em" rspace="0.167em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.2.cmml">argmin</mo><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.3.cmml">𝑽</mi></msub><msub id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.2.cmml">min</mi><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.3.cmml">𝑼</mi></msub></mrow><mo lspace="0.167em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3a" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.cmml">⁡</mo><mfrac id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.cmml"><mn id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.2.cmml">1</mn><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.cmml"><mn id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.1.cmml">​</mo><msub id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.2.cmml">N</mi><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1.cmml">​</mo><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1a" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1.cmml">​</mo><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.4" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1b" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1.cmml">​</mo><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.5" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1c" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1.cmml">​</mo><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.6" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.6.cmml">n</mi></mrow></msub></mrow></mfrac></mrow><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.2.cmml">​</mo><msubsup id="S5.I1.i2.p1.1.m1.2.2.1.1.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.cmml"><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.2.1.cmml">∥</mo><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.cmml"><mover accent="true" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.2.cmml">𝑨</mi><mo id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.1.cmml">~</mo></mover><mo id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.2.cmml">𝑼</mi><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.1" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.1.cmml">​</mo><msup id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2.cmml">𝑽</mi><mo id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.cmml">⊤</mo></msup></mrow></mrow><mo fence="true" lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.2.1.cmml">∥</mo></mrow><mi id="S5.I1.i2.p1.1.m1.2.2.1.1.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.3.cmml">F</mi><mn id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.3.cmml">2</mn></msubsup></mrow><mo id="S5.I1.i2.p1.1.m1.2.2.1.2" xref="S5.I1.i2.p1.1.m1.2.2.1.2.cmml">+</mo><mrow id="S5.I1.i2.p1.1.m1.2.2.1.3" xref="S5.I1.i2.p1.1.m1.2.2.1.3.cmml"><mi id="S5.I1.i2.p1.1.m1.2.2.1.3.2" xref="S5.I1.i2.p1.1.m1.2.2.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.3.1" xref="S5.I1.i2.p1.1.m1.2.2.1.3.1.cmml">​</mo><msub id="S5.I1.i2.p1.1.m1.2.2.1.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.cmml"><mrow id="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.2" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.1.cmml"><mo fence="true" rspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.2.1" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.1.1.cmml">∥</mo><mi id="S5.I1.i2.p1.1.m1.1.1" xref="S5.I1.i2.p1.1.m1.1.1.cmml">𝑽</mi><mo fence="true" lspace="0em" id="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.2.2" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.1.1.cmml">∥</mo></mrow><mn id="S5.I1.i2.p1.1.m1.2.2.1.3.3.3" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.3.cmml">1</mn></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.I1.i2.p1.1.m1.2b"><apply id="S5.I1.i2.p1.1.m1.2.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2"><eq id="S5.I1.i2.p1.1.m1.2.2.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.2"></eq><ci id="S5.I1.i2.p1.1.m1.2.2.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.3">𝑽</ci><apply id="S5.I1.i2.p1.1.m1.2.2.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1"><plus id="S5.I1.i2.p1.1.m1.2.2.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.2"></plus><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1"><times id="S5.I1.i2.p1.1.m1.2.2.1.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.2"></times><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3"><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1"><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1">subscript</csymbol><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.2">argmin</ci><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.1.3">𝑽</ci></apply><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2">subscript</csymbol><min id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.2"></min><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.1.2.3">𝑼</ci></apply></apply><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2"><divide id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2"></divide><cn type="integer" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.2">1</cn><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3"><times id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.1"></times><cn type="integer" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.2">2</cn><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3">subscript</csymbol><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.2">𝑁</ci><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3"><times id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.1"></times><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.2">𝑡</ci><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.3">𝑟</ci><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.4.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.4">𝑎</ci><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.5.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.5">𝑖</ci><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.6.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.3.2.3.3.3.6">𝑛</ci></apply></apply></apply></apply></apply><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1">subscript</csymbol><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1">superscript</csymbol><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1"><csymbol cd="latexml" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.2">delimited-∥∥</csymbol><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1"><minus id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.1"></minus><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2"><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.1">~</ci><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.2.2">𝑨</ci></apply><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3"><times id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.1"></times><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.2">𝑼</ci><apply id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3">superscript</csymbol><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.2">𝑽</ci><csymbol cd="latexml" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.1.1.1.3.3.3">top</csymbol></apply></apply></apply></apply><cn type="integer" id="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.1.3">2</cn></apply><ci id="S5.I1.i2.p1.1.m1.2.2.1.1.1.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.1.1.3">𝐹</ci></apply></apply><apply id="S5.I1.i2.p1.1.m1.2.2.1.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3"><times id="S5.I1.i2.p1.1.m1.2.2.1.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.1"></times><ci id="S5.I1.i2.p1.1.m1.2.2.1.3.2.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.2">𝜆</ci><apply id="S5.I1.i2.p1.1.m1.2.2.1.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3"><csymbol cd="ambiguous" id="S5.I1.i2.p1.1.m1.2.2.1.3.3.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3">subscript</csymbol><apply id="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.2"><csymbol cd="latexml" id="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.1.1.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.2.2.1">delimited-∥∥</csymbol><ci id="S5.I1.i2.p1.1.m1.1.1.cmml" xref="S5.I1.i2.p1.1.m1.1.1">𝑽</ci></apply><cn type="integer" id="S5.I1.i2.p1.1.m1.2.2.1.3.3.3.cmml" xref="S5.I1.i2.p1.1.m1.2.2.1.3.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i2.p1.1.m1.2c">\bm{V}=\operatorname*{argmin}_{\bm{V}}\min_{\bm{U}}\frac{1}{2N_{train}}\lVert\tilde{\bm{A}}-\bm{U}\bm{V}^{\top}\rVert^{2}_{F}+\lambda\lVert\bm{V}\rVert_{1}</annotation></semantics></math>, where <math id="S5.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="\bm{U}^{\top}\bm{U}=\bm{I}" display="inline"><semantics id="S5.I1.i2.p1.2.m2.1a"><mrow id="S5.I1.i2.p1.2.m2.1.1" xref="S5.I1.i2.p1.2.m2.1.1.cmml"><mrow id="S5.I1.i2.p1.2.m2.1.1.2" xref="S5.I1.i2.p1.2.m2.1.1.2.cmml"><msup id="S5.I1.i2.p1.2.m2.1.1.2.2" xref="S5.I1.i2.p1.2.m2.1.1.2.2.cmml"><mi id="S5.I1.i2.p1.2.m2.1.1.2.2.2" xref="S5.I1.i2.p1.2.m2.1.1.2.2.2.cmml">𝑼</mi><mo id="S5.I1.i2.p1.2.m2.1.1.2.2.3" xref="S5.I1.i2.p1.2.m2.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S5.I1.i2.p1.2.m2.1.1.2.1" xref="S5.I1.i2.p1.2.m2.1.1.2.1.cmml">​</mo><mi id="S5.I1.i2.p1.2.m2.1.1.2.3" xref="S5.I1.i2.p1.2.m2.1.1.2.3.cmml">𝑼</mi></mrow><mo id="S5.I1.i2.p1.2.m2.1.1.1" xref="S5.I1.i2.p1.2.m2.1.1.1.cmml">=</mo><mi id="S5.I1.i2.p1.2.m2.1.1.3" xref="S5.I1.i2.p1.2.m2.1.1.3.cmml">𝑰</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.I1.i2.p1.2.m2.1b"><apply id="S5.I1.i2.p1.2.m2.1.1.cmml" xref="S5.I1.i2.p1.2.m2.1.1"><eq id="S5.I1.i2.p1.2.m2.1.1.1.cmml" xref="S5.I1.i2.p1.2.m2.1.1.1"></eq><apply id="S5.I1.i2.p1.2.m2.1.1.2.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2"><times id="S5.I1.i2.p1.2.m2.1.1.2.1.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2.1"></times><apply id="S5.I1.i2.p1.2.m2.1.1.2.2.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2.2"><csymbol cd="ambiguous" id="S5.I1.i2.p1.2.m2.1.1.2.2.1.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2.2">superscript</csymbol><ci id="S5.I1.i2.p1.2.m2.1.1.2.2.2.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2.2.2">𝑼</ci><csymbol cd="latexml" id="S5.I1.i2.p1.2.m2.1.1.2.2.3.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2.2.3">top</csymbol></apply><ci id="S5.I1.i2.p1.2.m2.1.1.2.3.cmml" xref="S5.I1.i2.p1.2.m2.1.1.2.3">𝑼</ci></apply><ci id="S5.I1.i2.p1.2.m2.1.1.3.cmml" xref="S5.I1.i2.p1.2.m2.1.1.3">𝑰</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i2.p1.2.m2.1c">\bm{U}^{\top}\bm{U}=\bm{I}</annotation></semantics></math>. This is not the only dictionary learning formulation referred to as SparsePCA <cite class="ltx_cite ltx_citemacro_citep">(Zou &amp; Xue, <a href="#bib.bib75" title="" class="ltx_ref">2018</a>; Camacho et al., <a href="#bib.bib13" title="" class="ltx_ref">2020</a>)</cite>, but it is a good representative and is the only one with support in a major Python library.</p>
</div>
</li>
<li id="S5.I1.i3" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i3.p1" class="ltx_para">
<p id="S5.I1.i3.p1.1" class="ltx_p"><span id="S5.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">FastJL</span> <cite class="ltx_cite ltx_citemacro_citep">(Ailon &amp; Chazelle, <a href="#bib.bib3" title="" class="ltx_ref">2009</a>)</cite>. <math id="S5.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="\bm{V}" display="inline"><semantics id="S5.I1.i3.p1.1.m1.1a"><mi id="S5.I1.i3.p1.1.m1.1.1" xref="S5.I1.i3.p1.1.m1.1.1.cmml">𝑽</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i3.p1.1.m1.1b"><ci id="S5.I1.i3.p1.1.m1.1.1.cmml" xref="S5.I1.i3.p1.1.m1.1.1">𝑽</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i3.p1.1.m1.1c">\bm{V}</annotation></semantics></math> is set to Rademacher random variables composed with a Fast Hadamard Transform (FHT). For simplicity, we exclude the FHT in the timing.</p>
</div>
</li>
<li id="S5.I1.i4" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i4.p1" class="ltx_para">
<p id="S5.I1.i4.p1.2" class="ltx_p"><span id="S5.I1.i4.p1.2.1" class="ltx_text ltx_font_bold">HashJL</span> <cite class="ltx_cite ltx_citemacro_citep">(Dasgupta et al., <a href="#bib.bib17" title="" class="ltx_ref">2010</a>)</cite>. <math id="S5.I1.i4.p1.1.m1.1" class="ltx_Math" alttext="\bm{V}" display="inline"><semantics id="S5.I1.i4.p1.1.m1.1a"><mi id="S5.I1.i4.p1.1.m1.1.1" xref="S5.I1.i4.p1.1.m1.1.1.cmml">𝑽</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i4.p1.1.m1.1b"><ci id="S5.I1.i4.p1.1.m1.1.1.cmml" xref="S5.I1.i4.p1.1.m1.1.1">𝑽</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i4.p1.1.m1.1c">\bm{V}</annotation></semantics></math> is zero except for a <math id="S5.I1.i4.p1.2.m2.1" class="ltx_Math" alttext="\pm 1" display="inline"><semantics id="S5.I1.i4.p1.2.m2.1a"><mrow id="S5.I1.i4.p1.2.m2.1.1" xref="S5.I1.i4.p1.2.m2.1.1.cmml"><mo id="S5.I1.i4.p1.2.m2.1.1a" xref="S5.I1.i4.p1.2.m2.1.1.cmml">±</mo><mn id="S5.I1.i4.p1.2.m2.1.1.2" xref="S5.I1.i4.p1.2.m2.1.1.2.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.I1.i4.p1.2.m2.1b"><apply id="S5.I1.i4.p1.2.m2.1.1.cmml" xref="S5.I1.i4.p1.2.m2.1.1"><csymbol cd="latexml" id="S5.I1.i4.p1.2.m2.1.1.1.cmml" xref="S5.I1.i4.p1.2.m2.1.1">plus-or-minus</csymbol><cn type="integer" id="S5.I1.i4.p1.2.m2.1.1.2.cmml" xref="S5.I1.i4.p1.2.m2.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i4.p1.2.m2.1c">\pm 1</annotation></semantics></math> in each row, with both sign and position chosen uniformly at random.
</p>
</div>
</li>
<li id="S5.I1.i5" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i5.p1" class="ltx_para">
<p id="S5.I1.i5.p1.1" class="ltx_p"><span id="S5.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">ScalarQuantize</span>. The matrices are not projected, but instead linearly quantized to eight bits such that the smallest and largest entries map to
either 0 and 255 or -128 and 127, as appropriate.
We use FBGEMM <cite class="ltx_cite ltx_citemacro_citep">(Khudia et al., <a href="#bib.bib45" title="" class="ltx_ref">2018</a>)</cite> to perform the quantized matrix multiplies. We neglect the time required to convert from other formats to eight bits, reflecting the optimistic scenario in which the matrices are already of the appropriate types.
</p>
</div>
</li>
<li id="S5.I1.i6" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i6.p1" class="ltx_para">
<p id="S5.I1.i6.p1.1" class="ltx_p"><span id="S5.I1.i6.p1.1.1" class="ltx_text ltx_font_bold">Bolt</span> <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>. Bolt
is the most similar method to our own, differing only in the encoding function, the use of averaging instead of upcasting, and the optimization of centroids. </p>
</div>
</li>
<li id="S5.I1.i7" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i7.p1" class="ltx_para">
<p id="S5.I1.i7.p1.1" class="ltx_p"><span id="S5.I1.i7.p1.1.1" class="ltx_text ltx_font_bold">Exact Multiplication</span>. We simply compute the matrix product <math id="S5.I1.i7.p1.1.m1.1" class="ltx_Math" alttext="\bm{A}\bm{B}" display="inline"><semantics id="S5.I1.i7.p1.1.m1.1a"><mrow id="S5.I1.i7.p1.1.m1.1.1" xref="S5.I1.i7.p1.1.m1.1.1.cmml"><mi id="S5.I1.i7.p1.1.m1.1.1.2" xref="S5.I1.i7.p1.1.m1.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S5.I1.i7.p1.1.m1.1.1.1" xref="S5.I1.i7.p1.1.m1.1.1.1.cmml">​</mo><mi id="S5.I1.i7.p1.1.m1.1.1.3" xref="S5.I1.i7.p1.1.m1.1.1.3.cmml">𝑩</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.I1.i7.p1.1.m1.1b"><apply id="S5.I1.i7.p1.1.m1.1.1.cmml" xref="S5.I1.i7.p1.1.m1.1.1"><times id="S5.I1.i7.p1.1.m1.1.1.1.cmml" xref="S5.I1.i7.p1.1.m1.1.1.1"></times><ci id="S5.I1.i7.p1.1.m1.1.1.2.cmml" xref="S5.I1.i7.p1.1.m1.1.1.2">𝑨</ci><ci id="S5.I1.i7.p1.1.m1.1.1.3.cmml" xref="S5.I1.i7.p1.1.m1.1.1.3">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i7.p1.1.m1.1c">\bm{A}\bm{B}</annotation></semantics></math> using a modern BLAS implementation.</p>
</div>
</li>
<li id="S5.I1.i8" class="ltx_item" style="list-style-type:none;padding-top:2.3pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i8.p1" class="ltx_para">
<p id="S5.I1.i8.p1.1" class="ltx_p"><span id="S5.I1.i8.p1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Maddness<span id="S5.I1.i8.p1.1.1.1" class="ltx_text ltx_font_upright">-PQ</span></span>. A handicapped version of <span id="S5.I1.i8.p1.1.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.I1.i8.p1.1.2.1" class="ltx_text ltx_font_upright"> </span></span>without the prototype optimization step. The gap between <span id="S5.I1.i8.p1.1.3" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.I1.i8.p1.1.3.1" class="ltx_text ltx_font_upright"> </span></span>and <span id="S5.I1.i8.p1.1.4" class="ltx_text ltx_font_smallcaps">Maddness</span>-PQ is the gain from optimizing the prototypes.</p>
</div>
</li>
</ul>
<p id="S5.SS1.p1.5" class="ltx_p">We also compared to many additional methods (see Appendix <a href="#A5" title="Appendix E Additional Experimental Details ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>), but omit their results since they were not competitive with those listed here.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>How Fast is <span id="S5.SS2.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>?</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.5" class="ltx_p">We begin by profiling the raw speed of our method. In Figure <a href="#S5.F3" title="Figure 3 ‣ 5.2 How Fast is Maddness? ‣ 5 Experiments ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we time the <math id="S5.SS2.p1.1.m1.1" class="ltx_Math" alttext="g(\bm{A})" display="inline"><semantics id="S5.SS2.p1.1.m1.1a"><mrow id="S5.SS2.p1.1.m1.1.2" xref="S5.SS2.p1.1.m1.1.2.cmml"><mi id="S5.SS2.p1.1.m1.1.2.2" xref="S5.SS2.p1.1.m1.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S5.SS2.p1.1.m1.1.2.1" xref="S5.SS2.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S5.SS2.p1.1.m1.1.2.3.2" xref="S5.SS2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S5.SS2.p1.1.m1.1.2.3.2.1" xref="S5.SS2.p1.1.m1.1.2.cmml">(</mo><mi id="S5.SS2.p1.1.m1.1.1" xref="S5.SS2.p1.1.m1.1.1.cmml">𝑨</mi><mo stretchy="false" id="S5.SS2.p1.1.m1.1.2.3.2.2" xref="S5.SS2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.1.m1.1b"><apply id="S5.SS2.p1.1.m1.1.2.cmml" xref="S5.SS2.p1.1.m1.1.2"><times id="S5.SS2.p1.1.m1.1.2.1.cmml" xref="S5.SS2.p1.1.m1.1.2.1"></times><ci id="S5.SS2.p1.1.m1.1.2.2.cmml" xref="S5.SS2.p1.1.m1.1.2.2">𝑔</ci><ci id="S5.SS2.p1.1.m1.1.1.cmml" xref="S5.SS2.p1.1.m1.1.1">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.1.m1.1c">g(\bm{A})</annotation></semantics></math> functions for various vector quantization methods. The <math id="S5.SS2.p1.2.m2.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S5.SS2.p1.2.m2.1a"><mi id="S5.SS2.p1.2.m2.1.1" xref="S5.SS2.p1.2.m2.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.2.m2.1b"><ci id="S5.SS2.p1.2.m2.1.1.cmml" xref="S5.SS2.p1.2.m2.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.2.m2.1c">\bm{A}</annotation></semantics></math> matrices have <math id="S5.SS2.p1.3.m3.1" class="ltx_Math" alttext="2^{14}" display="inline"><semantics id="S5.SS2.p1.3.m3.1a"><msup id="S5.SS2.p1.3.m3.1.1" xref="S5.SS2.p1.3.m3.1.1.cmml"><mn id="S5.SS2.p1.3.m3.1.1.2" xref="S5.SS2.p1.3.m3.1.1.2.cmml">2</mn><mn id="S5.SS2.p1.3.m3.1.1.3" xref="S5.SS2.p1.3.m3.1.1.3.cmml">14</mn></msup><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.3.m3.1b"><apply id="S5.SS2.p1.3.m3.1.1.cmml" xref="S5.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS2.p1.3.m3.1.1.1.cmml" xref="S5.SS2.p1.3.m3.1.1">superscript</csymbol><cn type="integer" id="S5.SS2.p1.3.m3.1.1.2.cmml" xref="S5.SS2.p1.3.m3.1.1.2">2</cn><cn type="integer" id="S5.SS2.p1.3.m3.1.1.3.cmml" xref="S5.SS2.p1.3.m3.1.1.3">14</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.3.m3.1c">2^{14}</annotation></semantics></math> rows and varying numbers of columns <math id="S5.SS2.p1.4.m4.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S5.SS2.p1.4.m4.1a"><mi id="S5.SS2.p1.4.m4.1.1" xref="S5.SS2.p1.4.m4.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.4.m4.1b"><ci id="S5.SS2.p1.4.m4.1.1.cmml" xref="S5.SS2.p1.4.m4.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.4.m4.1c">D</annotation></semantics></math>. Following <cite class="ltx_cite ltx_citemacro_citet">Blalock &amp; Guttag (<a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>, we also vary the number of codebooks <math id="S5.SS2.p1.5.m5.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S5.SS2.p1.5.m5.1a"><mi id="S5.SS2.p1.5.m5.1.1" xref="S5.SS2.p1.5.m5.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.5.m5.1b"><ci id="S5.SS2.p1.5.m5.1.1.cmml" xref="S5.SS2.p1.5.m5.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.5.m5.1c">C</annotation></semantics></math>, profiling 8-, 16-, and 32-byte encodings. We measure in bytes rather than codebooks since PQ and OPQ use eight bits per codebook while Bolt and <span id="S5.SS2.p1.5.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.SS2.p1.5.1.1" class="ltx_text ltx_font_upright"> </span></span>use four.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.2" class="ltx_p"><span id="S5.SS2.p2.2.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.SS2.p2.2.1.1" class="ltx_text ltx_font_upright"> </span></span>is up to two orders of magnitude faster than existing methods, and its throughput increases with row length. This latter property is because its encoding cost per row is <math id="S5.SS2.p2.1.m1.1" class="ltx_Math" alttext="O(C)" display="inline"><semantics id="S5.SS2.p2.1.m1.1a"><mrow id="S5.SS2.p2.1.m1.1.2" xref="S5.SS2.p2.1.m1.1.2.cmml"><mi id="S5.SS2.p2.1.m1.1.2.2" xref="S5.SS2.p2.1.m1.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S5.SS2.p2.1.m1.1.2.1" xref="S5.SS2.p2.1.m1.1.2.1.cmml">​</mo><mrow id="S5.SS2.p2.1.m1.1.2.3.2" xref="S5.SS2.p2.1.m1.1.2.cmml"><mo stretchy="false" id="S5.SS2.p2.1.m1.1.2.3.2.1" xref="S5.SS2.p2.1.m1.1.2.cmml">(</mo><mi id="S5.SS2.p2.1.m1.1.1" xref="S5.SS2.p2.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="S5.SS2.p2.1.m1.1.2.3.2.2" xref="S5.SS2.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.1.m1.1b"><apply id="S5.SS2.p2.1.m1.1.2.cmml" xref="S5.SS2.p2.1.m1.1.2"><times id="S5.SS2.p2.1.m1.1.2.1.cmml" xref="S5.SS2.p2.1.m1.1.2.1"></times><ci id="S5.SS2.p2.1.m1.1.2.2.cmml" xref="S5.SS2.p2.1.m1.1.2.2">𝑂</ci><ci id="S5.SS2.p2.1.m1.1.1.cmml" xref="S5.SS2.p2.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.1.m1.1c">O(C)</annotation></semantics></math> rather than <math id="S5.SS2.p2.2.m2.1" class="ltx_Math" alttext="O(D)" display="inline"><semantics id="S5.SS2.p2.2.m2.1a"><mrow id="S5.SS2.p2.2.m2.1.2" xref="S5.SS2.p2.2.m2.1.2.cmml"><mi id="S5.SS2.p2.2.m2.1.2.2" xref="S5.SS2.p2.2.m2.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S5.SS2.p2.2.m2.1.2.1" xref="S5.SS2.p2.2.m2.1.2.1.cmml">​</mo><mrow id="S5.SS2.p2.2.m2.1.2.3.2" xref="S5.SS2.p2.2.m2.1.2.cmml"><mo stretchy="false" id="S5.SS2.p2.2.m2.1.2.3.2.1" xref="S5.SS2.p2.2.m2.1.2.cmml">(</mo><mi id="S5.SS2.p2.2.m2.1.1" xref="S5.SS2.p2.2.m2.1.1.cmml">D</mi><mo stretchy="false" id="S5.SS2.p2.2.m2.1.2.3.2.2" xref="S5.SS2.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.2.m2.1b"><apply id="S5.SS2.p2.2.m2.1.2.cmml" xref="S5.SS2.p2.2.m2.1.2"><times id="S5.SS2.p2.2.m2.1.2.1.cmml" xref="S5.SS2.p2.2.m2.1.2.1"></times><ci id="S5.SS2.p2.2.m2.1.2.2.cmml" xref="S5.SS2.p2.2.m2.1.2.2">𝑂</ci><ci id="S5.SS2.p2.2.m2.1.1.cmml" xref="S5.SS2.p2.2.m2.1.1">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.2.m2.1c">O(D)</annotation></semantics></math>.</p>
</div>
<figure id="S5.F3" class="ltx_figure"><img src="/html/2106.10860/assets/x3.png" id="S5.F3.1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="512" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span><span id="S5.F3.5.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.F3.5.1.1" class="ltx_text ltx_font_upright"> </span></span>encodes the <math id="S5.F3.3.2.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S5.F3.3.2.m1.1b"><mi id="S5.F3.3.2.m1.1.1" xref="S5.F3.3.2.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S5.F3.3.2.m1.1c"><ci id="S5.F3.3.2.m1.1.1.cmml" xref="S5.F3.3.2.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F3.3.2.m1.1d">\bm{A}</annotation></semantics></math> matrix orders of magnitude more quickly than existing vector quantization methods.</figcaption>
</figure>
<div id="S5.SS2.p3" class="ltx_para">
<p id="S5.SS2.p3.1" class="ltx_p">We also profile the speed of our aggregation function <math id="S5.SS2.p3.1.m1.2" class="ltx_Math" alttext="f(\cdot,\cdot)" display="inline"><semantics id="S5.SS2.p3.1.m1.2a"><mrow id="S5.SS2.p3.1.m1.2.3" xref="S5.SS2.p3.1.m1.2.3.cmml"><mi id="S5.SS2.p3.1.m1.2.3.2" xref="S5.SS2.p3.1.m1.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S5.SS2.p3.1.m1.2.3.1" xref="S5.SS2.p3.1.m1.2.3.1.cmml">​</mo><mrow id="S5.SS2.p3.1.m1.2.3.3.2" xref="S5.SS2.p3.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="S5.SS2.p3.1.m1.2.3.3.2.1" xref="S5.SS2.p3.1.m1.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S5.SS2.p3.1.m1.1.1" xref="S5.SS2.p3.1.m1.1.1.cmml">⋅</mo><mo rspace="0em" id="S5.SS2.p3.1.m1.2.3.3.2.2" xref="S5.SS2.p3.1.m1.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S5.SS2.p3.1.m1.2.2" xref="S5.SS2.p3.1.m1.2.2.cmml">⋅</mo><mo stretchy="false" id="S5.SS2.p3.1.m1.2.3.3.2.3" xref="S5.SS2.p3.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.1.m1.2b"><apply id="S5.SS2.p3.1.m1.2.3.cmml" xref="S5.SS2.p3.1.m1.2.3"><times id="S5.SS2.p3.1.m1.2.3.1.cmml" xref="S5.SS2.p3.1.m1.2.3.1"></times><ci id="S5.SS2.p3.1.m1.2.3.2.cmml" xref="S5.SS2.p3.1.m1.2.3.2">𝑓</ci><interval closure="open" id="S5.SS2.p3.1.m1.2.3.3.1.cmml" xref="S5.SS2.p3.1.m1.2.3.3.2"><ci id="S5.SS2.p3.1.m1.1.1.cmml" xref="S5.SS2.p3.1.m1.1.1">⋅</ci><ci id="S5.SS2.p3.1.m1.2.2.cmml" xref="S5.SS2.p3.1.m1.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.1.m1.2c">f(\cdot,\cdot)</annotation></semantics></math> using the same baselines as <cite class="ltx_cite ltx_citemacro_citet">Blalock &amp; Guttag (<a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>. As Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2 How Fast is Maddness? ‣ 5 Experiments ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows, our average-based, matrix-aware aggregation is significantly faster than the upcasting-based method of Bolt, its nearest rival.</p>
</div>
<figure id="S5.F4" class="ltx_figure"><img src="/html/2106.10860/assets/x4.png" id="S5.F4.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="226" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Given the preprocessed matrices, <span id="S5.F4.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.F4.3.1.1" class="ltx_text ltx_font_upright"> </span></span>computes the approximate output twice as fast as the fastest existing method.</figcaption>
</figure>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Softmax Classifier</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.5" class="ltx_p">As described in Section <a href="#S1" title="1 Introduction ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we approximated linear classifiers on the widely used CIFAR-10 and CIFAR-100 datasets <cite class="ltx_cite ltx_citemacro_citep">(Krizhevsky et al., <a href="#bib.bib46" title="" class="ltx_ref">2009</a>)</cite>. The classifiers use as input features the 512-dimensional activations of open-source, VGG-like neural networks trained on each dataset <cite class="ltx_cite ltx_citemacro_citep">(Geifman, <a href="#bib.bib30" title="" class="ltx_ref">2018</a>)</cite>. The matrices <math id="S5.SS3.p1.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S5.SS3.p1.1.m1.1a"><mi id="S5.SS3.p1.1.m1.1.1" xref="S5.SS3.p1.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><ci id="S5.SS3.p1.1.m1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">\bm{A}</annotation></semantics></math> are the <math id="S5.SS3.p1.2.m2.1" class="ltx_Math" alttext="10000\times 512" display="inline"><semantics id="S5.SS3.p1.2.m2.1a"><mrow id="S5.SS3.p1.2.m2.1.1" xref="S5.SS3.p1.2.m2.1.1.cmml"><mn id="S5.SS3.p1.2.m2.1.1.2" xref="S5.SS3.p1.2.m2.1.1.2.cmml">10000</mn><mo lspace="0.222em" rspace="0.222em" id="S5.SS3.p1.2.m2.1.1.1" xref="S5.SS3.p1.2.m2.1.1.1.cmml">×</mo><mn id="S5.SS3.p1.2.m2.1.1.3" xref="S5.SS3.p1.2.m2.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.2.m2.1b"><apply id="S5.SS3.p1.2.m2.1.1.cmml" xref="S5.SS3.p1.2.m2.1.1"><times id="S5.SS3.p1.2.m2.1.1.1.cmml" xref="S5.SS3.p1.2.m2.1.1.1"></times><cn type="integer" id="S5.SS3.p1.2.m2.1.1.2.cmml" xref="S5.SS3.p1.2.m2.1.1.2">10000</cn><cn type="integer" id="S5.SS3.p1.2.m2.1.1.3.cmml" xref="S5.SS3.p1.2.m2.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.2.m2.1c">10000\times 512</annotation></semantics></math>-dimensional floating point activations for the full test sets, and the matrices <math id="S5.SS3.p1.3.m3.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S5.SS3.p1.3.m3.1a"><mi id="S5.SS3.p1.3.m3.1.1" xref="S5.SS3.p1.3.m3.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.3.m3.1b"><ci id="S5.SS3.p1.3.m3.1.1.cmml" xref="S5.SS3.p1.3.m3.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.3.m3.1c">\bm{B}</annotation></semantics></math> are each network’s final dense layer. The <math id="S5.SS3.p1.4.m4.1" class="ltx_Math" alttext="50000\times 512" display="inline"><semantics id="S5.SS3.p1.4.m4.1a"><mrow id="S5.SS3.p1.4.m4.1.1" xref="S5.SS3.p1.4.m4.1.1.cmml"><mn id="S5.SS3.p1.4.m4.1.1.2" xref="S5.SS3.p1.4.m4.1.1.2.cmml">50000</mn><mo lspace="0.222em" rspace="0.222em" id="S5.SS3.p1.4.m4.1.1.1" xref="S5.SS3.p1.4.m4.1.1.1.cmml">×</mo><mn id="S5.SS3.p1.4.m4.1.1.3" xref="S5.SS3.p1.4.m4.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.4.m4.1b"><apply id="S5.SS3.p1.4.m4.1.1.cmml" xref="S5.SS3.p1.4.m4.1.1"><times id="S5.SS3.p1.4.m4.1.1.1.cmml" xref="S5.SS3.p1.4.m4.1.1.1"></times><cn type="integer" id="S5.SS3.p1.4.m4.1.1.2.cmml" xref="S5.SS3.p1.4.m4.1.1.2">50000</cn><cn type="integer" id="S5.SS3.p1.4.m4.1.1.3.cmml" xref="S5.SS3.p1.4.m4.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.4.m4.1c">50000\times 512</annotation></semantics></math>-dimensional activations from the training set serve as the training matrices <math id="S5.SS3.p1.5.m5.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="S5.SS3.p1.5.m5.1a"><mover accent="true" id="S5.SS3.p1.5.m5.1.1" xref="S5.SS3.p1.5.m5.1.1.cmml"><mi id="S5.SS3.p1.5.m5.1.1.2" xref="S5.SS3.p1.5.m5.1.1.2.cmml">𝑨</mi><mo id="S5.SS3.p1.5.m5.1.1.1" xref="S5.SS3.p1.5.m5.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.5.m5.1b"><apply id="S5.SS3.p1.5.m5.1.1.cmml" xref="S5.SS3.p1.5.m5.1.1"><ci id="S5.SS3.p1.5.m5.1.1.1.cmml" xref="S5.SS3.p1.5.m5.1.1.1">~</ci><ci id="S5.SS3.p1.5.m5.1.1.2.cmml" xref="S5.SS3.p1.5.m5.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.5.m5.1c">\tilde{\bm{A}}</annotation></semantics></math>.
As shown in Figure <a href="#S5.F5" title="Figure 5 ‣ 5.3 Softmax Classifier ‣ 5 Experiments ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, <span id="S5.SS3.p1.5.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.SS3.p1.5.1.1" class="ltx_text ltx_font_upright"> </span></span>significantly outperforms all existing methods, achieving virtually the same accuracy as exact multiplication more than an order of magnitude faster.</p>
</div>
<figure id="S5.F5" class="ltx_figure"><img src="/html/2106.10860/assets/x5.png" id="S5.F5.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="369" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span><span id="S5.F5.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.F5.3.1.1" class="ltx_text ltx_font_upright"> </span></span>achieves a far better speed-accuracy tradeoff than any existing method when approximating two softmax classifiers.</figcaption>
</figure>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.4" class="ltx_p">Moreover, our method achieves this performance despite having worse support from the hardware. More precisely, it obtains speedups much smaller than the level of compression it provides. For example, the third points from the right in both plots correspond to speedups of roughly <math id="S5.SS3.p2.1.m1.1" class="ltx_math_unparsed" alttext="100\times" display="inline"><semantics id="S5.SS3.p2.1.m1.1a"><mrow id="S5.SS3.p2.1.m1.1b"><mn id="S5.SS3.p2.1.m1.1.1">100</mn><mo lspace="0.222em" id="S5.SS3.p2.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S5.SS3.p2.1.m1.1c">100\times</annotation></semantics></math>. However, they are compressing each <math id="S5.SS3.p2.2.m2.1" class="ltx_Math" alttext="512\times 4\text{B}=2048\text{B}" display="inline"><semantics id="S5.SS3.p2.2.m2.1a"><mrow id="S5.SS3.p2.2.m2.1.1" xref="S5.SS3.p2.2.m2.1.1.cmml"><mrow id="S5.SS3.p2.2.m2.1.1.2" xref="S5.SS3.p2.2.m2.1.1.2.cmml"><mrow id="S5.SS3.p2.2.m2.1.1.2.2" xref="S5.SS3.p2.2.m2.1.1.2.2.cmml"><mn id="S5.SS3.p2.2.m2.1.1.2.2.2" xref="S5.SS3.p2.2.m2.1.1.2.2.2.cmml">512</mn><mo lspace="0.222em" rspace="0.222em" id="S5.SS3.p2.2.m2.1.1.2.2.1" xref="S5.SS3.p2.2.m2.1.1.2.2.1.cmml">×</mo><mn id="S5.SS3.p2.2.m2.1.1.2.2.3" xref="S5.SS3.p2.2.m2.1.1.2.2.3.cmml">4</mn></mrow><mo lspace="0em" rspace="0em" id="S5.SS3.p2.2.m2.1.1.2.1" xref="S5.SS3.p2.2.m2.1.1.2.1.cmml">​</mo><mtext id="S5.SS3.p2.2.m2.1.1.2.3" xref="S5.SS3.p2.2.m2.1.1.2.3a.cmml">B</mtext></mrow><mo id="S5.SS3.p2.2.m2.1.1.1" xref="S5.SS3.p2.2.m2.1.1.1.cmml">=</mo><mrow id="S5.SS3.p2.2.m2.1.1.3" xref="S5.SS3.p2.2.m2.1.1.3.cmml"><mn id="S5.SS3.p2.2.m2.1.1.3.2" xref="S5.SS3.p2.2.m2.1.1.3.2.cmml">2048</mn><mo lspace="0em" rspace="0em" id="S5.SS3.p2.2.m2.1.1.3.1" xref="S5.SS3.p2.2.m2.1.1.3.1.cmml">​</mo><mtext id="S5.SS3.p2.2.m2.1.1.3.3" xref="S5.SS3.p2.2.m2.1.1.3.3a.cmml">B</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.2.m2.1b"><apply id="S5.SS3.p2.2.m2.1.1.cmml" xref="S5.SS3.p2.2.m2.1.1"><eq id="S5.SS3.p2.2.m2.1.1.1.cmml" xref="S5.SS3.p2.2.m2.1.1.1"></eq><apply id="S5.SS3.p2.2.m2.1.1.2.cmml" xref="S5.SS3.p2.2.m2.1.1.2"><times id="S5.SS3.p2.2.m2.1.1.2.1.cmml" xref="S5.SS3.p2.2.m2.1.1.2.1"></times><apply id="S5.SS3.p2.2.m2.1.1.2.2.cmml" xref="S5.SS3.p2.2.m2.1.1.2.2"><times id="S5.SS3.p2.2.m2.1.1.2.2.1.cmml" xref="S5.SS3.p2.2.m2.1.1.2.2.1"></times><cn type="integer" id="S5.SS3.p2.2.m2.1.1.2.2.2.cmml" xref="S5.SS3.p2.2.m2.1.1.2.2.2">512</cn><cn type="integer" id="S5.SS3.p2.2.m2.1.1.2.2.3.cmml" xref="S5.SS3.p2.2.m2.1.1.2.2.3">4</cn></apply><ci id="S5.SS3.p2.2.m2.1.1.2.3a.cmml" xref="S5.SS3.p2.2.m2.1.1.2.3"><mtext id="S5.SS3.p2.2.m2.1.1.2.3.cmml" xref="S5.SS3.p2.2.m2.1.1.2.3">B</mtext></ci></apply><apply id="S5.SS3.p2.2.m2.1.1.3.cmml" xref="S5.SS3.p2.2.m2.1.1.3"><times id="S5.SS3.p2.2.m2.1.1.3.1.cmml" xref="S5.SS3.p2.2.m2.1.1.3.1"></times><cn type="integer" id="S5.SS3.p2.2.m2.1.1.3.2.cmml" xref="S5.SS3.p2.2.m2.1.1.3.2">2048</cn><ci id="S5.SS3.p2.2.m2.1.1.3.3a.cmml" xref="S5.SS3.p2.2.m2.1.1.3.3"><mtext id="S5.SS3.p2.2.m2.1.1.3.3.cmml" xref="S5.SS3.p2.2.m2.1.1.3.3">B</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.2.m2.1c">512\times 4\text{B}=2048\text{B}</annotation></semantics></math> row of the input down to a mere 4B, a savings of <math id="S5.SS3.p2.3.m3.1" class="ltx_math_unparsed" alttext="512\times" display="inline"><semantics id="S5.SS3.p2.3.m3.1a"><mrow id="S5.SS3.p2.3.m3.1b"><mn id="S5.SS3.p2.3.m3.1.1">512</mn><mo lspace="0.222em" id="S5.SS3.p2.3.m3.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S5.SS3.p2.3.m3.1c">512\times</annotation></semantics></math> (sizes not shown in figure). <span id="S5.SS3.p2.4.1" class="ltx_text ltx_font_italic">If the hardware could lookup-accumulate as many bytes per cycle as it can multiply-accumulate, our method could be over <math id="S5.SS3.p2.4.1.m1.1" class="ltx_math_unparsed" alttext="4\times" display="inline"><semantics id="S5.SS3.p2.4.1.m1.1a"><mrow id="S5.SS3.p2.4.1.m1.1b"><mn id="S5.SS3.p2.4.1.m1.1.1">4</mn><mo lspace="0.222em" id="S5.SS3.p2.4.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S5.SS3.p2.4.1.m1.1c">4\times</annotation></semantics></math> faster</span>. Combined with the fact that multiplexers require many fewer transistors than multipliers, this suggests that a hardware implementation of our method might offer large efficiency gains compared to existing accelerators.</p>
</div>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>Kernel-Based Classification</h3>

<div id="S5.SS4.p1" class="ltx_para">
<p id="S5.SS4.p1.2" class="ltx_p">To assess the efficacy of our method on a larger and more diverse set of datasets than CIFAR-10 and CIFAR-100, we trained kernel classifiers on the datasets from the UCR Time Series Archive <cite class="ltx_cite ltx_citemacro_citep">(Dau et al., <a href="#bib.bib18" title="" class="ltx_ref">2018</a>)</cite>. To enable meaningful speed comparison across datasets, we resampled the time series in all datasets to the median length and obtained the matrix <math id="S5.SS4.p1.1.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S5.SS4.p1.1.m1.1a"><mi id="S5.SS4.p1.1.m1.1.1" xref="S5.SS4.p1.1.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S5.SS4.p1.1.m1.1b"><ci id="S5.SS4.p1.1.m1.1.1.cmml" xref="S5.SS4.p1.1.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS4.p1.1.m1.1c">\bm{B}</annotation></semantics></math> for each dataset by running Stochastic Neighbor Compression <cite class="ltx_cite ltx_citemacro_citep">(Kusner et al., <a href="#bib.bib47" title="" class="ltx_ref">2014</a>)</cite> on the training set with an RBF kernel of bandwidth one.
We approximate the Euclidean distances used by the kernel via the identity <math id="S5.SS4.p1.2.m2.3" class="ltx_Math" alttext="\lVert\bm{x}-\bm{y}\rVert_{2}^{2}=\lVert\bm{x}\rVert_{2}^{2}-2\bm{x}^{\top}\bm{y}+\lVert\bm{y}\rVert_{2}^{2}" display="inline"><semantics id="S5.SS4.p1.2.m2.3a"><mrow id="S5.SS4.p1.2.m2.3.3" xref="S5.SS4.p1.2.m2.3.3.cmml"><msubsup id="S5.SS4.p1.2.m2.3.3.1" xref="S5.SS4.p1.2.m2.3.3.1.cmml"><mrow id="S5.SS4.p1.2.m2.3.3.1.1.1.1" xref="S5.SS4.p1.2.m2.3.3.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="S5.SS4.p1.2.m2.3.3.1.1.1.1.2" xref="S5.SS4.p1.2.m2.3.3.1.1.1.2.1.cmml">∥</mo><mrow id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.cmml"><mi id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.2" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.2.cmml">𝒙</mi><mo id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.1" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.1.cmml">−</mo><mi id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.3" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.3.cmml">𝒚</mi></mrow><mo fence="true" lspace="0em" id="S5.SS4.p1.2.m2.3.3.1.1.1.1.3" xref="S5.SS4.p1.2.m2.3.3.1.1.1.2.1.cmml">∥</mo></mrow><mn id="S5.SS4.p1.2.m2.3.3.1.1.3" xref="S5.SS4.p1.2.m2.3.3.1.1.3.cmml">2</mn><mn id="S5.SS4.p1.2.m2.3.3.1.3" xref="S5.SS4.p1.2.m2.3.3.1.3.cmml">2</mn></msubsup><mo rspace="0.1389em" id="S5.SS4.p1.2.m2.3.3.2" xref="S5.SS4.p1.2.m2.3.3.2.cmml">=</mo><mrow id="S5.SS4.p1.2.m2.3.3.3" xref="S5.SS4.p1.2.m2.3.3.3.cmml"><mrow id="S5.SS4.p1.2.m2.3.3.3.2" xref="S5.SS4.p1.2.m2.3.3.3.2.cmml"><msubsup id="S5.SS4.p1.2.m2.3.3.3.2.2" xref="S5.SS4.p1.2.m2.3.3.3.2.2.cmml"><mrow id="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.2" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.1.cmml"><mo fence="true" lspace="0.1389em" rspace="0em" id="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.2.1" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.1.1.cmml">∥</mo><mi id="S5.SS4.p1.2.m2.1.1" xref="S5.SS4.p1.2.m2.1.1.cmml">𝒙</mi><mo fence="true" lspace="0em" rspace="0em" id="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.2.2" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.1.1.cmml">∥</mo></mrow><mn id="S5.SS4.p1.2.m2.3.3.3.2.2.2.3" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.3.cmml">2</mn><mn id="S5.SS4.p1.2.m2.3.3.3.2.2.3" xref="S5.SS4.p1.2.m2.3.3.3.2.2.3.cmml">2</mn></msubsup><mo id="S5.SS4.p1.2.m2.3.3.3.2.1" xref="S5.SS4.p1.2.m2.3.3.3.2.1.cmml">−</mo><mrow id="S5.SS4.p1.2.m2.3.3.3.2.3" xref="S5.SS4.p1.2.m2.3.3.3.2.3.cmml"><mn id="S5.SS4.p1.2.m2.3.3.3.2.3.2" xref="S5.SS4.p1.2.m2.3.3.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S5.SS4.p1.2.m2.3.3.3.2.3.1" xref="S5.SS4.p1.2.m2.3.3.3.2.3.1.cmml">​</mo><msup id="S5.SS4.p1.2.m2.3.3.3.2.3.3" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3.cmml"><mi id="S5.SS4.p1.2.m2.3.3.3.2.3.3.2" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3.2.cmml">𝒙</mi><mo id="S5.SS4.p1.2.m2.3.3.3.2.3.3.3" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S5.SS4.p1.2.m2.3.3.3.2.3.1a" xref="S5.SS4.p1.2.m2.3.3.3.2.3.1.cmml">​</mo><mi id="S5.SS4.p1.2.m2.3.3.3.2.3.4" xref="S5.SS4.p1.2.m2.3.3.3.2.3.4.cmml">𝒚</mi></mrow></mrow><mo id="S5.SS4.p1.2.m2.3.3.3.1" xref="S5.SS4.p1.2.m2.3.3.3.1.cmml">+</mo><msubsup id="S5.SS4.p1.2.m2.3.3.3.3" xref="S5.SS4.p1.2.m2.3.3.3.3.cmml"><mrow id="S5.SS4.p1.2.m2.3.3.3.3.2.2.2" xref="S5.SS4.p1.2.m2.3.3.3.3.2.2.1.cmml"><mo fence="true" lspace="0em" rspace="0em" id="S5.SS4.p1.2.m2.3.3.3.3.2.2.2.1" xref="S5.SS4.p1.2.m2.3.3.3.3.2.2.1.1.cmml">∥</mo><mi id="S5.SS4.p1.2.m2.2.2" xref="S5.SS4.p1.2.m2.2.2.cmml">𝒚</mi><mo fence="true" lspace="0em" id="S5.SS4.p1.2.m2.3.3.3.3.2.2.2.2" xref="S5.SS4.p1.2.m2.3.3.3.3.2.2.1.1.cmml">∥</mo></mrow><mn id="S5.SS4.p1.2.m2.3.3.3.3.2.3" xref="S5.SS4.p1.2.m2.3.3.3.3.2.3.cmml">2</mn><mn id="S5.SS4.p1.2.m2.3.3.3.3.3" xref="S5.SS4.p1.2.m2.3.3.3.3.3.cmml">2</mn></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS4.p1.2.m2.3b"><apply id="S5.SS4.p1.2.m2.3.3.cmml" xref="S5.SS4.p1.2.m2.3.3"><eq id="S5.SS4.p1.2.m2.3.3.2.cmml" xref="S5.SS4.p1.2.m2.3.3.2"></eq><apply id="S5.SS4.p1.2.m2.3.3.1.cmml" xref="S5.SS4.p1.2.m2.3.3.1"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.1.2.cmml" xref="S5.SS4.p1.2.m2.3.3.1">superscript</csymbol><apply id="S5.SS4.p1.2.m2.3.3.1.1.cmml" xref="S5.SS4.p1.2.m2.3.3.1"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.1.1.2.cmml" xref="S5.SS4.p1.2.m2.3.3.1">subscript</csymbol><apply id="S5.SS4.p1.2.m2.3.3.1.1.1.2.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1"><csymbol cd="latexml" id="S5.SS4.p1.2.m2.3.3.1.1.1.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.2">delimited-∥∥</csymbol><apply id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1"><minus id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.1.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.1"></minus><ci id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.2.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.2">𝒙</ci><ci id="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.3.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.1.1.1.3">𝒚</ci></apply></apply><cn type="integer" id="S5.SS4.p1.2.m2.3.3.1.1.3.cmml" xref="S5.SS4.p1.2.m2.3.3.1.1.3">2</cn></apply><cn type="integer" id="S5.SS4.p1.2.m2.3.3.1.3.cmml" xref="S5.SS4.p1.2.m2.3.3.1.3">2</cn></apply><apply id="S5.SS4.p1.2.m2.3.3.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3"><plus id="S5.SS4.p1.2.m2.3.3.3.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.1"></plus><apply id="S5.SS4.p1.2.m2.3.3.3.2.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2"><minus id="S5.SS4.p1.2.m2.3.3.3.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.1"></minus><apply id="S5.SS4.p1.2.m2.3.3.3.2.2.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.3.2.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2">superscript</csymbol><apply id="S5.SS4.p1.2.m2.3.3.3.2.2.2.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.3.2.2.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2">subscript</csymbol><apply id="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.2"><csymbol cd="latexml" id="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.1.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.2.2.1">delimited-∥∥</csymbol><ci id="S5.SS4.p1.2.m2.1.1.cmml" xref="S5.SS4.p1.2.m2.1.1">𝒙</ci></apply><cn type="integer" id="S5.SS4.p1.2.m2.3.3.3.2.2.2.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2.2.3">2</cn></apply><cn type="integer" id="S5.SS4.p1.2.m2.3.3.3.2.2.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.2.3">2</cn></apply><apply id="S5.SS4.p1.2.m2.3.3.3.2.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3"><times id="S5.SS4.p1.2.m2.3.3.3.2.3.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.1"></times><cn type="integer" id="S5.SS4.p1.2.m2.3.3.3.2.3.2.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.2">2</cn><apply id="S5.SS4.p1.2.m2.3.3.3.2.3.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.3.2.3.3.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3">superscript</csymbol><ci id="S5.SS4.p1.2.m2.3.3.3.2.3.3.2.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3.2">𝒙</ci><csymbol cd="latexml" id="S5.SS4.p1.2.m2.3.3.3.2.3.3.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.3.3">top</csymbol></apply><ci id="S5.SS4.p1.2.m2.3.3.3.2.3.4.cmml" xref="S5.SS4.p1.2.m2.3.3.3.2.3.4">𝒚</ci></apply></apply><apply id="S5.SS4.p1.2.m2.3.3.3.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.3.3.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3">superscript</csymbol><apply id="S5.SS4.p1.2.m2.3.3.3.3.2.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3"><csymbol cd="ambiguous" id="S5.SS4.p1.2.m2.3.3.3.3.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3">subscript</csymbol><apply id="S5.SS4.p1.2.m2.3.3.3.3.2.2.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3.2.2.2"><csymbol cd="latexml" id="S5.SS4.p1.2.m2.3.3.3.3.2.2.1.1.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3.2.2.2.1">delimited-∥∥</csymbol><ci id="S5.SS4.p1.2.m2.2.2.cmml" xref="S5.SS4.p1.2.m2.2.2">𝒚</ci></apply><cn type="integer" id="S5.SS4.p1.2.m2.3.3.3.3.2.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3.2.3">2</cn></apply><cn type="integer" id="S5.SS4.p1.2.m2.3.3.3.3.3.cmml" xref="S5.SS4.p1.2.m2.3.3.3.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS4.p1.2.m2.3c">\lVert\bm{x}-\bm{y}\rVert_{2}^{2}=\lVert\bm{x}\rVert_{2}^{2}-2\bm{x}^{\top}\bm{y}+\lVert\bm{y}\rVert_{2}^{2}</annotation></semantics></math>, which consists only of dot products.
This is not the state-of-the-art means of classifying time series, but it does yield fixed-sized matrices and is representative of several modern techniques for constructing highly efficient classifiers <cite class="ltx_cite ltx_citemacro_citep">(Kusner et al., <a href="#bib.bib47" title="" class="ltx_ref">2014</a>; Wang et al., <a href="#bib.bib67" title="" class="ltx_ref">2016b</a>; Zhong et al., <a href="#bib.bib74" title="" class="ltx_ref">2017</a>; Gupta et al., <a href="#bib.bib33" title="" class="ltx_ref">2017</a>)</cite>.
Because Stochastic Neighbor Compression optimizes the classifiers to avoid redundancy, this task is quite difficult.</p>
</div>
<div id="S5.SS4.p2" class="ltx_para">
<p id="S5.SS4.p2.1" class="ltx_p">As shown in Figure <a href="#S5.F6" title="Figure 6 ‣ 5.4 Kernel-Based Classification ‣ 5 Experiments ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, <span id="S5.SS4.p2.1.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.SS4.p2.1.1.1" class="ltx_text ltx_font_upright"> </span></span>is significantly faster than alternatives at a given level of accuracy. A counter-intuitive result, however, is that optimization of the prototypes occasionally reduces accuracy—see the red line dipping below the blue one in the lowest subplot. Since the optimization strictly increases the expressive power, we believe that this is a product of overfitting and could be corrected by not fixing <math id="S5.SS4.p2.1.m1.1" class="ltx_Math" alttext="\lambda=1" display="inline"><semantics id="S5.SS4.p2.1.m1.1a"><mrow id="S5.SS4.p2.1.m1.1.1" xref="S5.SS4.p2.1.m1.1.1.cmml"><mi id="S5.SS4.p2.1.m1.1.1.2" xref="S5.SS4.p2.1.m1.1.1.2.cmml">λ</mi><mo id="S5.SS4.p2.1.m1.1.1.1" xref="S5.SS4.p2.1.m1.1.1.1.cmml">=</mo><mn id="S5.SS4.p2.1.m1.1.1.3" xref="S5.SS4.p2.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS4.p2.1.m1.1b"><apply id="S5.SS4.p2.1.m1.1.1.cmml" xref="S5.SS4.p2.1.m1.1.1"><eq id="S5.SS4.p2.1.m1.1.1.1.cmml" xref="S5.SS4.p2.1.m1.1.1.1"></eq><ci id="S5.SS4.p2.1.m1.1.1.2.cmml" xref="S5.SS4.p2.1.m1.1.1.2">𝜆</ci><cn type="integer" id="S5.SS4.p2.1.m1.1.1.3.cmml" xref="S5.SS4.p2.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS4.p2.1.m1.1c">\lambda=1</annotation></semantics></math> in the ridge regression.</p>
</div>
<figure id="S5.F6" class="ltx_figure"><img src="/html/2106.10860/assets/x6.png" id="S5.F6.1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="475" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Fraction of UCR datasets for which each method preserves a given fraction of the original accuracy.
<span id="S5.F6.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.F6.3.1.1" class="ltx_text ltx_font_upright"> </span></span>enables much greater speedups for a given level of accuracy degradation.</figcaption>
</figure>
</section>
<section id="S5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.5 </span>Image Filtering</h3>

<div id="S5.SS5.p1" class="ltx_para">
<p id="S5.SS5.p1.9" class="ltx_p">To test the extreme limits of <span id="S5.SS5.p1.9.1" class="ltx_text ltx_font_smallcaps">Maddness</span>, we benchmarked the various techniques’ ability to apply small filters to images (after an offline im2row transform to reduce the task to matrix multiplication). This task is extreme in that <math id="S5.SS5.p1.1.m1.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S5.SS5.p1.1.m1.1a"><mi id="S5.SS5.p1.1.m1.1.1" xref="S5.SS5.p1.1.m1.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.1.m1.1b"><ci id="S5.SS5.p1.1.m1.1.1.cmml" xref="S5.SS5.p1.1.m1.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.1.m1.1c">D</annotation></semantics></math> and <math id="S5.SS5.p1.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S5.SS5.p1.2.m2.1a"><mi id="S5.SS5.p1.2.m2.1.1" xref="S5.SS5.p1.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.2.m2.1b"><ci id="S5.SS5.p1.2.m2.1.1.cmml" xref="S5.SS5.p1.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.2.m2.1c">M</annotation></semantics></math> are tiny, affording almost no opportunity to amortize preprocessing costs. As representative example filters, we chose <math id="S5.SS5.p1.3.m3.1" class="ltx_Math" alttext="3\times 3" display="inline"><semantics id="S5.SS5.p1.3.m3.1a"><mrow id="S5.SS5.p1.3.m3.1.1" xref="S5.SS5.p1.3.m3.1.1.cmml"><mn id="S5.SS5.p1.3.m3.1.1.2" xref="S5.SS5.p1.3.m3.1.1.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="S5.SS5.p1.3.m3.1.1.1" xref="S5.SS5.p1.3.m3.1.1.1.cmml">×</mo><mn id="S5.SS5.p1.3.m3.1.1.3" xref="S5.SS5.p1.3.m3.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.3.m3.1b"><apply id="S5.SS5.p1.3.m3.1.1.cmml" xref="S5.SS5.p1.3.m3.1.1"><times id="S5.SS5.p1.3.m3.1.1.1.cmml" xref="S5.SS5.p1.3.m3.1.1.1"></times><cn type="integer" id="S5.SS5.p1.3.m3.1.1.2.cmml" xref="S5.SS5.p1.3.m3.1.1.2">3</cn><cn type="integer" id="S5.SS5.p1.3.m3.1.1.3.cmml" xref="S5.SS5.p1.3.m3.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.3.m3.1c">3\times 3</annotation></semantics></math> Sobel kernels and <math id="S5.SS5.p1.4.m4.1" class="ltx_Math" alttext="5\times 5" display="inline"><semantics id="S5.SS5.p1.4.m4.1a"><mrow id="S5.SS5.p1.4.m4.1.1" xref="S5.SS5.p1.4.m4.1.1.cmml"><mn id="S5.SS5.p1.4.m4.1.1.2" xref="S5.SS5.p1.4.m4.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="S5.SS5.p1.4.m4.1.1.1" xref="S5.SS5.p1.4.m4.1.1.1.cmml">×</mo><mn id="S5.SS5.p1.4.m4.1.1.3" xref="S5.SS5.p1.4.m4.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.4.m4.1b"><apply id="S5.SS5.p1.4.m4.1.1.cmml" xref="S5.SS5.p1.4.m4.1.1"><times id="S5.SS5.p1.4.m4.1.1.1.cmml" xref="S5.SS5.p1.4.m4.1.1.1"></times><cn type="integer" id="S5.SS5.p1.4.m4.1.1.2.cmml" xref="S5.SS5.p1.4.m4.1.1.2">5</cn><cn type="integer" id="S5.SS5.p1.4.m4.1.1.3.cmml" xref="S5.SS5.p1.4.m4.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.4.m4.1c">5\times 5</annotation></semantics></math> Gaussian kernels. These are common high-pass and low-pass filters, respectively. We took the first 10 images from the first 50 classes of the Caltech101 dataset <cite class="ltx_cite ltx_citemacro_citep">(Fei-Fei et al., <a href="#bib.bib26" title="" class="ltx_ref">2004</a>)</cite> as a single training set, and the first 10 images from the remaining 51 classes as 510 test sets. We constructed the <math id="S5.SS5.p1.5.m5.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S5.SS5.p1.5.m5.1a"><mi id="S5.SS5.p1.5.m5.1.1" xref="S5.SS5.p1.5.m5.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.5.m5.1b"><ci id="S5.SS5.p1.5.m5.1.1.cmml" xref="S5.SS5.p1.5.m5.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.5.m5.1c">\bm{A}</annotation></semantics></math> matrices by extracting each patch of each image as one row. The <math id="S5.SS5.p1.6.m6.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S5.SS5.p1.6.m6.1a"><mi id="S5.SS5.p1.6.m6.1.1" xref="S5.SS5.p1.6.m6.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.6.m6.1b"><ci id="S5.SS5.p1.6.m6.1.1.cmml" xref="S5.SS5.p1.6.m6.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.6.m6.1c">\bm{B}</annotation></semantics></math> matrices have two columns, corresponding to one pair of Sobel or Gaussian filters (since using these filters in pairs is common). We report the normalized mean-squared error (NMSE), defined as <math id="S5.SS5.p1.7.m7.4" class="ltx_Math" alttext="\lVert\bm{\hat{C}}_{i,j}-\bm{A}\bm{B}\rVert^{2}_{F}/\lVert\bm{A}\bm{B}\rVert^{2}_{F}" display="inline"><semantics id="S5.SS5.p1.7.m7.4a"><mrow id="S5.SS5.p1.7.m7.4.4" xref="S5.SS5.p1.7.m7.4.4.cmml"><msubsup id="S5.SS5.p1.7.m7.3.3.1" xref="S5.SS5.p1.7.m7.3.3.1.cmml"><mrow id="S5.SS5.p1.7.m7.3.3.1.1.1.1" xref="S5.SS5.p1.7.m7.3.3.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="S5.SS5.p1.7.m7.3.3.1.1.1.1.2" xref="S5.SS5.p1.7.m7.3.3.1.1.1.2.1.cmml">∥</mo><mrow id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.cmml"><msub id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.cmml"><mover accent="true" id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.cmml"><mi id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.2" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.2.cmml">𝑪</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.1" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.1.cmml">^</mo></mover><mrow id="S5.SS5.p1.7.m7.2.2.2.4" xref="S5.SS5.p1.7.m7.2.2.2.3.cmml"><mi id="S5.SS5.p1.7.m7.1.1.1.1" xref="S5.SS5.p1.7.m7.1.1.1.1.cmml">i</mi><mo id="S5.SS5.p1.7.m7.2.2.2.4.1" xref="S5.SS5.p1.7.m7.2.2.2.3.cmml">,</mo><mi id="S5.SS5.p1.7.m7.2.2.2.2" xref="S5.SS5.p1.7.m7.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.1" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.1.cmml">−</mo><mrow id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.cmml"><mi id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.2" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.1" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.1.cmml">​</mo><mi id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.3" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.3.cmml">𝑩</mi></mrow></mrow><mo fence="true" lspace="0em" id="S5.SS5.p1.7.m7.3.3.1.1.1.1.3" xref="S5.SS5.p1.7.m7.3.3.1.1.1.2.1.cmml">∥</mo></mrow><mi id="S5.SS5.p1.7.m7.3.3.1.3" xref="S5.SS5.p1.7.m7.3.3.1.3.cmml">F</mi><mn id="S5.SS5.p1.7.m7.3.3.1.1.3" xref="S5.SS5.p1.7.m7.3.3.1.1.3.cmml">2</mn></msubsup><mo id="S5.SS5.p1.7.m7.4.4.3" xref="S5.SS5.p1.7.m7.4.4.3.cmml">/</mo><msubsup id="S5.SS5.p1.7.m7.4.4.2" xref="S5.SS5.p1.7.m7.4.4.2.cmml"><mrow id="S5.SS5.p1.7.m7.4.4.2.1.1.1" xref="S5.SS5.p1.7.m7.4.4.2.1.1.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="S5.SS5.p1.7.m7.4.4.2.1.1.1.2" xref="S5.SS5.p1.7.m7.4.4.2.1.1.2.1.cmml">∥</mo><mrow id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.cmml"><mi id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.2" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.1" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.1.cmml">​</mo><mi id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.3" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.3.cmml">𝑩</mi></mrow><mo fence="true" lspace="0em" id="S5.SS5.p1.7.m7.4.4.2.1.1.1.3" xref="S5.SS5.p1.7.m7.4.4.2.1.1.2.1.cmml">∥</mo></mrow><mi id="S5.SS5.p1.7.m7.4.4.2.3" xref="S5.SS5.p1.7.m7.4.4.2.3.cmml">F</mi><mn id="S5.SS5.p1.7.m7.4.4.2.1.3" xref="S5.SS5.p1.7.m7.4.4.2.1.3.cmml">2</mn></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.7.m7.4b"><apply id="S5.SS5.p1.7.m7.4.4.cmml" xref="S5.SS5.p1.7.m7.4.4"><divide id="S5.SS5.p1.7.m7.4.4.3.cmml" xref="S5.SS5.p1.7.m7.4.4.3"></divide><apply id="S5.SS5.p1.7.m7.3.3.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1"><csymbol cd="ambiguous" id="S5.SS5.p1.7.m7.3.3.1.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1">subscript</csymbol><apply id="S5.SS5.p1.7.m7.3.3.1.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1"><csymbol cd="ambiguous" id="S5.SS5.p1.7.m7.3.3.1.1.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1">superscript</csymbol><apply id="S5.SS5.p1.7.m7.3.3.1.1.1.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1"><csymbol cd="latexml" id="S5.SS5.p1.7.m7.3.3.1.1.1.2.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.2">delimited-∥∥</csymbol><apply id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1"><minus id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.1"></minus><apply id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2">subscript</csymbol><apply id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2"><ci id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.1">bold-^</ci><ci id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.2.2.2">𝑪</ci></apply><list id="S5.SS5.p1.7.m7.2.2.2.3.cmml" xref="S5.SS5.p1.7.m7.2.2.2.4"><ci id="S5.SS5.p1.7.m7.1.1.1.1.cmml" xref="S5.SS5.p1.7.m7.1.1.1.1">𝑖</ci><ci id="S5.SS5.p1.7.m7.2.2.2.2.cmml" xref="S5.SS5.p1.7.m7.2.2.2.2">𝑗</ci></list></apply><apply id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3"><times id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.1.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.1"></times><ci id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.2.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.2">𝑨</ci><ci id="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.3.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.1.1.1.3.3">𝑩</ci></apply></apply></apply><cn type="integer" id="S5.SS5.p1.7.m7.3.3.1.1.3.cmml" xref="S5.SS5.p1.7.m7.3.3.1.1.3">2</cn></apply><ci id="S5.SS5.p1.7.m7.3.3.1.3.cmml" xref="S5.SS5.p1.7.m7.3.3.1.3">𝐹</ci></apply><apply id="S5.SS5.p1.7.m7.4.4.2.cmml" xref="S5.SS5.p1.7.m7.4.4.2"><csymbol cd="ambiguous" id="S5.SS5.p1.7.m7.4.4.2.2.cmml" xref="S5.SS5.p1.7.m7.4.4.2">subscript</csymbol><apply id="S5.SS5.p1.7.m7.4.4.2.1.cmml" xref="S5.SS5.p1.7.m7.4.4.2"><csymbol cd="ambiguous" id="S5.SS5.p1.7.m7.4.4.2.1.2.cmml" xref="S5.SS5.p1.7.m7.4.4.2">superscript</csymbol><apply id="S5.SS5.p1.7.m7.4.4.2.1.1.2.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1"><csymbol cd="latexml" id="S5.SS5.p1.7.m7.4.4.2.1.1.2.1.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.2">delimited-∥∥</csymbol><apply id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1"><times id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.1.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.1"></times><ci id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.2.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.2">𝑨</ci><ci id="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.3.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.1.1.1.3">𝑩</ci></apply></apply><cn type="integer" id="S5.SS5.p1.7.m7.4.4.2.1.3.cmml" xref="S5.SS5.p1.7.m7.4.4.2.1.3">2</cn></apply><ci id="S5.SS5.p1.7.m7.4.4.2.3.cmml" xref="S5.SS5.p1.7.m7.4.4.2.3">𝐹</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.7.m7.4c">\lVert\bm{\hat{C}}_{i,j}-\bm{A}\bm{B}\rVert^{2}_{F}/\lVert\bm{A}\bm{B}\rVert^{2}_{F}</annotation></semantics></math>, where <math id="S5.SS5.p1.8.m8.1" class="ltx_Math" alttext="\hat{\bm{C}}" display="inline"><semantics id="S5.SS5.p1.8.m8.1a"><mover accent="true" id="S5.SS5.p1.8.m8.1.1" xref="S5.SS5.p1.8.m8.1.1.cmml"><mi id="S5.SS5.p1.8.m8.1.1.2" xref="S5.SS5.p1.8.m8.1.1.2.cmml">𝑪</mi><mo id="S5.SS5.p1.8.m8.1.1.1" xref="S5.SS5.p1.8.m8.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.8.m8.1b"><apply id="S5.SS5.p1.8.m8.1.1.cmml" xref="S5.SS5.p1.8.m8.1.1"><ci id="S5.SS5.p1.8.m8.1.1.1.cmml" xref="S5.SS5.p1.8.m8.1.1.1">^</ci><ci id="S5.SS5.p1.8.m8.1.1.2.cmml" xref="S5.SS5.p1.8.m8.1.1.2">𝑪</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.8.m8.1c">\hat{\bm{C}}</annotation></semantics></math> is the method’s estimate of <math id="S5.SS5.p1.9.m9.1" class="ltx_Math" alttext="\bm{A}\bm{B}" display="inline"><semantics id="S5.SS5.p1.9.m9.1a"><mrow id="S5.SS5.p1.9.m9.1.1" xref="S5.SS5.p1.9.m9.1.1.cmml"><mi id="S5.SS5.p1.9.m9.1.1.2" xref="S5.SS5.p1.9.m9.1.1.2.cmml">𝑨</mi><mo lspace="0em" rspace="0em" id="S5.SS5.p1.9.m9.1.1.1" xref="S5.SS5.p1.9.m9.1.1.1.cmml">​</mo><mi id="S5.SS5.p1.9.m9.1.1.3" xref="S5.SS5.p1.9.m9.1.1.3.cmml">𝑩</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.9.m9.1b"><apply id="S5.SS5.p1.9.m9.1.1.cmml" xref="S5.SS5.p1.9.m9.1.1"><times id="S5.SS5.p1.9.m9.1.1.1.cmml" xref="S5.SS5.p1.9.m9.1.1.1"></times><ci id="S5.SS5.p1.9.m9.1.1.2.cmml" xref="S5.SS5.p1.9.m9.1.1.2">𝑨</ci><ci id="S5.SS5.p1.9.m9.1.1.3.cmml" xref="S5.SS5.p1.9.m9.1.1.3">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.9.m9.1c">\bm{A}\bm{B}</annotation></semantics></math>. An NMSE of 0 is perfect and an NMSE of 1 corresponds to always predicting 0.</p>
</div>
<div id="S5.SS5.p2" class="ltx_para">
<p id="S5.SS5.p2.2" class="ltx_p">In Figure <a href="#S5.F7" title="Figure 7 ‣ 5.5 Image Filtering ‣ 5 Experiments ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, we see that it is only <span id="S5.SS5.p2.2.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.SS5.p2.2.1.1" class="ltx_text ltx_font_upright"> </span></span>that offers any advantage over exact matrix products. This is likely because two columns afford almost no time to preprocess <math id="S5.SS5.p2.1.m1.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S5.SS5.p2.1.m1.1a"><mi id="S5.SS5.p2.1.m1.1.1" xref="S5.SS5.p2.1.m1.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p2.1.m1.1b"><ci id="S5.SS5.p2.1.m1.1.1.cmml" xref="S5.SS5.p2.1.m1.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p2.1.m1.1c">\bm{A}</annotation></semantics></math>; indeed, rival vector quantization methods cannot logically do less work than brute force in this setting, and dense linear methods can only save work by embedding rows of <math id="S5.SS5.p2.2.m2.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S5.SS5.p2.2.m2.1a"><mi id="S5.SS5.p2.2.m2.1.1" xref="S5.SS5.p2.2.m2.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p2.2.m2.1b"><ci id="S5.SS5.p2.2.m2.1.1.cmml" xref="S5.SS5.p2.2.m2.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p2.2.m2.1c">\bm{A}</annotation></semantics></math> in one-dimensional space.
<span id="S5.SS5.p2.2.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.SS5.p2.2.2.1" class="ltx_text ltx_font_upright"> </span></span>performs much worse on the high-pass filters (top) than the low-pass filters (bottom). This is likely because the former produce outputs with variance that is orders of magnitude lower than that of the original image, making the NMSE denominator tiny.</p>
</div>
<figure id="S5.F7" class="ltx_figure"><img src="/html/2106.10860/assets/x7.png" id="S5.F7.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="308" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Despite there being only two columns in the matrix <math id="S5.F7.3.2.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S5.F7.3.2.m1.1b"><mi id="S5.F7.3.2.m1.1.1" xref="S5.F7.3.2.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S5.F7.3.2.m1.1c"><ci id="S5.F7.3.2.m1.1.1.cmml" xref="S5.F7.3.2.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F7.3.2.m1.1d">\bm{B}</annotation></semantics></math>, <span id="S5.F7.6.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.F7.6.1.1" class="ltx_text ltx_font_upright"> </span></span>still achieves a significant speedup with reasonable accuracy. Methods that are Pareto dominated by exact matrix multiplication on both tasks are not shown; this includes all methods but <span id="S5.F7.7.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="S5.F7.7.2.1" class="ltx_text ltx_font_upright"> </span></span>and SparsePCA.</figcaption>
</figure>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Discussion and Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Because our work draws on a number of different fields but does not fit cleanly into any of them, it is useful to discuss what we have and have not demonstrated, as well as possible implications and extensions of our work.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">Our main empirical finding is that our proposed method, <span id="S6.p2.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>, achieves order-of-magnitude speedups compared to existing AMM methods and up to two-order-of-magnitude speedups compared to the dense baseline. It also compresses matrices by up to three orders of magnitude. These results are evaluated on a CPU, and are obtainable only when there is a training set for one matrix. We also claim superior performance only when one matrix is larger than the other, and both matrices are tall—the regime wherein our extremely fast (but less accurate) encoding function is beneficial. Our method also loses utility when the larger matrix is known ahead of time; this assumption is common in similarity search, and eliminates the need for a fast encoding function entirely. Our approximate integer summation and fused table lookups would likely be useful independent of any of these assumptions, but demonstrating this is future work.</p>
</div>
<div id="S6.p3" class="ltx_para">
<p id="S6.p3.1" class="ltx_p">We also have several theoretical findings, taking the form of guarantees regarding the errors introduced by our method and its constituent subroutines. While we do obtain an overall generalization guarantee, this guarantee is not tight. In particular, it should grow looser with the large matrix’s Frobenius norm and tighter as its singular values become more concentrated; at present, however, it simply grows looser as the largest singular value grows. The missing step is a guarantee that our encoding function will yield lower quantization errors when the singular values are more concentrated, which is its behavior in practice.</p>
</div>
<div id="S6.p4" class="ltx_para">
<p id="S6.p4.1" class="ltx_p">We have not demonstrated results using GPUs or other accelerators. While such accelerators are a small minority of hardware, they are often used in machine learning. Our method is not inherently tied to CPUs, but the differing performance characteristics of accelerators mean that adapting our method to them would require both algorithmic and implementation work, with the details depending on the device. We also have not evaluated a multi-CPU-threaded extension of our algorithm, though this is because our method is intended to serve as the low-level, compute-bound, block matrix product routine called by individual threads.</p>
</div>
<div id="S6.p5" class="ltx_para">
<p id="S6.p5.1" class="ltx_p">Finally, we have not demonstrated results using convolutional layers in neural networks, or results accelerating full networks. The weight reuse in convolutional layers presents many opportunities for algorithmic optimizations, and we hope to exploit them using a specialized extension of our method in future work. Accelerating overall networks will require two significant undertakings: first, the engineering work of building and integrating custom operators, data layouts, etc., into existing frameworks and networks; and second, the research necessary to determine when, how, and to what extent to include approximate kernels inspired by our approach. A particular difficulty with the latter is that our hash function is not differentiable.</p>
</div>
<div id="S6.p6" class="ltx_para">
<p id="S6.p6.1" class="ltx_p">We believe that accelerating full networks with our ideas is a promising direction, particularly for inference. This is especially true at the hardware level—our method requires only multi<span id="S6.p6.1.1" class="ltx_text ltx_font_italic">plexers</span>, not multi<span id="S6.p6.1.2" class="ltx_text ltx_font_italic">pliers</span>, and can therefore be implemented easily and with far less power than current matrix product logic. Moreover, our encoded representation and lookup tables have contiguous and uniformly-sized elements, making our approximate GEMM inner loops nearly identical to their dense counterparts–i.e., there is no need for complex access patterns or sparsity handling.</p>
</div>
<div id="S6.p7" class="ltx_para">
<p id="S6.p7.1" class="ltx_p">In summary, we introduced <span id="S6.p7.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>, an algorithm that achieves up to a <math id="S6.p7.1.m1.1" class="ltx_math_unparsed" alttext="10\times" display="inline"><semantics id="S6.p7.1.m1.1a"><mrow id="S6.p7.1.m1.1b"><mn id="S6.p7.1.m1.1.1">10</mn><mo lspace="0.222em" id="S6.p7.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.p7.1.m1.1c">10\times</annotation></semantics></math> better speed-quality tradeoff than existing methods for the well-studied problem of approximate matrix multiplication (AMM), as measured on a large, diverse, and challenging set of real-world matrices. Our approach is a significant departure from existing AMM work in that it relies on hashing and table lookups rather than multiply-add operations. Our results suggest that future methods similar to our own might hold promise for accelerating convolution, deep learning, and other workloads bottlenecked by linear transforms.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Abadi et al. (2016)</span>
<span class="ltx_bibblock">
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M.,
Ghemawat, S., et al.

</span>
<span class="ltx_bibblock">Tensorflow: A system for large-scale machine learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">OSDI</em>, volume 16, pp.  265–283, 2016.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Achlioptas (2001)</span>
<span class="ltx_bibblock">
Achlioptas, D.

</span>
<span class="ltx_bibblock">Database-friendly random projections.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART
symposium on Principles of database systems</em>, pp.  274–281, 2001.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ailon &amp; Chazelle (2009)</span>
<span class="ltx_bibblock">
Ailon, N. and Chazelle, B.

</span>
<span class="ltx_bibblock">The Fast Johnson-Lindenstrauss Transform and Approximate Nearest
Neighbors.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">SIAM Journal on Computing (SICOMP)</em>, 39(1):302–322, 2009.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1137/060673096</span>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">André et al. (2017)</span>
<span class="ltx_bibblock">
André, F., Kermarrec, A.-M., and Le Scouarnec, N.

</span>
<span class="ltx_bibblock">Accelerated nearest neighbor search with quick adc.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2017 ACM on International Conference on
Multimedia Retrieval</em>, pp.  159–166, 2017.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">André et al. (2019)</span>
<span class="ltx_bibblock">
André, F., Kermarrec, A.-M., and Le Scouarnec, N.

</span>
<span class="ltx_bibblock">Quicker adc: Unlocking the hidden potential of product quantization
with simd.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">IEEE transactions on pattern analysis and machine
intelligence</em>, 2019.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Babenko &amp; Lempitsky (2014)</span>
<span class="ltx_bibblock">
Babenko, A. and Lempitsky, V.

</span>
<span class="ltx_bibblock">Additive quantization for extreme vector compression.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition</em>, pp.  931–938, 2014.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Babenko &amp; Lempitsky (2015)</span>
<span class="ltx_bibblock">
Babenko, A. and Lempitsky, V.

</span>
<span class="ltx_bibblock">Tree Quantization for Large-Scale Similarity Search and
Classification.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, pp.  1–9, 2015.

</span>
<span class="ltx_bibblock">URL
<a href="papers3://publication/uuid/F4762974-BB97-4208-B035-508945A90EFC" title="" class="ltx_ref ltx_url ltx_font_typewriter">papers3://publication/uuid/F4762974-BB97-4208-B035-508945A90EFC</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Babenko et al. (2016)</span>
<span class="ltx_bibblock">
Babenko, A., Arandjelović, R., and Lempitsky, V.

</span>
<span class="ltx_bibblock">Pairwise quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1606.01550</em>, 2016.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bakhtiary et al. (2015)</span>
<span class="ltx_bibblock">
Bakhtiary, A. H., Lapedriza, A., and Masip, D.

</span>
<span class="ltx_bibblock">Speeding up neural networks for large scale classification using wta
hashing.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1504.07488</em>, 2015.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bartlett &amp; Mendelson (2002)</span>
<span class="ltx_bibblock">
Bartlett, P. L. and Mendelson, S.

</span>
<span class="ltx_bibblock">Rademacher and gaussian complexities: Risk bounds and structural
results.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 3(Nov):463–482, 2002.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blalock &amp; Guttag (2017)</span>
<span class="ltx_bibblock">
Blalock, D. W. and Guttag, J. V.

</span>
<span class="ltx_bibblock">Bolt: Accelerated data mining with fast vector compression.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 23rd ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining</em>, pp.  727–735. ACM, 2017.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blalock et al. (2020)</span>
<span class="ltx_bibblock">
Blalock, D. W., Ortiz, J. J. G., Frankle, J., and Guttag, J. V.

</span>
<span class="ltx_bibblock">What is the state of neural network pruning?

</span>
<span class="ltx_bibblock">In Dhillon, I. S., Papailiopoulos, D. S., and Sze, V. (eds.),
<em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin,
TX, USA, March 2-4, 2020</em>. mlsys.org, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlsys.org/book/296.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlsys.org/book/296.pdf</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Camacho et al. (2020)</span>
<span class="ltx_bibblock">
Camacho, J., Smilde, A., Saccenti, E., and Westerhuis, J.

</span>
<span class="ltx_bibblock">All sparse pca models are wrong, but some are useful. part i:
Computation of scores, residuals and explained variance.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Chemometrics and Intelligent Laboratory Systems</em>, 196:103907, 2020.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2019)</span>
<span class="ltx_bibblock">
Chen, B., Medini, T., and Shrivastava, A.

</span>
<span class="ltx_bibblock">Slide: In defense of smart algorithms over hardware acceleration for
large-scale deep learning systems.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1903.03129</em>, 2019.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2015)</span>
<span class="ltx_bibblock">
Chen, W., Wilson, J. T., Tyree, S., Weinberger, K. Q., and Chen, Y.

</span>
<span class="ltx_bibblock">Compressing neural networks with the hashing trick.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">ICML</em>, pp.  2285–2294, 2015.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2016)</span>
<span class="ltx_bibblock">
Chen, Y.-H., Emer, J., and Sze, V.

</span>
<span class="ltx_bibblock">Eyeriss: A spatial architecture for energy-efficient dataflow for
convolutional neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">ACM SIGARCH Computer Architecture News</em>, 44(3):367–379, 2016.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dasgupta et al. (2010)</span>
<span class="ltx_bibblock">
Dasgupta, A., Kumar, R., and Sarlós, T.

</span>
<span class="ltx_bibblock">A sparse johnson: Lindenstrauss transform.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the forty-second ACM symposium on Theory of
computing</em>, pp.  341–350, 2010.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dau et al. (2018)</span>
<span class="ltx_bibblock">
Dau, H. A., Keogh, E., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S.,
Ratanamahatana, C. A., Yanping, Hu, B., Begum, N., Bagnall, A., Mueen, A.,
Batista, G., and Hexagon-ML.

</span>
<span class="ltx_bibblock">The ucr time series classification archive, October 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.cs.ucr.edu/~eamonn/time_series_data_2018/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.cs.ucr.edu/~eamonn/time_series_data_2018/</a>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dean et al. (2013)</span>
<span class="ltx_bibblock">
Dean, T., Ruzon, M. A., Segal, M., Shlens, J., Vijayanarasimhan, S., and
Yagnik, J.

</span>
<span class="ltx_bibblock">Fast, accurate detection of 100,000 object classes on a single
machine.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition</em>, pp.  1814–1821, 2013.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Desai et al. (2016)</span>
<span class="ltx_bibblock">
Desai, A., Ghashami, M., and Phillips, J. M.

</span>
<span class="ltx_bibblock">Improved practical matrix sketching with guarantees.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Knowledge and Data Engineering</em>,
28(7):1678–1690, 2016.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Drineas et al. (2006a)</span>
<span class="ltx_bibblock">
Drineas, P., Kannan, R., and Mahoney, M. W.

</span>
<span class="ltx_bibblock">Fast Monte Carlo Algorithms for Matrices I: Approximating
Matrix Multiplication.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">SIAM Journal on Computing</em>, 36(1):132–157,
January 2006a.

</span>
<span class="ltx_bibblock">ISSN 0097-5397, 1095-7111.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1137/S0097539704442684</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://epubs.siam.org/doi/10.1137/S0097539704442684" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://epubs.siam.org/doi/10.1137/S0097539704442684</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Drineas et al. (2006b)</span>
<span class="ltx_bibblock">
Drineas, P., Kannan, R., and Mahoney, M. W.

</span>
<span class="ltx_bibblock">Fast Monte Carlo Algorithms for Matrices II: Computing a
Low-Rank Approximation to a Matrix.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">SIAM Journal on Computing</em>, 36(1):158–183,
January 2006b.

</span>
<span class="ltx_bibblock">ISSN 0097-5397, 1095-7111.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1137/S0097539704442696</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://epubs.siam.org/doi/10.1137/S0097539704442696" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://epubs.siam.org/doi/10.1137/S0097539704442696</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Drineas et al. (2006c)</span>
<span class="ltx_bibblock">
Drineas, P., Kannan, R., and Mahoney, M. W.

</span>
<span class="ltx_bibblock">Fast Monte Carlo Algorithms for Matrices III: Computing a
Compressed Approximate Matrix Decomposition.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">SIAM Journal on Computing</em>, 36(1):184–206,
January 2006c.

</span>
<span class="ltx_bibblock">ISSN 0097-5397, 1095-7111.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1137/S0097539704442702</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://epubs.siam.org/doi/10.1137/S0097539704442702" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://epubs.siam.org/doi/10.1137/S0097539704442702</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dutta et al. (2016)</span>
<span class="ltx_bibblock">
Dutta, S., Cadambe, V., and Grover, P.

</span>
<span class="ltx_bibblock">Short-dot: Computing large linear transforms distributedly using
coded short dot products.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Advances In Neural Information Processing Systems</em>, pp. 2100–2108, 2016.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Eckart &amp; Young (1936)</span>
<span class="ltx_bibblock">
Eckart, C. and Young, G.

</span>
<span class="ltx_bibblock">The approximation of one matrix by another of lower rank.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Psychometrika</em>, 1(3):211–218, 1936.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fei-Fei et al. (2004)</span>
<span class="ltx_bibblock">
Fei-Fei, L., Fergus, R., and Perona, P.

</span>
<span class="ltx_bibblock">Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categories.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">2004 conference on computer vision and pattern recognition
workshop</em>, pp.  178–178. IEEE, 2004.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Francis &amp; Raimond (2018a)</span>
<span class="ltx_bibblock">
Francis, D. P. and Raimond, K.

</span>
<span class="ltx_bibblock">An improvement of the parameterized frequent directions algorithm.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Data Mining and Knowledge Discovery</em>, 32(2):453–482, March 2018a.

</span>
<span class="ltx_bibblock">ISSN 1384-5810, 1573-756X.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1007/s10618-017-0542-x</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://link.springer.com/10.1007/s10618-017-0542-x" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://link.springer.com/10.1007/s10618-017-0542-x</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Francis &amp; Raimond (2018b)</span>
<span class="ltx_bibblock">
Francis, D. P. and Raimond, K.

</span>
<span class="ltx_bibblock">A practical streaming approximate matrix multiplication algorithm.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Journal of King Saud University - Computer and Information
Sciences</em>, September 2018b.

</span>
<span class="ltx_bibblock">ISSN 13191578.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1016/j.jksuci.2018.09.010</span>.

</span>
<span class="ltx_bibblock">URL
<a target="_blank" href="https://linkinghub.elsevier.com/retrieve/pii/S1319157818306396" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://linkinghub.elsevier.com/retrieve/pii/S1319157818306396</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ge et al. (2014)</span>
<span class="ltx_bibblock">
Ge, T., He, K., Ke, Q., and Sun, J.

</span>
<span class="ltx_bibblock">Optimized product quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">IEEE transactions on pattern analysis and machine
intelligence</em>, 36(4):744–755, 2014.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geifman (2018)</span>
<span class="ltx_bibblock">
Geifman, Y.

</span>
<span class="ltx_bibblock">cifar-vgg, 3 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/geifmany/cifar-vgg" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/geifmany/cifar-vgg</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ghashami et al. (2016)</span>
<span class="ltx_bibblock">
Ghashami, M., Liberty, E., Phillips, J. M., and Woodruff, D. P.

</span>
<span class="ltx_bibblock">Frequent Directions: Simple and Deterministic Matrix
Sketching.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">SIAM Journal on Computing</em>, 45(5):1762–1792, January 2016.

</span>
<span class="ltx_bibblock">ISSN 0097-5397, 1095-7111.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1137/15M1009718</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://epubs.siam.org/doi/10.1137/15M1009718" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://epubs.siam.org/doi/10.1137/15M1009718</a>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guennebaud et al. (2010)</span>
<span class="ltx_bibblock">
Guennebaud, G., Jacob, B., et al.

</span>
<span class="ltx_bibblock">Eigen v3.

</span>
<span class="ltx_bibblock">http://eigen.tuxfamily.org, 2010.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gupta et al. (2017)</span>
<span class="ltx_bibblock">
Gupta, C., Suggala, A. S., Goyal, A., Simhadri, H. V., Paranjape, B., Kumar,
A., Goyal, S., Udupa, R., Varma, M., and Jain, P.

</span>
<span class="ltx_bibblock">Protonn: Compressed and accurate knn for resource-scarce devices.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 34th International Conference on Machine
Learning-Volume 70</em>, pp.  1331–1340. JMLR. org, 2017.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2016)</span>
<span class="ltx_bibblock">
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally,
W. J.

</span>
<span class="ltx_bibblock">Eie: efficient inference engine on compressed deep neural network.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 43rd International Symposium on Computer
Architecture</em>, pp.  243–254. IEEE Press, 2016.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2016a)</span>
<span class="ltx_bibblock">
He, K., Zhang, X., Ren, S., and Sun, J.

</span>
<span class="ltx_bibblock">Deep residual learning for image recognition.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and
pattern recognition</em>, pp.  770–778, 2016a.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2016b)</span>
<span class="ltx_bibblock">
He, K., Zhang, X., Ren, S., and Sun, J.

</span>
<span class="ltx_bibblock">Identity mappings in deep residual networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">European conference on computer vision</em>, pp.  630–645.
Springer, 2016b.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2017)</span>
<span class="ltx_bibblock">
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q.

</span>
<span class="ltx_bibblock">Densely connected convolutional networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and
pattern recognition</em>, pp.  4700–4708, 2017.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang (2019)</span>
<span class="ltx_bibblock">
Huang, Z.

</span>
<span class="ltx_bibblock">Near Optimal Frequent Directions for Sketching Dense and
Sparse Matrices.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 20(1):23, February 2019.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Indyk &amp; Motwani (1998)</span>
<span class="ltx_bibblock">
Indyk, P. and Motwani, R.

</span>
<span class="ltx_bibblock">Approximate nearest neighbors: towards removing the curse of
dimensionality.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Proceedings of the thirtieth annual ACM symposium on Theory
of computing</em>, pp.  604–613, 1998.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Irony et al. (2004)</span>
<span class="ltx_bibblock">
Irony, D., Toledo, S., and Tiskin, A.

</span>
<span class="ltx_bibblock">Communication lower bounds for distributed-memory matrix
multiplication.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Journal of Parallel and Distributed Computing</em>, 64(9):1017–1026, 2004.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jegou et al. (2011)</span>
<span class="ltx_bibblock">
Jegou, H., Douze, M., and Schmid, C.

</span>
<span class="ltx_bibblock">Product quantization for nearest neighbor search.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">IEEE transactions on pattern analysis and machine
intelligence</em>, 33(1):117–128, 2011.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ji et al. (2012)</span>
<span class="ltx_bibblock">
Ji, J., Li, J., Yan, S., Zhang, B., and Tian, Q.

</span>
<span class="ltx_bibblock">Super-bit locality-sensitive hashing.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, pp. 108–116, 2012.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jouppi et al. (2017)</span>
<span class="ltx_bibblock">
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R.,
Bates, S., Bhatia, S., Boden, N., Borchers, A., et al.

</span>
<span class="ltx_bibblock">In-datacenter performance analysis of a tensor processing unit.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 44th Annual International Symposium on
Computer Architecture</em>, pp.  1–12. ACM, 2017.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kakade et al. (2009)</span>
<span class="ltx_bibblock">
Kakade, S. M., Sridharan, K., and Tewari, A.

</span>
<span class="ltx_bibblock">On the complexity of linear prediction: Risk bounds, margin bounds,
and regularization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, pp. 793–800, 2009.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khudia et al. (2018)</span>
<span class="ltx_bibblock">
Khudia, D., Basu, P., and Deng, S.

</span>
<span class="ltx_bibblock">Open-sourcing fbgemm for state-of-the-art server-side inference,
2018.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krizhevsky et al. (2009)</span>
<span class="ltx_bibblock">
Krizhevsky, A., Hinton, G., et al.

</span>
<span class="ltx_bibblock">Learning multiple layers of features from tiny images.

</span>
<span class="ltx_bibblock">Technical report, Citeseer, 2009.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kusner et al. (2014)</span>
<span class="ltx_bibblock">
Kusner, M., Tyree, S., Weinberger, K., and Agrawal, K.

</span>
<span class="ltx_bibblock">Stochastic neighbor compression.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 622–630, 2014.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kyrillidis et al. (2014)</span>
<span class="ltx_bibblock">
Kyrillidis, A., Vlachos, M., and Zouzias, A.

</span>
<span class="ltx_bibblock">Approximate Matrix Multiplication with Application to Linear
Embeddings.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv:1403.7683 [cs, math, stat]</em>, March 2014.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1403.7683" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1403.7683</a>.

</span>
<span class="ltx_bibblock">arXiv: 1403.7683.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liberty (2012)</span>
<span class="ltx_bibblock">
Liberty, E.

</span>
<span class="ltx_bibblock">Simple and Deterministic Matrix Sketching.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">arXiv:1206.0594 [cs]</em>, June 2012.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1206.0594" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1206.0594</a>.

</span>
<span class="ltx_bibblock">arXiv: 1206.0594.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2016)</span>
<span class="ltx_bibblock">
Liu, S., Shao, J., and Lu, H.

</span>
<span class="ltx_bibblock">Generalized Residual Vector Quantization for Large Scale Data.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Proceedings - IEEE International Conference on Multimedia and
Expo</em>, 2016-Augus, 2016.

</span>
<span class="ltx_bibblock">ISSN 1945788X.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/ICME.2016.7552944</span>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Luo et al. (2019)</span>
<span class="ltx_bibblock">
Luo, L., Chen, C., Zhang, Z., Li, W.-J., and Zhang, T.

</span>
<span class="ltx_bibblock">Robust Frequent Directions with Application in Online
Learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 20(1):41, February 2019.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mairal et al. (2009)</span>
<span class="ltx_bibblock">
Mairal, J., Bach, F., Ponce, J., and Sapiro, G.

</span>
<span class="ltx_bibblock">Online dictionary learning for sparse coding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 26th annual international conference on
machine learning</em>, pp.  689–696, 2009.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Manne &amp; Pal (2014)</span>
<span class="ltx_bibblock">
Manne, S. and Pal, M.

</span>
<span class="ltx_bibblock">Fast Approximate Matrix Multiplication by Solving Linear
Systems.

</span>
<span class="ltx_bibblock"><em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">arXiv:1408.4230 [cs]</em>, August 2014.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1408.4230" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1408.4230</a>.

</span>
<span class="ltx_bibblock">arXiv: 1408.4230.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Martinez et al. (2014)</span>
<span class="ltx_bibblock">
Martinez, J., Hoos, H. H., and Little, J. J.

</span>
<span class="ltx_bibblock">Stacked quantizers for compositional vector compression.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1411.2173</em>, 2014.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Martinez et al. (2016)</span>
<span class="ltx_bibblock">
Martinez, J., Clement, J., Hoos, H. H., and Little, J. J.

</span>
<span class="ltx_bibblock">Revisiting additive quantization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">European Conference on Computer Vision</em>, pp.  137–153.
Springer, 2016.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mroueh et al. (2016)</span>
<span class="ltx_bibblock">
Mroueh, Y., Marcheret, E., and Goel, V.

</span>
<span class="ltx_bibblock">Co-Occuring Directions Sketching for Approximate Matrix
Multiply.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">arXiv:1610.07686 [cs]</em>, October 2016.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1610.07686" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1610.07686</a>.

</span>
<span class="ltx_bibblock">arXiv: 1610.07686.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nelson &amp; Nguyên (2013)</span>
<span class="ltx_bibblock">
Nelson, J. and Nguyên, H. L.

</span>
<span class="ltx_bibblock">Osnap: Faster numerical linear algebra algorithms via sparser
subspace embeddings.

</span>
<span class="ltx_bibblock">In <em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">2013 ieee 54th annual symposium on foundations of computer
science</em>, pp.  117–126. IEEE, 2013.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pagh (2013)</span>
<span class="ltx_bibblock">
Pagh, R.

</span>
<span class="ltx_bibblock">Compressed matrix multiplication.

</span>
<span class="ltx_bibblock"><em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">ACM Transactions on Computation Theory</em>, 5(3):1–17, August 2013.

</span>
<span class="ltx_bibblock">ISSN 19423454.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1145/2493252.2493254</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dl.acm.org/citation.cfm?doid=2493252.2493254" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dl.acm.org/citation.cfm?doid=2493252.2493254</a>.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Parashar et al. (2017)</span>
<span class="ltx_bibblock">
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany,
B., Emer, J., Keckler, S. W., and Dally, W. J.

</span>
<span class="ltx_bibblock">Scnn: An accelerator for compressed-sparse convolutional neural
networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">ACM SIGARCH Computer Architecture News</em>, 45(2):27–40, 2017.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paszke et al. (2017)</span>
<span class="ltx_bibblock">
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z.,
Desmaison, A., Antiga, L., and Lerer, A.

</span>
<span class="ltx_bibblock">Automatic differentiation in pytorch.

</span>
<span class="ltx_bibblock">2017.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sarlos (2006)</span>
<span class="ltx_bibblock">
Sarlos, T.

</span>
<span class="ltx_bibblock">Improved Approximation Algorithms for Large Matrices via
Random Projections.

</span>
<span class="ltx_bibblock">In <em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">2006 47th Annual IEEE Symposium on Foundations of
Computer Science (FOCS’06)</em>, pp.  143–152, Berkeley, CA, October
2006. IEEE.

</span>
<span class="ltx_bibblock">ISBN 978-0-7695-2720-8.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/FOCS.2006.37</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://ieeexplore.ieee.org/document/4031351/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ieeexplore.ieee.org/document/4031351/</a>.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Spring &amp; Shrivastava (2017)</span>
<span class="ltx_bibblock">
Spring, R. and Shrivastava, A.

</span>
<span class="ltx_bibblock">Scalable and sustainable deep learning via randomized hashing.

</span>
<span class="ltx_bibblock">In <em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 23rd ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining</em>, pp.  445–454, 2017.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Teng &amp; Chu (2019)</span>
<span class="ltx_bibblock">
Teng, D. and Chu, D.

</span>
<span class="ltx_bibblock">A Fast Frequent Directions Algorithm for Low Rank
Approximation.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Pattern Analysis and Machine
Intelligence</em>, 41(6):1279–1293, June 2019.

</span>
<span class="ltx_bibblock">ISSN 0162-8828, 2160-9292, 1939-3539.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/TPAMI.2018.2839198</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://ieeexplore.ieee.org/document/8362693/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ieeexplore.ieee.org/document/8362693/</a>.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2014a)</span>
<span class="ltx_bibblock">
Wang, J., Shen, H. T., Song, J., and Ji, J.

</span>
<span class="ltx_bibblock">Hashing for similarity search: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1408.2927</em>, 2014a.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2014b)</span>
<span class="ltx_bibblock">
Wang, J., Shen, H. T., Yan, S., Yu, N., Li, S., and Wang, J.

</span>
<span class="ltx_bibblock">Optimized distances for binary code ranking.

</span>
<span class="ltx_bibblock">In <em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 22nd ACM international conference on
Multimedia</em>, pp.  517–526, 2014b.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2016a)</span>
<span class="ltx_bibblock">
Wang, J., Liu, W., Kumar, S., and Chang, S.-F.

</span>
<span class="ltx_bibblock">Learning to hash for indexing big data—a survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib66.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE</em>, 104(1):34–57,
2016a.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2016b)</span>
<span class="ltx_bibblock">
Wang, W., Chen, C., Chen, W., Rai, P., and Carin, L.

</span>
<span class="ltx_bibblock">Deep metric learning with data summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">Joint European Conference on Machine Learning and Knowledge
Discovery in Databases</em>, pp.  777–794. Springer, 2016b.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2019)</span>
<span class="ltx_bibblock">
Wu, C.-J., Brooks, D., Chen, K., Chen, D., Choudhury, S., Dukhan, M.,
Hazelwood, K., Isaac, E., Jia, Y., Jia, B., et al.

</span>
<span class="ltx_bibblock">Machine learning at facebook: Understanding inference at the edge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">2019 IEEE International Symposium on High Performance
Computer Architecture (HPCA)</em>, pp.  331–344. IEEE, 2019.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ye et al. (2016)</span>
<span class="ltx_bibblock">
Ye, Q., Luo, L., and Zhang, Z.

</span>
<span class="ltx_bibblock">Frequent Direction Algorithms for Approximate Matrix
Multiplication with Applications in CCA.

</span>
<span class="ltx_bibblock">In <em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">IJCAI</em>, pp.  7, 2016.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2017)</span>
<span class="ltx_bibblock">
Yu, Q., Maddah-Ali, M., and Avestimehr, S.

</span>
<span class="ltx_bibblock">Polynomial codes: an optimal design for high-dimensional coded matrix
multiplication.

</span>
<span class="ltx_bibblock">In <em id="bib.bib70.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, pp. 4403–4413, 2017.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2020)</span>
<span class="ltx_bibblock">
Yu, Q., Ali, M., and Avestimehr, A. S.

</span>
<span class="ltx_bibblock">Straggler mitigation in distributed matrix multiplication:
Fundamental limits and optimal coding.

</span>
<span class="ltx_bibblock"><em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Information Theory</em>, 2020.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2014)</span>
<span class="ltx_bibblock">
Zhang, T., Du, C., and Wang, J.

</span>
<span class="ltx_bibblock">Composite Quantization for Approximate Nearest Neighbor Search.

</span>
<span class="ltx_bibblock"><em id="bib.bib72.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 31st International Conference on Machine
Learning (ICML-14)</em>, 32:838–846, 2014.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2015)</span>
<span class="ltx_bibblock">
Zhang, T., Qi, G.-J., Tang, J., and Wang, J.

</span>
<span class="ltx_bibblock">Sparse composite quantization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib73.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition</em>, pp.  4548–4556, 2015.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2017)</span>
<span class="ltx_bibblock">
Zhong, K., Guo, R., Kumar, S., Yan, B., Simcha, D., and Dhillon, I.

</span>
<span class="ltx_bibblock">Fast classification with binary prototypes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib74.1.1" class="ltx_emph ltx_font_italic">Artificial Intelligence and Statistics</em>, pp.  1255–1263,
2017.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zou &amp; Xue (2018)</span>
<span class="ltx_bibblock">
Zou, H. and Xue, L.

</span>
<span class="ltx_bibblock">A selective overview of sparse principal component analysis.

</span>
<span class="ltx_bibblock"><em id="bib.bib75.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE</em>, 106(8):1311–1320, 2018.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Quantizing Lookup Tables</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.8" class="ltx_p">Since the lookup table entries naturally occupy more than 8 bits even for 8-bit data (since products of 8-bit values require 16 bits), some means of quantizing these entries is necessary to enable vectorization. Unfortunately, existing quantization methods are not applicable to our problem setting. The scheme of <cite class="ltx_cite ltx_citemacro_citet">Blalock &amp; Guttag (<a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite> requires knowledge of <math id="A1.p1.1.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A1.p1.1.m1.1a"><mi id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1b"><ci id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1c">\bm{B}</annotation></semantics></math> at training time, while the scheme of <cite class="ltx_cite ltx_citemacro_citet">André et al. (<a href="#bib.bib4" title="" class="ltx_ref">2017</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">André et al. (<a href="#bib.bib5" title="" class="ltx_ref">2019</a>)</cite> is only applicable for nearest-neighbor search. We instead use the following approach, where <math id="A1.p1.2.m2.1" class="ltx_Math" alttext="\bm{T}\in\mathbb{R}^{M\times C\times K}" display="inline"><semantics id="A1.p1.2.m2.1a"><mrow id="A1.p1.2.m2.1.1" xref="A1.p1.2.m2.1.1.cmml"><mi id="A1.p1.2.m2.1.1.2" xref="A1.p1.2.m2.1.1.2.cmml">𝑻</mi><mo id="A1.p1.2.m2.1.1.1" xref="A1.p1.2.m2.1.1.1.cmml">∈</mo><msup id="A1.p1.2.m2.1.1.3" xref="A1.p1.2.m2.1.1.3.cmml"><mi id="A1.p1.2.m2.1.1.3.2" xref="A1.p1.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="A1.p1.2.m2.1.1.3.3" xref="A1.p1.2.m2.1.1.3.3.cmml"><mi id="A1.p1.2.m2.1.1.3.3.2" xref="A1.p1.2.m2.1.1.3.3.2.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="A1.p1.2.m2.1.1.3.3.1" xref="A1.p1.2.m2.1.1.3.3.1.cmml">×</mo><mi id="A1.p1.2.m2.1.1.3.3.3" xref="A1.p1.2.m2.1.1.3.3.3.cmml">C</mi><mo lspace="0.222em" rspace="0.222em" id="A1.p1.2.m2.1.1.3.3.1a" xref="A1.p1.2.m2.1.1.3.3.1.cmml">×</mo><mi id="A1.p1.2.m2.1.1.3.3.4" xref="A1.p1.2.m2.1.1.3.3.4.cmml">K</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.1b"><apply id="A1.p1.2.m2.1.1.cmml" xref="A1.p1.2.m2.1.1"><in id="A1.p1.2.m2.1.1.1.cmml" xref="A1.p1.2.m2.1.1.1"></in><ci id="A1.p1.2.m2.1.1.2.cmml" xref="A1.p1.2.m2.1.1.2">𝑻</ci><apply id="A1.p1.2.m2.1.1.3.cmml" xref="A1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="A1.p1.2.m2.1.1.3.1.cmml" xref="A1.p1.2.m2.1.1.3">superscript</csymbol><ci id="A1.p1.2.m2.1.1.3.2.cmml" xref="A1.p1.2.m2.1.1.3.2">ℝ</ci><apply id="A1.p1.2.m2.1.1.3.3.cmml" xref="A1.p1.2.m2.1.1.3.3"><times id="A1.p1.2.m2.1.1.3.3.1.cmml" xref="A1.p1.2.m2.1.1.3.3.1"></times><ci id="A1.p1.2.m2.1.1.3.3.2.cmml" xref="A1.p1.2.m2.1.1.3.3.2">𝑀</ci><ci id="A1.p1.2.m2.1.1.3.3.3.cmml" xref="A1.p1.2.m2.1.1.3.3.3">𝐶</ci><ci id="A1.p1.2.m2.1.1.3.3.4.cmml" xref="A1.p1.2.m2.1.1.3.3.4">𝐾</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.1c">\bm{T}\in\mathbb{R}^{M\times C\times K}</annotation></semantics></math> is the tensor of lookup tables for all <math id="A1.p1.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="A1.p1.3.m3.1a"><mi id="A1.p1.3.m3.1.1" xref="A1.p1.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="A1.p1.3.m3.1b"><ci id="A1.p1.3.m3.1.1.cmml" xref="A1.p1.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.3.m3.1c">M</annotation></semantics></math> columns of <math id="A1.p1.4.m4.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A1.p1.4.m4.1a"><mi id="A1.p1.4.m4.1.1" xref="A1.p1.4.m4.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A1.p1.4.m4.1b"><ci id="A1.p1.4.m4.1.1.cmml" xref="A1.p1.4.m4.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.4.m4.1c">\bm{B}</annotation></semantics></math>, <math id="A1.p1.5.m5.1" class="ltx_Math" alttext="\bm{T}^{q}" display="inline"><semantics id="A1.p1.5.m5.1a"><msup id="A1.p1.5.m5.1.1" xref="A1.p1.5.m5.1.1.cmml"><mi id="A1.p1.5.m5.1.1.2" xref="A1.p1.5.m5.1.1.2.cmml">𝑻</mi><mi id="A1.p1.5.m5.1.1.3" xref="A1.p1.5.m5.1.1.3.cmml">q</mi></msup><annotation-xml encoding="MathML-Content" id="A1.p1.5.m5.1b"><apply id="A1.p1.5.m5.1.1.cmml" xref="A1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A1.p1.5.m5.1.1.1.cmml" xref="A1.p1.5.m5.1.1">superscript</csymbol><ci id="A1.p1.5.m5.1.1.2.cmml" xref="A1.p1.5.m5.1.1.2">𝑻</ci><ci id="A1.p1.5.m5.1.1.3.cmml" xref="A1.p1.5.m5.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.5.m5.1c">\bm{T}^{q}</annotation></semantics></math> is the quantized version of <math id="A1.p1.6.m6.1" class="ltx_Math" alttext="\bm{T}" display="inline"><semantics id="A1.p1.6.m6.1a"><mi id="A1.p1.6.m6.1.1" xref="A1.p1.6.m6.1.1.cmml">𝑻</mi><annotation-xml encoding="MathML-Content" id="A1.p1.6.m6.1b"><ci id="A1.p1.6.m6.1.1.cmml" xref="A1.p1.6.m6.1.1">𝑻</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.6.m6.1c">\bm{T}</annotation></semantics></math>, <math id="A1.p1.7.m7.1" class="ltx_Math" alttext="\bm{\delta}\in\mathbb{R}^{C}" display="inline"><semantics id="A1.p1.7.m7.1a"><mrow id="A1.p1.7.m7.1.1" xref="A1.p1.7.m7.1.1.cmml"><mi id="A1.p1.7.m7.1.1.2" xref="A1.p1.7.m7.1.1.2.cmml">𝜹</mi><mo id="A1.p1.7.m7.1.1.1" xref="A1.p1.7.m7.1.1.1.cmml">∈</mo><msup id="A1.p1.7.m7.1.1.3" xref="A1.p1.7.m7.1.1.3.cmml"><mi id="A1.p1.7.m7.1.1.3.2" xref="A1.p1.7.m7.1.1.3.2.cmml">ℝ</mi><mi id="A1.p1.7.m7.1.1.3.3" xref="A1.p1.7.m7.1.1.3.3.cmml">C</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.7.m7.1b"><apply id="A1.p1.7.m7.1.1.cmml" xref="A1.p1.7.m7.1.1"><in id="A1.p1.7.m7.1.1.1.cmml" xref="A1.p1.7.m7.1.1.1"></in><ci id="A1.p1.7.m7.1.1.2.cmml" xref="A1.p1.7.m7.1.1.2">𝜹</ci><apply id="A1.p1.7.m7.1.1.3.cmml" xref="A1.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="A1.p1.7.m7.1.1.3.1.cmml" xref="A1.p1.7.m7.1.1.3">superscript</csymbol><ci id="A1.p1.7.m7.1.1.3.2.cmml" xref="A1.p1.7.m7.1.1.3.2">ℝ</ci><ci id="A1.p1.7.m7.1.1.3.3.cmml" xref="A1.p1.7.m7.1.1.3.3">𝐶</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.7.m7.1c">\bm{\delta}\in\mathbb{R}^{C}</annotation></semantics></math> is a vector of table-specific offsets, and <math id="A1.p1.8.m8.1" class="ltx_Math" alttext="\alpha^{-1}" display="inline"><semantics id="A1.p1.8.m8.1a"><msup id="A1.p1.8.m8.1.1" xref="A1.p1.8.m8.1.1.cmml"><mi id="A1.p1.8.m8.1.1.2" xref="A1.p1.8.m8.1.1.2.cmml">α</mi><mrow id="A1.p1.8.m8.1.1.3" xref="A1.p1.8.m8.1.1.3.cmml"><mo id="A1.p1.8.m8.1.1.3a" xref="A1.p1.8.m8.1.1.3.cmml">−</mo><mn id="A1.p1.8.m8.1.1.3.2" xref="A1.p1.8.m8.1.1.3.2.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A1.p1.8.m8.1b"><apply id="A1.p1.8.m8.1.1.cmml" xref="A1.p1.8.m8.1.1"><csymbol cd="ambiguous" id="A1.p1.8.m8.1.1.1.cmml" xref="A1.p1.8.m8.1.1">superscript</csymbol><ci id="A1.p1.8.m8.1.1.2.cmml" xref="A1.p1.8.m8.1.1.2">𝛼</ci><apply id="A1.p1.8.m8.1.1.3.cmml" xref="A1.p1.8.m8.1.1.3"><minus id="A1.p1.8.m8.1.1.3.1.cmml" xref="A1.p1.8.m8.1.1.3"></minus><cn type="integer" id="A1.p1.8.m8.1.1.3.2.cmml" xref="A1.p1.8.m8.1.1.3.2">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.8.m8.1c">\alpha^{-1}</annotation></semantics></math> is an overall scale factor:</p>
<table id="A6.EGx11" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E12"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E12.m1.1" class="ltx_Math" alttext="\displaystyle\bm{\delta}_{c}" display="inline"><semantics id="A1.E12.m1.1a"><msub id="A1.E12.m1.1.1" xref="A1.E12.m1.1.1.cmml"><mi id="A1.E12.m1.1.1.2" xref="A1.E12.m1.1.1.2.cmml">𝜹</mi><mi id="A1.E12.m1.1.1.3" xref="A1.E12.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="A1.E12.m1.1b"><apply id="A1.E12.m1.1.1.cmml" xref="A1.E12.m1.1.1"><csymbol cd="ambiguous" id="A1.E12.m1.1.1.1.cmml" xref="A1.E12.m1.1.1">subscript</csymbol><ci id="A1.E12.m1.1.1.2.cmml" xref="A1.E12.m1.1.1.2">𝜹</ci><ci id="A1.E12.m1.1.1.3.cmml" xref="A1.E12.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E12.m1.1c">\displaystyle\bm{\delta}_{c}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E12.m2.5" class="ltx_Math" alttext="\displaystyle\triangleq\min_{m,k}\text{ }\bm{T}_{m,c,k}" display="inline"><semantics id="A1.E12.m2.5a"><mrow id="A1.E12.m2.5.6" xref="A1.E12.m2.5.6.cmml"><mi id="A1.E12.m2.5.6.2" xref="A1.E12.m2.5.6.2.cmml"></mi><mo id="A1.E12.m2.5.6.1" xref="A1.E12.m2.5.6.1.cmml">≜</mo><mrow id="A1.E12.m2.5.6.3" xref="A1.E12.m2.5.6.3.cmml"><munder id="A1.E12.m2.5.6.3.1" xref="A1.E12.m2.5.6.3.1.cmml"><mi id="A1.E12.m2.5.6.3.1.2" xref="A1.E12.m2.5.6.3.1.2.cmml">min</mi><mrow id="A1.E12.m2.2.2.2.4" xref="A1.E12.m2.2.2.2.3.cmml"><mi id="A1.E12.m2.1.1.1.1" xref="A1.E12.m2.1.1.1.1.cmml">m</mi><mo id="A1.E12.m2.2.2.2.4.1" xref="A1.E12.m2.2.2.2.3.cmml">,</mo><mi id="A1.E12.m2.2.2.2.2" xref="A1.E12.m2.2.2.2.2.cmml">k</mi></mrow></munder><mo lspace="0.167em" id="A1.E12.m2.5.6.3a" xref="A1.E12.m2.5.6.3.cmml">⁡</mo><mrow id="A1.E12.m2.5.6.3.2" xref="A1.E12.m2.5.6.3.2.cmml"><mtext id="A1.E12.m2.5.6.3.2.2" xref="A1.E12.m2.5.6.3.2.2a.cmml"> </mtext><mo lspace="0em" rspace="0em" id="A1.E12.m2.5.6.3.2.1" xref="A1.E12.m2.5.6.3.2.1.cmml">​</mo><msub id="A1.E12.m2.5.6.3.2.3" xref="A1.E12.m2.5.6.3.2.3.cmml"><mi id="A1.E12.m2.5.6.3.2.3.2" xref="A1.E12.m2.5.6.3.2.3.2.cmml">𝑻</mi><mrow id="A1.E12.m2.5.5.3.5" xref="A1.E12.m2.5.5.3.4.cmml"><mi id="A1.E12.m2.3.3.1.1" xref="A1.E12.m2.3.3.1.1.cmml">m</mi><mo id="A1.E12.m2.5.5.3.5.1" xref="A1.E12.m2.5.5.3.4.cmml">,</mo><mi id="A1.E12.m2.4.4.2.2" xref="A1.E12.m2.4.4.2.2.cmml">c</mi><mo id="A1.E12.m2.5.5.3.5.2" xref="A1.E12.m2.5.5.3.4.cmml">,</mo><mi id="A1.E12.m2.5.5.3.3" xref="A1.E12.m2.5.5.3.3.cmml">k</mi></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E12.m2.5b"><apply id="A1.E12.m2.5.6.cmml" xref="A1.E12.m2.5.6"><ci id="A1.E12.m2.5.6.1.cmml" xref="A1.E12.m2.5.6.1">≜</ci><csymbol cd="latexml" id="A1.E12.m2.5.6.2.cmml" xref="A1.E12.m2.5.6.2">absent</csymbol><apply id="A1.E12.m2.5.6.3.cmml" xref="A1.E12.m2.5.6.3"><apply id="A1.E12.m2.5.6.3.1.cmml" xref="A1.E12.m2.5.6.3.1"><csymbol cd="ambiguous" id="A1.E12.m2.5.6.3.1.1.cmml" xref="A1.E12.m2.5.6.3.1">subscript</csymbol><min id="A1.E12.m2.5.6.3.1.2.cmml" xref="A1.E12.m2.5.6.3.1.2"></min><list id="A1.E12.m2.2.2.2.3.cmml" xref="A1.E12.m2.2.2.2.4"><ci id="A1.E12.m2.1.1.1.1.cmml" xref="A1.E12.m2.1.1.1.1">𝑚</ci><ci id="A1.E12.m2.2.2.2.2.cmml" xref="A1.E12.m2.2.2.2.2">𝑘</ci></list></apply><apply id="A1.E12.m2.5.6.3.2.cmml" xref="A1.E12.m2.5.6.3.2"><times id="A1.E12.m2.5.6.3.2.1.cmml" xref="A1.E12.m2.5.6.3.2.1"></times><ci id="A1.E12.m2.5.6.3.2.2a.cmml" xref="A1.E12.m2.5.6.3.2.2"><mtext id="A1.E12.m2.5.6.3.2.2.cmml" xref="A1.E12.m2.5.6.3.2.2"> </mtext></ci><apply id="A1.E12.m2.5.6.3.2.3.cmml" xref="A1.E12.m2.5.6.3.2.3"><csymbol cd="ambiguous" id="A1.E12.m2.5.6.3.2.3.1.cmml" xref="A1.E12.m2.5.6.3.2.3">subscript</csymbol><ci id="A1.E12.m2.5.6.3.2.3.2.cmml" xref="A1.E12.m2.5.6.3.2.3.2">𝑻</ci><list id="A1.E12.m2.5.5.3.4.cmml" xref="A1.E12.m2.5.5.3.5"><ci id="A1.E12.m2.3.3.1.1.cmml" xref="A1.E12.m2.3.3.1.1">𝑚</ci><ci id="A1.E12.m2.4.4.2.2.cmml" xref="A1.E12.m2.4.4.2.2">𝑐</ci><ci id="A1.E12.m2.5.5.3.3.cmml" xref="A1.E12.m2.5.5.3.3">𝑘</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E12.m2.5c">\displaystyle\triangleq\min_{m,k}\text{ }\bm{T}_{m,c,k}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(12)</span></td>
</tr></tbody>
<tbody id="A1.E13"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E13.m1.1" class="ltx_Math" alttext="\displaystyle\alpha^{-1}\triangleq 2^{l}\text{, }l" display="inline"><semantics id="A1.E13.m1.1a"><mrow id="A1.E13.m1.1.1" xref="A1.E13.m1.1.1.cmml"><msup id="A1.E13.m1.1.1.2" xref="A1.E13.m1.1.1.2.cmml"><mi id="A1.E13.m1.1.1.2.2" xref="A1.E13.m1.1.1.2.2.cmml">α</mi><mrow id="A1.E13.m1.1.1.2.3" xref="A1.E13.m1.1.1.2.3.cmml"><mo id="A1.E13.m1.1.1.2.3a" xref="A1.E13.m1.1.1.2.3.cmml">−</mo><mn id="A1.E13.m1.1.1.2.3.2" xref="A1.E13.m1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo id="A1.E13.m1.1.1.1" xref="A1.E13.m1.1.1.1.cmml">≜</mo><mrow id="A1.E13.m1.1.1.3" xref="A1.E13.m1.1.1.3.cmml"><msup id="A1.E13.m1.1.1.3.2" xref="A1.E13.m1.1.1.3.2.cmml"><mn id="A1.E13.m1.1.1.3.2.2" xref="A1.E13.m1.1.1.3.2.2.cmml">2</mn><mi id="A1.E13.m1.1.1.3.2.3" xref="A1.E13.m1.1.1.3.2.3.cmml">l</mi></msup><mo lspace="0em" rspace="0em" id="A1.E13.m1.1.1.3.1" xref="A1.E13.m1.1.1.3.1.cmml">​</mo><mtext id="A1.E13.m1.1.1.3.3" xref="A1.E13.m1.1.1.3.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="A1.E13.m1.1.1.3.1a" xref="A1.E13.m1.1.1.3.1.cmml">​</mo><mi id="A1.E13.m1.1.1.3.4" xref="A1.E13.m1.1.1.3.4.cmml">l</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E13.m1.1b"><apply id="A1.E13.m1.1.1.cmml" xref="A1.E13.m1.1.1"><ci id="A1.E13.m1.1.1.1.cmml" xref="A1.E13.m1.1.1.1">≜</ci><apply id="A1.E13.m1.1.1.2.cmml" xref="A1.E13.m1.1.1.2"><csymbol cd="ambiguous" id="A1.E13.m1.1.1.2.1.cmml" xref="A1.E13.m1.1.1.2">superscript</csymbol><ci id="A1.E13.m1.1.1.2.2.cmml" xref="A1.E13.m1.1.1.2.2">𝛼</ci><apply id="A1.E13.m1.1.1.2.3.cmml" xref="A1.E13.m1.1.1.2.3"><minus id="A1.E13.m1.1.1.2.3.1.cmml" xref="A1.E13.m1.1.1.2.3"></minus><cn type="integer" id="A1.E13.m1.1.1.2.3.2.cmml" xref="A1.E13.m1.1.1.2.3.2">1</cn></apply></apply><apply id="A1.E13.m1.1.1.3.cmml" xref="A1.E13.m1.1.1.3"><times id="A1.E13.m1.1.1.3.1.cmml" xref="A1.E13.m1.1.1.3.1"></times><apply id="A1.E13.m1.1.1.3.2.cmml" xref="A1.E13.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.E13.m1.1.1.3.2.1.cmml" xref="A1.E13.m1.1.1.3.2">superscript</csymbol><cn type="integer" id="A1.E13.m1.1.1.3.2.2.cmml" xref="A1.E13.m1.1.1.3.2.2">2</cn><ci id="A1.E13.m1.1.1.3.2.3.cmml" xref="A1.E13.m1.1.1.3.2.3">𝑙</ci></apply><ci id="A1.E13.m1.1.1.3.3a.cmml" xref="A1.E13.m1.1.1.3.3"><mtext id="A1.E13.m1.1.1.3.3.cmml" xref="A1.E13.m1.1.1.3.3">, </mtext></ci><ci id="A1.E13.m1.1.1.3.4.cmml" xref="A1.E13.m1.1.1.3.4">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E13.m1.1c">\displaystyle\alpha^{-1}\triangleq 2^{l}\text{, }l</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E13.m2.9" class="ltx_Math" alttext="\displaystyle=\max_{c}\left\lfloor\log_{2}\left(\frac{255}{\max_{m,k}(\bm{T}_{m,c,k}-\delta_{c})}\right)\right\rfloor" display="inline"><semantics id="A1.E13.m2.9a"><mrow id="A1.E13.m2.9.9" xref="A1.E13.m2.9.9.cmml"><mi id="A1.E13.m2.9.9.4" xref="A1.E13.m2.9.9.4.cmml"></mi><mo id="A1.E13.m2.9.9.3" xref="A1.E13.m2.9.9.3.cmml">=</mo><mrow id="A1.E13.m2.9.9.2.2" xref="A1.E13.m2.9.9.2.3.cmml"><munder id="A1.E13.m2.8.8.1.1.1" xref="A1.E13.m2.8.8.1.1.1.cmml"><mi id="A1.E13.m2.8.8.1.1.1.2" xref="A1.E13.m2.8.8.1.1.1.2.cmml">max</mi><mi id="A1.E13.m2.8.8.1.1.1.3" xref="A1.E13.m2.8.8.1.1.1.3.cmml">c</mi></munder><mo id="A1.E13.m2.9.9.2.2a" xref="A1.E13.m2.9.9.2.3.cmml">⁡</mo><mrow id="A1.E13.m2.9.9.2.2.2" xref="A1.E13.m2.9.9.2.3.cmml"><mo id="A1.E13.m2.9.9.2.2.2.2" xref="A1.E13.m2.9.9.2.3.cmml">⌊</mo><mrow id="A1.E13.m2.9.9.2.2.2.1.1" xref="A1.E13.m2.9.9.2.2.2.1.2.cmml"><msub id="A1.E13.m2.9.9.2.2.2.1.1.1" xref="A1.E13.m2.9.9.2.2.2.1.1.1.cmml"><mi id="A1.E13.m2.9.9.2.2.2.1.1.1.2" xref="A1.E13.m2.9.9.2.2.2.1.1.1.2.cmml">log</mi><mn id="A1.E13.m2.9.9.2.2.2.1.1.1.3" xref="A1.E13.m2.9.9.2.2.2.1.1.1.3.cmml">2</mn></msub><mo id="A1.E13.m2.9.9.2.2.2.1.1a" xref="A1.E13.m2.9.9.2.2.2.1.2.cmml">⁡</mo><mrow id="A1.E13.m2.9.9.2.2.2.1.1.2" xref="A1.E13.m2.9.9.2.2.2.1.2.cmml"><mo id="A1.E13.m2.9.9.2.2.2.1.1.2.1" xref="A1.E13.m2.9.9.2.2.2.1.2.cmml">(</mo><mstyle displaystyle="true" id="A1.E13.m2.7.7" xref="A1.E13.m2.7.7.cmml"><mfrac id="A1.E13.m2.7.7a" xref="A1.E13.m2.7.7.cmml"><mn id="A1.E13.m2.7.7.9" xref="A1.E13.m2.7.7.9.cmml">255</mn><mrow id="A1.E13.m2.7.7.7.7" xref="A1.E13.m2.7.7.7.8.cmml"><msub id="A1.E13.m2.6.6.6.6.1" xref="A1.E13.m2.6.6.6.6.1.cmml"><mi id="A1.E13.m2.6.6.6.6.1.2" xref="A1.E13.m2.6.6.6.6.1.2.cmml">max</mi><mrow id="A1.E13.m2.2.2.2.2.2.4" xref="A1.E13.m2.2.2.2.2.2.3.cmml"><mi id="A1.E13.m2.1.1.1.1.1.1" xref="A1.E13.m2.1.1.1.1.1.1.cmml">m</mi><mo id="A1.E13.m2.2.2.2.2.2.4.1" xref="A1.E13.m2.2.2.2.2.2.3.cmml">,</mo><mi id="A1.E13.m2.2.2.2.2.2.2" xref="A1.E13.m2.2.2.2.2.2.2.cmml">k</mi></mrow></msub><mo id="A1.E13.m2.7.7.7.7a" xref="A1.E13.m2.7.7.7.8.cmml">⁡</mo><mrow id="A1.E13.m2.7.7.7.7.2" xref="A1.E13.m2.7.7.7.8.cmml"><mo stretchy="false" id="A1.E13.m2.7.7.7.7.2.2" xref="A1.E13.m2.7.7.7.8.cmml">(</mo><mrow id="A1.E13.m2.7.7.7.7.2.1" xref="A1.E13.m2.7.7.7.7.2.1.cmml"><msub id="A1.E13.m2.7.7.7.7.2.1.2" xref="A1.E13.m2.7.7.7.7.2.1.2.cmml"><mi id="A1.E13.m2.7.7.7.7.2.1.2.2" xref="A1.E13.m2.7.7.7.7.2.1.2.2.cmml">𝑻</mi><mrow id="A1.E13.m2.5.5.5.5.3.5" xref="A1.E13.m2.5.5.5.5.3.4.cmml"><mi id="A1.E13.m2.3.3.3.3.1.1" xref="A1.E13.m2.3.3.3.3.1.1.cmml">m</mi><mo id="A1.E13.m2.5.5.5.5.3.5.1" xref="A1.E13.m2.5.5.5.5.3.4.cmml">,</mo><mi id="A1.E13.m2.4.4.4.4.2.2" xref="A1.E13.m2.4.4.4.4.2.2.cmml">c</mi><mo id="A1.E13.m2.5.5.5.5.3.5.2" xref="A1.E13.m2.5.5.5.5.3.4.cmml">,</mo><mi id="A1.E13.m2.5.5.5.5.3.3" xref="A1.E13.m2.5.5.5.5.3.3.cmml">k</mi></mrow></msub><mo id="A1.E13.m2.7.7.7.7.2.1.1" xref="A1.E13.m2.7.7.7.7.2.1.1.cmml">−</mo><msub id="A1.E13.m2.7.7.7.7.2.1.3" xref="A1.E13.m2.7.7.7.7.2.1.3.cmml"><mi id="A1.E13.m2.7.7.7.7.2.1.3.2" xref="A1.E13.m2.7.7.7.7.2.1.3.2.cmml">δ</mi><mi id="A1.E13.m2.7.7.7.7.2.1.3.3" xref="A1.E13.m2.7.7.7.7.2.1.3.3.cmml">c</mi></msub></mrow><mo stretchy="false" id="A1.E13.m2.7.7.7.7.2.3" xref="A1.E13.m2.7.7.7.8.cmml">)</mo></mrow></mrow></mfrac></mstyle><mo id="A1.E13.m2.9.9.2.2.2.1.1.2.2" xref="A1.E13.m2.9.9.2.2.2.1.2.cmml">)</mo></mrow></mrow><mo id="A1.E13.m2.9.9.2.2.2.3" xref="A1.E13.m2.9.9.2.3.cmml">⌋</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E13.m2.9b"><apply id="A1.E13.m2.9.9.cmml" xref="A1.E13.m2.9.9"><eq id="A1.E13.m2.9.9.3.cmml" xref="A1.E13.m2.9.9.3"></eq><csymbol cd="latexml" id="A1.E13.m2.9.9.4.cmml" xref="A1.E13.m2.9.9.4">absent</csymbol><apply id="A1.E13.m2.9.9.2.3.cmml" xref="A1.E13.m2.9.9.2.2"><apply id="A1.E13.m2.8.8.1.1.1.cmml" xref="A1.E13.m2.8.8.1.1.1"><csymbol cd="ambiguous" id="A1.E13.m2.8.8.1.1.1.1.cmml" xref="A1.E13.m2.8.8.1.1.1">subscript</csymbol><max id="A1.E13.m2.8.8.1.1.1.2.cmml" xref="A1.E13.m2.8.8.1.1.1.2"></max><ci id="A1.E13.m2.8.8.1.1.1.3.cmml" xref="A1.E13.m2.8.8.1.1.1.3">𝑐</ci></apply><apply id="A1.E13.m2.9.9.2.2.2.1.2.cmml" xref="A1.E13.m2.9.9.2.2.2.1.1"><apply id="A1.E13.m2.9.9.2.2.2.1.1.1.cmml" xref="A1.E13.m2.9.9.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.E13.m2.9.9.2.2.2.1.1.1.1.cmml" xref="A1.E13.m2.9.9.2.2.2.1.1.1">subscript</csymbol><log id="A1.E13.m2.9.9.2.2.2.1.1.1.2.cmml" xref="A1.E13.m2.9.9.2.2.2.1.1.1.2"></log><cn type="integer" id="A1.E13.m2.9.9.2.2.2.1.1.1.3.cmml" xref="A1.E13.m2.9.9.2.2.2.1.1.1.3">2</cn></apply><apply id="A1.E13.m2.7.7.cmml" xref="A1.E13.m2.7.7"><divide id="A1.E13.m2.7.7.8.cmml" xref="A1.E13.m2.7.7"></divide><cn type="integer" id="A1.E13.m2.7.7.9.cmml" xref="A1.E13.m2.7.7.9">255</cn><apply id="A1.E13.m2.7.7.7.8.cmml" xref="A1.E13.m2.7.7.7.7"><apply id="A1.E13.m2.6.6.6.6.1.cmml" xref="A1.E13.m2.6.6.6.6.1"><csymbol cd="ambiguous" id="A1.E13.m2.6.6.6.6.1.1.cmml" xref="A1.E13.m2.6.6.6.6.1">subscript</csymbol><max id="A1.E13.m2.6.6.6.6.1.2.cmml" xref="A1.E13.m2.6.6.6.6.1.2"></max><list id="A1.E13.m2.2.2.2.2.2.3.cmml" xref="A1.E13.m2.2.2.2.2.2.4"><ci id="A1.E13.m2.1.1.1.1.1.1.cmml" xref="A1.E13.m2.1.1.1.1.1.1">𝑚</ci><ci id="A1.E13.m2.2.2.2.2.2.2.cmml" xref="A1.E13.m2.2.2.2.2.2.2">𝑘</ci></list></apply><apply id="A1.E13.m2.7.7.7.7.2.1.cmml" xref="A1.E13.m2.7.7.7.7.2.1"><minus id="A1.E13.m2.7.7.7.7.2.1.1.cmml" xref="A1.E13.m2.7.7.7.7.2.1.1"></minus><apply id="A1.E13.m2.7.7.7.7.2.1.2.cmml" xref="A1.E13.m2.7.7.7.7.2.1.2"><csymbol cd="ambiguous" id="A1.E13.m2.7.7.7.7.2.1.2.1.cmml" xref="A1.E13.m2.7.7.7.7.2.1.2">subscript</csymbol><ci id="A1.E13.m2.7.7.7.7.2.1.2.2.cmml" xref="A1.E13.m2.7.7.7.7.2.1.2.2">𝑻</ci><list id="A1.E13.m2.5.5.5.5.3.4.cmml" xref="A1.E13.m2.5.5.5.5.3.5"><ci id="A1.E13.m2.3.3.3.3.1.1.cmml" xref="A1.E13.m2.3.3.3.3.1.1">𝑚</ci><ci id="A1.E13.m2.4.4.4.4.2.2.cmml" xref="A1.E13.m2.4.4.4.4.2.2">𝑐</ci><ci id="A1.E13.m2.5.5.5.5.3.3.cmml" xref="A1.E13.m2.5.5.5.5.3.3">𝑘</ci></list></apply><apply id="A1.E13.m2.7.7.7.7.2.1.3.cmml" xref="A1.E13.m2.7.7.7.7.2.1.3"><csymbol cd="ambiguous" id="A1.E13.m2.7.7.7.7.2.1.3.1.cmml" xref="A1.E13.m2.7.7.7.7.2.1.3">subscript</csymbol><ci id="A1.E13.m2.7.7.7.7.2.1.3.2.cmml" xref="A1.E13.m2.7.7.7.7.2.1.3.2">𝛿</ci><ci id="A1.E13.m2.7.7.7.7.2.1.3.3.cmml" xref="A1.E13.m2.7.7.7.7.2.1.3.3">𝑐</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E13.m2.9c">\displaystyle=\max_{c}\left\lfloor\log_{2}\left(\frac{255}{\max_{m,k}(\bm{T}_{m,c,k}-\delta_{c})}\right)\right\rfloor</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(13)</span></td>
</tr></tbody>
<tbody id="A1.E14"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E14.m1.3" class="ltx_Math" alttext="\displaystyle\bm{T}^{q}_{m,c,k}" display="inline"><semantics id="A1.E14.m1.3a"><msubsup id="A1.E14.m1.3.4" xref="A1.E14.m1.3.4.cmml"><mi id="A1.E14.m1.3.4.2.2" xref="A1.E14.m1.3.4.2.2.cmml">𝑻</mi><mrow id="A1.E14.m1.3.3.3.5" xref="A1.E14.m1.3.3.3.4.cmml"><mi id="A1.E14.m1.1.1.1.1" xref="A1.E14.m1.1.1.1.1.cmml">m</mi><mo id="A1.E14.m1.3.3.3.5.1" xref="A1.E14.m1.3.3.3.4.cmml">,</mo><mi id="A1.E14.m1.2.2.2.2" xref="A1.E14.m1.2.2.2.2.cmml">c</mi><mo id="A1.E14.m1.3.3.3.5.2" xref="A1.E14.m1.3.3.3.4.cmml">,</mo><mi id="A1.E14.m1.3.3.3.3" xref="A1.E14.m1.3.3.3.3.cmml">k</mi></mrow><mi id="A1.E14.m1.3.4.2.3" xref="A1.E14.m1.3.4.2.3.cmml">q</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.E14.m1.3b"><apply id="A1.E14.m1.3.4.cmml" xref="A1.E14.m1.3.4"><csymbol cd="ambiguous" id="A1.E14.m1.3.4.1.cmml" xref="A1.E14.m1.3.4">subscript</csymbol><apply id="A1.E14.m1.3.4.2.cmml" xref="A1.E14.m1.3.4"><csymbol cd="ambiguous" id="A1.E14.m1.3.4.2.1.cmml" xref="A1.E14.m1.3.4">superscript</csymbol><ci id="A1.E14.m1.3.4.2.2.cmml" xref="A1.E14.m1.3.4.2.2">𝑻</ci><ci id="A1.E14.m1.3.4.2.3.cmml" xref="A1.E14.m1.3.4.2.3">𝑞</ci></apply><list id="A1.E14.m1.3.3.3.4.cmml" xref="A1.E14.m1.3.3.3.5"><ci id="A1.E14.m1.1.1.1.1.cmml" xref="A1.E14.m1.1.1.1.1">𝑚</ci><ci id="A1.E14.m1.2.2.2.2.cmml" xref="A1.E14.m1.2.2.2.2">𝑐</ci><ci id="A1.E14.m1.3.3.3.3.cmml" xref="A1.E14.m1.3.3.3.3">𝑘</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E14.m1.3c">\displaystyle\bm{T}^{q}_{m,c,k}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E14.m2.4" class="ltx_Math" alttext="\displaystyle\triangleq\alpha^{-1}(\bm{T}_{m,c,k}-\delta_{c})." display="inline"><semantics id="A1.E14.m2.4a"><mrow id="A1.E14.m2.4.4.1" xref="A1.E14.m2.4.4.1.1.cmml"><mrow id="A1.E14.m2.4.4.1.1" xref="A1.E14.m2.4.4.1.1.cmml"><mi id="A1.E14.m2.4.4.1.1.3" xref="A1.E14.m2.4.4.1.1.3.cmml"></mi><mo id="A1.E14.m2.4.4.1.1.2" xref="A1.E14.m2.4.4.1.1.2.cmml">≜</mo><mrow id="A1.E14.m2.4.4.1.1.1" xref="A1.E14.m2.4.4.1.1.1.cmml"><msup id="A1.E14.m2.4.4.1.1.1.3" xref="A1.E14.m2.4.4.1.1.1.3.cmml"><mi id="A1.E14.m2.4.4.1.1.1.3.2" xref="A1.E14.m2.4.4.1.1.1.3.2.cmml">α</mi><mrow id="A1.E14.m2.4.4.1.1.1.3.3" xref="A1.E14.m2.4.4.1.1.1.3.3.cmml"><mo id="A1.E14.m2.4.4.1.1.1.3.3a" xref="A1.E14.m2.4.4.1.1.1.3.3.cmml">−</mo><mn id="A1.E14.m2.4.4.1.1.1.3.3.2" xref="A1.E14.m2.4.4.1.1.1.3.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A1.E14.m2.4.4.1.1.1.2" xref="A1.E14.m2.4.4.1.1.1.2.cmml">​</mo><mrow id="A1.E14.m2.4.4.1.1.1.1.1" xref="A1.E14.m2.4.4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E14.m2.4.4.1.1.1.1.1.2" xref="A1.E14.m2.4.4.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E14.m2.4.4.1.1.1.1.1.1" xref="A1.E14.m2.4.4.1.1.1.1.1.1.cmml"><msub id="A1.E14.m2.4.4.1.1.1.1.1.1.2" xref="A1.E14.m2.4.4.1.1.1.1.1.1.2.cmml"><mi id="A1.E14.m2.4.4.1.1.1.1.1.1.2.2" xref="A1.E14.m2.4.4.1.1.1.1.1.1.2.2.cmml">𝑻</mi><mrow id="A1.E14.m2.3.3.3.5" xref="A1.E14.m2.3.3.3.4.cmml"><mi id="A1.E14.m2.1.1.1.1" xref="A1.E14.m2.1.1.1.1.cmml">m</mi><mo id="A1.E14.m2.3.3.3.5.1" xref="A1.E14.m2.3.3.3.4.cmml">,</mo><mi id="A1.E14.m2.2.2.2.2" xref="A1.E14.m2.2.2.2.2.cmml">c</mi><mo id="A1.E14.m2.3.3.3.5.2" xref="A1.E14.m2.3.3.3.4.cmml">,</mo><mi id="A1.E14.m2.3.3.3.3" xref="A1.E14.m2.3.3.3.3.cmml">k</mi></mrow></msub><mo id="A1.E14.m2.4.4.1.1.1.1.1.1.1" xref="A1.E14.m2.4.4.1.1.1.1.1.1.1.cmml">−</mo><msub id="A1.E14.m2.4.4.1.1.1.1.1.1.3" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3.cmml"><mi id="A1.E14.m2.4.4.1.1.1.1.1.1.3.2" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3.2.cmml">δ</mi><mi id="A1.E14.m2.4.4.1.1.1.1.1.1.3.3" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3.3.cmml">c</mi></msub></mrow><mo stretchy="false" id="A1.E14.m2.4.4.1.1.1.1.1.3" xref="A1.E14.m2.4.4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="A1.E14.m2.4.4.1.2" xref="A1.E14.m2.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.E14.m2.4b"><apply id="A1.E14.m2.4.4.1.1.cmml" xref="A1.E14.m2.4.4.1"><ci id="A1.E14.m2.4.4.1.1.2.cmml" xref="A1.E14.m2.4.4.1.1.2">≜</ci><csymbol cd="latexml" id="A1.E14.m2.4.4.1.1.3.cmml" xref="A1.E14.m2.4.4.1.1.3">absent</csymbol><apply id="A1.E14.m2.4.4.1.1.1.cmml" xref="A1.E14.m2.4.4.1.1.1"><times id="A1.E14.m2.4.4.1.1.1.2.cmml" xref="A1.E14.m2.4.4.1.1.1.2"></times><apply id="A1.E14.m2.4.4.1.1.1.3.cmml" xref="A1.E14.m2.4.4.1.1.1.3"><csymbol cd="ambiguous" id="A1.E14.m2.4.4.1.1.1.3.1.cmml" xref="A1.E14.m2.4.4.1.1.1.3">superscript</csymbol><ci id="A1.E14.m2.4.4.1.1.1.3.2.cmml" xref="A1.E14.m2.4.4.1.1.1.3.2">𝛼</ci><apply id="A1.E14.m2.4.4.1.1.1.3.3.cmml" xref="A1.E14.m2.4.4.1.1.1.3.3"><minus id="A1.E14.m2.4.4.1.1.1.3.3.1.cmml" xref="A1.E14.m2.4.4.1.1.1.3.3"></minus><cn type="integer" id="A1.E14.m2.4.4.1.1.1.3.3.2.cmml" xref="A1.E14.m2.4.4.1.1.1.3.3.2">1</cn></apply></apply><apply id="A1.E14.m2.4.4.1.1.1.1.1.1.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1"><minus id="A1.E14.m2.4.4.1.1.1.1.1.1.1.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.1"></minus><apply id="A1.E14.m2.4.4.1.1.1.1.1.1.2.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E14.m2.4.4.1.1.1.1.1.1.2.1.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.2">subscript</csymbol><ci id="A1.E14.m2.4.4.1.1.1.1.1.1.2.2.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.2.2">𝑻</ci><list id="A1.E14.m2.3.3.3.4.cmml" xref="A1.E14.m2.3.3.3.5"><ci id="A1.E14.m2.1.1.1.1.cmml" xref="A1.E14.m2.1.1.1.1">𝑚</ci><ci id="A1.E14.m2.2.2.2.2.cmml" xref="A1.E14.m2.2.2.2.2">𝑐</ci><ci id="A1.E14.m2.3.3.3.3.cmml" xref="A1.E14.m2.3.3.3.3">𝑘</ci></list></apply><apply id="A1.E14.m2.4.4.1.1.1.1.1.1.3.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E14.m2.4.4.1.1.1.1.1.1.3.1.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.E14.m2.4.4.1.1.1.1.1.1.3.2.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3.2">𝛿</ci><ci id="A1.E14.m2.4.4.1.1.1.1.1.1.3.3.cmml" xref="A1.E14.m2.4.4.1.1.1.1.1.1.3.3">𝑐</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E14.m2.4c">\displaystyle\triangleq\alpha^{-1}(\bm{T}_{m,c,k}-\delta_{c}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(14)</span></td>
</tr></tbody>
</table>
<p id="A1.p1.11" class="ltx_p">This is similar to equations <a href="#A2.E15" title="In Appendix B Quantization and MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> and <a href="#A2.E16" title="In Appendix B Quantization and MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>, but with the scale factor pooled across all codebooks instead of unique to each input column. The <math id="A1.p1.9.m1.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="A1.p1.9.m1.1a"><mi id="A1.p1.9.m1.1.1" xref="A1.p1.9.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="A1.p1.9.m1.1b"><ci id="A1.p1.9.m1.1.1.cmml" xref="A1.p1.9.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.9.m1.1c">\alpha</annotation></semantics></math> used here is the same as that in equation <a href="#S1.E2" title="In 1.1 Problem Formulation ‣ 1 Introduction ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, and the matrix <math id="A1.p1.10.m2.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="A1.p1.10.m2.1a"><mi id="A1.p1.10.m2.1.1" xref="A1.p1.10.m2.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="A1.p1.10.m2.1b"><ci id="A1.p1.10.m2.1.1.cmml" xref="A1.p1.10.m2.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.10.m2.1c">\beta</annotation></semantics></math> in equation <a href="#S1.E2" title="In 1.1 Problem Formulation ‣ 1 Introduction ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> has entries equal to <math id="A1.p1.11.m3.1" class="ltx_Math" alttext="\sum_{c}\bm{\delta}_{c}" display="inline"><semantics id="A1.p1.11.m3.1a"><mrow id="A1.p1.11.m3.1.1" xref="A1.p1.11.m3.1.1.cmml"><msub id="A1.p1.11.m3.1.1.1" xref="A1.p1.11.m3.1.1.1.cmml"><mo id="A1.p1.11.m3.1.1.1.2" xref="A1.p1.11.m3.1.1.1.2.cmml">∑</mo><mi id="A1.p1.11.m3.1.1.1.3" xref="A1.p1.11.m3.1.1.1.3.cmml">c</mi></msub><msub id="A1.p1.11.m3.1.1.2" xref="A1.p1.11.m3.1.1.2.cmml"><mi id="A1.p1.11.m3.1.1.2.2" xref="A1.p1.11.m3.1.1.2.2.cmml">𝜹</mi><mi id="A1.p1.11.m3.1.1.2.3" xref="A1.p1.11.m3.1.1.2.3.cmml">c</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.11.m3.1b"><apply id="A1.p1.11.m3.1.1.cmml" xref="A1.p1.11.m3.1.1"><apply id="A1.p1.11.m3.1.1.1.cmml" xref="A1.p1.11.m3.1.1.1"><csymbol cd="ambiguous" id="A1.p1.11.m3.1.1.1.1.cmml" xref="A1.p1.11.m3.1.1.1">subscript</csymbol><sum id="A1.p1.11.m3.1.1.1.2.cmml" xref="A1.p1.11.m3.1.1.1.2"></sum><ci id="A1.p1.11.m3.1.1.1.3.cmml" xref="A1.p1.11.m3.1.1.1.3">𝑐</ci></apply><apply id="A1.p1.11.m3.1.1.2.cmml" xref="A1.p1.11.m3.1.1.2"><csymbol cd="ambiguous" id="A1.p1.11.m3.1.1.2.1.cmml" xref="A1.p1.11.m3.1.1.2">subscript</csymbol><ci id="A1.p1.11.m3.1.1.2.2.cmml" xref="A1.p1.11.m3.1.1.2.2">𝜹</ci><ci id="A1.p1.11.m3.1.1.2.3.cmml" xref="A1.p1.11.m3.1.1.2.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.11.m3.1c">\sum_{c}\bm{\delta}_{c}</annotation></semantics></math> (plus the debiasing constant from our averaging-based aggregation).</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Quantization and <span id="A2.1.1" class="ltx_text ltx_font_smallcaps">MaddnessHash</span>
</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">The use of at most 16 leaves is so that the resulting codes use 4 bits. This allows the use of these same shuffle instructions to accelerate the table lookups as in <cite class="ltx_cite ltx_citemacro_citet">Blalock &amp; Guttag (<a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>.</p>
</div>
<div id="A2.p2" class="ltx_para">
<p id="A2.p2.3" class="ltx_p">The only subtlety in vectorizing our hash function is that one must execute line 4 using shuffle instructions such as <span id="A2.p2.3.1" class="ltx_text ltx_font_typewriter">vpshufb</span> on x86, <span id="A2.p2.3.2" class="ltx_text ltx_font_typewriter">vtbl</span> on ARM, or <span id="A2.p2.3.3" class="ltx_text ltx_font_typewriter">vtbl</span> on PowerPC. In order to do this, the split values and scalars <math id="A2.p2.1.m1.1" class="ltx_Math" alttext="x_{j^{t}}" display="inline"><semantics id="A2.p2.1.m1.1a"><msub id="A2.p2.1.m1.1.1" xref="A2.p2.1.m1.1.1.cmml"><mi id="A2.p2.1.m1.1.1.2" xref="A2.p2.1.m1.1.1.2.cmml">x</mi><msup id="A2.p2.1.m1.1.1.3" xref="A2.p2.1.m1.1.1.3.cmml"><mi id="A2.p2.1.m1.1.1.3.2" xref="A2.p2.1.m1.1.1.3.2.cmml">j</mi><mi id="A2.p2.1.m1.1.1.3.3" xref="A2.p2.1.m1.1.1.3.3.cmml">t</mi></msup></msub><annotation-xml encoding="MathML-Content" id="A2.p2.1.m1.1b"><apply id="A2.p2.1.m1.1.1.cmml" xref="A2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A2.p2.1.m1.1.1.1.cmml" xref="A2.p2.1.m1.1.1">subscript</csymbol><ci id="A2.p2.1.m1.1.1.2.cmml" xref="A2.p2.1.m1.1.1.2">𝑥</ci><apply id="A2.p2.1.m1.1.1.3.cmml" xref="A2.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.p2.1.m1.1.1.3.1.cmml" xref="A2.p2.1.m1.1.1.3">superscript</csymbol><ci id="A2.p2.1.m1.1.1.3.2.cmml" xref="A2.p2.1.m1.1.1.3.2">𝑗</ci><ci id="A2.p2.1.m1.1.1.3.3.cmml" xref="A2.p2.1.m1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.1.m1.1c">x_{j^{t}}</annotation></semantics></math> must be 8-bit integers. We quantize them by learning for each split index <math id="A2.p2.2.m2.1" class="ltx_Math" alttext="j" display="inline"><semantics id="A2.p2.2.m2.1a"><mi id="A2.p2.2.m2.1.1" xref="A2.p2.2.m2.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="A2.p2.2.m2.1b"><ci id="A2.p2.2.m2.1.1.cmml" xref="A2.p2.2.m2.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.2.m2.1c">j</annotation></semantics></math> a pair of scalars <math id="A2.p2.3.m3.2" class="ltx_Math" alttext="(\gamma_{j},\delta_{j})" display="inline"><semantics id="A2.p2.3.m3.2a"><mrow id="A2.p2.3.m3.2.2.2" xref="A2.p2.3.m3.2.2.3.cmml"><mo stretchy="false" id="A2.p2.3.m3.2.2.2.3" xref="A2.p2.3.m3.2.2.3.cmml">(</mo><msub id="A2.p2.3.m3.1.1.1.1" xref="A2.p2.3.m3.1.1.1.1.cmml"><mi id="A2.p2.3.m3.1.1.1.1.2" xref="A2.p2.3.m3.1.1.1.1.2.cmml">γ</mi><mi id="A2.p2.3.m3.1.1.1.1.3" xref="A2.p2.3.m3.1.1.1.1.3.cmml">j</mi></msub><mo id="A2.p2.3.m3.2.2.2.4" xref="A2.p2.3.m3.2.2.3.cmml">,</mo><msub id="A2.p2.3.m3.2.2.2.2" xref="A2.p2.3.m3.2.2.2.2.cmml"><mi id="A2.p2.3.m3.2.2.2.2.2" xref="A2.p2.3.m3.2.2.2.2.2.cmml">δ</mi><mi id="A2.p2.3.m3.2.2.2.2.3" xref="A2.p2.3.m3.2.2.2.2.3.cmml">j</mi></msub><mo stretchy="false" id="A2.p2.3.m3.2.2.2.5" xref="A2.p2.3.m3.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.p2.3.m3.2b"><interval closure="open" id="A2.p2.3.m3.2.2.3.cmml" xref="A2.p2.3.m3.2.2.2"><apply id="A2.p2.3.m3.1.1.1.1.cmml" xref="A2.p2.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="A2.p2.3.m3.1.1.1.1.1.cmml" xref="A2.p2.3.m3.1.1.1.1">subscript</csymbol><ci id="A2.p2.3.m3.1.1.1.1.2.cmml" xref="A2.p2.3.m3.1.1.1.1.2">𝛾</ci><ci id="A2.p2.3.m3.1.1.1.1.3.cmml" xref="A2.p2.3.m3.1.1.1.1.3">𝑗</ci></apply><apply id="A2.p2.3.m3.2.2.2.2.cmml" xref="A2.p2.3.m3.2.2.2.2"><csymbol cd="ambiguous" id="A2.p2.3.m3.2.2.2.2.1.cmml" xref="A2.p2.3.m3.2.2.2.2">subscript</csymbol><ci id="A2.p2.3.m3.2.2.2.2.2.cmml" xref="A2.p2.3.m3.2.2.2.2.2">𝛿</ci><ci id="A2.p2.3.m3.2.2.2.2.3.cmml" xref="A2.p2.3.m3.2.2.2.2.3">𝑗</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.3.m3.2c">(\gamma_{j},\delta_{j})</annotation></semantics></math>, where</p>
<table id="A6.EGx12" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A2.E15"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A2.E15.m1.1" class="ltx_Math" alttext="\displaystyle\delta_{j}" display="inline"><semantics id="A2.E15.m1.1a"><msub id="A2.E15.m1.1.1" xref="A2.E15.m1.1.1.cmml"><mi id="A2.E15.m1.1.1.2" xref="A2.E15.m1.1.1.2.cmml">δ</mi><mi id="A2.E15.m1.1.1.3" xref="A2.E15.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="A2.E15.m1.1b"><apply id="A2.E15.m1.1.1.cmml" xref="A2.E15.m1.1.1"><csymbol cd="ambiguous" id="A2.E15.m1.1.1.1.cmml" xref="A2.E15.m1.1.1">subscript</csymbol><ci id="A2.E15.m1.1.1.2.cmml" xref="A2.E15.m1.1.1.2">𝛿</ci><ci id="A2.E15.m1.1.1.3.cmml" xref="A2.E15.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.E15.m1.1c">\displaystyle\delta_{j}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.E15.m2.1" class="ltx_Math" alttext="\displaystyle\triangleq\min_{i}\bm{v}^{j}_{i}" display="inline"><semantics id="A2.E15.m2.1a"><mrow id="A2.E15.m2.1.1" xref="A2.E15.m2.1.1.cmml"><mi id="A2.E15.m2.1.1.2" xref="A2.E15.m2.1.1.2.cmml"></mi><mo id="A2.E15.m2.1.1.1" xref="A2.E15.m2.1.1.1.cmml">≜</mo><mrow id="A2.E15.m2.1.1.3" xref="A2.E15.m2.1.1.3.cmml"><munder id="A2.E15.m2.1.1.3.1" xref="A2.E15.m2.1.1.3.1.cmml"><mi id="A2.E15.m2.1.1.3.1.2" xref="A2.E15.m2.1.1.3.1.2.cmml">min</mi><mi id="A2.E15.m2.1.1.3.1.3" xref="A2.E15.m2.1.1.3.1.3.cmml">i</mi></munder><mo lspace="0.167em" id="A2.E15.m2.1.1.3a" xref="A2.E15.m2.1.1.3.cmml">⁡</mo><msubsup id="A2.E15.m2.1.1.3.2" xref="A2.E15.m2.1.1.3.2.cmml"><mi id="A2.E15.m2.1.1.3.2.2.2" xref="A2.E15.m2.1.1.3.2.2.2.cmml">𝒗</mi><mi id="A2.E15.m2.1.1.3.2.3" xref="A2.E15.m2.1.1.3.2.3.cmml">i</mi><mi id="A2.E15.m2.1.1.3.2.2.3" xref="A2.E15.m2.1.1.3.2.2.3.cmml">j</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.E15.m2.1b"><apply id="A2.E15.m2.1.1.cmml" xref="A2.E15.m2.1.1"><ci id="A2.E15.m2.1.1.1.cmml" xref="A2.E15.m2.1.1.1">≜</ci><csymbol cd="latexml" id="A2.E15.m2.1.1.2.cmml" xref="A2.E15.m2.1.1.2">absent</csymbol><apply id="A2.E15.m2.1.1.3.cmml" xref="A2.E15.m2.1.1.3"><apply id="A2.E15.m2.1.1.3.1.cmml" xref="A2.E15.m2.1.1.3.1"><csymbol cd="ambiguous" id="A2.E15.m2.1.1.3.1.1.cmml" xref="A2.E15.m2.1.1.3.1">subscript</csymbol><min id="A2.E15.m2.1.1.3.1.2.cmml" xref="A2.E15.m2.1.1.3.1.2"></min><ci id="A2.E15.m2.1.1.3.1.3.cmml" xref="A2.E15.m2.1.1.3.1.3">𝑖</ci></apply><apply id="A2.E15.m2.1.1.3.2.cmml" xref="A2.E15.m2.1.1.3.2"><csymbol cd="ambiguous" id="A2.E15.m2.1.1.3.2.1.cmml" xref="A2.E15.m2.1.1.3.2">subscript</csymbol><apply id="A2.E15.m2.1.1.3.2.2.cmml" xref="A2.E15.m2.1.1.3.2"><csymbol cd="ambiguous" id="A2.E15.m2.1.1.3.2.2.1.cmml" xref="A2.E15.m2.1.1.3.2">superscript</csymbol><ci id="A2.E15.m2.1.1.3.2.2.2.cmml" xref="A2.E15.m2.1.1.3.2.2.2">𝒗</ci><ci id="A2.E15.m2.1.1.3.2.2.3.cmml" xref="A2.E15.m2.1.1.3.2.2.3">𝑗</ci></apply><ci id="A2.E15.m2.1.1.3.2.3.cmml" xref="A2.E15.m2.1.1.3.2.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.E15.m2.1c">\displaystyle\triangleq\min_{i}\bm{v}^{j}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(15)</span></td>
</tr></tbody>
<tbody id="A2.E16"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A2.E16.m1.1" class="ltx_Math" alttext="\displaystyle\gamma_{j}" display="inline"><semantics id="A2.E16.m1.1a"><msub id="A2.E16.m1.1.1" xref="A2.E16.m1.1.1.cmml"><mi id="A2.E16.m1.1.1.2" xref="A2.E16.m1.1.1.2.cmml">γ</mi><mi id="A2.E16.m1.1.1.3" xref="A2.E16.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="A2.E16.m1.1b"><apply id="A2.E16.m1.1.1.cmml" xref="A2.E16.m1.1.1"><csymbol cd="ambiguous" id="A2.E16.m1.1.1.1.cmml" xref="A2.E16.m1.1.1">subscript</csymbol><ci id="A2.E16.m1.1.1.2.cmml" xref="A2.E16.m1.1.1.2">𝛾</ci><ci id="A2.E16.m1.1.1.3.cmml" xref="A2.E16.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.E16.m1.1c">\displaystyle\gamma_{j}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.E16.m2.3" class="ltx_Math" alttext="\displaystyle\triangleq 2^{l},l=\left\lfloor\log_{2}\left(\frac{255}{\max_{i}\bm{v}^{j}_{i}-\delta_{j}}\right)\right\rfloor" display="inline"><semantics id="A2.E16.m2.3a"><mrow id="A2.E16.m2.3.3.2" xref="A2.E16.m2.3.3.3.cmml"><mrow id="A2.E16.m2.2.2.1.1" xref="A2.E16.m2.2.2.1.1.cmml"><mi id="A2.E16.m2.2.2.1.1.2" xref="A2.E16.m2.2.2.1.1.2.cmml"></mi><mo id="A2.E16.m2.2.2.1.1.1" xref="A2.E16.m2.2.2.1.1.1.cmml">≜</mo><msup id="A2.E16.m2.2.2.1.1.3" xref="A2.E16.m2.2.2.1.1.3.cmml"><mn id="A2.E16.m2.2.2.1.1.3.2" xref="A2.E16.m2.2.2.1.1.3.2.cmml">2</mn><mi id="A2.E16.m2.2.2.1.1.3.3" xref="A2.E16.m2.2.2.1.1.3.3.cmml">l</mi></msup></mrow><mo id="A2.E16.m2.3.3.2.3" xref="A2.E16.m2.3.3.3a.cmml">,</mo><mrow id="A2.E16.m2.3.3.2.2" xref="A2.E16.m2.3.3.2.2.cmml"><mi id="A2.E16.m2.3.3.2.2.3" xref="A2.E16.m2.3.3.2.2.3.cmml">l</mi><mo id="A2.E16.m2.3.3.2.2.2" xref="A2.E16.m2.3.3.2.2.2.cmml">=</mo><mrow id="A2.E16.m2.3.3.2.2.1.1" xref="A2.E16.m2.3.3.2.2.1.2.cmml"><mo id="A2.E16.m2.3.3.2.2.1.1.2" xref="A2.E16.m2.3.3.2.2.1.2.1.cmml">⌊</mo><mrow id="A2.E16.m2.3.3.2.2.1.1.1.1" xref="A2.E16.m2.3.3.2.2.1.1.1.2.cmml"><msub id="A2.E16.m2.3.3.2.2.1.1.1.1.1" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1.cmml"><mi id="A2.E16.m2.3.3.2.2.1.1.1.1.1.2" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1.2.cmml">log</mi><mn id="A2.E16.m2.3.3.2.2.1.1.1.1.1.3" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A2.E16.m2.3.3.2.2.1.1.1.1a" xref="A2.E16.m2.3.3.2.2.1.1.1.2.cmml">⁡</mo><mrow id="A2.E16.m2.3.3.2.2.1.1.1.1.2" xref="A2.E16.m2.3.3.2.2.1.1.1.2.cmml"><mo id="A2.E16.m2.3.3.2.2.1.1.1.1.2.1" xref="A2.E16.m2.3.3.2.2.1.1.1.2.cmml">(</mo><mstyle displaystyle="true" id="A2.E16.m2.1.1" xref="A2.E16.m2.1.1.cmml"><mfrac id="A2.E16.m2.1.1a" xref="A2.E16.m2.1.1.cmml"><mn id="A2.E16.m2.1.1.2" xref="A2.E16.m2.1.1.2.cmml">255</mn><mrow id="A2.E16.m2.1.1.3" xref="A2.E16.m2.1.1.3.cmml"><mrow id="A2.E16.m2.1.1.3.2" xref="A2.E16.m2.1.1.3.2.cmml"><msub id="A2.E16.m2.1.1.3.2.1" xref="A2.E16.m2.1.1.3.2.1.cmml"><mi id="A2.E16.m2.1.1.3.2.1.2" xref="A2.E16.m2.1.1.3.2.1.2.cmml">max</mi><mi id="A2.E16.m2.1.1.3.2.1.3" xref="A2.E16.m2.1.1.3.2.1.3.cmml">i</mi></msub><mo lspace="0.167em" id="A2.E16.m2.1.1.3.2a" xref="A2.E16.m2.1.1.3.2.cmml">⁡</mo><msubsup id="A2.E16.m2.1.1.3.2.2" xref="A2.E16.m2.1.1.3.2.2.cmml"><mi id="A2.E16.m2.1.1.3.2.2.2.2" xref="A2.E16.m2.1.1.3.2.2.2.2.cmml">𝒗</mi><mi id="A2.E16.m2.1.1.3.2.2.3" xref="A2.E16.m2.1.1.3.2.2.3.cmml">i</mi><mi id="A2.E16.m2.1.1.3.2.2.2.3" xref="A2.E16.m2.1.1.3.2.2.2.3.cmml">j</mi></msubsup></mrow><mo id="A2.E16.m2.1.1.3.1" xref="A2.E16.m2.1.1.3.1.cmml">−</mo><msub id="A2.E16.m2.1.1.3.3" xref="A2.E16.m2.1.1.3.3.cmml"><mi id="A2.E16.m2.1.1.3.3.2" xref="A2.E16.m2.1.1.3.3.2.cmml">δ</mi><mi id="A2.E16.m2.1.1.3.3.3" xref="A2.E16.m2.1.1.3.3.3.cmml">j</mi></msub></mrow></mfrac></mstyle><mo id="A2.E16.m2.3.3.2.2.1.1.1.1.2.2" xref="A2.E16.m2.3.3.2.2.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A2.E16.m2.3.3.2.2.1.1.3" xref="A2.E16.m2.3.3.2.2.1.2.1.cmml">⌋</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.E16.m2.3b"><apply id="A2.E16.m2.3.3.3.cmml" xref="A2.E16.m2.3.3.2"><csymbol cd="ambiguous" id="A2.E16.m2.3.3.3a.cmml" xref="A2.E16.m2.3.3.2.3">formulae-sequence</csymbol><apply id="A2.E16.m2.2.2.1.1.cmml" xref="A2.E16.m2.2.2.1.1"><ci id="A2.E16.m2.2.2.1.1.1.cmml" xref="A2.E16.m2.2.2.1.1.1">≜</ci><csymbol cd="latexml" id="A2.E16.m2.2.2.1.1.2.cmml" xref="A2.E16.m2.2.2.1.1.2">absent</csymbol><apply id="A2.E16.m2.2.2.1.1.3.cmml" xref="A2.E16.m2.2.2.1.1.3"><csymbol cd="ambiguous" id="A2.E16.m2.2.2.1.1.3.1.cmml" xref="A2.E16.m2.2.2.1.1.3">superscript</csymbol><cn type="integer" id="A2.E16.m2.2.2.1.1.3.2.cmml" xref="A2.E16.m2.2.2.1.1.3.2">2</cn><ci id="A2.E16.m2.2.2.1.1.3.3.cmml" xref="A2.E16.m2.2.2.1.1.3.3">𝑙</ci></apply></apply><apply id="A2.E16.m2.3.3.2.2.cmml" xref="A2.E16.m2.3.3.2.2"><eq id="A2.E16.m2.3.3.2.2.2.cmml" xref="A2.E16.m2.3.3.2.2.2"></eq><ci id="A2.E16.m2.3.3.2.2.3.cmml" xref="A2.E16.m2.3.3.2.2.3">𝑙</ci><apply id="A2.E16.m2.3.3.2.2.1.2.cmml" xref="A2.E16.m2.3.3.2.2.1.1"><floor id="A2.E16.m2.3.3.2.2.1.2.1.cmml" xref="A2.E16.m2.3.3.2.2.1.1.2"></floor><apply id="A2.E16.m2.3.3.2.2.1.1.1.2.cmml" xref="A2.E16.m2.3.3.2.2.1.1.1.1"><apply id="A2.E16.m2.3.3.2.2.1.1.1.1.1.cmml" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.E16.m2.3.3.2.2.1.1.1.1.1.1.cmml" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1">subscript</csymbol><log id="A2.E16.m2.3.3.2.2.1.1.1.1.1.2.cmml" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1.2"></log><cn type="integer" id="A2.E16.m2.3.3.2.2.1.1.1.1.1.3.cmml" xref="A2.E16.m2.3.3.2.2.1.1.1.1.1.3">2</cn></apply><apply id="A2.E16.m2.1.1.cmml" xref="A2.E16.m2.1.1"><divide id="A2.E16.m2.1.1.1.cmml" xref="A2.E16.m2.1.1"></divide><cn type="integer" id="A2.E16.m2.1.1.2.cmml" xref="A2.E16.m2.1.1.2">255</cn><apply id="A2.E16.m2.1.1.3.cmml" xref="A2.E16.m2.1.1.3"><minus id="A2.E16.m2.1.1.3.1.cmml" xref="A2.E16.m2.1.1.3.1"></minus><apply id="A2.E16.m2.1.1.3.2.cmml" xref="A2.E16.m2.1.1.3.2"><apply id="A2.E16.m2.1.1.3.2.1.cmml" xref="A2.E16.m2.1.1.3.2.1"><csymbol cd="ambiguous" id="A2.E16.m2.1.1.3.2.1.1.cmml" xref="A2.E16.m2.1.1.3.2.1">subscript</csymbol><max id="A2.E16.m2.1.1.3.2.1.2.cmml" xref="A2.E16.m2.1.1.3.2.1.2"></max><ci id="A2.E16.m2.1.1.3.2.1.3.cmml" xref="A2.E16.m2.1.1.3.2.1.3">𝑖</ci></apply><apply id="A2.E16.m2.1.1.3.2.2.cmml" xref="A2.E16.m2.1.1.3.2.2"><csymbol cd="ambiguous" id="A2.E16.m2.1.1.3.2.2.1.cmml" xref="A2.E16.m2.1.1.3.2.2">subscript</csymbol><apply id="A2.E16.m2.1.1.3.2.2.2.cmml" xref="A2.E16.m2.1.1.3.2.2"><csymbol cd="ambiguous" id="A2.E16.m2.1.1.3.2.2.2.1.cmml" xref="A2.E16.m2.1.1.3.2.2">superscript</csymbol><ci id="A2.E16.m2.1.1.3.2.2.2.2.cmml" xref="A2.E16.m2.1.1.3.2.2.2.2">𝒗</ci><ci id="A2.E16.m2.1.1.3.2.2.2.3.cmml" xref="A2.E16.m2.1.1.3.2.2.2.3">𝑗</ci></apply><ci id="A2.E16.m2.1.1.3.2.2.3.cmml" xref="A2.E16.m2.1.1.3.2.2.3">𝑖</ci></apply></apply><apply id="A2.E16.m2.1.1.3.3.cmml" xref="A2.E16.m2.1.1.3.3"><csymbol cd="ambiguous" id="A2.E16.m2.1.1.3.3.1.cmml" xref="A2.E16.m2.1.1.3.3">subscript</csymbol><ci id="A2.E16.m2.1.1.3.3.2.cmml" xref="A2.E16.m2.1.1.3.3.2">𝛿</ci><ci id="A2.E16.m2.1.1.3.3.3.cmml" xref="A2.E16.m2.1.1.3.3.3">𝑗</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.E16.m2.3c">\displaystyle\triangleq 2^{l},l=\left\lfloor\log_{2}\left(\frac{255}{\max_{i}\bm{v}^{j}_{i}-\delta_{j}}\right)\right\rfloor</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(16)</span></td>
</tr></tbody>
</table>
<p id="A2.p2.7" class="ltx_p">This restriction of <math id="A2.p2.4.m1.1" class="ltx_Math" alttext="\gamma_{j}" display="inline"><semantics id="A2.p2.4.m1.1a"><msub id="A2.p2.4.m1.1.1" xref="A2.p2.4.m1.1.1.cmml"><mi id="A2.p2.4.m1.1.1.2" xref="A2.p2.4.m1.1.1.2.cmml">γ</mi><mi id="A2.p2.4.m1.1.1.3" xref="A2.p2.4.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="A2.p2.4.m1.1b"><apply id="A2.p2.4.m1.1.1.cmml" xref="A2.p2.4.m1.1.1"><csymbol cd="ambiguous" id="A2.p2.4.m1.1.1.1.cmml" xref="A2.p2.4.m1.1.1">subscript</csymbol><ci id="A2.p2.4.m1.1.1.2.cmml" xref="A2.p2.4.m1.1.1.2">𝛾</ci><ci id="A2.p2.4.m1.1.1.3.cmml" xref="A2.p2.4.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.4.m1.1c">\gamma_{j}</annotation></semantics></math> to powers of two allows one to quantize <math id="A2.p2.5.m2.1" class="ltx_Math" alttext="x_{j^{t}}" display="inline"><semantics id="A2.p2.5.m2.1a"><msub id="A2.p2.5.m2.1.1" xref="A2.p2.5.m2.1.1.cmml"><mi id="A2.p2.5.m2.1.1.2" xref="A2.p2.5.m2.1.1.2.cmml">x</mi><msup id="A2.p2.5.m2.1.1.3" xref="A2.p2.5.m2.1.1.3.cmml"><mi id="A2.p2.5.m2.1.1.3.2" xref="A2.p2.5.m2.1.1.3.2.cmml">j</mi><mi id="A2.p2.5.m2.1.1.3.3" xref="A2.p2.5.m2.1.1.3.3.cmml">t</mi></msup></msub><annotation-xml encoding="MathML-Content" id="A2.p2.5.m2.1b"><apply id="A2.p2.5.m2.1.1.cmml" xref="A2.p2.5.m2.1.1"><csymbol cd="ambiguous" id="A2.p2.5.m2.1.1.1.cmml" xref="A2.p2.5.m2.1.1">subscript</csymbol><ci id="A2.p2.5.m2.1.1.2.cmml" xref="A2.p2.5.m2.1.1.2">𝑥</ci><apply id="A2.p2.5.m2.1.1.3.cmml" xref="A2.p2.5.m2.1.1.3"><csymbol cd="ambiguous" id="A2.p2.5.m2.1.1.3.1.cmml" xref="A2.p2.5.m2.1.1.3">superscript</csymbol><ci id="A2.p2.5.m2.1.1.3.2.cmml" xref="A2.p2.5.m2.1.1.3.2">𝑗</ci><ci id="A2.p2.5.m2.1.1.3.3.cmml" xref="A2.p2.5.m2.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.5.m2.1c">x_{j^{t}}</annotation></semantics></math> values with only shifts instead of multiplies. The <math id="A2.p2.6.m3.1" class="ltx_Math" alttext="\bm{v}" display="inline"><semantics id="A2.p2.6.m3.1a"><mi id="A2.p2.6.m3.1.1" xref="A2.p2.6.m3.1.1.cmml">𝒗</mi><annotation-xml encoding="MathML-Content" id="A2.p2.6.m3.1b"><ci id="A2.p2.6.m3.1.1.cmml" xref="A2.p2.6.m3.1.1">𝒗</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.6.m3.1c">\bm{v}</annotation></semantics></math> values can be quantized at the end of the training phase, while the <math id="A2.p2.7.m4.1" class="ltx_Math" alttext="x_{j^{t}}" display="inline"><semantics id="A2.p2.7.m4.1a"><msub id="A2.p2.7.m4.1.1" xref="A2.p2.7.m4.1.1.cmml"><mi id="A2.p2.7.m4.1.1.2" xref="A2.p2.7.m4.1.1.2.cmml">x</mi><msup id="A2.p2.7.m4.1.1.3" xref="A2.p2.7.m4.1.1.3.cmml"><mi id="A2.p2.7.m4.1.1.3.2" xref="A2.p2.7.m4.1.1.3.2.cmml">j</mi><mi id="A2.p2.7.m4.1.1.3.3" xref="A2.p2.7.m4.1.1.3.3.cmml">t</mi></msup></msub><annotation-xml encoding="MathML-Content" id="A2.p2.7.m4.1b"><apply id="A2.p2.7.m4.1.1.cmml" xref="A2.p2.7.m4.1.1"><csymbol cd="ambiguous" id="A2.p2.7.m4.1.1.1.cmml" xref="A2.p2.7.m4.1.1">subscript</csymbol><ci id="A2.p2.7.m4.1.1.2.cmml" xref="A2.p2.7.m4.1.1.2">𝑥</ci><apply id="A2.p2.7.m4.1.1.3.cmml" xref="A2.p2.7.m4.1.1.3"><csymbol cd="ambiguous" id="A2.p2.7.m4.1.1.3.1.cmml" xref="A2.p2.7.m4.1.1.3">superscript</csymbol><ci id="A2.p2.7.m4.1.1.3.2.cmml" xref="A2.p2.7.m4.1.1.3.2">𝑗</ci><ci id="A2.p2.7.m4.1.1.3.3.cmml" xref="A2.p2.7.m4.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.7.m4.1c">x_{j^{t}}</annotation></semantics></math> values must be quantized within Algorithm <a href="#alg1" title="Algorithm 1 ‣ 4.1 Hash Function Family, 𝒈⁢(⋅) ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> before line 5.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Subroutines for Training <span id="A3.1.1" class="ltx_text ltx_font_smallcaps">MaddnessHash</span>
</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">The <span id="A3.p1.1.1" class="ltx_text ltx_font_typewriter">optimal_split_threshold</span> algorithm (Algorithm <a href="#alg3" title="Algorithm 3 ‣ Appendix C Subroutines for Training MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) finds the best threshold at which to split a bucket within a given dimension. To do this, it uses the <span id="A3.p1.1.2" class="ltx_text ltx_font_typewriter">cumulative_sse</span> function (Algorithm <a href="#alg4" title="Algorithm 4 ‣ Appendix C Subroutines for Training MaddnessHash ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>) to help evaluate the loss associated with the resulting child buckets.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p">These algorithms exploit the fact that the sum of squared errors can be computed using only the sum of values and sum of squared values, both of which can be updated in <math id="A3.p2.1.m1.1" class="ltx_Math" alttext="O(1)" display="inline"><semantics id="A3.p2.1.m1.1a"><mrow id="A3.p2.1.m1.1.2" xref="A3.p2.1.m1.1.2.cmml"><mi id="A3.p2.1.m1.1.2.2" xref="A3.p2.1.m1.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="A3.p2.1.m1.1.2.1" xref="A3.p2.1.m1.1.2.1.cmml">​</mo><mrow id="A3.p2.1.m1.1.2.3.2" xref="A3.p2.1.m1.1.2.cmml"><mo stretchy="false" id="A3.p2.1.m1.1.2.3.2.1" xref="A3.p2.1.m1.1.2.cmml">(</mo><mn id="A3.p2.1.m1.1.1" xref="A3.p2.1.m1.1.1.cmml">1</mn><mo stretchy="false" id="A3.p2.1.m1.1.2.3.2.2" xref="A3.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.1b"><apply id="A3.p2.1.m1.1.2.cmml" xref="A3.p2.1.m1.1.2"><times id="A3.p2.1.m1.1.2.1.cmml" xref="A3.p2.1.m1.1.2.1"></times><ci id="A3.p2.1.m1.1.2.2.cmml" xref="A3.p2.1.m1.1.2.2">𝑂</ci><cn type="integer" id="A3.p2.1.m1.1.1.cmml" xref="A3.p2.1.m1.1.1">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.1c">O(1)</annotation></semantics></math> time when a vector is moved from one side of the split to the other.</p>
</div>
<figure id="alg3" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg3.2.1.1" class="ltx_text ltx_font_bold">Algorithm 3</span> </span> Optimal Split Threshold Within a Bucket</figcaption>
<div id="alg3.3" class="ltx_listing ltx_listing">
<div id="alg3.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span>  <span id="alg3.l1.2" class="ltx_text ltx_font_bold">Input:</span> bucket <math id="alg3.l1.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="alg3.l1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg3.l1.m1.1.1" xref="alg3.l1.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m1.1b"><ci id="alg3.l1.m1.1.1.cmml" xref="alg3.l1.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m1.1c">\mathcal{B}</annotation></semantics></math>, index <math id="alg3.l1.m2.1" class="ltx_Math" alttext="j" display="inline"><semantics id="alg3.l1.m2.1a"><mi id="alg3.l1.m2.1.1" xref="alg3.l1.m2.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m2.1b"><ci id="alg3.l1.m2.1.1.cmml" xref="alg3.l1.m2.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m2.1c">j</annotation></semantics></math>

</div>
<div id="alg3.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>  <math id="alg3.l2.m1.1" class="ltx_Math" alttext="\bm{X}\leftarrow\texttt{as\_2d\_array}(\mathcal{B})" display="inline"><semantics id="alg3.l2.m1.1a"><mrow id="alg3.l2.m1.1.2" xref="alg3.l2.m1.1.2.cmml"><mi id="alg3.l2.m1.1.2.2" xref="alg3.l2.m1.1.2.2.cmml">𝑿</mi><mo stretchy="false" id="alg3.l2.m1.1.2.1" xref="alg3.l2.m1.1.2.1.cmml">←</mo><mrow id="alg3.l2.m1.1.2.3" xref="alg3.l2.m1.1.2.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l2.m1.1.2.3.2" xref="alg3.l2.m1.1.2.3.2a.cmml">as_2d_array</mtext><mo lspace="0em" rspace="0em" id="alg3.l2.m1.1.2.3.1" xref="alg3.l2.m1.1.2.3.1.cmml">​</mo><mrow id="alg3.l2.m1.1.2.3.3.2" xref="alg3.l2.m1.1.2.3.cmml"><mo stretchy="false" id="alg3.l2.m1.1.2.3.3.2.1" xref="alg3.l2.m1.1.2.3.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="alg3.l2.m1.1.1" xref="alg3.l2.m1.1.1.cmml">ℬ</mi><mo stretchy="false" id="alg3.l2.m1.1.2.3.3.2.2" xref="alg3.l2.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l2.m1.1b"><apply id="alg3.l2.m1.1.2.cmml" xref="alg3.l2.m1.1.2"><ci id="alg3.l2.m1.1.2.1.cmml" xref="alg3.l2.m1.1.2.1">←</ci><ci id="alg3.l2.m1.1.2.2.cmml" xref="alg3.l2.m1.1.2.2">𝑿</ci><apply id="alg3.l2.m1.1.2.3.cmml" xref="alg3.l2.m1.1.2.3"><times id="alg3.l2.m1.1.2.3.1.cmml" xref="alg3.l2.m1.1.2.3.1"></times><ci id="alg3.l2.m1.1.2.3.2a.cmml" xref="alg3.l2.m1.1.2.3.2"><mtext class="ltx_mathvariant_monospace" id="alg3.l2.m1.1.2.3.2.cmml" xref="alg3.l2.m1.1.2.3.2">as_2d_array</mtext></ci><ci id="alg3.l2.m1.1.1.cmml" xref="alg3.l2.m1.1.1">ℬ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l2.m1.1c">\bm{X}\leftarrow\texttt{as\_2d\_array}(\mathcal{B})</annotation></semantics></math> 

</div>
<div id="alg3.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>  <math id="alg3.l3.m1.1" class="ltx_Math" alttext="\bm{X}^{sort}=\texttt{sort\_rows\_based\_on\_col}(\bm{X}\text{, }j)" display="inline"><semantics id="alg3.l3.m1.1a"><mrow id="alg3.l3.m1.1.1" xref="alg3.l3.m1.1.1.cmml"><msup id="alg3.l3.m1.1.1.3" xref="alg3.l3.m1.1.1.3.cmml"><mi id="alg3.l3.m1.1.1.3.2" xref="alg3.l3.m1.1.1.3.2.cmml">𝑿</mi><mrow id="alg3.l3.m1.1.1.3.3" xref="alg3.l3.m1.1.1.3.3.cmml"><mi id="alg3.l3.m1.1.1.3.3.2" xref="alg3.l3.m1.1.1.3.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg3.l3.m1.1.1.3.3.1" xref="alg3.l3.m1.1.1.3.3.1.cmml">​</mo><mi id="alg3.l3.m1.1.1.3.3.3" xref="alg3.l3.m1.1.1.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg3.l3.m1.1.1.3.3.1a" xref="alg3.l3.m1.1.1.3.3.1.cmml">​</mo><mi id="alg3.l3.m1.1.1.3.3.4" xref="alg3.l3.m1.1.1.3.3.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg3.l3.m1.1.1.3.3.1b" xref="alg3.l3.m1.1.1.3.3.1.cmml">​</mo><mi id="alg3.l3.m1.1.1.3.3.5" xref="alg3.l3.m1.1.1.3.3.5.cmml">t</mi></mrow></msup><mo id="alg3.l3.m1.1.1.2" xref="alg3.l3.m1.1.1.2.cmml">=</mo><mrow id="alg3.l3.m1.1.1.1" xref="alg3.l3.m1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l3.m1.1.1.1.3" xref="alg3.l3.m1.1.1.1.3a.cmml">sort_rows_based_on_col</mtext><mo lspace="0em" rspace="0em" id="alg3.l3.m1.1.1.1.2" xref="alg3.l3.m1.1.1.1.2.cmml">​</mo><mrow id="alg3.l3.m1.1.1.1.1.1" xref="alg3.l3.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg3.l3.m1.1.1.1.1.1.2" xref="alg3.l3.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg3.l3.m1.1.1.1.1.1.1" xref="alg3.l3.m1.1.1.1.1.1.1.cmml"><mi id="alg3.l3.m1.1.1.1.1.1.1.2" xref="alg3.l3.m1.1.1.1.1.1.1.2.cmml">𝑿</mi><mo lspace="0em" rspace="0em" id="alg3.l3.m1.1.1.1.1.1.1.1" xref="alg3.l3.m1.1.1.1.1.1.1.1.cmml">​</mo><mtext id="alg3.l3.m1.1.1.1.1.1.1.3" xref="alg3.l3.m1.1.1.1.1.1.1.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg3.l3.m1.1.1.1.1.1.1.1a" xref="alg3.l3.m1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg3.l3.m1.1.1.1.1.1.1.4" xref="alg3.l3.m1.1.1.1.1.1.1.4.cmml">j</mi></mrow><mo stretchy="false" id="alg3.l3.m1.1.1.1.1.1.3" xref="alg3.l3.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l3.m1.1b"><apply id="alg3.l3.m1.1.1.cmml" xref="alg3.l3.m1.1.1"><eq id="alg3.l3.m1.1.1.2.cmml" xref="alg3.l3.m1.1.1.2"></eq><apply id="alg3.l3.m1.1.1.3.cmml" xref="alg3.l3.m1.1.1.3"><csymbol cd="ambiguous" id="alg3.l3.m1.1.1.3.1.cmml" xref="alg3.l3.m1.1.1.3">superscript</csymbol><ci id="alg3.l3.m1.1.1.3.2.cmml" xref="alg3.l3.m1.1.1.3.2">𝑿</ci><apply id="alg3.l3.m1.1.1.3.3.cmml" xref="alg3.l3.m1.1.1.3.3"><times id="alg3.l3.m1.1.1.3.3.1.cmml" xref="alg3.l3.m1.1.1.3.3.1"></times><ci id="alg3.l3.m1.1.1.3.3.2.cmml" xref="alg3.l3.m1.1.1.3.3.2">𝑠</ci><ci id="alg3.l3.m1.1.1.3.3.3.cmml" xref="alg3.l3.m1.1.1.3.3.3">𝑜</ci><ci id="alg3.l3.m1.1.1.3.3.4.cmml" xref="alg3.l3.m1.1.1.3.3.4">𝑟</ci><ci id="alg3.l3.m1.1.1.3.3.5.cmml" xref="alg3.l3.m1.1.1.3.3.5">𝑡</ci></apply></apply><apply id="alg3.l3.m1.1.1.1.cmml" xref="alg3.l3.m1.1.1.1"><times id="alg3.l3.m1.1.1.1.2.cmml" xref="alg3.l3.m1.1.1.1.2"></times><ci id="alg3.l3.m1.1.1.1.3a.cmml" xref="alg3.l3.m1.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="alg3.l3.m1.1.1.1.3.cmml" xref="alg3.l3.m1.1.1.1.3">sort_rows_based_on_col</mtext></ci><apply id="alg3.l3.m1.1.1.1.1.1.1.cmml" xref="alg3.l3.m1.1.1.1.1.1"><times id="alg3.l3.m1.1.1.1.1.1.1.1.cmml" xref="alg3.l3.m1.1.1.1.1.1.1.1"></times><ci id="alg3.l3.m1.1.1.1.1.1.1.2.cmml" xref="alg3.l3.m1.1.1.1.1.1.1.2">𝑿</ci><ci id="alg3.l3.m1.1.1.1.1.1.1.3a.cmml" xref="alg3.l3.m1.1.1.1.1.1.1.3"><mtext id="alg3.l3.m1.1.1.1.1.1.1.3.cmml" xref="alg3.l3.m1.1.1.1.1.1.1.3">, </mtext></ci><ci id="alg3.l3.m1.1.1.1.1.1.1.4.cmml" xref="alg3.l3.m1.1.1.1.1.1.1.4">𝑗</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l3.m1.1c">\bm{X}^{sort}=\texttt{sort\_rows\_based\_on\_col}(\bm{X}\text{, }j)</annotation></semantics></math>

</div>
<div id="alg3.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span>  <math id="alg3.l4.m1.2" class="ltx_Math" alttext="\texttt{sses\_head}\leftarrow\texttt{cumulative\_sse}(\bm{X}^{sort},\texttt{false})" display="inline"><semantics id="alg3.l4.m1.2a"><mrow id="alg3.l4.m1.2.2" xref="alg3.l4.m1.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l4.m1.2.2.3" xref="alg3.l4.m1.2.2.3a.cmml">sses_head</mtext><mo stretchy="false" id="alg3.l4.m1.2.2.2" xref="alg3.l4.m1.2.2.2.cmml">←</mo><mrow id="alg3.l4.m1.2.2.1" xref="alg3.l4.m1.2.2.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l4.m1.2.2.1.3" xref="alg3.l4.m1.2.2.1.3a.cmml">cumulative_sse</mtext><mo lspace="0em" rspace="0em" id="alg3.l4.m1.2.2.1.2" xref="alg3.l4.m1.2.2.1.2.cmml">​</mo><mrow id="alg3.l4.m1.2.2.1.1.1" xref="alg3.l4.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="alg3.l4.m1.2.2.1.1.1.2" xref="alg3.l4.m1.2.2.1.1.2.cmml">(</mo><msup id="alg3.l4.m1.2.2.1.1.1.1" xref="alg3.l4.m1.2.2.1.1.1.1.cmml"><mi id="alg3.l4.m1.2.2.1.1.1.1.2" xref="alg3.l4.m1.2.2.1.1.1.1.2.cmml">𝑿</mi><mrow id="alg3.l4.m1.2.2.1.1.1.1.3" xref="alg3.l4.m1.2.2.1.1.1.1.3.cmml"><mi id="alg3.l4.m1.2.2.1.1.1.1.3.2" xref="alg3.l4.m1.2.2.1.1.1.1.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg3.l4.m1.2.2.1.1.1.1.3.1" xref="alg3.l4.m1.2.2.1.1.1.1.3.1.cmml">​</mo><mi id="alg3.l4.m1.2.2.1.1.1.1.3.3" xref="alg3.l4.m1.2.2.1.1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg3.l4.m1.2.2.1.1.1.1.3.1a" xref="alg3.l4.m1.2.2.1.1.1.1.3.1.cmml">​</mo><mi id="alg3.l4.m1.2.2.1.1.1.1.3.4" xref="alg3.l4.m1.2.2.1.1.1.1.3.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg3.l4.m1.2.2.1.1.1.1.3.1b" xref="alg3.l4.m1.2.2.1.1.1.1.3.1.cmml">​</mo><mi id="alg3.l4.m1.2.2.1.1.1.1.3.5" xref="alg3.l4.m1.2.2.1.1.1.1.3.5.cmml">t</mi></mrow></msup><mo id="alg3.l4.m1.2.2.1.1.1.3" xref="alg3.l4.m1.2.2.1.1.2.cmml">,</mo><mtext class="ltx_mathvariant_monospace" id="alg3.l4.m1.1.1" xref="alg3.l4.m1.1.1a.cmml">false</mtext><mo stretchy="false" id="alg3.l4.m1.2.2.1.1.1.4" xref="alg3.l4.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l4.m1.2b"><apply id="alg3.l4.m1.2.2.cmml" xref="alg3.l4.m1.2.2"><ci id="alg3.l4.m1.2.2.2.cmml" xref="alg3.l4.m1.2.2.2">←</ci><ci id="alg3.l4.m1.2.2.3a.cmml" xref="alg3.l4.m1.2.2.3"><mtext class="ltx_mathvariant_monospace" id="alg3.l4.m1.2.2.3.cmml" xref="alg3.l4.m1.2.2.3">sses_head</mtext></ci><apply id="alg3.l4.m1.2.2.1.cmml" xref="alg3.l4.m1.2.2.1"><times id="alg3.l4.m1.2.2.1.2.cmml" xref="alg3.l4.m1.2.2.1.2"></times><ci id="alg3.l4.m1.2.2.1.3a.cmml" xref="alg3.l4.m1.2.2.1.3"><mtext class="ltx_mathvariant_monospace" id="alg3.l4.m1.2.2.1.3.cmml" xref="alg3.l4.m1.2.2.1.3">cumulative_sse</mtext></ci><interval closure="open" id="alg3.l4.m1.2.2.1.1.2.cmml" xref="alg3.l4.m1.2.2.1.1.1"><apply id="alg3.l4.m1.2.2.1.1.1.1.cmml" xref="alg3.l4.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l4.m1.2.2.1.1.1.1.1.cmml" xref="alg3.l4.m1.2.2.1.1.1.1">superscript</csymbol><ci id="alg3.l4.m1.2.2.1.1.1.1.2.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.2">𝑿</ci><apply id="alg3.l4.m1.2.2.1.1.1.1.3.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.3"><times id="alg3.l4.m1.2.2.1.1.1.1.3.1.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.3.1"></times><ci id="alg3.l4.m1.2.2.1.1.1.1.3.2.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.3.2">𝑠</ci><ci id="alg3.l4.m1.2.2.1.1.1.1.3.3.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.3.3">𝑜</ci><ci id="alg3.l4.m1.2.2.1.1.1.1.3.4.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.3.4">𝑟</ci><ci id="alg3.l4.m1.2.2.1.1.1.1.3.5.cmml" xref="alg3.l4.m1.2.2.1.1.1.1.3.5">𝑡</ci></apply></apply><ci id="alg3.l4.m1.1.1a.cmml" xref="alg3.l4.m1.1.1"><mtext class="ltx_mathvariant_monospace" id="alg3.l4.m1.1.1.cmml" xref="alg3.l4.m1.1.1">false</mtext></ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l4.m1.2c">\texttt{sses\_head}\leftarrow\texttt{cumulative\_sse}(\bm{X}^{sort},\texttt{false})</annotation></semantics></math>

</div>
<div id="alg3.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>  <math id="alg3.l5.m1.2" class="ltx_Math" alttext="\texttt{sses\_tail}\leftarrow\texttt{cumulative\_sse}(\bm{X}^{sort},\texttt{true})" display="inline"><semantics id="alg3.l5.m1.2a"><mrow id="alg3.l5.m1.2.2" xref="alg3.l5.m1.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l5.m1.2.2.3" xref="alg3.l5.m1.2.2.3a.cmml">sses_tail</mtext><mo stretchy="false" id="alg3.l5.m1.2.2.2" xref="alg3.l5.m1.2.2.2.cmml">←</mo><mrow id="alg3.l5.m1.2.2.1" xref="alg3.l5.m1.2.2.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l5.m1.2.2.1.3" xref="alg3.l5.m1.2.2.1.3a.cmml">cumulative_sse</mtext><mo lspace="0em" rspace="0em" id="alg3.l5.m1.2.2.1.2" xref="alg3.l5.m1.2.2.1.2.cmml">​</mo><mrow id="alg3.l5.m1.2.2.1.1.1" xref="alg3.l5.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="alg3.l5.m1.2.2.1.1.1.2" xref="alg3.l5.m1.2.2.1.1.2.cmml">(</mo><msup id="alg3.l5.m1.2.2.1.1.1.1" xref="alg3.l5.m1.2.2.1.1.1.1.cmml"><mi id="alg3.l5.m1.2.2.1.1.1.1.2" xref="alg3.l5.m1.2.2.1.1.1.1.2.cmml">𝑿</mi><mrow id="alg3.l5.m1.2.2.1.1.1.1.3" xref="alg3.l5.m1.2.2.1.1.1.1.3.cmml"><mi id="alg3.l5.m1.2.2.1.1.1.1.3.2" xref="alg3.l5.m1.2.2.1.1.1.1.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg3.l5.m1.2.2.1.1.1.1.3.1" xref="alg3.l5.m1.2.2.1.1.1.1.3.1.cmml">​</mo><mi id="alg3.l5.m1.2.2.1.1.1.1.3.3" xref="alg3.l5.m1.2.2.1.1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg3.l5.m1.2.2.1.1.1.1.3.1a" xref="alg3.l5.m1.2.2.1.1.1.1.3.1.cmml">​</mo><mi id="alg3.l5.m1.2.2.1.1.1.1.3.4" xref="alg3.l5.m1.2.2.1.1.1.1.3.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg3.l5.m1.2.2.1.1.1.1.3.1b" xref="alg3.l5.m1.2.2.1.1.1.1.3.1.cmml">​</mo><mi id="alg3.l5.m1.2.2.1.1.1.1.3.5" xref="alg3.l5.m1.2.2.1.1.1.1.3.5.cmml">t</mi></mrow></msup><mo id="alg3.l5.m1.2.2.1.1.1.3" xref="alg3.l5.m1.2.2.1.1.2.cmml">,</mo><mtext class="ltx_mathvariant_monospace" id="alg3.l5.m1.1.1" xref="alg3.l5.m1.1.1a.cmml">true</mtext><mo stretchy="false" id="alg3.l5.m1.2.2.1.1.1.4" xref="alg3.l5.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l5.m1.2b"><apply id="alg3.l5.m1.2.2.cmml" xref="alg3.l5.m1.2.2"><ci id="alg3.l5.m1.2.2.2.cmml" xref="alg3.l5.m1.2.2.2">←</ci><ci id="alg3.l5.m1.2.2.3a.cmml" xref="alg3.l5.m1.2.2.3"><mtext class="ltx_mathvariant_monospace" id="alg3.l5.m1.2.2.3.cmml" xref="alg3.l5.m1.2.2.3">sses_tail</mtext></ci><apply id="alg3.l5.m1.2.2.1.cmml" xref="alg3.l5.m1.2.2.1"><times id="alg3.l5.m1.2.2.1.2.cmml" xref="alg3.l5.m1.2.2.1.2"></times><ci id="alg3.l5.m1.2.2.1.3a.cmml" xref="alg3.l5.m1.2.2.1.3"><mtext class="ltx_mathvariant_monospace" id="alg3.l5.m1.2.2.1.3.cmml" xref="alg3.l5.m1.2.2.1.3">cumulative_sse</mtext></ci><interval closure="open" id="alg3.l5.m1.2.2.1.1.2.cmml" xref="alg3.l5.m1.2.2.1.1.1"><apply id="alg3.l5.m1.2.2.1.1.1.1.cmml" xref="alg3.l5.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l5.m1.2.2.1.1.1.1.1.cmml" xref="alg3.l5.m1.2.2.1.1.1.1">superscript</csymbol><ci id="alg3.l5.m1.2.2.1.1.1.1.2.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.2">𝑿</ci><apply id="alg3.l5.m1.2.2.1.1.1.1.3.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.3"><times id="alg3.l5.m1.2.2.1.1.1.1.3.1.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.3.1"></times><ci id="alg3.l5.m1.2.2.1.1.1.1.3.2.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.3.2">𝑠</ci><ci id="alg3.l5.m1.2.2.1.1.1.1.3.3.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.3.3">𝑜</ci><ci id="alg3.l5.m1.2.2.1.1.1.1.3.4.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.3.4">𝑟</ci><ci id="alg3.l5.m1.2.2.1.1.1.1.3.5.cmml" xref="alg3.l5.m1.2.2.1.1.1.1.3.5">𝑡</ci></apply></apply><ci id="alg3.l5.m1.1.1a.cmml" xref="alg3.l5.m1.1.1"><mtext class="ltx_mathvariant_monospace" id="alg3.l5.m1.1.1.cmml" xref="alg3.l5.m1.1.1">true</mtext></ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l5.m1.2c">\texttt{sses\_tail}\leftarrow\texttt{cumulative\_sse}(\bm{X}^{sort},\texttt{true})</annotation></semantics></math>

</div>
<div id="alg3.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>  <math id="alg3.l6.m1.1" class="ltx_Math" alttext="\texttt{losses}\leftarrow\texttt{sses\_head}" display="inline"><semantics id="alg3.l6.m1.1a"><mrow id="alg3.l6.m1.1.1" xref="alg3.l6.m1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l6.m1.1.1.2" xref="alg3.l6.m1.1.1.2a.cmml">losses</mtext><mo stretchy="false" id="alg3.l6.m1.1.1.1" xref="alg3.l6.m1.1.1.1.cmml">←</mo><mtext class="ltx_mathvariant_monospace" id="alg3.l6.m1.1.1.3" xref="alg3.l6.m1.1.1.3a.cmml">sses_head</mtext></mrow><annotation-xml encoding="MathML-Content" id="alg3.l6.m1.1b"><apply id="alg3.l6.m1.1.1.cmml" xref="alg3.l6.m1.1.1"><ci id="alg3.l6.m1.1.1.1.cmml" xref="alg3.l6.m1.1.1.1">←</ci><ci id="alg3.l6.m1.1.1.2a.cmml" xref="alg3.l6.m1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="alg3.l6.m1.1.1.2.cmml" xref="alg3.l6.m1.1.1.2">losses</mtext></ci><ci id="alg3.l6.m1.1.1.3a.cmml" xref="alg3.l6.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="alg3.l6.m1.1.1.3.cmml" xref="alg3.l6.m1.1.1.3">sses_head</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l6.m1.1c">\texttt{losses}\leftarrow\texttt{sses\_head}</annotation></semantics></math>

</div>
<div id="alg3.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>  <math id="alg3.l7.m1.1" class="ltx_Math" alttext="\texttt{losses}_{1:N-1}\leftarrow\texttt{losses}_{1:N-1}+\texttt{sses\_tail}_{2:N}" display="inline"><semantics id="alg3.l7.m1.1a"><mrow id="alg3.l7.m1.1.1" xref="alg3.l7.m1.1.1.cmml"><msub id="alg3.l7.m1.1.1.2" xref="alg3.l7.m1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l7.m1.1.1.2.2" xref="alg3.l7.m1.1.1.2.2a.cmml">losses</mtext><mrow id="alg3.l7.m1.1.1.2.3" xref="alg3.l7.m1.1.1.2.3.cmml"><mn id="alg3.l7.m1.1.1.2.3.2" xref="alg3.l7.m1.1.1.2.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="alg3.l7.m1.1.1.2.3.1" xref="alg3.l7.m1.1.1.2.3.1.cmml">:</mo><mrow id="alg3.l7.m1.1.1.2.3.3" xref="alg3.l7.m1.1.1.2.3.3.cmml"><mi id="alg3.l7.m1.1.1.2.3.3.2" xref="alg3.l7.m1.1.1.2.3.3.2.cmml">N</mi><mo id="alg3.l7.m1.1.1.2.3.3.1" xref="alg3.l7.m1.1.1.2.3.3.1.cmml">−</mo><mn id="alg3.l7.m1.1.1.2.3.3.3" xref="alg3.l7.m1.1.1.2.3.3.3.cmml">1</mn></mrow></mrow></msub><mo stretchy="false" id="alg3.l7.m1.1.1.1" xref="alg3.l7.m1.1.1.1.cmml">←</mo><mrow id="alg3.l7.m1.1.1.3" xref="alg3.l7.m1.1.1.3.cmml"><msub id="alg3.l7.m1.1.1.3.2" xref="alg3.l7.m1.1.1.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l7.m1.1.1.3.2.2" xref="alg3.l7.m1.1.1.3.2.2a.cmml">losses</mtext><mrow id="alg3.l7.m1.1.1.3.2.3" xref="alg3.l7.m1.1.1.3.2.3.cmml"><mn id="alg3.l7.m1.1.1.3.2.3.2" xref="alg3.l7.m1.1.1.3.2.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="alg3.l7.m1.1.1.3.2.3.1" xref="alg3.l7.m1.1.1.3.2.3.1.cmml">:</mo><mrow id="alg3.l7.m1.1.1.3.2.3.3" xref="alg3.l7.m1.1.1.3.2.3.3.cmml"><mi id="alg3.l7.m1.1.1.3.2.3.3.2" xref="alg3.l7.m1.1.1.3.2.3.3.2.cmml">N</mi><mo id="alg3.l7.m1.1.1.3.2.3.3.1" xref="alg3.l7.m1.1.1.3.2.3.3.1.cmml">−</mo><mn id="alg3.l7.m1.1.1.3.2.3.3.3" xref="alg3.l7.m1.1.1.3.2.3.3.3.cmml">1</mn></mrow></mrow></msub><mo id="alg3.l7.m1.1.1.3.1" xref="alg3.l7.m1.1.1.3.1.cmml">+</mo><msub id="alg3.l7.m1.1.1.3.3" xref="alg3.l7.m1.1.1.3.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l7.m1.1.1.3.3.2" xref="alg3.l7.m1.1.1.3.3.2a.cmml">sses_tail</mtext><mrow id="alg3.l7.m1.1.1.3.3.3" xref="alg3.l7.m1.1.1.3.3.3.cmml"><mn id="alg3.l7.m1.1.1.3.3.3.2" xref="alg3.l7.m1.1.1.3.3.3.2.cmml">2</mn><mo lspace="0.278em" rspace="0.278em" id="alg3.l7.m1.1.1.3.3.3.1" xref="alg3.l7.m1.1.1.3.3.3.1.cmml">:</mo><mi id="alg3.l7.m1.1.1.3.3.3.3" xref="alg3.l7.m1.1.1.3.3.3.3.cmml">N</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l7.m1.1b"><apply id="alg3.l7.m1.1.1.cmml" xref="alg3.l7.m1.1.1"><ci id="alg3.l7.m1.1.1.1.cmml" xref="alg3.l7.m1.1.1.1">←</ci><apply id="alg3.l7.m1.1.1.2.cmml" xref="alg3.l7.m1.1.1.2"><csymbol cd="ambiguous" id="alg3.l7.m1.1.1.2.1.cmml" xref="alg3.l7.m1.1.1.2">subscript</csymbol><ci id="alg3.l7.m1.1.1.2.2a.cmml" xref="alg3.l7.m1.1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="alg3.l7.m1.1.1.2.2.cmml" xref="alg3.l7.m1.1.1.2.2">losses</mtext></ci><apply id="alg3.l7.m1.1.1.2.3.cmml" xref="alg3.l7.m1.1.1.2.3"><ci id="alg3.l7.m1.1.1.2.3.1.cmml" xref="alg3.l7.m1.1.1.2.3.1">:</ci><cn type="integer" id="alg3.l7.m1.1.1.2.3.2.cmml" xref="alg3.l7.m1.1.1.2.3.2">1</cn><apply id="alg3.l7.m1.1.1.2.3.3.cmml" xref="alg3.l7.m1.1.1.2.3.3"><minus id="alg3.l7.m1.1.1.2.3.3.1.cmml" xref="alg3.l7.m1.1.1.2.3.3.1"></minus><ci id="alg3.l7.m1.1.1.2.3.3.2.cmml" xref="alg3.l7.m1.1.1.2.3.3.2">𝑁</ci><cn type="integer" id="alg3.l7.m1.1.1.2.3.3.3.cmml" xref="alg3.l7.m1.1.1.2.3.3.3">1</cn></apply></apply></apply><apply id="alg3.l7.m1.1.1.3.cmml" xref="alg3.l7.m1.1.1.3"><plus id="alg3.l7.m1.1.1.3.1.cmml" xref="alg3.l7.m1.1.1.3.1"></plus><apply id="alg3.l7.m1.1.1.3.2.cmml" xref="alg3.l7.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg3.l7.m1.1.1.3.2.1.cmml" xref="alg3.l7.m1.1.1.3.2">subscript</csymbol><ci id="alg3.l7.m1.1.1.3.2.2a.cmml" xref="alg3.l7.m1.1.1.3.2.2"><mtext class="ltx_mathvariant_monospace" id="alg3.l7.m1.1.1.3.2.2.cmml" xref="alg3.l7.m1.1.1.3.2.2">losses</mtext></ci><apply id="alg3.l7.m1.1.1.3.2.3.cmml" xref="alg3.l7.m1.1.1.3.2.3"><ci id="alg3.l7.m1.1.1.3.2.3.1.cmml" xref="alg3.l7.m1.1.1.3.2.3.1">:</ci><cn type="integer" id="alg3.l7.m1.1.1.3.2.3.2.cmml" xref="alg3.l7.m1.1.1.3.2.3.2">1</cn><apply id="alg3.l7.m1.1.1.3.2.3.3.cmml" xref="alg3.l7.m1.1.1.3.2.3.3"><minus id="alg3.l7.m1.1.1.3.2.3.3.1.cmml" xref="alg3.l7.m1.1.1.3.2.3.3.1"></minus><ci id="alg3.l7.m1.1.1.3.2.3.3.2.cmml" xref="alg3.l7.m1.1.1.3.2.3.3.2">𝑁</ci><cn type="integer" id="alg3.l7.m1.1.1.3.2.3.3.3.cmml" xref="alg3.l7.m1.1.1.3.2.3.3.3">1</cn></apply></apply></apply><apply id="alg3.l7.m1.1.1.3.3.cmml" xref="alg3.l7.m1.1.1.3.3"><csymbol cd="ambiguous" id="alg3.l7.m1.1.1.3.3.1.cmml" xref="alg3.l7.m1.1.1.3.3">subscript</csymbol><ci id="alg3.l7.m1.1.1.3.3.2a.cmml" xref="alg3.l7.m1.1.1.3.3.2"><mtext class="ltx_mathvariant_monospace" id="alg3.l7.m1.1.1.3.3.2.cmml" xref="alg3.l7.m1.1.1.3.3.2">sses_tail</mtext></ci><apply id="alg3.l7.m1.1.1.3.3.3.cmml" xref="alg3.l7.m1.1.1.3.3.3"><ci id="alg3.l7.m1.1.1.3.3.3.1.cmml" xref="alg3.l7.m1.1.1.3.3.3.1">:</ci><cn type="integer" id="alg3.l7.m1.1.1.3.3.3.2.cmml" xref="alg3.l7.m1.1.1.3.3.3.2">2</cn><ci id="alg3.l7.m1.1.1.3.3.3.3.cmml" xref="alg3.l7.m1.1.1.3.3.3.3">𝑁</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l7.m1.1c">\texttt{losses}_{1:N-1}\leftarrow\texttt{losses}_{1:N-1}+\texttt{sses\_tail}_{2:N}</annotation></semantics></math>

</div>
<div id="alg3.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>  <math id="alg3.l8.m1.1" class="ltx_Math" alttext="n^{\ast}\leftarrow\operatorname*{argmin}_{n}\texttt{losses}_{n}" display="inline"><semantics id="alg3.l8.m1.1a"><mrow id="alg3.l8.m1.1.1" xref="alg3.l8.m1.1.1.cmml"><msup id="alg3.l8.m1.1.1.2" xref="alg3.l8.m1.1.1.2.cmml"><mi id="alg3.l8.m1.1.1.2.2" xref="alg3.l8.m1.1.1.2.2.cmml">n</mi><mo id="alg3.l8.m1.1.1.2.3" xref="alg3.l8.m1.1.1.2.3.cmml">∗</mo></msup><mo rspace="0.1389em" stretchy="false" id="alg3.l8.m1.1.1.1" xref="alg3.l8.m1.1.1.1.cmml">←</mo><mrow id="alg3.l8.m1.1.1.3" xref="alg3.l8.m1.1.1.3.cmml"><msub id="alg3.l8.m1.1.1.3.1" xref="alg3.l8.m1.1.1.3.1.cmml"><mo lspace="0.1389em" rspace="0.167em" id="alg3.l8.m1.1.1.3.1.2" xref="alg3.l8.m1.1.1.3.1.2.cmml">argmin</mo><mi id="alg3.l8.m1.1.1.3.1.3" xref="alg3.l8.m1.1.1.3.1.3.cmml">n</mi></msub><msub id="alg3.l8.m1.1.1.3.2" xref="alg3.l8.m1.1.1.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg3.l8.m1.1.1.3.2.2" xref="alg3.l8.m1.1.1.3.2.2a.cmml">losses</mtext><mi id="alg3.l8.m1.1.1.3.2.3" xref="alg3.l8.m1.1.1.3.2.3.cmml">n</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l8.m1.1b"><apply id="alg3.l8.m1.1.1.cmml" xref="alg3.l8.m1.1.1"><ci id="alg3.l8.m1.1.1.1.cmml" xref="alg3.l8.m1.1.1.1">←</ci><apply id="alg3.l8.m1.1.1.2.cmml" xref="alg3.l8.m1.1.1.2"><csymbol cd="ambiguous" id="alg3.l8.m1.1.1.2.1.cmml" xref="alg3.l8.m1.1.1.2">superscript</csymbol><ci id="alg3.l8.m1.1.1.2.2.cmml" xref="alg3.l8.m1.1.1.2.2">𝑛</ci><ci id="alg3.l8.m1.1.1.2.3.cmml" xref="alg3.l8.m1.1.1.2.3">∗</ci></apply><apply id="alg3.l8.m1.1.1.3.cmml" xref="alg3.l8.m1.1.1.3"><apply id="alg3.l8.m1.1.1.3.1.cmml" xref="alg3.l8.m1.1.1.3.1"><csymbol cd="ambiguous" id="alg3.l8.m1.1.1.3.1.1.cmml" xref="alg3.l8.m1.1.1.3.1">subscript</csymbol><ci id="alg3.l8.m1.1.1.3.1.2.cmml" xref="alg3.l8.m1.1.1.3.1.2">argmin</ci><ci id="alg3.l8.m1.1.1.3.1.3.cmml" xref="alg3.l8.m1.1.1.3.1.3">𝑛</ci></apply><apply id="alg3.l8.m1.1.1.3.2.cmml" xref="alg3.l8.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg3.l8.m1.1.1.3.2.1.cmml" xref="alg3.l8.m1.1.1.3.2">subscript</csymbol><ci id="alg3.l8.m1.1.1.3.2.2a.cmml" xref="alg3.l8.m1.1.1.3.2.2"><mtext class="ltx_mathvariant_monospace" id="alg3.l8.m1.1.1.3.2.2.cmml" xref="alg3.l8.m1.1.1.3.2.2">losses</mtext></ci><ci id="alg3.l8.m1.1.1.3.2.3.cmml" xref="alg3.l8.m1.1.1.3.2.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l8.m1.1c">n^{\ast}\leftarrow\operatorname*{argmin}_{n}\texttt{losses}_{n}</annotation></semantics></math>

</div>
<div id="alg3.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg3.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>  <math id="alg3.l9.m1.1" class="ltx_Math" alttext="\textbf{return }(\bm{X}^{sort}_{n^{\ast}\text{, }j}+\bm{X}^{sort}_{n^{\ast}+1\text{, }j})/2\text{, }\texttt{losses}_{n^{\ast}}" display="inline"><semantics id="alg3.l9.m1.1a"><mrow id="alg3.l9.m1.1.1" xref="alg3.l9.m1.1.1.cmml"><mrow id="alg3.l9.m1.1.1.1" xref="alg3.l9.m1.1.1.1.cmml"><mrow id="alg3.l9.m1.1.1.1.1" xref="alg3.l9.m1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="alg3.l9.m1.1.1.1.1.3" xref="alg3.l9.m1.1.1.1.1.3a.cmml">return </mtext><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.2" xref="alg3.l9.m1.1.1.1.1.2.cmml">​</mo><mrow id="alg3.l9.m1.1.1.1.1.1.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg3.l9.m1.1.1.1.1.1.1.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg3.l9.m1.1.1.1.1.1.1.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.cmml"><msubsup id="alg3.l9.m1.1.1.1.1.1.1.1.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.cmml"><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.2.cmml">𝑿</mi><mrow id="alg3.l9.m1.1.1.1.1.1.1.1.2.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.cmml"><msup id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.cmml"><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.2.cmml">n</mi><mo id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.3.cmml">∗</mo></msup><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.1.cmml">​</mo><mtext id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.1a" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.4" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.4.cmml">j</mi></mrow><mrow id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1a" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.4" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1b" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.5" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.5.cmml">t</mi></mrow></msubsup><mo id="alg3.l9.m1.1.1.1.1.1.1.1.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.1.cmml">+</mo><msubsup id="alg3.l9.m1.1.1.1.1.1.1.1.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.cmml"><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.2.cmml">𝑿</mi><mrow id="alg3.l9.m1.1.1.1.1.1.1.1.3.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.cmml"><msup id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.cmml"><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.2.cmml">n</mi><mo id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.3.cmml">∗</mo></msup><mo id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.1.cmml">+</mo><mrow id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.cmml"><mn id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.1.cmml">​</mo><mtext id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.3a.cmml">, </mtext><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.1a" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.4" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.4.cmml">j</mi></mrow></mrow><mrow id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.cmml"><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.2" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1a" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.4" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1b" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.5" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.5.cmml">t</mi></mrow></msubsup></mrow><mo stretchy="false" id="alg3.l9.m1.1.1.1.1.1.1.3" xref="alg3.l9.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="alg3.l9.m1.1.1.1.2" xref="alg3.l9.m1.1.1.1.2.cmml">/</mo><mn id="alg3.l9.m1.1.1.1.3" xref="alg3.l9.m1.1.1.1.3.cmml">2</mn></mrow><mo lspace="0em" rspace="0em" id="alg3.l9.m1.1.1.2" xref="alg3.l9.m1.1.1.2.cmml">​</mo><msub id="alg3.l9.m1.1.1.3" xref="alg3.l9.m1.1.1.3.cmml"><mrow id="alg3.l9.m1.1.1.3.2" xref="alg3.l9.m1.1.1.3.2c.cmml"><mtext id="alg3.l9.m1.1.1.3.2a" xref="alg3.l9.m1.1.1.3.2c.cmml">, </mtext><mtext class="ltx_mathvariant_monospace" id="alg3.l9.m1.1.1.3.2b" xref="alg3.l9.m1.1.1.3.2c.cmml">losses</mtext></mrow><msup id="alg3.l9.m1.1.1.3.3" xref="alg3.l9.m1.1.1.3.3.cmml"><mi id="alg3.l9.m1.1.1.3.3.2" xref="alg3.l9.m1.1.1.3.3.2.cmml">n</mi><mo id="alg3.l9.m1.1.1.3.3.3" xref="alg3.l9.m1.1.1.3.3.3.cmml">∗</mo></msup></msub></mrow><annotation-xml encoding="MathML-Content" id="alg3.l9.m1.1b"><apply id="alg3.l9.m1.1.1.cmml" xref="alg3.l9.m1.1.1"><times id="alg3.l9.m1.1.1.2.cmml" xref="alg3.l9.m1.1.1.2"></times><apply id="alg3.l9.m1.1.1.1.cmml" xref="alg3.l9.m1.1.1.1"><divide id="alg3.l9.m1.1.1.1.2.cmml" xref="alg3.l9.m1.1.1.1.2"></divide><apply id="alg3.l9.m1.1.1.1.1.cmml" xref="alg3.l9.m1.1.1.1.1"><times id="alg3.l9.m1.1.1.1.1.2.cmml" xref="alg3.l9.m1.1.1.1.1.2"></times><ci id="alg3.l9.m1.1.1.1.1.3a.cmml" xref="alg3.l9.m1.1.1.1.1.3"><mtext class="ltx_mathvariant_bold" id="alg3.l9.m1.1.1.1.1.3.cmml" xref="alg3.l9.m1.1.1.1.1.3">return </mtext></ci><apply id="alg3.l9.m1.1.1.1.1.1.1.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1"><plus id="alg3.l9.m1.1.1.1.1.1.1.1.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.1"></plus><apply id="alg3.l9.m1.1.1.1.1.1.1.1.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.1.1.1.1.1.2.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2">subscript</csymbol><apply id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.2">𝑿</ci><apply id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3"><times id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.1"></times><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.2">𝑠</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.3">𝑜</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.4.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.4">𝑟</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.5.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.2.3.5">𝑡</ci></apply></apply><apply id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3"><times id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.1"></times><apply id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2">superscript</csymbol><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.2">𝑛</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.2.3">∗</ci></apply><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.3a.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.3"><mtext mathsize="70%" id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.3">, </mtext></ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.2.3.4.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.2.3.4">𝑗</ci></apply></apply><apply id="alg3.l9.m1.1.1.1.1.1.1.1.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.1.1.1.1.1.3.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.2">𝑿</ci><apply id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3"><times id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.1"></times><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.2">𝑠</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.3">𝑜</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.4.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.4">𝑟</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.5.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.2.3.5">𝑡</ci></apply></apply><apply id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3"><plus id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.1"></plus><apply id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2">superscript</csymbol><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.2">𝑛</ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.2.3">∗</ci></apply><apply id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3"><times id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.1.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.1"></times><cn type="integer" id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.2.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.2">1</cn><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.3a.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.3"><mtext mathsize="70%" id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.3.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.3">, </mtext></ci><ci id="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.4.cmml" xref="alg3.l9.m1.1.1.1.1.1.1.1.3.3.3.4">𝑗</ci></apply></apply></apply></apply></apply><cn type="integer" id="alg3.l9.m1.1.1.1.3.cmml" xref="alg3.l9.m1.1.1.1.3">2</cn></apply><apply id="alg3.l9.m1.1.1.3.cmml" xref="alg3.l9.m1.1.1.3"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.3.1.cmml" xref="alg3.l9.m1.1.1.3">subscript</csymbol><ci id="alg3.l9.m1.1.1.3.2c.cmml" xref="alg3.l9.m1.1.1.3.2"><mrow id="alg3.l9.m1.1.1.3.2.cmml" xref="alg3.l9.m1.1.1.3.2"><mtext id="alg3.l9.m1.1.1.3.2a.cmml" xref="alg3.l9.m1.1.1.3.2">, </mtext><mtext class="ltx_mathvariant_monospace" id="alg3.l9.m1.1.1.3.2b.cmml" xref="alg3.l9.m1.1.1.3.2">losses</mtext></mrow></ci><apply id="alg3.l9.m1.1.1.3.3.cmml" xref="alg3.l9.m1.1.1.3.3"><csymbol cd="ambiguous" id="alg3.l9.m1.1.1.3.3.1.cmml" xref="alg3.l9.m1.1.1.3.3">superscript</csymbol><ci id="alg3.l9.m1.1.1.3.3.2.cmml" xref="alg3.l9.m1.1.1.3.3.2">𝑛</ci><ci id="alg3.l9.m1.1.1.3.3.3.cmml" xref="alg3.l9.m1.1.1.3.3.3">∗</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l9.m1.1c">\textbf{return }(\bm{X}^{sort}_{n^{\ast}\text{, }j}+\bm{X}^{sort}_{n^{\ast}+1\text{, }j})/2\text{, }\texttt{losses}_{n^{\ast}}</annotation></semantics></math>

</div>
</div>
</figure>
<figure id="alg4" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg4.2.1.1" class="ltx_text ltx_font_bold">Algorithm 4</span> </span> Cumulative SSE</figcaption>
<div id="alg4.3" class="ltx_listing ltx_listing">
<div id="alg4.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span>  <span id="alg4.l1.2" class="ltx_text ltx_font_bold">Input:</span> 2D array <math id="alg4.l1.m1.1" class="ltx_Math" alttext="\bm{X}" display="inline"><semantics id="alg4.l1.m1.1a"><mi id="alg4.l1.m1.1.1" xref="alg4.l1.m1.1.1.cmml">𝑿</mi><annotation-xml encoding="MathML-Content" id="alg4.l1.m1.1b"><ci id="alg4.l1.m1.1.1.cmml" xref="alg4.l1.m1.1.1">𝑿</ci></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m1.1c">\bm{X}</annotation></semantics></math>, boolean <span id="alg4.l1.3" class="ltx_text ltx_font_typewriter">reverse</span>

</div>
<div id="alg4.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>  <math id="alg4.l2.m1.3" class="ltx_Math" alttext="N,D\leftarrow\texttt{shape}(\bm{X})" display="inline"><semantics id="alg4.l2.m1.3a"><mrow id="alg4.l2.m1.3.4" xref="alg4.l2.m1.3.4.cmml"><mrow id="alg4.l2.m1.3.4.2.2" xref="alg4.l2.m1.3.4.2.1.cmml"><mi id="alg4.l2.m1.2.2" xref="alg4.l2.m1.2.2.cmml">N</mi><mo id="alg4.l2.m1.3.4.2.2.1" xref="alg4.l2.m1.3.4.2.1.cmml">,</mo><mi id="alg4.l2.m1.3.3" xref="alg4.l2.m1.3.3.cmml">D</mi></mrow><mo stretchy="false" id="alg4.l2.m1.3.4.1" xref="alg4.l2.m1.3.4.1.cmml">←</mo><mrow id="alg4.l2.m1.3.4.3" xref="alg4.l2.m1.3.4.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l2.m1.3.4.3.2" xref="alg4.l2.m1.3.4.3.2a.cmml">shape</mtext><mo lspace="0em" rspace="0em" id="alg4.l2.m1.3.4.3.1" xref="alg4.l2.m1.3.4.3.1.cmml">​</mo><mrow id="alg4.l2.m1.3.4.3.3.2" xref="alg4.l2.m1.3.4.3.cmml"><mo stretchy="false" id="alg4.l2.m1.3.4.3.3.2.1" xref="alg4.l2.m1.3.4.3.cmml">(</mo><mi id="alg4.l2.m1.1.1" xref="alg4.l2.m1.1.1.cmml">𝑿</mi><mo stretchy="false" id="alg4.l2.m1.3.4.3.3.2.2" xref="alg4.l2.m1.3.4.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l2.m1.3b"><apply id="alg4.l2.m1.3.4.cmml" xref="alg4.l2.m1.3.4"><ci id="alg4.l2.m1.3.4.1.cmml" xref="alg4.l2.m1.3.4.1">←</ci><list id="alg4.l2.m1.3.4.2.1.cmml" xref="alg4.l2.m1.3.4.2.2"><ci id="alg4.l2.m1.2.2.cmml" xref="alg4.l2.m1.2.2">𝑁</ci><ci id="alg4.l2.m1.3.3.cmml" xref="alg4.l2.m1.3.3">𝐷</ci></list><apply id="alg4.l2.m1.3.4.3.cmml" xref="alg4.l2.m1.3.4.3"><times id="alg4.l2.m1.3.4.3.1.cmml" xref="alg4.l2.m1.3.4.3.1"></times><ci id="alg4.l2.m1.3.4.3.2a.cmml" xref="alg4.l2.m1.3.4.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l2.m1.3.4.3.2.cmml" xref="alg4.l2.m1.3.4.3.2">shape</mtext></ci><ci id="alg4.l2.m1.1.1.cmml" xref="alg4.l2.m1.1.1">𝑿</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l2.m1.3c">N,D\leftarrow\texttt{shape}(\bm{X})</annotation></semantics></math>

</div>
<div id="alg4.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>  <span id="alg4.l3.2" class="ltx_text ltx_font_bold">if</span> <span id="alg4.l3.3" class="ltx_text ltx_font_typewriter">reverse</span> <span id="alg4.l3.4" class="ltx_text ltx_font_bold">then</span>



</div>
<div id="alg4.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span>     <math id="alg4.l4.m1.6" class="ltx_Math" alttext="\forall_{i}\texttt{ swap}(\bm{X}_{i,d},\bm{X}_{N-i+1,d})" display="inline"><semantics id="alg4.l4.m1.6a"><mrow id="alg4.l4.m1.6.6" xref="alg4.l4.m1.6.6.cmml"><msub id="alg4.l4.m1.6.6.3" xref="alg4.l4.m1.6.6.3.cmml"><mo id="alg4.l4.m1.6.6.3.2" xref="alg4.l4.m1.6.6.3.2.cmml">∀</mo><mi id="alg4.l4.m1.6.6.3.3" xref="alg4.l4.m1.6.6.3.3.cmml">i</mi></msub><mrow id="alg4.l4.m1.6.6.2" xref="alg4.l4.m1.6.6.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l4.m1.6.6.2.4" xref="alg4.l4.m1.6.6.2.4a.cmml"> swap</mtext><mo lspace="0em" rspace="0em" id="alg4.l4.m1.6.6.2.3" xref="alg4.l4.m1.6.6.2.3.cmml">​</mo><mrow id="alg4.l4.m1.6.6.2.2.2" xref="alg4.l4.m1.6.6.2.2.3.cmml"><mo stretchy="false" id="alg4.l4.m1.6.6.2.2.2.3" xref="alg4.l4.m1.6.6.2.2.3.cmml">(</mo><msub id="alg4.l4.m1.5.5.1.1.1.1" xref="alg4.l4.m1.5.5.1.1.1.1.cmml"><mi id="alg4.l4.m1.5.5.1.1.1.1.2" xref="alg4.l4.m1.5.5.1.1.1.1.2.cmml">𝑿</mi><mrow id="alg4.l4.m1.2.2.2.4" xref="alg4.l4.m1.2.2.2.3.cmml"><mi id="alg4.l4.m1.1.1.1.1" xref="alg4.l4.m1.1.1.1.1.cmml">i</mi><mo id="alg4.l4.m1.2.2.2.4.1" xref="alg4.l4.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l4.m1.2.2.2.2" xref="alg4.l4.m1.2.2.2.2.cmml">d</mi></mrow></msub><mo id="alg4.l4.m1.6.6.2.2.2.4" xref="alg4.l4.m1.6.6.2.2.3.cmml">,</mo><msub id="alg4.l4.m1.6.6.2.2.2.2" xref="alg4.l4.m1.6.6.2.2.2.2.cmml"><mi id="alg4.l4.m1.6.6.2.2.2.2.2" xref="alg4.l4.m1.6.6.2.2.2.2.2.cmml">𝑿</mi><mrow id="alg4.l4.m1.4.4.2.2" xref="alg4.l4.m1.4.4.2.3.cmml"><mrow id="alg4.l4.m1.4.4.2.2.1" xref="alg4.l4.m1.4.4.2.2.1.cmml"><mrow id="alg4.l4.m1.4.4.2.2.1.2" xref="alg4.l4.m1.4.4.2.2.1.2.cmml"><mi id="alg4.l4.m1.4.4.2.2.1.2.2" xref="alg4.l4.m1.4.4.2.2.1.2.2.cmml">N</mi><mo id="alg4.l4.m1.4.4.2.2.1.2.1" xref="alg4.l4.m1.4.4.2.2.1.2.1.cmml">−</mo><mi id="alg4.l4.m1.4.4.2.2.1.2.3" xref="alg4.l4.m1.4.4.2.2.1.2.3.cmml">i</mi></mrow><mo id="alg4.l4.m1.4.4.2.2.1.1" xref="alg4.l4.m1.4.4.2.2.1.1.cmml">+</mo><mn id="alg4.l4.m1.4.4.2.2.1.3" xref="alg4.l4.m1.4.4.2.2.1.3.cmml">1</mn></mrow><mo id="alg4.l4.m1.4.4.2.2.2" xref="alg4.l4.m1.4.4.2.3.cmml">,</mo><mi id="alg4.l4.m1.3.3.1.1" xref="alg4.l4.m1.3.3.1.1.cmml">d</mi></mrow></msub><mo stretchy="false" id="alg4.l4.m1.6.6.2.2.2.5" xref="alg4.l4.m1.6.6.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l4.m1.6b"><apply id="alg4.l4.m1.6.6.cmml" xref="alg4.l4.m1.6.6"><apply id="alg4.l4.m1.6.6.3.cmml" xref="alg4.l4.m1.6.6.3"><csymbol cd="ambiguous" id="alg4.l4.m1.6.6.3.1.cmml" xref="alg4.l4.m1.6.6.3">subscript</csymbol><csymbol cd="latexml" id="alg4.l4.m1.6.6.3.2.cmml" xref="alg4.l4.m1.6.6.3.2">for-all</csymbol><ci id="alg4.l4.m1.6.6.3.3.cmml" xref="alg4.l4.m1.6.6.3.3">𝑖</ci></apply><apply id="alg4.l4.m1.6.6.2.cmml" xref="alg4.l4.m1.6.6.2"><times id="alg4.l4.m1.6.6.2.3.cmml" xref="alg4.l4.m1.6.6.2.3"></times><ci id="alg4.l4.m1.6.6.2.4a.cmml" xref="alg4.l4.m1.6.6.2.4"><mtext class="ltx_mathvariant_monospace" id="alg4.l4.m1.6.6.2.4.cmml" xref="alg4.l4.m1.6.6.2.4"> swap</mtext></ci><interval closure="open" id="alg4.l4.m1.6.6.2.2.3.cmml" xref="alg4.l4.m1.6.6.2.2.2"><apply id="alg4.l4.m1.5.5.1.1.1.1.cmml" xref="alg4.l4.m1.5.5.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l4.m1.5.5.1.1.1.1.1.cmml" xref="alg4.l4.m1.5.5.1.1.1.1">subscript</csymbol><ci id="alg4.l4.m1.5.5.1.1.1.1.2.cmml" xref="alg4.l4.m1.5.5.1.1.1.1.2">𝑿</ci><list id="alg4.l4.m1.2.2.2.3.cmml" xref="alg4.l4.m1.2.2.2.4"><ci id="alg4.l4.m1.1.1.1.1.cmml" xref="alg4.l4.m1.1.1.1.1">𝑖</ci><ci id="alg4.l4.m1.2.2.2.2.cmml" xref="alg4.l4.m1.2.2.2.2">𝑑</ci></list></apply><apply id="alg4.l4.m1.6.6.2.2.2.2.cmml" xref="alg4.l4.m1.6.6.2.2.2.2"><csymbol cd="ambiguous" id="alg4.l4.m1.6.6.2.2.2.2.1.cmml" xref="alg4.l4.m1.6.6.2.2.2.2">subscript</csymbol><ci id="alg4.l4.m1.6.6.2.2.2.2.2.cmml" xref="alg4.l4.m1.6.6.2.2.2.2.2">𝑿</ci><list id="alg4.l4.m1.4.4.2.3.cmml" xref="alg4.l4.m1.4.4.2.2"><apply id="alg4.l4.m1.4.4.2.2.1.cmml" xref="alg4.l4.m1.4.4.2.2.1"><plus id="alg4.l4.m1.4.4.2.2.1.1.cmml" xref="alg4.l4.m1.4.4.2.2.1.1"></plus><apply id="alg4.l4.m1.4.4.2.2.1.2.cmml" xref="alg4.l4.m1.4.4.2.2.1.2"><minus id="alg4.l4.m1.4.4.2.2.1.2.1.cmml" xref="alg4.l4.m1.4.4.2.2.1.2.1"></minus><ci id="alg4.l4.m1.4.4.2.2.1.2.2.cmml" xref="alg4.l4.m1.4.4.2.2.1.2.2">𝑁</ci><ci id="alg4.l4.m1.4.4.2.2.1.2.3.cmml" xref="alg4.l4.m1.4.4.2.2.1.2.3">𝑖</ci></apply><cn type="integer" id="alg4.l4.m1.4.4.2.2.1.3.cmml" xref="alg4.l4.m1.4.4.2.2.1.3">1</cn></apply><ci id="alg4.l4.m1.3.3.1.1.cmml" xref="alg4.l4.m1.3.3.1.1">𝑑</ci></list></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l4.m1.6c">\forall_{i}\texttt{ swap}(\bm{X}_{i,d},\bm{X}_{N-i+1,d})</annotation></semantics></math>

</div>
<div id="alg4.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>  <span id="alg4.l5.2" class="ltx_text ltx_font_bold">end</span> <span id="alg4.l5.3" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg4.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>  <math id="alg4.l6.m1.1" class="ltx_Math" alttext="\texttt{out}\leftarrow\texttt{empty}(N)" display="inline"><semantics id="alg4.l6.m1.1a"><mrow id="alg4.l6.m1.1.2" xref="alg4.l6.m1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l6.m1.1.2.2" xref="alg4.l6.m1.1.2.2a.cmml">out</mtext><mo stretchy="false" id="alg4.l6.m1.1.2.1" xref="alg4.l6.m1.1.2.1.cmml">←</mo><mrow id="alg4.l6.m1.1.2.3" xref="alg4.l6.m1.1.2.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l6.m1.1.2.3.2" xref="alg4.l6.m1.1.2.3.2a.cmml">empty</mtext><mo lspace="0em" rspace="0em" id="alg4.l6.m1.1.2.3.1" xref="alg4.l6.m1.1.2.3.1.cmml">​</mo><mrow id="alg4.l6.m1.1.2.3.3.2" xref="alg4.l6.m1.1.2.3.cmml"><mo stretchy="false" id="alg4.l6.m1.1.2.3.3.2.1" xref="alg4.l6.m1.1.2.3.cmml">(</mo><mi id="alg4.l6.m1.1.1" xref="alg4.l6.m1.1.1.cmml">N</mi><mo stretchy="false" id="alg4.l6.m1.1.2.3.3.2.2" xref="alg4.l6.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l6.m1.1b"><apply id="alg4.l6.m1.1.2.cmml" xref="alg4.l6.m1.1.2"><ci id="alg4.l6.m1.1.2.1.cmml" xref="alg4.l6.m1.1.2.1">←</ci><ci id="alg4.l6.m1.1.2.2a.cmml" xref="alg4.l6.m1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l6.m1.1.2.2.cmml" xref="alg4.l6.m1.1.2.2">out</mtext></ci><apply id="alg4.l6.m1.1.2.3.cmml" xref="alg4.l6.m1.1.2.3"><times id="alg4.l6.m1.1.2.3.1.cmml" xref="alg4.l6.m1.1.2.3.1"></times><ci id="alg4.l6.m1.1.2.3.2a.cmml" xref="alg4.l6.m1.1.2.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l6.m1.1.2.3.2.cmml" xref="alg4.l6.m1.1.2.3.2">empty</mtext></ci><ci id="alg4.l6.m1.1.1.cmml" xref="alg4.l6.m1.1.1">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l6.m1.1c">\texttt{out}\leftarrow\texttt{empty}(N)</annotation></semantics></math>

</div>
<div id="alg4.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>  <math id="alg4.l7.m1.1" class="ltx_Math" alttext="\texttt{cumX}\leftarrow\texttt{empty}(D)" display="inline"><semantics id="alg4.l7.m1.1a"><mrow id="alg4.l7.m1.1.2" xref="alg4.l7.m1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l7.m1.1.2.2" xref="alg4.l7.m1.1.2.2a.cmml">cumX</mtext><mo stretchy="false" id="alg4.l7.m1.1.2.1" xref="alg4.l7.m1.1.2.1.cmml">←</mo><mrow id="alg4.l7.m1.1.2.3" xref="alg4.l7.m1.1.2.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l7.m1.1.2.3.2" xref="alg4.l7.m1.1.2.3.2a.cmml">empty</mtext><mo lspace="0em" rspace="0em" id="alg4.l7.m1.1.2.3.1" xref="alg4.l7.m1.1.2.3.1.cmml">​</mo><mrow id="alg4.l7.m1.1.2.3.3.2" xref="alg4.l7.m1.1.2.3.cmml"><mo stretchy="false" id="alg4.l7.m1.1.2.3.3.2.1" xref="alg4.l7.m1.1.2.3.cmml">(</mo><mi id="alg4.l7.m1.1.1" xref="alg4.l7.m1.1.1.cmml">D</mi><mo stretchy="false" id="alg4.l7.m1.1.2.3.3.2.2" xref="alg4.l7.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l7.m1.1b"><apply id="alg4.l7.m1.1.2.cmml" xref="alg4.l7.m1.1.2"><ci id="alg4.l7.m1.1.2.1.cmml" xref="alg4.l7.m1.1.2.1">←</ci><ci id="alg4.l7.m1.1.2.2a.cmml" xref="alg4.l7.m1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l7.m1.1.2.2.cmml" xref="alg4.l7.m1.1.2.2">cumX</mtext></ci><apply id="alg4.l7.m1.1.2.3.cmml" xref="alg4.l7.m1.1.2.3"><times id="alg4.l7.m1.1.2.3.1.cmml" xref="alg4.l7.m1.1.2.3.1"></times><ci id="alg4.l7.m1.1.2.3.2a.cmml" xref="alg4.l7.m1.1.2.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l7.m1.1.2.3.2.cmml" xref="alg4.l7.m1.1.2.3.2">empty</mtext></ci><ci id="alg4.l7.m1.1.1.cmml" xref="alg4.l7.m1.1.1">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l7.m1.1c">\texttt{cumX}\leftarrow\texttt{empty}(D)</annotation></semantics></math>

</div>
<div id="alg4.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>  <math id="alg4.l8.m1.1" class="ltx_Math" alttext="\texttt{cumX2}\leftarrow\texttt{empty}(D)" display="inline"><semantics id="alg4.l8.m1.1a"><mrow id="alg4.l8.m1.1.2" xref="alg4.l8.m1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l8.m1.1.2.2" xref="alg4.l8.m1.1.2.2a.cmml">cumX2</mtext><mo stretchy="false" id="alg4.l8.m1.1.2.1" xref="alg4.l8.m1.1.2.1.cmml">←</mo><mrow id="alg4.l8.m1.1.2.3" xref="alg4.l8.m1.1.2.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l8.m1.1.2.3.2" xref="alg4.l8.m1.1.2.3.2a.cmml">empty</mtext><mo lspace="0em" rspace="0em" id="alg4.l8.m1.1.2.3.1" xref="alg4.l8.m1.1.2.3.1.cmml">​</mo><mrow id="alg4.l8.m1.1.2.3.3.2" xref="alg4.l8.m1.1.2.3.cmml"><mo stretchy="false" id="alg4.l8.m1.1.2.3.3.2.1" xref="alg4.l8.m1.1.2.3.cmml">(</mo><mi id="alg4.l8.m1.1.1" xref="alg4.l8.m1.1.1.cmml">D</mi><mo stretchy="false" id="alg4.l8.m1.1.2.3.3.2.2" xref="alg4.l8.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l8.m1.1b"><apply id="alg4.l8.m1.1.2.cmml" xref="alg4.l8.m1.1.2"><ci id="alg4.l8.m1.1.2.1.cmml" xref="alg4.l8.m1.1.2.1">←</ci><ci id="alg4.l8.m1.1.2.2a.cmml" xref="alg4.l8.m1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l8.m1.1.2.2.cmml" xref="alg4.l8.m1.1.2.2">cumX2</mtext></ci><apply id="alg4.l8.m1.1.2.3.cmml" xref="alg4.l8.m1.1.2.3"><times id="alg4.l8.m1.1.2.3.1.cmml" xref="alg4.l8.m1.1.2.3.1"></times><ci id="alg4.l8.m1.1.2.3.2a.cmml" xref="alg4.l8.m1.1.2.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l8.m1.1.2.3.2.cmml" xref="alg4.l8.m1.1.2.3.2">empty</mtext></ci><ci id="alg4.l8.m1.1.1.cmml" xref="alg4.l8.m1.1.1">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l8.m1.1c">\texttt{cumX2}\leftarrow\texttt{empty}(D)</annotation></semantics></math>

<span id="alg4.l8.2" class="ltx_text" style="color:#009900;">
// Initialize first row of output and cumulative values
</span>


</div>
<div id="alg4.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>  <math id="alg4.l9.m1.1" class="ltx_Math" alttext="\texttt{out}_{1}\leftarrow 0" display="inline"><semantics id="alg4.l9.m1.1a"><mrow id="alg4.l9.m1.1.1" xref="alg4.l9.m1.1.1.cmml"><msub id="alg4.l9.m1.1.1.2" xref="alg4.l9.m1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l9.m1.1.1.2.2" xref="alg4.l9.m1.1.1.2.2a.cmml">out</mtext><mn id="alg4.l9.m1.1.1.2.3" xref="alg4.l9.m1.1.1.2.3.cmml">1</mn></msub><mo stretchy="false" id="alg4.l9.m1.1.1.1" xref="alg4.l9.m1.1.1.1.cmml">←</mo><mn id="alg4.l9.m1.1.1.3" xref="alg4.l9.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg4.l9.m1.1b"><apply id="alg4.l9.m1.1.1.cmml" xref="alg4.l9.m1.1.1"><ci id="alg4.l9.m1.1.1.1.cmml" xref="alg4.l9.m1.1.1.1">←</ci><apply id="alg4.l9.m1.1.1.2.cmml" xref="alg4.l9.m1.1.1.2"><csymbol cd="ambiguous" id="alg4.l9.m1.1.1.2.1.cmml" xref="alg4.l9.m1.1.1.2">subscript</csymbol><ci id="alg4.l9.m1.1.1.2.2a.cmml" xref="alg4.l9.m1.1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l9.m1.1.1.2.2.cmml" xref="alg4.l9.m1.1.1.2.2">out</mtext></ci><cn type="integer" id="alg4.l9.m1.1.1.2.3.cmml" xref="alg4.l9.m1.1.1.2.3">1</cn></apply><cn type="integer" id="alg4.l9.m1.1.1.3.cmml" xref="alg4.l9.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l9.m1.1c">\texttt{out}_{1}\leftarrow 0</annotation></semantics></math>

</div>
<div id="alg4.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span>  <span id="alg4.l10.2" class="ltx_text ltx_font_bold">for</span> <math id="alg4.l10.m1.1" class="ltx_Math" alttext="d\leftarrow 1\textbf{ to }D" display="inline"><semantics id="alg4.l10.m1.1a"><mrow id="alg4.l10.m1.1.1" xref="alg4.l10.m1.1.1.cmml"><mi id="alg4.l10.m1.1.1.2" xref="alg4.l10.m1.1.1.2.cmml">d</mi><mo stretchy="false" id="alg4.l10.m1.1.1.1" xref="alg4.l10.m1.1.1.1.cmml">←</mo><mrow id="alg4.l10.m1.1.1.3" xref="alg4.l10.m1.1.1.3.cmml"><mn id="alg4.l10.m1.1.1.3.2" xref="alg4.l10.m1.1.1.3.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.3.1" xref="alg4.l10.m1.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg4.l10.m1.1.1.3.3" xref="alg4.l10.m1.1.1.3.3a.cmml"> to </mtext><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.3.1a" xref="alg4.l10.m1.1.1.3.1.cmml">​</mo><mi id="alg4.l10.m1.1.1.3.4" xref="alg4.l10.m1.1.1.3.4.cmml">D</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l10.m1.1b"><apply id="alg4.l10.m1.1.1.cmml" xref="alg4.l10.m1.1.1"><ci id="alg4.l10.m1.1.1.1.cmml" xref="alg4.l10.m1.1.1.1">←</ci><ci id="alg4.l10.m1.1.1.2.cmml" xref="alg4.l10.m1.1.1.2">𝑑</ci><apply id="alg4.l10.m1.1.1.3.cmml" xref="alg4.l10.m1.1.1.3"><times id="alg4.l10.m1.1.1.3.1.cmml" xref="alg4.l10.m1.1.1.3.1"></times><cn type="integer" id="alg4.l10.m1.1.1.3.2.cmml" xref="alg4.l10.m1.1.1.3.2">1</cn><ci id="alg4.l10.m1.1.1.3.3a.cmml" xref="alg4.l10.m1.1.1.3.3"><mtext class="ltx_mathvariant_bold" id="alg4.l10.m1.1.1.3.3.cmml" xref="alg4.l10.m1.1.1.3.3"> to </mtext></ci><ci id="alg4.l10.m1.1.1.3.4.cmml" xref="alg4.l10.m1.1.1.3.4">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l10.m1.1c">d\leftarrow 1\textbf{ to }D</annotation></semantics></math> <span id="alg4.l10.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg4.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span>     <math id="alg4.l11.m1.2" class="ltx_Math" alttext="\texttt{cumX}_{d}\leftarrow X_{1,d}" display="inline"><semantics id="alg4.l11.m1.2a"><mrow id="alg4.l11.m1.2.3" xref="alg4.l11.m1.2.3.cmml"><msub id="alg4.l11.m1.2.3.2" xref="alg4.l11.m1.2.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l11.m1.2.3.2.2" xref="alg4.l11.m1.2.3.2.2a.cmml">cumX</mtext><mi id="alg4.l11.m1.2.3.2.3" xref="alg4.l11.m1.2.3.2.3.cmml">d</mi></msub><mo stretchy="false" id="alg4.l11.m1.2.3.1" xref="alg4.l11.m1.2.3.1.cmml">←</mo><msub id="alg4.l11.m1.2.3.3" xref="alg4.l11.m1.2.3.3.cmml"><mi id="alg4.l11.m1.2.3.3.2" xref="alg4.l11.m1.2.3.3.2.cmml">X</mi><mrow id="alg4.l11.m1.2.2.2.4" xref="alg4.l11.m1.2.2.2.3.cmml"><mn id="alg4.l11.m1.1.1.1.1" xref="alg4.l11.m1.1.1.1.1.cmml">1</mn><mo id="alg4.l11.m1.2.2.2.4.1" xref="alg4.l11.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l11.m1.2.2.2.2" xref="alg4.l11.m1.2.2.2.2.cmml">d</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg4.l11.m1.2b"><apply id="alg4.l11.m1.2.3.cmml" xref="alg4.l11.m1.2.3"><ci id="alg4.l11.m1.2.3.1.cmml" xref="alg4.l11.m1.2.3.1">←</ci><apply id="alg4.l11.m1.2.3.2.cmml" xref="alg4.l11.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l11.m1.2.3.2.1.cmml" xref="alg4.l11.m1.2.3.2">subscript</csymbol><ci id="alg4.l11.m1.2.3.2.2a.cmml" xref="alg4.l11.m1.2.3.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l11.m1.2.3.2.2.cmml" xref="alg4.l11.m1.2.3.2.2">cumX</mtext></ci><ci id="alg4.l11.m1.2.3.2.3.cmml" xref="alg4.l11.m1.2.3.2.3">𝑑</ci></apply><apply id="alg4.l11.m1.2.3.3.cmml" xref="alg4.l11.m1.2.3.3"><csymbol cd="ambiguous" id="alg4.l11.m1.2.3.3.1.cmml" xref="alg4.l11.m1.2.3.3">subscript</csymbol><ci id="alg4.l11.m1.2.3.3.2.cmml" xref="alg4.l11.m1.2.3.3.2">𝑋</ci><list id="alg4.l11.m1.2.2.2.3.cmml" xref="alg4.l11.m1.2.2.2.4"><cn type="integer" id="alg4.l11.m1.1.1.1.1.cmml" xref="alg4.l11.m1.1.1.1.1">1</cn><ci id="alg4.l11.m1.2.2.2.2.cmml" xref="alg4.l11.m1.2.2.2.2">𝑑</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l11.m1.2c">\texttt{cumX}_{d}\leftarrow X_{1,d}</annotation></semantics></math>

</div>
<div id="alg4.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span>     <math id="alg4.l12.m1.3" class="ltx_Math" alttext="\texttt{cumX2}_{d}\leftarrow(X_{1,d})^{2}" display="inline"><semantics id="alg4.l12.m1.3a"><mrow id="alg4.l12.m1.3.3" xref="alg4.l12.m1.3.3.cmml"><msub id="alg4.l12.m1.3.3.3" xref="alg4.l12.m1.3.3.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l12.m1.3.3.3.2" xref="alg4.l12.m1.3.3.3.2a.cmml">cumX2</mtext><mi id="alg4.l12.m1.3.3.3.3" xref="alg4.l12.m1.3.3.3.3.cmml">d</mi></msub><mo stretchy="false" id="alg4.l12.m1.3.3.2" xref="alg4.l12.m1.3.3.2.cmml">←</mo><msup id="alg4.l12.m1.3.3.1" xref="alg4.l12.m1.3.3.1.cmml"><mrow id="alg4.l12.m1.3.3.1.1.1" xref="alg4.l12.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="alg4.l12.m1.3.3.1.1.1.2" xref="alg4.l12.m1.3.3.1.1.1.1.cmml">(</mo><msub id="alg4.l12.m1.3.3.1.1.1.1" xref="alg4.l12.m1.3.3.1.1.1.1.cmml"><mi id="alg4.l12.m1.3.3.1.1.1.1.2" xref="alg4.l12.m1.3.3.1.1.1.1.2.cmml">X</mi><mrow id="alg4.l12.m1.2.2.2.4" xref="alg4.l12.m1.2.2.2.3.cmml"><mn id="alg4.l12.m1.1.1.1.1" xref="alg4.l12.m1.1.1.1.1.cmml">1</mn><mo id="alg4.l12.m1.2.2.2.4.1" xref="alg4.l12.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l12.m1.2.2.2.2" xref="alg4.l12.m1.2.2.2.2.cmml">d</mi></mrow></msub><mo stretchy="false" id="alg4.l12.m1.3.3.1.1.1.3" xref="alg4.l12.m1.3.3.1.1.1.1.cmml">)</mo></mrow><mn id="alg4.l12.m1.3.3.1.3" xref="alg4.l12.m1.3.3.1.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="alg4.l12.m1.3b"><apply id="alg4.l12.m1.3.3.cmml" xref="alg4.l12.m1.3.3"><ci id="alg4.l12.m1.3.3.2.cmml" xref="alg4.l12.m1.3.3.2">←</ci><apply id="alg4.l12.m1.3.3.3.cmml" xref="alg4.l12.m1.3.3.3"><csymbol cd="ambiguous" id="alg4.l12.m1.3.3.3.1.cmml" xref="alg4.l12.m1.3.3.3">subscript</csymbol><ci id="alg4.l12.m1.3.3.3.2a.cmml" xref="alg4.l12.m1.3.3.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l12.m1.3.3.3.2.cmml" xref="alg4.l12.m1.3.3.3.2">cumX2</mtext></ci><ci id="alg4.l12.m1.3.3.3.3.cmml" xref="alg4.l12.m1.3.3.3.3">𝑑</ci></apply><apply id="alg4.l12.m1.3.3.1.cmml" xref="alg4.l12.m1.3.3.1"><csymbol cd="ambiguous" id="alg4.l12.m1.3.3.1.2.cmml" xref="alg4.l12.m1.3.3.1">superscript</csymbol><apply id="alg4.l12.m1.3.3.1.1.1.1.cmml" xref="alg4.l12.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="alg4.l12.m1.3.3.1.1.1.1.1.cmml" xref="alg4.l12.m1.3.3.1.1.1">subscript</csymbol><ci id="alg4.l12.m1.3.3.1.1.1.1.2.cmml" xref="alg4.l12.m1.3.3.1.1.1.1.2">𝑋</ci><list id="alg4.l12.m1.2.2.2.3.cmml" xref="alg4.l12.m1.2.2.2.4"><cn type="integer" id="alg4.l12.m1.1.1.1.1.cmml" xref="alg4.l12.m1.1.1.1.1">1</cn><ci id="alg4.l12.m1.2.2.2.2.cmml" xref="alg4.l12.m1.2.2.2.2">𝑑</ci></list></apply><cn type="integer" id="alg4.l12.m1.3.3.1.3.cmml" xref="alg4.l12.m1.3.3.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l12.m1.3c">\texttt{cumX2}_{d}\leftarrow(X_{1,d})^{2}</annotation></semantics></math>

</div>
<div id="alg4.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span>  <span id="alg4.l13.2" class="ltx_text ltx_font_bold">end</span> <span id="alg4.l13.3" class="ltx_text ltx_font_bold">for</span>
<span id="alg4.l13.4" class="ltx_text" style="color:#009900;">
// Compute remaining output rows
</span>


</div>
<div id="alg4.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span>  <span id="alg4.l14.2" class="ltx_text ltx_font_bold">for</span> <math id="alg4.l14.m1.1" class="ltx_Math" alttext="n\leftarrow 2\textbf{ to }N" display="inline"><semantics id="alg4.l14.m1.1a"><mrow id="alg4.l14.m1.1.1" xref="alg4.l14.m1.1.1.cmml"><mi id="alg4.l14.m1.1.1.2" xref="alg4.l14.m1.1.1.2.cmml">n</mi><mo stretchy="false" id="alg4.l14.m1.1.1.1" xref="alg4.l14.m1.1.1.1.cmml">←</mo><mrow id="alg4.l14.m1.1.1.3" xref="alg4.l14.m1.1.1.3.cmml"><mn id="alg4.l14.m1.1.1.3.2" xref="alg4.l14.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="alg4.l14.m1.1.1.3.1" xref="alg4.l14.m1.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg4.l14.m1.1.1.3.3" xref="alg4.l14.m1.1.1.3.3a.cmml"> to </mtext><mo lspace="0em" rspace="0em" id="alg4.l14.m1.1.1.3.1a" xref="alg4.l14.m1.1.1.3.1.cmml">​</mo><mi id="alg4.l14.m1.1.1.3.4" xref="alg4.l14.m1.1.1.3.4.cmml">N</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l14.m1.1b"><apply id="alg4.l14.m1.1.1.cmml" xref="alg4.l14.m1.1.1"><ci id="alg4.l14.m1.1.1.1.cmml" xref="alg4.l14.m1.1.1.1">←</ci><ci id="alg4.l14.m1.1.1.2.cmml" xref="alg4.l14.m1.1.1.2">𝑛</ci><apply id="alg4.l14.m1.1.1.3.cmml" xref="alg4.l14.m1.1.1.3"><times id="alg4.l14.m1.1.1.3.1.cmml" xref="alg4.l14.m1.1.1.3.1"></times><cn type="integer" id="alg4.l14.m1.1.1.3.2.cmml" xref="alg4.l14.m1.1.1.3.2">2</cn><ci id="alg4.l14.m1.1.1.3.3a.cmml" xref="alg4.l14.m1.1.1.3.3"><mtext class="ltx_mathvariant_bold" id="alg4.l14.m1.1.1.3.3.cmml" xref="alg4.l14.m1.1.1.3.3"> to </mtext></ci><ci id="alg4.l14.m1.1.1.3.4.cmml" xref="alg4.l14.m1.1.1.3.4">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l14.m1.1c">n\leftarrow 2\textbf{ to }N</annotation></semantics></math> <span id="alg4.l14.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg4.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span>     <math id="alg4.l15.m1.1" class="ltx_Math" alttext="\texttt{out}_{n}\leftarrow 0" display="inline"><semantics id="alg4.l15.m1.1a"><mrow id="alg4.l15.m1.1.1" xref="alg4.l15.m1.1.1.cmml"><msub id="alg4.l15.m1.1.1.2" xref="alg4.l15.m1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l15.m1.1.1.2.2" xref="alg4.l15.m1.1.1.2.2a.cmml">out</mtext><mi id="alg4.l15.m1.1.1.2.3" xref="alg4.l15.m1.1.1.2.3.cmml">n</mi></msub><mo stretchy="false" id="alg4.l15.m1.1.1.1" xref="alg4.l15.m1.1.1.1.cmml">←</mo><mn id="alg4.l15.m1.1.1.3" xref="alg4.l15.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg4.l15.m1.1b"><apply id="alg4.l15.m1.1.1.cmml" xref="alg4.l15.m1.1.1"><ci id="alg4.l15.m1.1.1.1.cmml" xref="alg4.l15.m1.1.1.1">←</ci><apply id="alg4.l15.m1.1.1.2.cmml" xref="alg4.l15.m1.1.1.2"><csymbol cd="ambiguous" id="alg4.l15.m1.1.1.2.1.cmml" xref="alg4.l15.m1.1.1.2">subscript</csymbol><ci id="alg4.l15.m1.1.1.2.2a.cmml" xref="alg4.l15.m1.1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l15.m1.1.1.2.2.cmml" xref="alg4.l15.m1.1.1.2.2">out</mtext></ci><ci id="alg4.l15.m1.1.1.2.3.cmml" xref="alg4.l15.m1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="alg4.l15.m1.1.1.3.cmml" xref="alg4.l15.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l15.m1.1c">\texttt{out}_{n}\leftarrow 0</annotation></semantics></math>

</div>
<div id="alg4.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span>     <span id="alg4.l16.2" class="ltx_text ltx_font_bold">for</span> <math id="alg4.l16.m1.1" class="ltx_Math" alttext="d\leftarrow 1\textbf{ to }D" display="inline"><semantics id="alg4.l16.m1.1a"><mrow id="alg4.l16.m1.1.1" xref="alg4.l16.m1.1.1.cmml"><mi id="alg4.l16.m1.1.1.2" xref="alg4.l16.m1.1.1.2.cmml">d</mi><mo stretchy="false" id="alg4.l16.m1.1.1.1" xref="alg4.l16.m1.1.1.1.cmml">←</mo><mrow id="alg4.l16.m1.1.1.3" xref="alg4.l16.m1.1.1.3.cmml"><mn id="alg4.l16.m1.1.1.3.2" xref="alg4.l16.m1.1.1.3.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg4.l16.m1.1.1.3.1" xref="alg4.l16.m1.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg4.l16.m1.1.1.3.3" xref="alg4.l16.m1.1.1.3.3a.cmml"> to </mtext><mo lspace="0em" rspace="0em" id="alg4.l16.m1.1.1.3.1a" xref="alg4.l16.m1.1.1.3.1.cmml">​</mo><mi id="alg4.l16.m1.1.1.3.4" xref="alg4.l16.m1.1.1.3.4.cmml">D</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l16.m1.1b"><apply id="alg4.l16.m1.1.1.cmml" xref="alg4.l16.m1.1.1"><ci id="alg4.l16.m1.1.1.1.cmml" xref="alg4.l16.m1.1.1.1">←</ci><ci id="alg4.l16.m1.1.1.2.cmml" xref="alg4.l16.m1.1.1.2">𝑑</ci><apply id="alg4.l16.m1.1.1.3.cmml" xref="alg4.l16.m1.1.1.3"><times id="alg4.l16.m1.1.1.3.1.cmml" xref="alg4.l16.m1.1.1.3.1"></times><cn type="integer" id="alg4.l16.m1.1.1.3.2.cmml" xref="alg4.l16.m1.1.1.3.2">1</cn><ci id="alg4.l16.m1.1.1.3.3a.cmml" xref="alg4.l16.m1.1.1.3.3"><mtext class="ltx_mathvariant_bold" id="alg4.l16.m1.1.1.3.3.cmml" xref="alg4.l16.m1.1.1.3.3"> to </mtext></ci><ci id="alg4.l16.m1.1.1.3.4.cmml" xref="alg4.l16.m1.1.1.3.4">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l16.m1.1c">d\leftarrow 1\textbf{ to }D</annotation></semantics></math> <span id="alg4.l16.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg4.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span>        <math id="alg4.l17.m1.2" class="ltx_Math" alttext="\texttt{cumX}_{d}\leftarrow\texttt{cumX}_{d}+X_{1,d}" display="inline"><semantics id="alg4.l17.m1.2a"><mrow id="alg4.l17.m1.2.3" xref="alg4.l17.m1.2.3.cmml"><msub id="alg4.l17.m1.2.3.2" xref="alg4.l17.m1.2.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l17.m1.2.3.2.2" xref="alg4.l17.m1.2.3.2.2a.cmml">cumX</mtext><mi id="alg4.l17.m1.2.3.2.3" xref="alg4.l17.m1.2.3.2.3.cmml">d</mi></msub><mo stretchy="false" id="alg4.l17.m1.2.3.1" xref="alg4.l17.m1.2.3.1.cmml">←</mo><mrow id="alg4.l17.m1.2.3.3" xref="alg4.l17.m1.2.3.3.cmml"><msub id="alg4.l17.m1.2.3.3.2" xref="alg4.l17.m1.2.3.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l17.m1.2.3.3.2.2" xref="alg4.l17.m1.2.3.3.2.2a.cmml">cumX</mtext><mi id="alg4.l17.m1.2.3.3.2.3" xref="alg4.l17.m1.2.3.3.2.3.cmml">d</mi></msub><mo id="alg4.l17.m1.2.3.3.1" xref="alg4.l17.m1.2.3.3.1.cmml">+</mo><msub id="alg4.l17.m1.2.3.3.3" xref="alg4.l17.m1.2.3.3.3.cmml"><mi id="alg4.l17.m1.2.3.3.3.2" xref="alg4.l17.m1.2.3.3.3.2.cmml">X</mi><mrow id="alg4.l17.m1.2.2.2.4" xref="alg4.l17.m1.2.2.2.3.cmml"><mn id="alg4.l17.m1.1.1.1.1" xref="alg4.l17.m1.1.1.1.1.cmml">1</mn><mo id="alg4.l17.m1.2.2.2.4.1" xref="alg4.l17.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l17.m1.2.2.2.2" xref="alg4.l17.m1.2.2.2.2.cmml">d</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l17.m1.2b"><apply id="alg4.l17.m1.2.3.cmml" xref="alg4.l17.m1.2.3"><ci id="alg4.l17.m1.2.3.1.cmml" xref="alg4.l17.m1.2.3.1">←</ci><apply id="alg4.l17.m1.2.3.2.cmml" xref="alg4.l17.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.2.1.cmml" xref="alg4.l17.m1.2.3.2">subscript</csymbol><ci id="alg4.l17.m1.2.3.2.2a.cmml" xref="alg4.l17.m1.2.3.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l17.m1.2.3.2.2.cmml" xref="alg4.l17.m1.2.3.2.2">cumX</mtext></ci><ci id="alg4.l17.m1.2.3.2.3.cmml" xref="alg4.l17.m1.2.3.2.3">𝑑</ci></apply><apply id="alg4.l17.m1.2.3.3.cmml" xref="alg4.l17.m1.2.3.3"><plus id="alg4.l17.m1.2.3.3.1.cmml" xref="alg4.l17.m1.2.3.3.1"></plus><apply id="alg4.l17.m1.2.3.3.2.cmml" xref="alg4.l17.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.3.2.1.cmml" xref="alg4.l17.m1.2.3.3.2">subscript</csymbol><ci id="alg4.l17.m1.2.3.3.2.2a.cmml" xref="alg4.l17.m1.2.3.3.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l17.m1.2.3.3.2.2.cmml" xref="alg4.l17.m1.2.3.3.2.2">cumX</mtext></ci><ci id="alg4.l17.m1.2.3.3.2.3.cmml" xref="alg4.l17.m1.2.3.3.2.3">𝑑</ci></apply><apply id="alg4.l17.m1.2.3.3.3.cmml" xref="alg4.l17.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.3.3.1.cmml" xref="alg4.l17.m1.2.3.3.3">subscript</csymbol><ci id="alg4.l17.m1.2.3.3.3.2.cmml" xref="alg4.l17.m1.2.3.3.3.2">𝑋</ci><list id="alg4.l17.m1.2.2.2.3.cmml" xref="alg4.l17.m1.2.2.2.4"><cn type="integer" id="alg4.l17.m1.1.1.1.1.cmml" xref="alg4.l17.m1.1.1.1.1">1</cn><ci id="alg4.l17.m1.2.2.2.2.cmml" xref="alg4.l17.m1.2.2.2.2">𝑑</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l17.m1.2c">\texttt{cumX}_{d}\leftarrow\texttt{cumX}_{d}+X_{1,d}</annotation></semantics></math>

</div>
<div id="alg4.l18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l18.1.1.1" class="ltx_text" style="font-size:80%;">18:</span></span>        <math id="alg4.l18.m1.3" class="ltx_Math" alttext="\texttt{cumX2}_{d}\leftarrow\texttt{cumX2}_{d}+(X_{1,d})^{2}" display="inline"><semantics id="alg4.l18.m1.3a"><mrow id="alg4.l18.m1.3.3" xref="alg4.l18.m1.3.3.cmml"><msub id="alg4.l18.m1.3.3.3" xref="alg4.l18.m1.3.3.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l18.m1.3.3.3.2" xref="alg4.l18.m1.3.3.3.2a.cmml">cumX2</mtext><mi id="alg4.l18.m1.3.3.3.3" xref="alg4.l18.m1.3.3.3.3.cmml">d</mi></msub><mo stretchy="false" id="alg4.l18.m1.3.3.2" xref="alg4.l18.m1.3.3.2.cmml">←</mo><mrow id="alg4.l18.m1.3.3.1" xref="alg4.l18.m1.3.3.1.cmml"><msub id="alg4.l18.m1.3.3.1.3" xref="alg4.l18.m1.3.3.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l18.m1.3.3.1.3.2" xref="alg4.l18.m1.3.3.1.3.2a.cmml">cumX2</mtext><mi id="alg4.l18.m1.3.3.1.3.3" xref="alg4.l18.m1.3.3.1.3.3.cmml">d</mi></msub><mo id="alg4.l18.m1.3.3.1.2" xref="alg4.l18.m1.3.3.1.2.cmml">+</mo><msup id="alg4.l18.m1.3.3.1.1" xref="alg4.l18.m1.3.3.1.1.cmml"><mrow id="alg4.l18.m1.3.3.1.1.1.1" xref="alg4.l18.m1.3.3.1.1.1.1.1.cmml"><mo stretchy="false" id="alg4.l18.m1.3.3.1.1.1.1.2" xref="alg4.l18.m1.3.3.1.1.1.1.1.cmml">(</mo><msub id="alg4.l18.m1.3.3.1.1.1.1.1" xref="alg4.l18.m1.3.3.1.1.1.1.1.cmml"><mi id="alg4.l18.m1.3.3.1.1.1.1.1.2" xref="alg4.l18.m1.3.3.1.1.1.1.1.2.cmml">X</mi><mrow id="alg4.l18.m1.2.2.2.4" xref="alg4.l18.m1.2.2.2.3.cmml"><mn id="alg4.l18.m1.1.1.1.1" xref="alg4.l18.m1.1.1.1.1.cmml">1</mn><mo id="alg4.l18.m1.2.2.2.4.1" xref="alg4.l18.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l18.m1.2.2.2.2" xref="alg4.l18.m1.2.2.2.2.cmml">d</mi></mrow></msub><mo stretchy="false" id="alg4.l18.m1.3.3.1.1.1.1.3" xref="alg4.l18.m1.3.3.1.1.1.1.1.cmml">)</mo></mrow><mn id="alg4.l18.m1.3.3.1.1.3" xref="alg4.l18.m1.3.3.1.1.3.cmml">2</mn></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l18.m1.3b"><apply id="alg4.l18.m1.3.3.cmml" xref="alg4.l18.m1.3.3"><ci id="alg4.l18.m1.3.3.2.cmml" xref="alg4.l18.m1.3.3.2">←</ci><apply id="alg4.l18.m1.3.3.3.cmml" xref="alg4.l18.m1.3.3.3"><csymbol cd="ambiguous" id="alg4.l18.m1.3.3.3.1.cmml" xref="alg4.l18.m1.3.3.3">subscript</csymbol><ci id="alg4.l18.m1.3.3.3.2a.cmml" xref="alg4.l18.m1.3.3.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l18.m1.3.3.3.2.cmml" xref="alg4.l18.m1.3.3.3.2">cumX2</mtext></ci><ci id="alg4.l18.m1.3.3.3.3.cmml" xref="alg4.l18.m1.3.3.3.3">𝑑</ci></apply><apply id="alg4.l18.m1.3.3.1.cmml" xref="alg4.l18.m1.3.3.1"><plus id="alg4.l18.m1.3.3.1.2.cmml" xref="alg4.l18.m1.3.3.1.2"></plus><apply id="alg4.l18.m1.3.3.1.3.cmml" xref="alg4.l18.m1.3.3.1.3"><csymbol cd="ambiguous" id="alg4.l18.m1.3.3.1.3.1.cmml" xref="alg4.l18.m1.3.3.1.3">subscript</csymbol><ci id="alg4.l18.m1.3.3.1.3.2a.cmml" xref="alg4.l18.m1.3.3.1.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l18.m1.3.3.1.3.2.cmml" xref="alg4.l18.m1.3.3.1.3.2">cumX2</mtext></ci><ci id="alg4.l18.m1.3.3.1.3.3.cmml" xref="alg4.l18.m1.3.3.1.3.3">𝑑</ci></apply><apply id="alg4.l18.m1.3.3.1.1.cmml" xref="alg4.l18.m1.3.3.1.1"><csymbol cd="ambiguous" id="alg4.l18.m1.3.3.1.1.2.cmml" xref="alg4.l18.m1.3.3.1.1">superscript</csymbol><apply id="alg4.l18.m1.3.3.1.1.1.1.1.cmml" xref="alg4.l18.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l18.m1.3.3.1.1.1.1.1.1.cmml" xref="alg4.l18.m1.3.3.1.1.1.1">subscript</csymbol><ci id="alg4.l18.m1.3.3.1.1.1.1.1.2.cmml" xref="alg4.l18.m1.3.3.1.1.1.1.1.2">𝑋</ci><list id="alg4.l18.m1.2.2.2.3.cmml" xref="alg4.l18.m1.2.2.2.4"><cn type="integer" id="alg4.l18.m1.1.1.1.1.cmml" xref="alg4.l18.m1.1.1.1.1">1</cn><ci id="alg4.l18.m1.2.2.2.2.cmml" xref="alg4.l18.m1.2.2.2.2">𝑑</ci></list></apply><cn type="integer" id="alg4.l18.m1.3.3.1.1.3.cmml" xref="alg4.l18.m1.3.3.1.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l18.m1.3c">\texttt{cumX2}_{d}\leftarrow\texttt{cumX2}_{d}+(X_{1,d})^{2}</annotation></semantics></math>

</div>
<div id="alg4.l19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l19.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span>        <math id="alg4.l19.m1.1" class="ltx_Math" alttext="\texttt{out}_{n}\leftarrow\texttt{out}_{n}+\texttt{cumX2}_{d}-(\texttt{cumX}_{d}\times\texttt{cumX}_{d}/n)" display="inline"><semantics id="alg4.l19.m1.1a"><mrow id="alg4.l19.m1.1.1" xref="alg4.l19.m1.1.1.cmml"><msub id="alg4.l19.m1.1.1.3" xref="alg4.l19.m1.1.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.3.2" xref="alg4.l19.m1.1.1.3.2a.cmml">out</mtext><mi id="alg4.l19.m1.1.1.3.3" xref="alg4.l19.m1.1.1.3.3.cmml">n</mi></msub><mo stretchy="false" id="alg4.l19.m1.1.1.2" xref="alg4.l19.m1.1.1.2.cmml">←</mo><mrow id="alg4.l19.m1.1.1.1" xref="alg4.l19.m1.1.1.1.cmml"><mrow id="alg4.l19.m1.1.1.1.3" xref="alg4.l19.m1.1.1.1.3.cmml"><msub id="alg4.l19.m1.1.1.1.3.2" xref="alg4.l19.m1.1.1.1.3.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.3.2.2" xref="alg4.l19.m1.1.1.1.3.2.2a.cmml">out</mtext><mi id="alg4.l19.m1.1.1.1.3.2.3" xref="alg4.l19.m1.1.1.1.3.2.3.cmml">n</mi></msub><mo id="alg4.l19.m1.1.1.1.3.1" xref="alg4.l19.m1.1.1.1.3.1.cmml">+</mo><msub id="alg4.l19.m1.1.1.1.3.3" xref="alg4.l19.m1.1.1.1.3.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.3.3.2" xref="alg4.l19.m1.1.1.1.3.3.2a.cmml">cumX2</mtext><mi id="alg4.l19.m1.1.1.1.3.3.3" xref="alg4.l19.m1.1.1.1.3.3.3.cmml">d</mi></msub></mrow><mo id="alg4.l19.m1.1.1.1.2" xref="alg4.l19.m1.1.1.1.2.cmml">−</mo><mrow id="alg4.l19.m1.1.1.1.1.1" xref="alg4.l19.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg4.l19.m1.1.1.1.1.1.2" xref="alg4.l19.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg4.l19.m1.1.1.1.1.1.1" xref="alg4.l19.m1.1.1.1.1.1.1.cmml"><mrow id="alg4.l19.m1.1.1.1.1.1.1.2" xref="alg4.l19.m1.1.1.1.1.1.1.2.cmml"><msub id="alg4.l19.m1.1.1.1.1.1.1.2.2" xref="alg4.l19.m1.1.1.1.1.1.1.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.1.1.1.2.2.2" xref="alg4.l19.m1.1.1.1.1.1.1.2.2.2a.cmml">cumX</mtext><mi id="alg4.l19.m1.1.1.1.1.1.1.2.2.3" xref="alg4.l19.m1.1.1.1.1.1.1.2.2.3.cmml">d</mi></msub><mo lspace="0.222em" rspace="0.222em" id="alg4.l19.m1.1.1.1.1.1.1.2.1" xref="alg4.l19.m1.1.1.1.1.1.1.2.1.cmml">×</mo><msub id="alg4.l19.m1.1.1.1.1.1.1.2.3" xref="alg4.l19.m1.1.1.1.1.1.1.2.3.cmml"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.1.1.1.2.3.2" xref="alg4.l19.m1.1.1.1.1.1.1.2.3.2a.cmml">cumX</mtext><mi id="alg4.l19.m1.1.1.1.1.1.1.2.3.3" xref="alg4.l19.m1.1.1.1.1.1.1.2.3.3.cmml">d</mi></msub></mrow><mo id="alg4.l19.m1.1.1.1.1.1.1.1" xref="alg4.l19.m1.1.1.1.1.1.1.1.cmml">/</mo><mi id="alg4.l19.m1.1.1.1.1.1.1.3" xref="alg4.l19.m1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="alg4.l19.m1.1.1.1.1.1.3" xref="alg4.l19.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l19.m1.1b"><apply id="alg4.l19.m1.1.1.cmml" xref="alg4.l19.m1.1.1"><ci id="alg4.l19.m1.1.1.2.cmml" xref="alg4.l19.m1.1.1.2">←</ci><apply id="alg4.l19.m1.1.1.3.cmml" xref="alg4.l19.m1.1.1.3"><csymbol cd="ambiguous" id="alg4.l19.m1.1.1.3.1.cmml" xref="alg4.l19.m1.1.1.3">subscript</csymbol><ci id="alg4.l19.m1.1.1.3.2a.cmml" xref="alg4.l19.m1.1.1.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.3.2.cmml" xref="alg4.l19.m1.1.1.3.2">out</mtext></ci><ci id="alg4.l19.m1.1.1.3.3.cmml" xref="alg4.l19.m1.1.1.3.3">𝑛</ci></apply><apply id="alg4.l19.m1.1.1.1.cmml" xref="alg4.l19.m1.1.1.1"><minus id="alg4.l19.m1.1.1.1.2.cmml" xref="alg4.l19.m1.1.1.1.2"></minus><apply id="alg4.l19.m1.1.1.1.3.cmml" xref="alg4.l19.m1.1.1.1.3"><plus id="alg4.l19.m1.1.1.1.3.1.cmml" xref="alg4.l19.m1.1.1.1.3.1"></plus><apply id="alg4.l19.m1.1.1.1.3.2.cmml" xref="alg4.l19.m1.1.1.1.3.2"><csymbol cd="ambiguous" id="alg4.l19.m1.1.1.1.3.2.1.cmml" xref="alg4.l19.m1.1.1.1.3.2">subscript</csymbol><ci id="alg4.l19.m1.1.1.1.3.2.2a.cmml" xref="alg4.l19.m1.1.1.1.3.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.3.2.2.cmml" xref="alg4.l19.m1.1.1.1.3.2.2">out</mtext></ci><ci id="alg4.l19.m1.1.1.1.3.2.3.cmml" xref="alg4.l19.m1.1.1.1.3.2.3">𝑛</ci></apply><apply id="alg4.l19.m1.1.1.1.3.3.cmml" xref="alg4.l19.m1.1.1.1.3.3"><csymbol cd="ambiguous" id="alg4.l19.m1.1.1.1.3.3.1.cmml" xref="alg4.l19.m1.1.1.1.3.3">subscript</csymbol><ci id="alg4.l19.m1.1.1.1.3.3.2a.cmml" xref="alg4.l19.m1.1.1.1.3.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.3.3.2.cmml" xref="alg4.l19.m1.1.1.1.3.3.2">cumX2</mtext></ci><ci id="alg4.l19.m1.1.1.1.3.3.3.cmml" xref="alg4.l19.m1.1.1.1.3.3.3">𝑑</ci></apply></apply><apply id="alg4.l19.m1.1.1.1.1.1.1.cmml" xref="alg4.l19.m1.1.1.1.1.1"><divide id="alg4.l19.m1.1.1.1.1.1.1.1.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.1"></divide><apply id="alg4.l19.m1.1.1.1.1.1.1.2.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2"><times id="alg4.l19.m1.1.1.1.1.1.1.2.1.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.1"></times><apply id="alg4.l19.m1.1.1.1.1.1.1.2.2.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="alg4.l19.m1.1.1.1.1.1.1.2.2.1.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="alg4.l19.m1.1.1.1.1.1.1.2.2.2a.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.2.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.1.1.1.2.2.2.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.2.2">cumX</mtext></ci><ci id="alg4.l19.m1.1.1.1.1.1.1.2.2.3.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.2.3">𝑑</ci></apply><apply id="alg4.l19.m1.1.1.1.1.1.1.2.3.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="alg4.l19.m1.1.1.1.1.1.1.2.3.1.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="alg4.l19.m1.1.1.1.1.1.1.2.3.2a.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.3.2"><mtext class="ltx_mathvariant_monospace" id="alg4.l19.m1.1.1.1.1.1.1.2.3.2.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.3.2">cumX</mtext></ci><ci id="alg4.l19.m1.1.1.1.1.1.1.2.3.3.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.2.3.3">𝑑</ci></apply></apply><ci id="alg4.l19.m1.1.1.1.1.1.1.3.cmml" xref="alg4.l19.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l19.m1.1c">\texttt{out}_{n}\leftarrow\texttt{out}_{n}+\texttt{cumX2}_{d}-(\texttt{cumX}_{d}\times\texttt{cumX}_{d}/n)</annotation></semantics></math>

</div>
<div id="alg4.l20" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l20.1.1.1" class="ltx_text" style="font-size:80%;">20:</span></span>     <span id="alg4.l20.2" class="ltx_text ltx_font_bold">end</span> <span id="alg4.l20.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg4.l21" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l21.1.1.1" class="ltx_text" style="font-size:80%;">21:</span></span>  <span id="alg4.l21.2" class="ltx_text ltx_font_bold">end</span> <span id="alg4.l21.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg4.l22" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l22.1.1.1" class="ltx_text" style="font-size:80%;">22:</span></span>  <span id="alg4.l22.2" class="ltx_text ltx_font_bold">return </span> <span id="alg4.l22.3" class="ltx_text ltx_font_typewriter">out</span>

</div>
</div>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Aggregation Using Pairwise Averages</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.6" class="ltx_p">Recall that we estimate sums of low-bitwidth integers by averaging pairs of values, then pairs of pairs, and so on. One could reduce all <math id="A4.p1.1.m1.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A4.p1.1.m1.1a"><mi id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><ci id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">C</annotation></semantics></math> values this way, but we find that one obtains a better speed-accuracy tradeoff by computing the average of blocks of <math id="A4.p1.2.m2.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A4.p1.2.m2.1a"><mi id="A4.p1.2.m2.1.1" xref="A4.p1.2.m2.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A4.p1.2.m2.1b"><ci id="A4.p1.2.m2.1.1.cmml" xref="A4.p1.2.m2.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.2.m2.1c">U</annotation></semantics></math> values and then upcasting to obtain exact sums of these averages. Multiplying this sum of averages by <math id="A4.p1.3.m3.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A4.p1.3.m3.1a"><mi id="A4.p1.3.m3.1.1" xref="A4.p1.3.m3.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A4.p1.3.m3.1b"><ci id="A4.p1.3.m3.1.1.cmml" xref="A4.p1.3.m3.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.3.m3.1c">U</annotation></semantics></math> and adding in a bias correction term gives one the overall estimate of the sum. One could tune <math id="A4.p1.4.m4.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A4.p1.4.m4.1a"><mi id="A4.p1.4.m4.1.1" xref="A4.p1.4.m4.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A4.p1.4.m4.1b"><ci id="A4.p1.4.m4.1.1.cmml" xref="A4.p1.4.m4.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.4.m4.1c">U</annotation></semantics></math> for a particular problem and hardware, but we simply set <math id="A4.p1.5.m5.1" class="ltx_Math" alttext="U=16" display="inline"><semantics id="A4.p1.5.m5.1a"><mrow id="A4.p1.5.m5.1.1" xref="A4.p1.5.m5.1.1.cmml"><mi id="A4.p1.5.m5.1.1.2" xref="A4.p1.5.m5.1.1.2.cmml">U</mi><mo id="A4.p1.5.m5.1.1.1" xref="A4.p1.5.m5.1.1.1.cmml">=</mo><mn id="A4.p1.5.m5.1.1.3" xref="A4.p1.5.m5.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.5.m5.1b"><apply id="A4.p1.5.m5.1.1.cmml" xref="A4.p1.5.m5.1.1"><eq id="A4.p1.5.m5.1.1.1.cmml" xref="A4.p1.5.m5.1.1.1"></eq><ci id="A4.p1.5.m5.1.1.2.cmml" xref="A4.p1.5.m5.1.1.2">𝑈</ci><cn type="integer" id="A4.p1.5.m5.1.1.3.cmml" xref="A4.p1.5.m5.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.5.m5.1c">U=16</annotation></semantics></math> in all our experiments. Having a larger <math id="A4.p1.6.m6.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A4.p1.6.m6.1a"><mi id="A4.p1.6.m6.1.1" xref="A4.p1.6.m6.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A4.p1.6.m6.1b"><ci id="A4.p1.6.m6.1.1.cmml" xref="A4.p1.6.m6.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.6.m6.1c">U</annotation></semantics></math> imposes less overhead because upcasting happens less often, but there are sharp diminishing returns to this; once upcasting is rare, doing it even less often is of little help thanks to Amdahl’s law.</p>
</div>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.1" class="ltx_p">Because of our assumption that we are operating on matrices, rather than a matrix and a vector, we can also improve on the aggregation of existing methods <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>; André et al., <a href="#bib.bib4" title="" class="ltx_ref">2017</a>, <a href="#bib.bib5" title="" class="ltx_ref">2019</a>)</cite> by fusing the aggregation of two or more output columns to hide read latency. Conceptually, this amounts to tiling the loop over output columns and alternating reads between the two corresponding tables within the innermost loop. This fusion does not change the output of the computation—only how efficiently it runs.</p>
</div>
<div id="A4.p3" class="ltx_para">
<p id="A4.p3.1" class="ltx_p">Having addressed these practical details, we may now proceed to the analysis of our estimator’s bias.</p>
</div>
<div id="A4.Thmdefinition1" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A4.Thmdefinition1.1.1.1" class="ltx_text ltx_font_bold">Definition D.1</span></span><span id="A4.Thmdefinition1.2.2" class="ltx_text ltx_font_bold"> </span>(Averaging Integer Sum Estimator)<span id="A4.Thmdefinition1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A4.Thmdefinition1.p1" class="ltx_para">
<p id="A4.Thmdefinition1.p1.2" class="ltx_p"><span id="A4.Thmdefinition1.p1.2.2" class="ltx_text ltx_font_italic">Let <math id="A4.Thmdefinition1.p1.1.1.m1.4" class="ltx_Math" alttext="\bm{x}\in\{0,1\}^{C},C\text{ \% }U=0,U=2^{p},p\geq 0" display="inline"><semantics id="A4.Thmdefinition1.p1.1.1.m1.4a"><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.3.cmml"><mrow id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.cmml"><mi id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.2" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.2.cmml">𝐱</mi><mo id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.1.cmml">∈</mo><msup id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.cmml"><mrow id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.1.cmml"><mo stretchy="false" id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.2.1" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.1.cmml">{</mo><mn id="A4.Thmdefinition1.p1.1.1.m1.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.1.1.cmml">0</mn><mo id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.1.cmml">,</mo><mn id="A4.Thmdefinition1.p1.1.1.m1.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.2.2.cmml">1</mn><mo stretchy="false" id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.2.3" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.1.cmml">}</mo></mrow><mi id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.3" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.3.cmml">C</mi></msup></mrow><mo id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.3a.cmml">,</mo><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.3.cmml"><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.cmml"><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.cmml"><mi id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.2.cmml">C</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.1" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.1.cmml">​</mo><mtext class="ltx_mathvariant_italic" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.3a.cmml"> % </mtext><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.1a" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.4" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.4.cmml">U</mi></mrow><mo id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.1.cmml">=</mo><mn id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.3.cmml">0</mn></mrow><mo id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.3a.cmml">,</mo><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.3.cmml"><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.cmml"><mi id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.2.cmml">U</mi><mo id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.1" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.1.cmml">=</mo><msup id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.cmml"><mn id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.2.cmml">2</mn><mi id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.3.cmml">p</mi></msup></mrow><mo id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.3a.cmml">,</mo><mrow id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.cmml"><mi id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.2" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.2.cmml">p</mi><mo id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.1" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.1.cmml">≥</mo><mn id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.3" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.3.cmml">0</mn></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.1.1.m1.4b"><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.1.1.m1.4.4.3a.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.3">formulae-sequence</csymbol><apply id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1"><in id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.1"></in><ci id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.2">𝐱</ci><apply id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3">superscript</csymbol><set id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.2.2"><cn type="integer" id="A4.Thmdefinition1.p1.1.1.m1.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.1.1">0</cn><cn type="integer" id="A4.Thmdefinition1.p1.1.1.m1.2.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.2.2">1</cn></set><ci id="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.3.3.1.1.3.3">𝐶</ci></apply></apply><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.3a.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.3">formulae-sequence</csymbol><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1"><eq id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.1"></eq><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2"><times id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.1"></times><ci id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.2">𝐶</ci><ci id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.3a.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.3"><mtext class="ltx_mathvariant_italic" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.3"> % </mtext></ci><ci id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.4.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.2.4">𝑈</ci></apply><cn type="integer" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.1.1.3">0</cn></apply><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.3a.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1"><eq id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.1"></eq><ci id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.2">𝑈</ci><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3">superscript</csymbol><cn type="integer" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.2">2</cn><ci id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.1.1.3.3">𝑝</ci></apply></apply><apply id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2"><geq id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.1.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.1"></geq><ci id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.2.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.2">𝑝</ci><cn type="integer" id="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.3.cmml" xref="A4.Thmdefinition1.p1.1.1.m1.4.4.2.2.2.2.2.2.3">0</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.1.1.m1.4c">\bm{x}\in\{0,1\}^{C},C\text{ \% }U=0,U=2^{p},p\geq 0</annotation></semantics></math>. The Averaging Integer Sum Estimator (AISE) <math id="A4.Thmdefinition1.p1.2.2.m2.1" class="ltx_Math" alttext="\hat{s}(\bm{x})" display="inline"><semantics id="A4.Thmdefinition1.p1.2.2.m2.1a"><mrow id="A4.Thmdefinition1.p1.2.2.m2.1.2" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.cmml"><mover accent="true" id="A4.Thmdefinition1.p1.2.2.m2.1.2.2" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.2.cmml"><mi id="A4.Thmdefinition1.p1.2.2.m2.1.2.2.2" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.2.2.cmml">s</mi><mo id="A4.Thmdefinition1.p1.2.2.m2.1.2.2.1" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.2.2.m2.1.2.1" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.1.cmml">​</mo><mrow id="A4.Thmdefinition1.p1.2.2.m2.1.2.3.2" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.cmml"><mo stretchy="false" id="A4.Thmdefinition1.p1.2.2.m2.1.2.3.2.1" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.cmml">(</mo><mi id="A4.Thmdefinition1.p1.2.2.m2.1.1" xref="A4.Thmdefinition1.p1.2.2.m2.1.1.cmml">𝐱</mi><mo stretchy="false" id="A4.Thmdefinition1.p1.2.2.m2.1.2.3.2.2" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.2.2.m2.1b"><apply id="A4.Thmdefinition1.p1.2.2.m2.1.2.cmml" xref="A4.Thmdefinition1.p1.2.2.m2.1.2"><times id="A4.Thmdefinition1.p1.2.2.m2.1.2.1.cmml" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.1"></times><apply id="A4.Thmdefinition1.p1.2.2.m2.1.2.2.cmml" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.2"><ci id="A4.Thmdefinition1.p1.2.2.m2.1.2.2.1.cmml" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.2.1">^</ci><ci id="A4.Thmdefinition1.p1.2.2.m2.1.2.2.2.cmml" xref="A4.Thmdefinition1.p1.2.2.m2.1.2.2.2">𝑠</ci></apply><ci id="A4.Thmdefinition1.p1.2.2.m2.1.1.cmml" xref="A4.Thmdefinition1.p1.2.2.m2.1.1">𝐱</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.2.2.m2.1c">\hat{s}(\bm{x})</annotation></semantics></math> is defined as:</span></p>
<table id="A6.EGx13" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A4.E17"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E17.m1.1" class="ltx_Math" alttext="\displaystyle\hat{s}(\bm{x})" display="inline"><semantics id="A4.E17.m1.1a"><mrow id="A4.E17.m1.1.2" xref="A4.E17.m1.1.2.cmml"><mover accent="true" id="A4.E17.m1.1.2.2" xref="A4.E17.m1.1.2.2.cmml"><mi id="A4.E17.m1.1.2.2.2" xref="A4.E17.m1.1.2.2.2.cmml">s</mi><mo id="A4.E17.m1.1.2.2.1" xref="A4.E17.m1.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A4.E17.m1.1.2.1" xref="A4.E17.m1.1.2.1.cmml">​</mo><mrow id="A4.E17.m1.1.2.3.2" xref="A4.E17.m1.1.2.cmml"><mo stretchy="false" id="A4.E17.m1.1.2.3.2.1" xref="A4.E17.m1.1.2.cmml">(</mo><mi id="A4.E17.m1.1.1" xref="A4.E17.m1.1.1.cmml">𝒙</mi><mo stretchy="false" id="A4.E17.m1.1.2.3.2.2" xref="A4.E17.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E17.m1.1b"><apply id="A4.E17.m1.1.2.cmml" xref="A4.E17.m1.1.2"><times id="A4.E17.m1.1.2.1.cmml" xref="A4.E17.m1.1.2.1"></times><apply id="A4.E17.m1.1.2.2.cmml" xref="A4.E17.m1.1.2.2"><ci id="A4.E17.m1.1.2.2.1.cmml" xref="A4.E17.m1.1.2.2.1">^</ci><ci id="A4.E17.m1.1.2.2.2.cmml" xref="A4.E17.m1.1.2.2.2">𝑠</ci></apply><ci id="A4.E17.m1.1.1.cmml" xref="A4.E17.m1.1.1">𝒙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E17.m1.1c">\displaystyle\hat{s}(\bm{x})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A4.E17.m2.1" class="ltx_Math" alttext="\displaystyle\triangleq\sum_{k=1}^{C/U}\hat{s}_{U}(\bm{x}_{i_{k}:j_{k}})" display="inline"><semantics id="A4.E17.m2.1a"><mrow id="A4.E17.m2.1.1" xref="A4.E17.m2.1.1.cmml"><mi id="A4.E17.m2.1.1.3" xref="A4.E17.m2.1.1.3.cmml"></mi><mo id="A4.E17.m2.1.1.2" xref="A4.E17.m2.1.1.2.cmml">≜</mo><mrow id="A4.E17.m2.1.1.1" xref="A4.E17.m2.1.1.1.cmml"><mstyle displaystyle="true" id="A4.E17.m2.1.1.1.2" xref="A4.E17.m2.1.1.1.2.cmml"><munderover id="A4.E17.m2.1.1.1.2a" xref="A4.E17.m2.1.1.1.2.cmml"><mo movablelimits="false" id="A4.E17.m2.1.1.1.2.2.2" xref="A4.E17.m2.1.1.1.2.2.2.cmml">∑</mo><mrow id="A4.E17.m2.1.1.1.2.2.3" xref="A4.E17.m2.1.1.1.2.2.3.cmml"><mi id="A4.E17.m2.1.1.1.2.2.3.2" xref="A4.E17.m2.1.1.1.2.2.3.2.cmml">k</mi><mo id="A4.E17.m2.1.1.1.2.2.3.1" xref="A4.E17.m2.1.1.1.2.2.3.1.cmml">=</mo><mn id="A4.E17.m2.1.1.1.2.2.3.3" xref="A4.E17.m2.1.1.1.2.2.3.3.cmml">1</mn></mrow><mrow id="A4.E17.m2.1.1.1.2.3" xref="A4.E17.m2.1.1.1.2.3.cmml"><mi id="A4.E17.m2.1.1.1.2.3.2" xref="A4.E17.m2.1.1.1.2.3.2.cmml">C</mi><mo id="A4.E17.m2.1.1.1.2.3.1" xref="A4.E17.m2.1.1.1.2.3.1.cmml">/</mo><mi id="A4.E17.m2.1.1.1.2.3.3" xref="A4.E17.m2.1.1.1.2.3.3.cmml">U</mi></mrow></munderover></mstyle><mrow id="A4.E17.m2.1.1.1.1" xref="A4.E17.m2.1.1.1.1.cmml"><msub id="A4.E17.m2.1.1.1.1.3" xref="A4.E17.m2.1.1.1.1.3.cmml"><mover accent="true" id="A4.E17.m2.1.1.1.1.3.2" xref="A4.E17.m2.1.1.1.1.3.2.cmml"><mi id="A4.E17.m2.1.1.1.1.3.2.2" xref="A4.E17.m2.1.1.1.1.3.2.2.cmml">s</mi><mo id="A4.E17.m2.1.1.1.1.3.2.1" xref="A4.E17.m2.1.1.1.1.3.2.1.cmml">^</mo></mover><mi id="A4.E17.m2.1.1.1.1.3.3" xref="A4.E17.m2.1.1.1.1.3.3.cmml">U</mi></msub><mo lspace="0em" rspace="0em" id="A4.E17.m2.1.1.1.1.2" xref="A4.E17.m2.1.1.1.1.2.cmml">​</mo><mrow id="A4.E17.m2.1.1.1.1.1.1" xref="A4.E17.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.E17.m2.1.1.1.1.1.1.2" xref="A4.E17.m2.1.1.1.1.1.1.1.cmml">(</mo><msub id="A4.E17.m2.1.1.1.1.1.1.1" xref="A4.E17.m2.1.1.1.1.1.1.1.cmml"><mi id="A4.E17.m2.1.1.1.1.1.1.1.2" xref="A4.E17.m2.1.1.1.1.1.1.1.2.cmml">𝒙</mi><mrow id="A4.E17.m2.1.1.1.1.1.1.1.3" xref="A4.E17.m2.1.1.1.1.1.1.1.3.cmml"><msub id="A4.E17.m2.1.1.1.1.1.1.1.3.2" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2.cmml"><mi id="A4.E17.m2.1.1.1.1.1.1.1.3.2.2" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2.2.cmml">i</mi><mi id="A4.E17.m2.1.1.1.1.1.1.1.3.2.3" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2.3.cmml">k</mi></msub><mo lspace="0.278em" rspace="0.278em" id="A4.E17.m2.1.1.1.1.1.1.1.3.1" xref="A4.E17.m2.1.1.1.1.1.1.1.3.1.cmml">:</mo><msub id="A4.E17.m2.1.1.1.1.1.1.1.3.3" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3.cmml"><mi id="A4.E17.m2.1.1.1.1.1.1.1.3.3.2" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3.2.cmml">j</mi><mi id="A4.E17.m2.1.1.1.1.1.1.1.3.3.3" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3.3.cmml">k</mi></msub></mrow></msub><mo stretchy="false" id="A4.E17.m2.1.1.1.1.1.1.3" xref="A4.E17.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E17.m2.1b"><apply id="A4.E17.m2.1.1.cmml" xref="A4.E17.m2.1.1"><ci id="A4.E17.m2.1.1.2.cmml" xref="A4.E17.m2.1.1.2">≜</ci><csymbol cd="latexml" id="A4.E17.m2.1.1.3.cmml" xref="A4.E17.m2.1.1.3">absent</csymbol><apply id="A4.E17.m2.1.1.1.cmml" xref="A4.E17.m2.1.1.1"><apply id="A4.E17.m2.1.1.1.2.cmml" xref="A4.E17.m2.1.1.1.2"><csymbol cd="ambiguous" id="A4.E17.m2.1.1.1.2.1.cmml" xref="A4.E17.m2.1.1.1.2">superscript</csymbol><apply id="A4.E17.m2.1.1.1.2.2.cmml" xref="A4.E17.m2.1.1.1.2"><csymbol cd="ambiguous" id="A4.E17.m2.1.1.1.2.2.1.cmml" xref="A4.E17.m2.1.1.1.2">subscript</csymbol><sum id="A4.E17.m2.1.1.1.2.2.2.cmml" xref="A4.E17.m2.1.1.1.2.2.2"></sum><apply id="A4.E17.m2.1.1.1.2.2.3.cmml" xref="A4.E17.m2.1.1.1.2.2.3"><eq id="A4.E17.m2.1.1.1.2.2.3.1.cmml" xref="A4.E17.m2.1.1.1.2.2.3.1"></eq><ci id="A4.E17.m2.1.1.1.2.2.3.2.cmml" xref="A4.E17.m2.1.1.1.2.2.3.2">𝑘</ci><cn type="integer" id="A4.E17.m2.1.1.1.2.2.3.3.cmml" xref="A4.E17.m2.1.1.1.2.2.3.3">1</cn></apply></apply><apply id="A4.E17.m2.1.1.1.2.3.cmml" xref="A4.E17.m2.1.1.1.2.3"><divide id="A4.E17.m2.1.1.1.2.3.1.cmml" xref="A4.E17.m2.1.1.1.2.3.1"></divide><ci id="A4.E17.m2.1.1.1.2.3.2.cmml" xref="A4.E17.m2.1.1.1.2.3.2">𝐶</ci><ci id="A4.E17.m2.1.1.1.2.3.3.cmml" xref="A4.E17.m2.1.1.1.2.3.3">𝑈</ci></apply></apply><apply id="A4.E17.m2.1.1.1.1.cmml" xref="A4.E17.m2.1.1.1.1"><times id="A4.E17.m2.1.1.1.1.2.cmml" xref="A4.E17.m2.1.1.1.1.2"></times><apply id="A4.E17.m2.1.1.1.1.3.cmml" xref="A4.E17.m2.1.1.1.1.3"><csymbol cd="ambiguous" id="A4.E17.m2.1.1.1.1.3.1.cmml" xref="A4.E17.m2.1.1.1.1.3">subscript</csymbol><apply id="A4.E17.m2.1.1.1.1.3.2.cmml" xref="A4.E17.m2.1.1.1.1.3.2"><ci id="A4.E17.m2.1.1.1.1.3.2.1.cmml" xref="A4.E17.m2.1.1.1.1.3.2.1">^</ci><ci id="A4.E17.m2.1.1.1.1.3.2.2.cmml" xref="A4.E17.m2.1.1.1.1.3.2.2">𝑠</ci></apply><ci id="A4.E17.m2.1.1.1.1.3.3.cmml" xref="A4.E17.m2.1.1.1.1.3.3">𝑈</ci></apply><apply id="A4.E17.m2.1.1.1.1.1.1.1.cmml" xref="A4.E17.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A4.E17.m2.1.1.1.1.1.1.1.1.cmml" xref="A4.E17.m2.1.1.1.1.1.1">subscript</csymbol><ci id="A4.E17.m2.1.1.1.1.1.1.1.2.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.2">𝒙</ci><apply id="A4.E17.m2.1.1.1.1.1.1.1.3.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3"><ci id="A4.E17.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.1">:</ci><apply id="A4.E17.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="A4.E17.m2.1.1.1.1.1.1.1.3.2.1.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="A4.E17.m2.1.1.1.1.1.1.1.3.2.2.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2.2">𝑖</ci><ci id="A4.E17.m2.1.1.1.1.1.1.1.3.2.3.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.2.3">𝑘</ci></apply><apply id="A4.E17.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A4.E17.m2.1.1.1.1.1.1.1.3.3.1.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="A4.E17.m2.1.1.1.1.1.1.1.3.3.2.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3.2">𝑗</ci><ci id="A4.E17.m2.1.1.1.1.1.1.1.3.3.3.cmml" xref="A4.E17.m2.1.1.1.1.1.1.1.3.3.3">𝑘</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E17.m2.1c">\displaystyle\triangleq\sum_{k=1}^{C/U}\hat{s}_{U}(\bm{x}_{i_{k}:j_{k}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(17)</span></td>
</tr></tbody>
<tbody id="A4.E18"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E18.m1.1" class="ltx_Math" alttext="\displaystyle\hat{s}_{U}(\bm{x})" display="inline"><semantics id="A4.E18.m1.1a"><mrow id="A4.E18.m1.1.2" xref="A4.E18.m1.1.2.cmml"><msub id="A4.E18.m1.1.2.2" xref="A4.E18.m1.1.2.2.cmml"><mover accent="true" id="A4.E18.m1.1.2.2.2" xref="A4.E18.m1.1.2.2.2.cmml"><mi id="A4.E18.m1.1.2.2.2.2" xref="A4.E18.m1.1.2.2.2.2.cmml">s</mi><mo id="A4.E18.m1.1.2.2.2.1" xref="A4.E18.m1.1.2.2.2.1.cmml">^</mo></mover><mi id="A4.E18.m1.1.2.2.3" xref="A4.E18.m1.1.2.2.3.cmml">U</mi></msub><mo lspace="0em" rspace="0em" id="A4.E18.m1.1.2.1" xref="A4.E18.m1.1.2.1.cmml">​</mo><mrow id="A4.E18.m1.1.2.3.2" xref="A4.E18.m1.1.2.cmml"><mo stretchy="false" id="A4.E18.m1.1.2.3.2.1" xref="A4.E18.m1.1.2.cmml">(</mo><mi id="A4.E18.m1.1.1" xref="A4.E18.m1.1.1.cmml">𝒙</mi><mo stretchy="false" id="A4.E18.m1.1.2.3.2.2" xref="A4.E18.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E18.m1.1b"><apply id="A4.E18.m1.1.2.cmml" xref="A4.E18.m1.1.2"><times id="A4.E18.m1.1.2.1.cmml" xref="A4.E18.m1.1.2.1"></times><apply id="A4.E18.m1.1.2.2.cmml" xref="A4.E18.m1.1.2.2"><csymbol cd="ambiguous" id="A4.E18.m1.1.2.2.1.cmml" xref="A4.E18.m1.1.2.2">subscript</csymbol><apply id="A4.E18.m1.1.2.2.2.cmml" xref="A4.E18.m1.1.2.2.2"><ci id="A4.E18.m1.1.2.2.2.1.cmml" xref="A4.E18.m1.1.2.2.2.1">^</ci><ci id="A4.E18.m1.1.2.2.2.2.cmml" xref="A4.E18.m1.1.2.2.2.2">𝑠</ci></apply><ci id="A4.E18.m1.1.2.2.3.cmml" xref="A4.E18.m1.1.2.2.3">𝑈</ci></apply><ci id="A4.E18.m1.1.1.cmml" xref="A4.E18.m1.1.1">𝒙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E18.m1.1c">\displaystyle\hat{s}_{U}(\bm{x})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A4.E18.m2.4" class="ltx_Math" alttext="\displaystyle\triangleq\begin{cases}x_{1}&amp;\bm{x}\in\mathbb{R}^{1}\\
\lfloor\frac{1}{2}(\hat{s}_{U}(\bm{x}_{left})+\hat{s}_{U}(\bm{x}_{right})+1)\rfloor&amp;\text{otherwise}\end{cases}" display="inline"><semantics id="A4.E18.m2.4a"><mrow id="A4.E18.m2.4.5" xref="A4.E18.m2.4.5.cmml"><mi id="A4.E18.m2.4.5.2" xref="A4.E18.m2.4.5.2.cmml"></mi><mo id="A4.E18.m2.4.5.1" xref="A4.E18.m2.4.5.1.cmml">≜</mo><mrow id="A4.E18.m2.4.4a" xref="A4.E18.m2.4.5.3.1.cmml"><mo id="A4.E18.m2.4.4a.5" xref="A4.E18.m2.4.5.3.1.1.cmml">{</mo><mtable columnspacing="5pt" rowspacing="0pt" id="A4.E18.m2.4.4.4a" xref="A4.E18.m2.4.5.3.1.cmml"><mtr id="A4.E18.m2.4.4.4aa" xref="A4.E18.m2.4.5.3.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="A4.E18.m2.4.4.4ab" xref="A4.E18.m2.4.5.3.1.cmml"><msub id="A4.E18.m2.1.1.1.1.1.1" xref="A4.E18.m2.1.1.1.1.1.1.cmml"><mi id="A4.E18.m2.1.1.1.1.1.1.2" xref="A4.E18.m2.1.1.1.1.1.1.2.cmml">x</mi><mn id="A4.E18.m2.1.1.1.1.1.1.3" xref="A4.E18.m2.1.1.1.1.1.1.3.cmml">1</mn></msub></mtd><mtd class="ltx_align_left" columnalign="left" id="A4.E18.m2.4.4.4ac" xref="A4.E18.m2.4.5.3.1.cmml"><mrow id="A4.E18.m2.2.2.2.2.2.1" xref="A4.E18.m2.2.2.2.2.2.1.cmml"><mi id="A4.E18.m2.2.2.2.2.2.1.2" xref="A4.E18.m2.2.2.2.2.2.1.2.cmml">𝒙</mi><mo id="A4.E18.m2.2.2.2.2.2.1.1" xref="A4.E18.m2.2.2.2.2.2.1.1.cmml">∈</mo><msup id="A4.E18.m2.2.2.2.2.2.1.3" xref="A4.E18.m2.2.2.2.2.2.1.3.cmml"><mi id="A4.E18.m2.2.2.2.2.2.1.3.2" xref="A4.E18.m2.2.2.2.2.2.1.3.2.cmml">ℝ</mi><mn id="A4.E18.m2.2.2.2.2.2.1.3.3" xref="A4.E18.m2.2.2.2.2.2.1.3.3.cmml">1</mn></msup></mrow></mtd></mtr><mtr id="A4.E18.m2.4.4.4ad" xref="A4.E18.m2.4.5.3.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="A4.E18.m2.4.4.4ae" xref="A4.E18.m2.4.5.3.1.cmml"><mrow id="A4.E18.m2.3.3.3.3.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.2.cmml"><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.2.1.cmml">⌊</mo><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.cmml"><mfrac id="A4.E18.m2.3.3.3.3.1.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3.cmml"><mn id="A4.E18.m2.3.3.3.3.1.1.1.1.3.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3.2.cmml">1</mn><mn id="A4.E18.m2.3.3.3.3.1.1.1.1.3.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.2.cmml">​</mo><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.cmml"><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.cmml"><msub id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.2.cmml">s</mi><mo id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.1.cmml">^</mo></mover><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.3.cmml">U</mi></msub><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.2.cmml">𝒙</mi><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1a" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.4" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1b" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.5" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml">t</mi></mrow></msub><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.3.cmml">+</mo><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.cmml"><msub id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.cmml"><mover accent="true" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.cmml"><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.2.cmml">s</mi><mo id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.1.cmml">^</mo></mover><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.3.cmml">U</mi></msub><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><msub id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.2.cmml">𝒙</mi><mrow id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.cmml"><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.2" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1a" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.4" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1b" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.5" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1c" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1.cmml">​</mo><mi id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.6" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.6.cmml">t</mi></mrow></msub><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.3a" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.3.cmml">+</mo><mn id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.4" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.4.cmml">1</mn></mrow><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.E18.m2.3.3.3.3.1.1.1.3" xref="A4.E18.m2.3.3.3.3.1.1.2.1.cmml">⌋</mo></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="A4.E18.m2.4.4.4af" xref="A4.E18.m2.4.5.3.1.cmml"><mtext class="ltx_mathvariant_italic" id="A4.E18.m2.4.4.4.4.2.1" xref="A4.E18.m2.4.4.4.4.2.1a.cmml">otherwise</mtext></mtd></mtr></mtable></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E18.m2.4b"><apply id="A4.E18.m2.4.5.cmml" xref="A4.E18.m2.4.5"><ci id="A4.E18.m2.4.5.1.cmml" xref="A4.E18.m2.4.5.1">≜</ci><csymbol cd="latexml" id="A4.E18.m2.4.5.2.cmml" xref="A4.E18.m2.4.5.2">absent</csymbol><apply id="A4.E18.m2.4.5.3.1.cmml" xref="A4.E18.m2.4.4a"><csymbol cd="latexml" id="A4.E18.m2.4.5.3.1.1.cmml" xref="A4.E18.m2.4.4a.5">cases</csymbol><apply id="A4.E18.m2.1.1.1.1.1.1.cmml" xref="A4.E18.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A4.E18.m2.1.1.1.1.1.1.1.cmml" xref="A4.E18.m2.1.1.1.1.1.1">subscript</csymbol><ci id="A4.E18.m2.1.1.1.1.1.1.2.cmml" xref="A4.E18.m2.1.1.1.1.1.1.2">𝑥</ci><cn type="integer" id="A4.E18.m2.1.1.1.1.1.1.3.cmml" xref="A4.E18.m2.1.1.1.1.1.1.3">1</cn></apply><apply id="A4.E18.m2.2.2.2.2.2.1.cmml" xref="A4.E18.m2.2.2.2.2.2.1"><in id="A4.E18.m2.2.2.2.2.2.1.1.cmml" xref="A4.E18.m2.2.2.2.2.2.1.1"></in><ci id="A4.E18.m2.2.2.2.2.2.1.2.cmml" xref="A4.E18.m2.2.2.2.2.2.1.2">𝒙</ci><apply id="A4.E18.m2.2.2.2.2.2.1.3.cmml" xref="A4.E18.m2.2.2.2.2.2.1.3"><csymbol cd="ambiguous" id="A4.E18.m2.2.2.2.2.2.1.3.1.cmml" xref="A4.E18.m2.2.2.2.2.2.1.3">superscript</csymbol><ci id="A4.E18.m2.2.2.2.2.2.1.3.2.cmml" xref="A4.E18.m2.2.2.2.2.2.1.3.2">ℝ</ci><cn type="integer" id="A4.E18.m2.2.2.2.2.2.1.3.3.cmml" xref="A4.E18.m2.2.2.2.2.2.1.3.3">1</cn></apply></apply><apply id="A4.E18.m2.3.3.3.3.1.1.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1"><floor id="A4.E18.m2.3.3.3.3.1.1.2.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.2"></floor><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1"><times id="A4.E18.m2.3.3.3.3.1.1.1.1.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.2"></times><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3"><divide id="A4.E18.m2.3.3.3.3.1.1.1.1.3.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3"></divide><cn type="integer" id="A4.E18.m2.3.3.3.3.1.1.1.1.3.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3.2">1</cn><cn type="integer" id="A4.E18.m2.3.3.3.3.1.1.1.1.3.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.3.3">2</cn></apply><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1"><plus id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.3"></plus><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1"><times id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.2"></times><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2"><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.1">^</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.2.2">𝑠</ci></apply><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.3.3">𝑈</ci></apply><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.2">𝒙</ci><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3"><times id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.1"></times><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑒</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.4">𝑓</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.1.1.1.1.3.5">𝑡</ci></apply></apply></apply><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2"><times id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.2"></times><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3">subscript</csymbol><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2"><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.1">^</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.2.2">𝑠</ci></apply><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.3.3">𝑈</ci></apply><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1">subscript</csymbol><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.2">𝒙</ci><apply id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3"><times id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.1"></times><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.2.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.2">𝑟</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.3.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.3">𝑖</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.4.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.4">𝑔</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.5.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.5">ℎ</ci><ci id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.6.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.2.1.1.1.3.6">𝑡</ci></apply></apply></apply><cn type="integer" id="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.4.cmml" xref="A4.E18.m2.3.3.3.3.1.1.1.1.1.1.1.4">1</cn></apply></apply></apply><ci id="A4.E18.m2.4.4.4.4.2.1a.cmml" xref="A4.E18.m2.4.4.4.4.2.1"><mtext class="ltx_mathvariant_italic" id="A4.E18.m2.4.4.4.4.2.1.cmml" xref="A4.E18.m2.4.4.4.4.2.1">otherwise</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E18.m2.4c">\displaystyle\triangleq\begin{cases}x_{1}&amp;\bm{x}\in\mathbb{R}^{1}\\
\lfloor\frac{1}{2}(\hat{s}_{U}(\bm{x}_{left})+\hat{s}_{U}(\bm{x}_{right})+1)\rfloor&amp;\text{otherwise}\end{cases}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(18)</span></td>
</tr></tbody>
</table>
<p id="A4.Thmdefinition1.p1.7" class="ltx_p"><span id="A4.Thmdefinition1.p1.7.5" class="ltx_text ltx_font_italic">where <math id="A4.Thmdefinition1.p1.3.1.m1.2" class="ltx_Math" alttext="i_{k}=(k-1)\cdot U+1,j_{k}=i_{U}+U" display="inline"><semantics id="A4.Thmdefinition1.p1.3.1.m1.2a"><mrow id="A4.Thmdefinition1.p1.3.1.m1.2.2.2" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.3.cmml"><mrow id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.cmml"><msub id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.cmml"><mi id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.2" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.2.cmml">i</mi><mi id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.3" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.3.cmml">k</mi></msub><mo id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.2" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.2.cmml">=</mo><mrow id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.cmml"><mrow id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.cmml"><mrow id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.2" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.2" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.2.cmml">k</mi><mo id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.1" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.3" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo rspace="0.055em" stretchy="false" id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.3" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.2" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.2.cmml">⋅</mo><mi id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.3" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.3.cmml">U</mi></mrow><mo id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.2" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.2.cmml">+</mo><mn id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.3" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.3.cmml">1</mn></mrow></mrow><mo id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.3" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.3a.cmml">,</mo><mrow id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.cmml"><msub id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.cmml"><mi id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.2" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.2.cmml">j</mi><mi id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.3" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.3.cmml">k</mi></msub><mo id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.1" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.1.cmml">=</mo><mrow id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.cmml"><msub id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.cmml"><mi id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.2" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.2.cmml">i</mi><mi id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.3" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.3.cmml">U</mi></msub><mo id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.1" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.1.cmml">+</mo><mi id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.3" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.3.cmml">U</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.3.1.m1.2b"><apply id="A4.Thmdefinition1.p1.3.1.m1.2.2.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.3.1.m1.2.2.3a.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1"><eq id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.2"></eq><apply id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3">subscript</csymbol><ci id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.2">𝑖</ci><ci id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.3.3">𝑘</ci></apply><apply id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1"><plus id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.2"></plus><apply id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1"><ci id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.2">⋅</ci><apply id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1"><minus id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.1"></minus><ci id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.2">𝑘</ci><cn type="integer" id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><ci id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.1.3">𝑈</ci></apply><cn type="integer" id="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.1.1.1.1.1.3">1</cn></apply></apply><apply id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2"><eq id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.1"></eq><apply id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2">subscript</csymbol><ci id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.2">𝑗</ci><ci id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.2.3">𝑘</ci></apply><apply id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3"><plus id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.1"></plus><apply id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.1.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2">subscript</csymbol><ci id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.2.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.2">𝑖</ci><ci id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.2.3">𝑈</ci></apply><ci id="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.3.cmml" xref="A4.Thmdefinition1.p1.3.1.m1.2.2.2.2.3.3">𝑈</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.3.1.m1.2c">i_{k}=(k-1)\cdot U+1,j_{k}=i_{U}+U</annotation></semantics></math> and <math id="A4.Thmdefinition1.p1.4.2.m2.1" class="ltx_Math" alttext="\bm{x}_{left}" display="inline"><semantics id="A4.Thmdefinition1.p1.4.2.m2.1a"><msub id="A4.Thmdefinition1.p1.4.2.m2.1.1" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.cmml"><mi id="A4.Thmdefinition1.p1.4.2.m2.1.1.2" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.2.cmml">𝐱</mi><mrow id="A4.Thmdefinition1.p1.4.2.m2.1.1.3" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.cmml"><mi id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.2" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.3" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1a" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.4" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1b" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.5" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.5.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.4.2.m2.1b"><apply id="A4.Thmdefinition1.p1.4.2.m2.1.1.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.4.2.m2.1.1.1.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1">subscript</csymbol><ci id="A4.Thmdefinition1.p1.4.2.m2.1.1.2.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.2">𝐱</ci><apply id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3"><times id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.1"></times><ci id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.2.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.2">𝑙</ci><ci id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.3.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.3">𝑒</ci><ci id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.4.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.4">𝑓</ci><ci id="A4.Thmdefinition1.p1.4.2.m2.1.1.3.5.cmml" xref="A4.Thmdefinition1.p1.4.2.m2.1.1.3.5">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.4.2.m2.1c">\bm{x}_{left}</annotation></semantics></math> and <math id="A4.Thmdefinition1.p1.5.3.m3.1" class="ltx_Math" alttext="\bm{x}_{right}" display="inline"><semantics id="A4.Thmdefinition1.p1.5.3.m3.1a"><msub id="A4.Thmdefinition1.p1.5.3.m3.1.1" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.cmml"><mi id="A4.Thmdefinition1.p1.5.3.m3.1.1.2" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.2.cmml">𝐱</mi><mrow id="A4.Thmdefinition1.p1.5.3.m3.1.1.3" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.cmml"><mi id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.2" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.3" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1a" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.4" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1b" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.5" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1c" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1.cmml">​</mo><mi id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.6" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.6.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.5.3.m3.1b"><apply id="A4.Thmdefinition1.p1.5.3.m3.1.1.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.5.3.m3.1.1.1.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1">subscript</csymbol><ci id="A4.Thmdefinition1.p1.5.3.m3.1.1.2.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.2">𝐱</ci><apply id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3"><times id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.1"></times><ci id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.2.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.2">𝑟</ci><ci id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.3.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.3">𝑖</ci><ci id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.4.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.4">𝑔</ci><ci id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.5.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.5">ℎ</ci><ci id="A4.Thmdefinition1.p1.5.3.m3.1.1.3.6.cmml" xref="A4.Thmdefinition1.p1.5.3.m3.1.1.3.6">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.5.3.m3.1c">\bm{x}_{right}</annotation></semantics></math> denote vectors formed by taking the initial and final <math id="A4.Thmdefinition1.p1.6.4.m4.1" class="ltx_Math" alttext="D/2" display="inline"><semantics id="A4.Thmdefinition1.p1.6.4.m4.1a"><mrow id="A4.Thmdefinition1.p1.6.4.m4.1.1" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.cmml"><mi id="A4.Thmdefinition1.p1.6.4.m4.1.1.2" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.2.cmml">D</mi><mo id="A4.Thmdefinition1.p1.6.4.m4.1.1.1" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.1.cmml">/</mo><mn id="A4.Thmdefinition1.p1.6.4.m4.1.1.3" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.6.4.m4.1b"><apply id="A4.Thmdefinition1.p1.6.4.m4.1.1.cmml" xref="A4.Thmdefinition1.p1.6.4.m4.1.1"><divide id="A4.Thmdefinition1.p1.6.4.m4.1.1.1.cmml" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.1"></divide><ci id="A4.Thmdefinition1.p1.6.4.m4.1.1.2.cmml" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.2">𝐷</ci><cn type="integer" id="A4.Thmdefinition1.p1.6.4.m4.1.1.3.cmml" xref="A4.Thmdefinition1.p1.6.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.6.4.m4.1c">D/2</annotation></semantics></math> indices of a given <math id="A4.Thmdefinition1.p1.7.5.m5.1" class="ltx_Math" alttext="\bm{x}\in\mathbb{R}^{D}" display="inline"><semantics id="A4.Thmdefinition1.p1.7.5.m5.1a"><mrow id="A4.Thmdefinition1.p1.7.5.m5.1.1" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.cmml"><mi id="A4.Thmdefinition1.p1.7.5.m5.1.1.2" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.2.cmml">𝐱</mi><mo id="A4.Thmdefinition1.p1.7.5.m5.1.1.1" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.1.cmml">∈</mo><msup id="A4.Thmdefinition1.p1.7.5.m5.1.1.3" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3.cmml"><mi id="A4.Thmdefinition1.p1.7.5.m5.1.1.3.2" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3.2.cmml">ℝ</mi><mi id="A4.Thmdefinition1.p1.7.5.m5.1.1.3.3" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition1.p1.7.5.m5.1b"><apply id="A4.Thmdefinition1.p1.7.5.m5.1.1.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1"><in id="A4.Thmdefinition1.p1.7.5.m5.1.1.1.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.1"></in><ci id="A4.Thmdefinition1.p1.7.5.m5.1.1.2.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.2">𝐱</ci><apply id="A4.Thmdefinition1.p1.7.5.m5.1.1.3.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3"><csymbol cd="ambiguous" id="A4.Thmdefinition1.p1.7.5.m5.1.1.3.1.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3">superscript</csymbol><ci id="A4.Thmdefinition1.p1.7.5.m5.1.1.3.2.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3.2">ℝ</ci><ci id="A4.Thmdefinition1.p1.7.5.m5.1.1.3.3.cmml" xref="A4.Thmdefinition1.p1.7.5.m5.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition1.p1.7.5.m5.1c">\bm{x}\in\mathbb{R}^{D}</annotation></semantics></math>.</span></p>
</div>
</div>
<div id="A4.Thmdefinition2" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A4.Thmdefinition2.1.1.1" class="ltx_text ltx_font_bold">Definition D.2</span></span><span id="A4.Thmdefinition2.2.2" class="ltx_text ltx_font_bold"> </span>(Pairwise Integer Sum and Sum Estimator)<span id="A4.Thmdefinition2.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A4.Thmdefinition2.p1" class="ltx_para">
<p id="A4.Thmdefinition2.p1.2" class="ltx_p"><span id="A4.Thmdefinition2.p1.2.2" class="ltx_text ltx_font_italic">For integers <math id="A4.Thmdefinition2.p1.1.1.m1.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A4.Thmdefinition2.p1.1.1.m1.1a"><mi id="A4.Thmdefinition2.p1.1.1.m1.1.1" xref="A4.Thmdefinition2.p1.1.1.m1.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition2.p1.1.1.m1.1b"><ci id="A4.Thmdefinition2.p1.1.1.m1.1.1.cmml" xref="A4.Thmdefinition2.p1.1.1.m1.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition2.p1.1.1.m1.1c">a</annotation></semantics></math> and <math id="A4.Thmdefinition2.p1.2.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A4.Thmdefinition2.p1.2.2.m2.1a"><mi id="A4.Thmdefinition2.p1.2.2.m2.1.1" xref="A4.Thmdefinition2.p1.2.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition2.p1.2.2.m2.1b"><ci id="A4.Thmdefinition2.p1.2.2.m2.1.1.cmml" xref="A4.Thmdefinition2.p1.2.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition2.p1.2.2.m2.1c">b</annotation></semantics></math>, define</span></p>
<table id="A6.EGx14" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A4.E19"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E19.m1.2" class="ltx_Math" alttext="\displaystyle s(a,b)" display="inline"><semantics id="A4.E19.m1.2a"><mrow id="A4.E19.m1.2.3" xref="A4.E19.m1.2.3.cmml"><mi id="A4.E19.m1.2.3.2" xref="A4.E19.m1.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A4.E19.m1.2.3.1" xref="A4.E19.m1.2.3.1.cmml">​</mo><mrow id="A4.E19.m1.2.3.3.2" xref="A4.E19.m1.2.3.3.1.cmml"><mo stretchy="false" id="A4.E19.m1.2.3.3.2.1" xref="A4.E19.m1.2.3.3.1.cmml">(</mo><mi id="A4.E19.m1.1.1" xref="A4.E19.m1.1.1.cmml">a</mi><mo id="A4.E19.m1.2.3.3.2.2" xref="A4.E19.m1.2.3.3.1.cmml">,</mo><mi id="A4.E19.m1.2.2" xref="A4.E19.m1.2.2.cmml">b</mi><mo stretchy="false" id="A4.E19.m1.2.3.3.2.3" xref="A4.E19.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E19.m1.2b"><apply id="A4.E19.m1.2.3.cmml" xref="A4.E19.m1.2.3"><times id="A4.E19.m1.2.3.1.cmml" xref="A4.E19.m1.2.3.1"></times><ci id="A4.E19.m1.2.3.2.cmml" xref="A4.E19.m1.2.3.2">𝑠</ci><interval closure="open" id="A4.E19.m1.2.3.3.1.cmml" xref="A4.E19.m1.2.3.3.2"><ci id="A4.E19.m1.1.1.cmml" xref="A4.E19.m1.1.1">𝑎</ci><ci id="A4.E19.m1.2.2.cmml" xref="A4.E19.m1.2.2">𝑏</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E19.m1.2c">\displaystyle s(a,b)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A4.E19.m2.1" class="ltx_Math" alttext="\displaystyle\triangleq a+b" display="inline"><semantics id="A4.E19.m2.1a"><mrow id="A4.E19.m2.1.1" xref="A4.E19.m2.1.1.cmml"><mi id="A4.E19.m2.1.1.2" xref="A4.E19.m2.1.1.2.cmml"></mi><mo id="A4.E19.m2.1.1.1" xref="A4.E19.m2.1.1.1.cmml">≜</mo><mrow id="A4.E19.m2.1.1.3" xref="A4.E19.m2.1.1.3.cmml"><mi id="A4.E19.m2.1.1.3.2" xref="A4.E19.m2.1.1.3.2.cmml">a</mi><mo id="A4.E19.m2.1.1.3.1" xref="A4.E19.m2.1.1.3.1.cmml">+</mo><mi id="A4.E19.m2.1.1.3.3" xref="A4.E19.m2.1.1.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E19.m2.1b"><apply id="A4.E19.m2.1.1.cmml" xref="A4.E19.m2.1.1"><ci id="A4.E19.m2.1.1.1.cmml" xref="A4.E19.m2.1.1.1">≜</ci><csymbol cd="latexml" id="A4.E19.m2.1.1.2.cmml" xref="A4.E19.m2.1.1.2">absent</csymbol><apply id="A4.E19.m2.1.1.3.cmml" xref="A4.E19.m2.1.1.3"><plus id="A4.E19.m2.1.1.3.1.cmml" xref="A4.E19.m2.1.1.3.1"></plus><ci id="A4.E19.m2.1.1.3.2.cmml" xref="A4.E19.m2.1.1.3.2">𝑎</ci><ci id="A4.E19.m2.1.1.3.3.cmml" xref="A4.E19.m2.1.1.3.3">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E19.m2.1c">\displaystyle\triangleq a+b</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(19)</span></td>
</tr></tbody>
<tbody id="A4.E20"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E20.m1.2" class="ltx_Math" alttext="\displaystyle\hat{s}(a,b)" display="inline"><semantics id="A4.E20.m1.2a"><mrow id="A4.E20.m1.2.3" xref="A4.E20.m1.2.3.cmml"><mover accent="true" id="A4.E20.m1.2.3.2" xref="A4.E20.m1.2.3.2.cmml"><mi id="A4.E20.m1.2.3.2.2" xref="A4.E20.m1.2.3.2.2.cmml">s</mi><mo id="A4.E20.m1.2.3.2.1" xref="A4.E20.m1.2.3.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A4.E20.m1.2.3.1" xref="A4.E20.m1.2.3.1.cmml">​</mo><mrow id="A4.E20.m1.2.3.3.2" xref="A4.E20.m1.2.3.3.1.cmml"><mo stretchy="false" id="A4.E20.m1.2.3.3.2.1" xref="A4.E20.m1.2.3.3.1.cmml">(</mo><mi id="A4.E20.m1.1.1" xref="A4.E20.m1.1.1.cmml">a</mi><mo id="A4.E20.m1.2.3.3.2.2" xref="A4.E20.m1.2.3.3.1.cmml">,</mo><mi id="A4.E20.m1.2.2" xref="A4.E20.m1.2.2.cmml">b</mi><mo stretchy="false" id="A4.E20.m1.2.3.3.2.3" xref="A4.E20.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E20.m1.2b"><apply id="A4.E20.m1.2.3.cmml" xref="A4.E20.m1.2.3"><times id="A4.E20.m1.2.3.1.cmml" xref="A4.E20.m1.2.3.1"></times><apply id="A4.E20.m1.2.3.2.cmml" xref="A4.E20.m1.2.3.2"><ci id="A4.E20.m1.2.3.2.1.cmml" xref="A4.E20.m1.2.3.2.1">^</ci><ci id="A4.E20.m1.2.3.2.2.cmml" xref="A4.E20.m1.2.3.2.2">𝑠</ci></apply><interval closure="open" id="A4.E20.m1.2.3.3.1.cmml" xref="A4.E20.m1.2.3.3.2"><ci id="A4.E20.m1.1.1.cmml" xref="A4.E20.m1.1.1">𝑎</ci><ci id="A4.E20.m1.2.2.cmml" xref="A4.E20.m1.2.2">𝑏</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E20.m1.2c">\displaystyle\hat{s}(a,b)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A4.E20.m2.2" class="ltx_Math" alttext="\displaystyle\triangleq 2\mu(a,b)" display="inline"><semantics id="A4.E20.m2.2a"><mrow id="A4.E20.m2.2.3" xref="A4.E20.m2.2.3.cmml"><mi id="A4.E20.m2.2.3.2" xref="A4.E20.m2.2.3.2.cmml"></mi><mo id="A4.E20.m2.2.3.1" xref="A4.E20.m2.2.3.1.cmml">≜</mo><mrow id="A4.E20.m2.2.3.3" xref="A4.E20.m2.2.3.3.cmml"><mn id="A4.E20.m2.2.3.3.2" xref="A4.E20.m2.2.3.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A4.E20.m2.2.3.3.1" xref="A4.E20.m2.2.3.3.1.cmml">​</mo><mi id="A4.E20.m2.2.3.3.3" xref="A4.E20.m2.2.3.3.3.cmml">μ</mi><mo lspace="0em" rspace="0em" id="A4.E20.m2.2.3.3.1a" xref="A4.E20.m2.2.3.3.1.cmml">​</mo><mrow id="A4.E20.m2.2.3.3.4.2" xref="A4.E20.m2.2.3.3.4.1.cmml"><mo stretchy="false" id="A4.E20.m2.2.3.3.4.2.1" xref="A4.E20.m2.2.3.3.4.1.cmml">(</mo><mi id="A4.E20.m2.1.1" xref="A4.E20.m2.1.1.cmml">a</mi><mo id="A4.E20.m2.2.3.3.4.2.2" xref="A4.E20.m2.2.3.3.4.1.cmml">,</mo><mi id="A4.E20.m2.2.2" xref="A4.E20.m2.2.2.cmml">b</mi><mo stretchy="false" id="A4.E20.m2.2.3.3.4.2.3" xref="A4.E20.m2.2.3.3.4.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E20.m2.2b"><apply id="A4.E20.m2.2.3.cmml" xref="A4.E20.m2.2.3"><ci id="A4.E20.m2.2.3.1.cmml" xref="A4.E20.m2.2.3.1">≜</ci><csymbol cd="latexml" id="A4.E20.m2.2.3.2.cmml" xref="A4.E20.m2.2.3.2">absent</csymbol><apply id="A4.E20.m2.2.3.3.cmml" xref="A4.E20.m2.2.3.3"><times id="A4.E20.m2.2.3.3.1.cmml" xref="A4.E20.m2.2.3.3.1"></times><cn type="integer" id="A4.E20.m2.2.3.3.2.cmml" xref="A4.E20.m2.2.3.3.2">2</cn><ci id="A4.E20.m2.2.3.3.3.cmml" xref="A4.E20.m2.2.3.3.3">𝜇</ci><interval closure="open" id="A4.E20.m2.2.3.3.4.1.cmml" xref="A4.E20.m2.2.3.3.4.2"><ci id="A4.E20.m2.1.1.cmml" xref="A4.E20.m2.1.1">𝑎</ci><ci id="A4.E20.m2.2.2.cmml" xref="A4.E20.m2.2.2">𝑏</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E20.m2.2c">\displaystyle\triangleq 2\mu(a,b)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(20)</span></td>
</tr></tbody>
</table>
<p id="A4.Thmdefinition2.p1.3" class="ltx_p"><span id="A4.Thmdefinition2.p1.3.1" class="ltx_text ltx_font_italic">where <math id="A4.Thmdefinition2.p1.3.1.m1.3" class="ltx_Math" alttext="\mu(a,b)\triangleq\lfloor\frac{1}{2}(a+b+1)\rfloor" display="inline"><semantics id="A4.Thmdefinition2.p1.3.1.m1.3a"><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.cmml"><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.cmml"><mi id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.2.cmml">μ</mi><mo lspace="0em" rspace="0em" id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.1.cmml">​</mo><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.1.cmml"><mo stretchy="false" id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.2.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.1.cmml">(</mo><mi id="A4.Thmdefinition2.p1.3.1.m1.1.1" xref="A4.Thmdefinition2.p1.3.1.m1.1.1.cmml">a</mi><mo id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.2.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.1.cmml">,</mo><mi id="A4.Thmdefinition2.p1.3.1.m1.2.2" xref="A4.Thmdefinition2.p1.3.1.m1.2.2.cmml">b</mi><mo stretchy="false" id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.2.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.1.cmml">)</mo></mrow></mrow><mo id="A4.Thmdefinition2.p1.3.1.m1.3.3.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.2.cmml">≜</mo><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.2.cmml"><mo stretchy="false" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.2.1.cmml">⌊</mo><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.cmml"><mfrac id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.cmml"><mn id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.2.cmml">1</mn><mn id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.cmml">(</mo><mrow id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.cmml"><mi id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.2" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.2.cmml">a</mi><mo id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.1" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.1.cmml">+</mo><mi id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.3.cmml">b</mi><mo id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.1a" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.1.cmml">+</mo><mn id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.4" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.4.cmml">1</mn></mrow><mo stretchy="false" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.3" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.2.1.cmml">⌋</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmdefinition2.p1.3.1.m1.3b"><apply id="A4.Thmdefinition2.p1.3.1.m1.3.3.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3"><ci id="A4.Thmdefinition2.p1.3.1.m1.3.3.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.2">≜</ci><apply id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3"><times id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.1"></times><ci id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.2">𝜇</ci><interval closure="open" id="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.3.3.2"><ci id="A4.Thmdefinition2.p1.3.1.m1.1.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.1.1">𝑎</ci><ci id="A4.Thmdefinition2.p1.3.1.m1.2.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.2.2">𝑏</ci></interval></apply><apply id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1"><floor id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.2.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.2"></floor><apply id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1"><times id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.2"></times><apply id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3"><divide id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3"></divide><cn type="integer" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.2">1</cn><cn type="integer" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.3.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.3.3">2</cn></apply><apply id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1"><plus id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.1.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.1"></plus><ci id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.2.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.2">𝑎</ci><ci id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.3.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.3">𝑏</ci><cn type="integer" id="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.4.cmml" xref="A4.Thmdefinition2.p1.3.1.m1.3.3.1.1.1.1.1.1.4">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmdefinition2.p1.3.1.m1.3c">\mu(a,b)\triangleq\lfloor\frac{1}{2}(a+b+1)\rfloor</annotation></semantics></math>.</span></p>
</div>
</div>
<div id="A4.Thmlemma1" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A4.Thmlemma1.1.1.1" class="ltx_text ltx_font_bold">Lemma D.1</span></span><span id="A4.Thmlemma1.2.2" class="ltx_text ltx_font_bold"> </span>(Bias when averaging one pair)<span id="A4.Thmlemma1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A4.Thmlemma1.p1" class="ltx_para">
<p id="A4.Thmlemma1.p1.4" class="ltx_p"><span id="A4.Thmlemma1.p1.4.4" class="ltx_text ltx_font_italic">Consider two scalars <math id="A4.Thmlemma1.p1.1.1.m1.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A4.Thmlemma1.p1.1.1.m1.1a"><mi id="A4.Thmlemma1.p1.1.1.m1.1.1" xref="A4.Thmlemma1.p1.1.1.m1.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A4.Thmlemma1.p1.1.1.m1.1b"><ci id="A4.Thmlemma1.p1.1.1.m1.1.1.cmml" xref="A4.Thmlemma1.p1.1.1.m1.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma1.p1.1.1.m1.1c">a</annotation></semantics></math> and <math id="A4.Thmlemma1.p1.2.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A4.Thmlemma1.p1.2.2.m2.1a"><mi id="A4.Thmlemma1.p1.2.2.m2.1.1" xref="A4.Thmlemma1.p1.2.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A4.Thmlemma1.p1.2.2.m2.1b"><ci id="A4.Thmlemma1.p1.2.2.m2.1.1.cmml" xref="A4.Thmlemma1.p1.2.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma1.p1.2.2.m2.1c">b</annotation></semantics></math>, with <math id="A4.Thmlemma1.p1.3.3.m3.2" class="ltx_Math" alttext="a,b\stackrel{{\scriptstyle iid}}{{\sim}}" display="inline"><semantics id="A4.Thmlemma1.p1.3.3.m3.2a"><mrow id="A4.Thmlemma1.p1.3.3.m3.2.3" xref="A4.Thmlemma1.p1.3.3.m3.2.3.cmml"><mrow id="A4.Thmlemma1.p1.3.3.m3.2.3.2.2" xref="A4.Thmlemma1.p1.3.3.m3.2.3.2.1.cmml"><mi id="A4.Thmlemma1.p1.3.3.m3.1.1" xref="A4.Thmlemma1.p1.3.3.m3.1.1.cmml">a</mi><mo id="A4.Thmlemma1.p1.3.3.m3.2.3.2.2.1" xref="A4.Thmlemma1.p1.3.3.m3.2.3.2.1.cmml">,</mo><mi id="A4.Thmlemma1.p1.3.3.m3.2.2" xref="A4.Thmlemma1.p1.3.3.m3.2.2.cmml">b</mi></mrow><mover id="A4.Thmlemma1.p1.3.3.m3.2.3.1" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.cmml"><mo id="A4.Thmlemma1.p1.3.3.m3.2.3.1.2" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.2.cmml">∼</mo><mrow id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.cmml"><mi id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.2" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.1" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.1.cmml">​</mo><mi id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.3" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.1a" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.1.cmml">​</mo><mi id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.4" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.4.cmml">d</mi></mrow></mover><mi id="A4.Thmlemma1.p1.3.3.m3.2.3.3" xref="A4.Thmlemma1.p1.3.3.m3.2.3.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmlemma1.p1.3.3.m3.2b"><apply id="A4.Thmlemma1.p1.3.3.m3.2.3.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3"><apply id="A4.Thmlemma1.p1.3.3.m3.2.3.1.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1"><csymbol cd="ambiguous" id="A4.Thmlemma1.p1.3.3.m3.2.3.1.1.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1">superscript</csymbol><csymbol cd="latexml" id="A4.Thmlemma1.p1.3.3.m3.2.3.1.2.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.2">similar-to</csymbol><apply id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3"><times id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.1.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.1"></times><ci id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.2.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.2">𝑖</ci><ci id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.3.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.3">𝑖</ci><ci id="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.4.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.1.3.4">𝑑</ci></apply></apply><list id="A4.Thmlemma1.p1.3.3.m3.2.3.2.1.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.2.2"><ci id="A4.Thmlemma1.p1.3.3.m3.1.1.cmml" xref="A4.Thmlemma1.p1.3.3.m3.1.1">𝑎</ci><ci id="A4.Thmlemma1.p1.3.3.m3.2.2.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.2">𝑏</ci></list><csymbol cd="latexml" id="A4.Thmlemma1.p1.3.3.m3.2.3.3.cmml" xref="A4.Thmlemma1.p1.3.3.m3.2.3.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma1.p1.3.3.m3.2c">a,b\stackrel{{\scriptstyle iid}}{{\sim}}</annotation></semantics></math> Bernoulli(.5). Define <math id="A4.Thmlemma1.p1.4.4.m4.6" class="ltx_Math" alttext="\varepsilon(a,b)\triangleq\hat{s}(a,b)-s(a,b)" display="inline"><semantics id="A4.Thmlemma1.p1.4.4.m4.6a"><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7" xref="A4.Thmlemma1.p1.4.4.m4.6.7.cmml"><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.cmml"><mi id="A4.Thmlemma1.p1.4.4.m4.6.7.2.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="A4.Thmlemma1.p1.4.4.m4.6.7.2.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.1.cmml">​</mo><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.1.cmml"><mo stretchy="false" id="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.2.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.1.cmml">(</mo><mi id="A4.Thmlemma1.p1.4.4.m4.1.1" xref="A4.Thmlemma1.p1.4.4.m4.1.1.cmml">a</mi><mo id="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.2.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.1.cmml">,</mo><mi id="A4.Thmlemma1.p1.4.4.m4.2.2" xref="A4.Thmlemma1.p1.4.4.m4.2.2.cmml">b</mi><mo stretchy="false" id="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.2.3" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.1.cmml">)</mo></mrow></mrow><mo id="A4.Thmlemma1.p1.4.4.m4.6.7.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.1.cmml">≜</mo><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.3" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.cmml"><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.cmml"><mover accent="true" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.cmml"><mi id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.2.cmml">s</mi><mo id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.1.cmml">​</mo><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.1.cmml"><mo stretchy="false" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.2.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.1.cmml">(</mo><mi id="A4.Thmlemma1.p1.4.4.m4.3.3" xref="A4.Thmlemma1.p1.4.4.m4.3.3.cmml">a</mi><mo id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.2.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.1.cmml">,</mo><mi id="A4.Thmlemma1.p1.4.4.m4.4.4" xref="A4.Thmlemma1.p1.4.4.m4.4.4.cmml">b</mi><mo stretchy="false" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.2.3" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.1.cmml">)</mo></mrow></mrow><mo id="A4.Thmlemma1.p1.4.4.m4.6.7.3.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.1.cmml">−</mo><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.cmml"><mi id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.1.cmml">​</mo><mrow id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.1.cmml"><mo stretchy="false" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.2.1" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.1.cmml">(</mo><mi id="A4.Thmlemma1.p1.4.4.m4.5.5" xref="A4.Thmlemma1.p1.4.4.m4.5.5.cmml">a</mi><mo id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.2.2" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.1.cmml">,</mo><mi id="A4.Thmlemma1.p1.4.4.m4.6.6" xref="A4.Thmlemma1.p1.4.4.m4.6.6.cmml">b</mi><mo stretchy="false" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.2.3" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmlemma1.p1.4.4.m4.6b"><apply id="A4.Thmlemma1.p1.4.4.m4.6.7.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7"><ci id="A4.Thmlemma1.p1.4.4.m4.6.7.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.1">≜</ci><apply id="A4.Thmlemma1.p1.4.4.m4.6.7.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2"><times id="A4.Thmlemma1.p1.4.4.m4.6.7.2.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.1"></times><ci id="A4.Thmlemma1.p1.4.4.m4.6.7.2.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.2">𝜀</ci><interval closure="open" id="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.2.3.2"><ci id="A4.Thmlemma1.p1.4.4.m4.1.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.1.1">𝑎</ci><ci id="A4.Thmlemma1.p1.4.4.m4.2.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.2.2">𝑏</ci></interval></apply><apply id="A4.Thmlemma1.p1.4.4.m4.6.7.3.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3"><minus id="A4.Thmlemma1.p1.4.4.m4.6.7.3.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.1"></minus><apply id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2"><times id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.1"></times><apply id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2"><ci id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.1">^</ci><ci id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.2.2">𝑠</ci></apply><interval closure="open" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.2.3.2"><ci id="A4.Thmlemma1.p1.4.4.m4.3.3.cmml" xref="A4.Thmlemma1.p1.4.4.m4.3.3">𝑎</ci><ci id="A4.Thmlemma1.p1.4.4.m4.4.4.cmml" xref="A4.Thmlemma1.p1.4.4.m4.4.4">𝑏</ci></interval></apply><apply id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3"><times id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.1"></times><ci id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.2.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.2">𝑠</ci><interval closure="open" id="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.1.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.7.3.3.3.2"><ci id="A4.Thmlemma1.p1.4.4.m4.5.5.cmml" xref="A4.Thmlemma1.p1.4.4.m4.5.5">𝑎</ci><ci id="A4.Thmlemma1.p1.4.4.m4.6.6.cmml" xref="A4.Thmlemma1.p1.4.4.m4.6.6">𝑏</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma1.p1.4.4.m4.6c">\varepsilon(a,b)\triangleq\hat{s}(a,b)-s(a,b)</annotation></semantics></math>. Then</span></p>
<table id="A4.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A4.Ex1.m1.3" class="ltx_Math" alttext="E[\varepsilon(a,b)]=\frac{1}{2}" display="block"><semantics id="A4.Ex1.m1.3a"><mrow id="A4.Ex1.m1.3.3" xref="A4.Ex1.m1.3.3.cmml"><mrow id="A4.Ex1.m1.3.3.1" xref="A4.Ex1.m1.3.3.1.cmml"><mi id="A4.Ex1.m1.3.3.1.3" xref="A4.Ex1.m1.3.3.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.Ex1.m1.3.3.1.2" xref="A4.Ex1.m1.3.3.1.2.cmml">​</mo><mrow id="A4.Ex1.m1.3.3.1.1.1" xref="A4.Ex1.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="A4.Ex1.m1.3.3.1.1.1.2" xref="A4.Ex1.m1.3.3.1.1.2.1.cmml">[</mo><mrow id="A4.Ex1.m1.3.3.1.1.1.1" xref="A4.Ex1.m1.3.3.1.1.1.1.cmml"><mi id="A4.Ex1.m1.3.3.1.1.1.1.2" xref="A4.Ex1.m1.3.3.1.1.1.1.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="A4.Ex1.m1.3.3.1.1.1.1.1" xref="A4.Ex1.m1.3.3.1.1.1.1.1.cmml">​</mo><mrow id="A4.Ex1.m1.3.3.1.1.1.1.3.2" xref="A4.Ex1.m1.3.3.1.1.1.1.3.1.cmml"><mo stretchy="false" id="A4.Ex1.m1.3.3.1.1.1.1.3.2.1" xref="A4.Ex1.m1.3.3.1.1.1.1.3.1.cmml">(</mo><mi id="A4.Ex1.m1.1.1" xref="A4.Ex1.m1.1.1.cmml">a</mi><mo id="A4.Ex1.m1.3.3.1.1.1.1.3.2.2" xref="A4.Ex1.m1.3.3.1.1.1.1.3.1.cmml">,</mo><mi id="A4.Ex1.m1.2.2" xref="A4.Ex1.m1.2.2.cmml">b</mi><mo stretchy="false" id="A4.Ex1.m1.3.3.1.1.1.1.3.2.3" xref="A4.Ex1.m1.3.3.1.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.Ex1.m1.3.3.1.1.1.3" xref="A4.Ex1.m1.3.3.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A4.Ex1.m1.3.3.2" xref="A4.Ex1.m1.3.3.2.cmml">=</mo><mfrac id="A4.Ex1.m1.3.3.3" xref="A4.Ex1.m1.3.3.3.cmml"><mn id="A4.Ex1.m1.3.3.3.2" xref="A4.Ex1.m1.3.3.3.2.cmml">1</mn><mn id="A4.Ex1.m1.3.3.3.3" xref="A4.Ex1.m1.3.3.3.3.cmml">2</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A4.Ex1.m1.3b"><apply id="A4.Ex1.m1.3.3.cmml" xref="A4.Ex1.m1.3.3"><eq id="A4.Ex1.m1.3.3.2.cmml" xref="A4.Ex1.m1.3.3.2"></eq><apply id="A4.Ex1.m1.3.3.1.cmml" xref="A4.Ex1.m1.3.3.1"><times id="A4.Ex1.m1.3.3.1.2.cmml" xref="A4.Ex1.m1.3.3.1.2"></times><ci id="A4.Ex1.m1.3.3.1.3.cmml" xref="A4.Ex1.m1.3.3.1.3">𝐸</ci><apply id="A4.Ex1.m1.3.3.1.1.2.cmml" xref="A4.Ex1.m1.3.3.1.1.1"><csymbol cd="latexml" id="A4.Ex1.m1.3.3.1.1.2.1.cmml" xref="A4.Ex1.m1.3.3.1.1.1.2">delimited-[]</csymbol><apply id="A4.Ex1.m1.3.3.1.1.1.1.cmml" xref="A4.Ex1.m1.3.3.1.1.1.1"><times id="A4.Ex1.m1.3.3.1.1.1.1.1.cmml" xref="A4.Ex1.m1.3.3.1.1.1.1.1"></times><ci id="A4.Ex1.m1.3.3.1.1.1.1.2.cmml" xref="A4.Ex1.m1.3.3.1.1.1.1.2">𝜀</ci><interval closure="open" id="A4.Ex1.m1.3.3.1.1.1.1.3.1.cmml" xref="A4.Ex1.m1.3.3.1.1.1.1.3.2"><ci id="A4.Ex1.m1.1.1.cmml" xref="A4.Ex1.m1.1.1">𝑎</ci><ci id="A4.Ex1.m1.2.2.cmml" xref="A4.Ex1.m1.2.2">𝑏</ci></interval></apply></apply></apply><apply id="A4.Ex1.m1.3.3.3.cmml" xref="A4.Ex1.m1.3.3.3"><divide id="A4.Ex1.m1.3.3.3.1.cmml" xref="A4.Ex1.m1.3.3.3"></divide><cn type="integer" id="A4.Ex1.m1.3.3.3.2.cmml" xref="A4.Ex1.m1.3.3.3.2">1</cn><cn type="integer" id="A4.Ex1.m1.3.3.3.3.cmml" xref="A4.Ex1.m1.3.3.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Ex1.m1.3c">E[\varepsilon(a,b)]=\frac{1}{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A4.1" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A4.1.p1" class="ltx_para">
<p id="A4.1.p1.8" class="ltx_p">The proof follows immediately from considering the four equiprobable realizations of the pair <math id="A4.1.p1.1.m1.2" class="ltx_Math" alttext="a,b" display="inline"><semantics id="A4.1.p1.1.m1.2a"><mrow id="A4.1.p1.1.m1.2.3.2" xref="A4.1.p1.1.m1.2.3.1.cmml"><mi id="A4.1.p1.1.m1.1.1" xref="A4.1.p1.1.m1.1.1.cmml">a</mi><mo id="A4.1.p1.1.m1.2.3.2.1" xref="A4.1.p1.1.m1.2.3.1.cmml">,</mo><mi id="A4.1.p1.1.m1.2.2" xref="A4.1.p1.1.m1.2.2.cmml">b</mi></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.1.m1.2b"><list id="A4.1.p1.1.m1.2.3.1.cmml" xref="A4.1.p1.1.m1.2.3.2"><ci id="A4.1.p1.1.m1.1.1.cmml" xref="A4.1.p1.1.m1.1.1">𝑎</ci><ci id="A4.1.p1.1.m1.2.2.cmml" xref="A4.1.p1.1.m1.2.2">𝑏</ci></list></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.1.m1.2c">a,b</annotation></semantics></math>. In the cases <math id="A4.1.p1.2.m2.2" class="ltx_Math" alttext="(0,0)" display="inline"><semantics id="A4.1.p1.2.m2.2a"><mrow id="A4.1.p1.2.m2.2.3.2" xref="A4.1.p1.2.m2.2.3.1.cmml"><mo stretchy="false" id="A4.1.p1.2.m2.2.3.2.1" xref="A4.1.p1.2.m2.2.3.1.cmml">(</mo><mn id="A4.1.p1.2.m2.1.1" xref="A4.1.p1.2.m2.1.1.cmml">0</mn><mo id="A4.1.p1.2.m2.2.3.2.2" xref="A4.1.p1.2.m2.2.3.1.cmml">,</mo><mn id="A4.1.p1.2.m2.2.2" xref="A4.1.p1.2.m2.2.2.cmml">0</mn><mo stretchy="false" id="A4.1.p1.2.m2.2.3.2.3" xref="A4.1.p1.2.m2.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.2.m2.2b"><interval closure="open" id="A4.1.p1.2.m2.2.3.1.cmml" xref="A4.1.p1.2.m2.2.3.2"><cn type="integer" id="A4.1.p1.2.m2.1.1.cmml" xref="A4.1.p1.2.m2.1.1">0</cn><cn type="integer" id="A4.1.p1.2.m2.2.2.cmml" xref="A4.1.p1.2.m2.2.2">0</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.2.m2.2c">(0,0)</annotation></semantics></math> and <math id="A4.1.p1.3.m3.2" class="ltx_Math" alttext="(1,1)" display="inline"><semantics id="A4.1.p1.3.m3.2a"><mrow id="A4.1.p1.3.m3.2.3.2" xref="A4.1.p1.3.m3.2.3.1.cmml"><mo stretchy="false" id="A4.1.p1.3.m3.2.3.2.1" xref="A4.1.p1.3.m3.2.3.1.cmml">(</mo><mn id="A4.1.p1.3.m3.1.1" xref="A4.1.p1.3.m3.1.1.cmml">1</mn><mo id="A4.1.p1.3.m3.2.3.2.2" xref="A4.1.p1.3.m3.2.3.1.cmml">,</mo><mn id="A4.1.p1.3.m3.2.2" xref="A4.1.p1.3.m3.2.2.cmml">1</mn><mo stretchy="false" id="A4.1.p1.3.m3.2.3.2.3" xref="A4.1.p1.3.m3.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.3.m3.2b"><interval closure="open" id="A4.1.p1.3.m3.2.3.1.cmml" xref="A4.1.p1.3.m3.2.3.2"><cn type="integer" id="A4.1.p1.3.m3.1.1.cmml" xref="A4.1.p1.3.m3.1.1">1</cn><cn type="integer" id="A4.1.p1.3.m3.2.2.cmml" xref="A4.1.p1.3.m3.2.2">1</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.3.m3.2c">(1,1)</annotation></semantics></math>, <math id="A4.1.p1.4.m4.4" class="ltx_Math" alttext="2\mu(a,b)=s(a,b)" display="inline"><semantics id="A4.1.p1.4.m4.4a"><mrow id="A4.1.p1.4.m4.4.5" xref="A4.1.p1.4.m4.4.5.cmml"><mrow id="A4.1.p1.4.m4.4.5.2" xref="A4.1.p1.4.m4.4.5.2.cmml"><mn id="A4.1.p1.4.m4.4.5.2.2" xref="A4.1.p1.4.m4.4.5.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A4.1.p1.4.m4.4.5.2.1" xref="A4.1.p1.4.m4.4.5.2.1.cmml">​</mo><mi id="A4.1.p1.4.m4.4.5.2.3" xref="A4.1.p1.4.m4.4.5.2.3.cmml">μ</mi><mo lspace="0em" rspace="0em" id="A4.1.p1.4.m4.4.5.2.1a" xref="A4.1.p1.4.m4.4.5.2.1.cmml">​</mo><mrow id="A4.1.p1.4.m4.4.5.2.4.2" xref="A4.1.p1.4.m4.4.5.2.4.1.cmml"><mo stretchy="false" id="A4.1.p1.4.m4.4.5.2.4.2.1" xref="A4.1.p1.4.m4.4.5.2.4.1.cmml">(</mo><mi id="A4.1.p1.4.m4.1.1" xref="A4.1.p1.4.m4.1.1.cmml">a</mi><mo id="A4.1.p1.4.m4.4.5.2.4.2.2" xref="A4.1.p1.4.m4.4.5.2.4.1.cmml">,</mo><mi id="A4.1.p1.4.m4.2.2" xref="A4.1.p1.4.m4.2.2.cmml">b</mi><mo stretchy="false" id="A4.1.p1.4.m4.4.5.2.4.2.3" xref="A4.1.p1.4.m4.4.5.2.4.1.cmml">)</mo></mrow></mrow><mo id="A4.1.p1.4.m4.4.5.1" xref="A4.1.p1.4.m4.4.5.1.cmml">=</mo><mrow id="A4.1.p1.4.m4.4.5.3" xref="A4.1.p1.4.m4.4.5.3.cmml"><mi id="A4.1.p1.4.m4.4.5.3.2" xref="A4.1.p1.4.m4.4.5.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A4.1.p1.4.m4.4.5.3.1" xref="A4.1.p1.4.m4.4.5.3.1.cmml">​</mo><mrow id="A4.1.p1.4.m4.4.5.3.3.2" xref="A4.1.p1.4.m4.4.5.3.3.1.cmml"><mo stretchy="false" id="A4.1.p1.4.m4.4.5.3.3.2.1" xref="A4.1.p1.4.m4.4.5.3.3.1.cmml">(</mo><mi id="A4.1.p1.4.m4.3.3" xref="A4.1.p1.4.m4.3.3.cmml">a</mi><mo id="A4.1.p1.4.m4.4.5.3.3.2.2" xref="A4.1.p1.4.m4.4.5.3.3.1.cmml">,</mo><mi id="A4.1.p1.4.m4.4.4" xref="A4.1.p1.4.m4.4.4.cmml">b</mi><mo stretchy="false" id="A4.1.p1.4.m4.4.5.3.3.2.3" xref="A4.1.p1.4.m4.4.5.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.4.m4.4b"><apply id="A4.1.p1.4.m4.4.5.cmml" xref="A4.1.p1.4.m4.4.5"><eq id="A4.1.p1.4.m4.4.5.1.cmml" xref="A4.1.p1.4.m4.4.5.1"></eq><apply id="A4.1.p1.4.m4.4.5.2.cmml" xref="A4.1.p1.4.m4.4.5.2"><times id="A4.1.p1.4.m4.4.5.2.1.cmml" xref="A4.1.p1.4.m4.4.5.2.1"></times><cn type="integer" id="A4.1.p1.4.m4.4.5.2.2.cmml" xref="A4.1.p1.4.m4.4.5.2.2">2</cn><ci id="A4.1.p1.4.m4.4.5.2.3.cmml" xref="A4.1.p1.4.m4.4.5.2.3">𝜇</ci><interval closure="open" id="A4.1.p1.4.m4.4.5.2.4.1.cmml" xref="A4.1.p1.4.m4.4.5.2.4.2"><ci id="A4.1.p1.4.m4.1.1.cmml" xref="A4.1.p1.4.m4.1.1">𝑎</ci><ci id="A4.1.p1.4.m4.2.2.cmml" xref="A4.1.p1.4.m4.2.2">𝑏</ci></interval></apply><apply id="A4.1.p1.4.m4.4.5.3.cmml" xref="A4.1.p1.4.m4.4.5.3"><times id="A4.1.p1.4.m4.4.5.3.1.cmml" xref="A4.1.p1.4.m4.4.5.3.1"></times><ci id="A4.1.p1.4.m4.4.5.3.2.cmml" xref="A4.1.p1.4.m4.4.5.3.2">𝑠</ci><interval closure="open" id="A4.1.p1.4.m4.4.5.3.3.1.cmml" xref="A4.1.p1.4.m4.4.5.3.3.2"><ci id="A4.1.p1.4.m4.3.3.cmml" xref="A4.1.p1.4.m4.3.3">𝑎</ci><ci id="A4.1.p1.4.m4.4.4.cmml" xref="A4.1.p1.4.m4.4.4">𝑏</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.4.m4.4c">2\mu(a,b)=s(a,b)</annotation></semantics></math>. In the cases <math id="A4.1.p1.5.m5.2" class="ltx_Math" alttext="(0,1)" display="inline"><semantics id="A4.1.p1.5.m5.2a"><mrow id="A4.1.p1.5.m5.2.3.2" xref="A4.1.p1.5.m5.2.3.1.cmml"><mo stretchy="false" id="A4.1.p1.5.m5.2.3.2.1" xref="A4.1.p1.5.m5.2.3.1.cmml">(</mo><mn id="A4.1.p1.5.m5.1.1" xref="A4.1.p1.5.m5.1.1.cmml">0</mn><mo id="A4.1.p1.5.m5.2.3.2.2" xref="A4.1.p1.5.m5.2.3.1.cmml">,</mo><mn id="A4.1.p1.5.m5.2.2" xref="A4.1.p1.5.m5.2.2.cmml">1</mn><mo stretchy="false" id="A4.1.p1.5.m5.2.3.2.3" xref="A4.1.p1.5.m5.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.5.m5.2b"><interval closure="open" id="A4.1.p1.5.m5.2.3.1.cmml" xref="A4.1.p1.5.m5.2.3.2"><cn type="integer" id="A4.1.p1.5.m5.1.1.cmml" xref="A4.1.p1.5.m5.1.1">0</cn><cn type="integer" id="A4.1.p1.5.m5.2.2.cmml" xref="A4.1.p1.5.m5.2.2">1</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.5.m5.2c">(0,1)</annotation></semantics></math> and <math id="A4.1.p1.6.m6.2" class="ltx_Math" alttext="(1,0)" display="inline"><semantics id="A4.1.p1.6.m6.2a"><mrow id="A4.1.p1.6.m6.2.3.2" xref="A4.1.p1.6.m6.2.3.1.cmml"><mo stretchy="false" id="A4.1.p1.6.m6.2.3.2.1" xref="A4.1.p1.6.m6.2.3.1.cmml">(</mo><mn id="A4.1.p1.6.m6.1.1" xref="A4.1.p1.6.m6.1.1.cmml">1</mn><mo id="A4.1.p1.6.m6.2.3.2.2" xref="A4.1.p1.6.m6.2.3.1.cmml">,</mo><mn id="A4.1.p1.6.m6.2.2" xref="A4.1.p1.6.m6.2.2.cmml">0</mn><mo stretchy="false" id="A4.1.p1.6.m6.2.3.2.3" xref="A4.1.p1.6.m6.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.6.m6.2b"><interval closure="open" id="A4.1.p1.6.m6.2.3.1.cmml" xref="A4.1.p1.6.m6.2.3.2"><cn type="integer" id="A4.1.p1.6.m6.1.1.cmml" xref="A4.1.p1.6.m6.1.1">1</cn><cn type="integer" id="A4.1.p1.6.m6.2.2.cmml" xref="A4.1.p1.6.m6.2.2">0</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.6.m6.2c">(1,0)</annotation></semantics></math>, <math id="A4.1.p1.7.m7.2" class="ltx_Math" alttext="2\mu(a,b)=2" display="inline"><semantics id="A4.1.p1.7.m7.2a"><mrow id="A4.1.p1.7.m7.2.3" xref="A4.1.p1.7.m7.2.3.cmml"><mrow id="A4.1.p1.7.m7.2.3.2" xref="A4.1.p1.7.m7.2.3.2.cmml"><mn id="A4.1.p1.7.m7.2.3.2.2" xref="A4.1.p1.7.m7.2.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A4.1.p1.7.m7.2.3.2.1" xref="A4.1.p1.7.m7.2.3.2.1.cmml">​</mo><mi id="A4.1.p1.7.m7.2.3.2.3" xref="A4.1.p1.7.m7.2.3.2.3.cmml">μ</mi><mo lspace="0em" rspace="0em" id="A4.1.p1.7.m7.2.3.2.1a" xref="A4.1.p1.7.m7.2.3.2.1.cmml">​</mo><mrow id="A4.1.p1.7.m7.2.3.2.4.2" xref="A4.1.p1.7.m7.2.3.2.4.1.cmml"><mo stretchy="false" id="A4.1.p1.7.m7.2.3.2.4.2.1" xref="A4.1.p1.7.m7.2.3.2.4.1.cmml">(</mo><mi id="A4.1.p1.7.m7.1.1" xref="A4.1.p1.7.m7.1.1.cmml">a</mi><mo id="A4.1.p1.7.m7.2.3.2.4.2.2" xref="A4.1.p1.7.m7.2.3.2.4.1.cmml">,</mo><mi id="A4.1.p1.7.m7.2.2" xref="A4.1.p1.7.m7.2.2.cmml">b</mi><mo stretchy="false" id="A4.1.p1.7.m7.2.3.2.4.2.3" xref="A4.1.p1.7.m7.2.3.2.4.1.cmml">)</mo></mrow></mrow><mo id="A4.1.p1.7.m7.2.3.1" xref="A4.1.p1.7.m7.2.3.1.cmml">=</mo><mn id="A4.1.p1.7.m7.2.3.3" xref="A4.1.p1.7.m7.2.3.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.7.m7.2b"><apply id="A4.1.p1.7.m7.2.3.cmml" xref="A4.1.p1.7.m7.2.3"><eq id="A4.1.p1.7.m7.2.3.1.cmml" xref="A4.1.p1.7.m7.2.3.1"></eq><apply id="A4.1.p1.7.m7.2.3.2.cmml" xref="A4.1.p1.7.m7.2.3.2"><times id="A4.1.p1.7.m7.2.3.2.1.cmml" xref="A4.1.p1.7.m7.2.3.2.1"></times><cn type="integer" id="A4.1.p1.7.m7.2.3.2.2.cmml" xref="A4.1.p1.7.m7.2.3.2.2">2</cn><ci id="A4.1.p1.7.m7.2.3.2.3.cmml" xref="A4.1.p1.7.m7.2.3.2.3">𝜇</ci><interval closure="open" id="A4.1.p1.7.m7.2.3.2.4.1.cmml" xref="A4.1.p1.7.m7.2.3.2.4.2"><ci id="A4.1.p1.7.m7.1.1.cmml" xref="A4.1.p1.7.m7.1.1">𝑎</ci><ci id="A4.1.p1.7.m7.2.2.cmml" xref="A4.1.p1.7.m7.2.2">𝑏</ci></interval></apply><cn type="integer" id="A4.1.p1.7.m7.2.3.3.cmml" xref="A4.1.p1.7.m7.2.3.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.7.m7.2c">2\mu(a,b)=2</annotation></semantics></math>, while <math id="A4.1.p1.8.m8.2" class="ltx_Math" alttext="s(a,b)=1" display="inline"><semantics id="A4.1.p1.8.m8.2a"><mrow id="A4.1.p1.8.m8.2.3" xref="A4.1.p1.8.m8.2.3.cmml"><mrow id="A4.1.p1.8.m8.2.3.2" xref="A4.1.p1.8.m8.2.3.2.cmml"><mi id="A4.1.p1.8.m8.2.3.2.2" xref="A4.1.p1.8.m8.2.3.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A4.1.p1.8.m8.2.3.2.1" xref="A4.1.p1.8.m8.2.3.2.1.cmml">​</mo><mrow id="A4.1.p1.8.m8.2.3.2.3.2" xref="A4.1.p1.8.m8.2.3.2.3.1.cmml"><mo stretchy="false" id="A4.1.p1.8.m8.2.3.2.3.2.1" xref="A4.1.p1.8.m8.2.3.2.3.1.cmml">(</mo><mi id="A4.1.p1.8.m8.1.1" xref="A4.1.p1.8.m8.1.1.cmml">a</mi><mo id="A4.1.p1.8.m8.2.3.2.3.2.2" xref="A4.1.p1.8.m8.2.3.2.3.1.cmml">,</mo><mi id="A4.1.p1.8.m8.2.2" xref="A4.1.p1.8.m8.2.2.cmml">b</mi><mo stretchy="false" id="A4.1.p1.8.m8.2.3.2.3.2.3" xref="A4.1.p1.8.m8.2.3.2.3.1.cmml">)</mo></mrow></mrow><mo id="A4.1.p1.8.m8.2.3.1" xref="A4.1.p1.8.m8.2.3.1.cmml">=</mo><mn id="A4.1.p1.8.m8.2.3.3" xref="A4.1.p1.8.m8.2.3.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.1.p1.8.m8.2b"><apply id="A4.1.p1.8.m8.2.3.cmml" xref="A4.1.p1.8.m8.2.3"><eq id="A4.1.p1.8.m8.2.3.1.cmml" xref="A4.1.p1.8.m8.2.3.1"></eq><apply id="A4.1.p1.8.m8.2.3.2.cmml" xref="A4.1.p1.8.m8.2.3.2"><times id="A4.1.p1.8.m8.2.3.2.1.cmml" xref="A4.1.p1.8.m8.2.3.2.1"></times><ci id="A4.1.p1.8.m8.2.3.2.2.cmml" xref="A4.1.p1.8.m8.2.3.2.2">𝑠</ci><interval closure="open" id="A4.1.p1.8.m8.2.3.2.3.1.cmml" xref="A4.1.p1.8.m8.2.3.2.3.2"><ci id="A4.1.p1.8.m8.1.1.cmml" xref="A4.1.p1.8.m8.1.1">𝑎</ci><ci id="A4.1.p1.8.m8.2.2.cmml" xref="A4.1.p1.8.m8.2.2">𝑏</ci></interval></apply><cn type="integer" id="A4.1.p1.8.m8.2.3.3.cmml" xref="A4.1.p1.8.m8.2.3.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.1.p1.8.m8.2c">s(a,b)=1</annotation></semantics></math>.
∎</p>
</div>
</div>
<div id="A4.Thmlemma2" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A4.Thmlemma2.1.1.1" class="ltx_text ltx_font_bold">Lemma D.2</span></span><span id="A4.Thmlemma2.2.2" class="ltx_text ltx_font_bold"> </span>(Variance of error when averaging one pair)<span id="A4.Thmlemma2.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A4.Thmlemma2.p1" class="ltx_para">
<p id="A4.Thmlemma2.p1.3" class="ltx_p"><span id="A4.Thmlemma2.p1.3.3" class="ltx_text ltx_font_italic">Consider two scalars <math id="A4.Thmlemma2.p1.1.1.m1.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A4.Thmlemma2.p1.1.1.m1.1a"><mi id="A4.Thmlemma2.p1.1.1.m1.1.1" xref="A4.Thmlemma2.p1.1.1.m1.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A4.Thmlemma2.p1.1.1.m1.1b"><ci id="A4.Thmlemma2.p1.1.1.m1.1.1.cmml" xref="A4.Thmlemma2.p1.1.1.m1.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma2.p1.1.1.m1.1c">a</annotation></semantics></math> and <math id="A4.Thmlemma2.p1.2.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A4.Thmlemma2.p1.2.2.m2.1a"><mi id="A4.Thmlemma2.p1.2.2.m2.1.1" xref="A4.Thmlemma2.p1.2.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A4.Thmlemma2.p1.2.2.m2.1b"><ci id="A4.Thmlemma2.p1.2.2.m2.1.1.cmml" xref="A4.Thmlemma2.p1.2.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma2.p1.2.2.m2.1c">b</annotation></semantics></math>, <math id="A4.Thmlemma2.p1.3.3.m3.2" class="ltx_Math" alttext="a,b\stackrel{{\scriptstyle iid}}{{\sim}}" display="inline"><semantics id="A4.Thmlemma2.p1.3.3.m3.2a"><mrow id="A4.Thmlemma2.p1.3.3.m3.2.3" xref="A4.Thmlemma2.p1.3.3.m3.2.3.cmml"><mrow id="A4.Thmlemma2.p1.3.3.m3.2.3.2.2" xref="A4.Thmlemma2.p1.3.3.m3.2.3.2.1.cmml"><mi id="A4.Thmlemma2.p1.3.3.m3.1.1" xref="A4.Thmlemma2.p1.3.3.m3.1.1.cmml">a</mi><mo id="A4.Thmlemma2.p1.3.3.m3.2.3.2.2.1" xref="A4.Thmlemma2.p1.3.3.m3.2.3.2.1.cmml">,</mo><mi id="A4.Thmlemma2.p1.3.3.m3.2.2" xref="A4.Thmlemma2.p1.3.3.m3.2.2.cmml">b</mi></mrow><mover id="A4.Thmlemma2.p1.3.3.m3.2.3.1" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.cmml"><mo id="A4.Thmlemma2.p1.3.3.m3.2.3.1.2" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.2.cmml">∼</mo><mrow id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.cmml"><mi id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.2" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.1" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.1.cmml">​</mo><mi id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.3" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.1a" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.1.cmml">​</mo><mi id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.4" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.4.cmml">d</mi></mrow></mover><mi id="A4.Thmlemma2.p1.3.3.m3.2.3.3" xref="A4.Thmlemma2.p1.3.3.m3.2.3.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A4.Thmlemma2.p1.3.3.m3.2b"><apply id="A4.Thmlemma2.p1.3.3.m3.2.3.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3"><apply id="A4.Thmlemma2.p1.3.3.m3.2.3.1.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1"><csymbol cd="ambiguous" id="A4.Thmlemma2.p1.3.3.m3.2.3.1.1.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1">superscript</csymbol><csymbol cd="latexml" id="A4.Thmlemma2.p1.3.3.m3.2.3.1.2.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.2">similar-to</csymbol><apply id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3"><times id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.1.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.1"></times><ci id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.2.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.2">𝑖</ci><ci id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.3.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.3">𝑖</ci><ci id="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.4.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.1.3.4">𝑑</ci></apply></apply><list id="A4.Thmlemma2.p1.3.3.m3.2.3.2.1.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.2.2"><ci id="A4.Thmlemma2.p1.3.3.m3.1.1.cmml" xref="A4.Thmlemma2.p1.3.3.m3.1.1">𝑎</ci><ci id="A4.Thmlemma2.p1.3.3.m3.2.2.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.2">𝑏</ci></list><csymbol cd="latexml" id="A4.Thmlemma2.p1.3.3.m3.2.3.3.cmml" xref="A4.Thmlemma2.p1.3.3.m3.2.3.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma2.p1.3.3.m3.2c">a,b\stackrel{{\scriptstyle iid}}{{\sim}}</annotation></semantics></math> Bernoulli(.5). Then</span></p>
<table id="A4.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A4.Ex2.m1.6" class="ltx_Math" alttext="E[\varepsilon(a,b)^{2}]-E[\varepsilon(x,y)]^{2}=\frac{1}{4}" display="block"><semantics id="A4.Ex2.m1.6a"><mrow id="A4.Ex2.m1.6.6" xref="A4.Ex2.m1.6.6.cmml"><mrow id="A4.Ex2.m1.6.6.2" xref="A4.Ex2.m1.6.6.2.cmml"><mrow id="A4.Ex2.m1.5.5.1.1" xref="A4.Ex2.m1.5.5.1.1.cmml"><mi id="A4.Ex2.m1.5.5.1.1.3" xref="A4.Ex2.m1.5.5.1.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.Ex2.m1.5.5.1.1.2" xref="A4.Ex2.m1.5.5.1.1.2.cmml">​</mo><mrow id="A4.Ex2.m1.5.5.1.1.1.1" xref="A4.Ex2.m1.5.5.1.1.1.2.cmml"><mo stretchy="false" id="A4.Ex2.m1.5.5.1.1.1.1.2" xref="A4.Ex2.m1.5.5.1.1.1.2.1.cmml">[</mo><mrow id="A4.Ex2.m1.5.5.1.1.1.1.1" xref="A4.Ex2.m1.5.5.1.1.1.1.1.cmml"><mi id="A4.Ex2.m1.5.5.1.1.1.1.1.2" xref="A4.Ex2.m1.5.5.1.1.1.1.1.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="A4.Ex2.m1.5.5.1.1.1.1.1.1" xref="A4.Ex2.m1.5.5.1.1.1.1.1.1.cmml">​</mo><msup id="A4.Ex2.m1.5.5.1.1.1.1.1.3" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.cmml"><mrow id="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.2" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.1.cmml"><mo stretchy="false" id="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.2.1" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.1.cmml">(</mo><mi id="A4.Ex2.m1.1.1" xref="A4.Ex2.m1.1.1.cmml">a</mi><mo id="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.2.2" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.1.cmml">,</mo><mi id="A4.Ex2.m1.2.2" xref="A4.Ex2.m1.2.2.cmml">b</mi><mo stretchy="false" id="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.2.3" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.1.cmml">)</mo></mrow><mn id="A4.Ex2.m1.5.5.1.1.1.1.1.3.3" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="A4.Ex2.m1.5.5.1.1.1.1.3" xref="A4.Ex2.m1.5.5.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A4.Ex2.m1.6.6.2.3" xref="A4.Ex2.m1.6.6.2.3.cmml">−</mo><mrow id="A4.Ex2.m1.6.6.2.2" xref="A4.Ex2.m1.6.6.2.2.cmml"><mi id="A4.Ex2.m1.6.6.2.2.3" xref="A4.Ex2.m1.6.6.2.2.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.Ex2.m1.6.6.2.2.2" xref="A4.Ex2.m1.6.6.2.2.2.cmml">​</mo><msup id="A4.Ex2.m1.6.6.2.2.1" xref="A4.Ex2.m1.6.6.2.2.1.cmml"><mrow id="A4.Ex2.m1.6.6.2.2.1.1.1" xref="A4.Ex2.m1.6.6.2.2.1.1.2.cmml"><mo stretchy="false" id="A4.Ex2.m1.6.6.2.2.1.1.1.2" xref="A4.Ex2.m1.6.6.2.2.1.1.2.1.cmml">[</mo><mrow id="A4.Ex2.m1.6.6.2.2.1.1.1.1" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.cmml"><mi id="A4.Ex2.m1.6.6.2.2.1.1.1.1.2" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="A4.Ex2.m1.6.6.2.2.1.1.1.1.1" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.1.cmml">​</mo><mrow id="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.2" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.1.cmml"><mo stretchy="false" id="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.2.1" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.1.cmml">(</mo><mi id="A4.Ex2.m1.3.3" xref="A4.Ex2.m1.3.3.cmml">x</mi><mo id="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.2.2" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.1.cmml">,</mo><mi id="A4.Ex2.m1.4.4" xref="A4.Ex2.m1.4.4.cmml">y</mi><mo stretchy="false" id="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.2.3" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.Ex2.m1.6.6.2.2.1.1.1.3" xref="A4.Ex2.m1.6.6.2.2.1.1.2.1.cmml">]</mo></mrow><mn id="A4.Ex2.m1.6.6.2.2.1.3" xref="A4.Ex2.m1.6.6.2.2.1.3.cmml">2</mn></msup></mrow></mrow><mo id="A4.Ex2.m1.6.6.3" xref="A4.Ex2.m1.6.6.3.cmml">=</mo><mfrac id="A4.Ex2.m1.6.6.4" xref="A4.Ex2.m1.6.6.4.cmml"><mn id="A4.Ex2.m1.6.6.4.2" xref="A4.Ex2.m1.6.6.4.2.cmml">1</mn><mn id="A4.Ex2.m1.6.6.4.3" xref="A4.Ex2.m1.6.6.4.3.cmml">4</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A4.Ex2.m1.6b"><apply id="A4.Ex2.m1.6.6.cmml" xref="A4.Ex2.m1.6.6"><eq id="A4.Ex2.m1.6.6.3.cmml" xref="A4.Ex2.m1.6.6.3"></eq><apply id="A4.Ex2.m1.6.6.2.cmml" xref="A4.Ex2.m1.6.6.2"><minus id="A4.Ex2.m1.6.6.2.3.cmml" xref="A4.Ex2.m1.6.6.2.3"></minus><apply id="A4.Ex2.m1.5.5.1.1.cmml" xref="A4.Ex2.m1.5.5.1.1"><times id="A4.Ex2.m1.5.5.1.1.2.cmml" xref="A4.Ex2.m1.5.5.1.1.2"></times><ci id="A4.Ex2.m1.5.5.1.1.3.cmml" xref="A4.Ex2.m1.5.5.1.1.3">𝐸</ci><apply id="A4.Ex2.m1.5.5.1.1.1.2.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1"><csymbol cd="latexml" id="A4.Ex2.m1.5.5.1.1.1.2.1.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.2">delimited-[]</csymbol><apply id="A4.Ex2.m1.5.5.1.1.1.1.1.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1"><times id="A4.Ex2.m1.5.5.1.1.1.1.1.1.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1.1"></times><ci id="A4.Ex2.m1.5.5.1.1.1.1.1.2.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1.2">𝜀</ci><apply id="A4.Ex2.m1.5.5.1.1.1.1.1.3.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A4.Ex2.m1.5.5.1.1.1.1.1.3.1.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3">superscript</csymbol><interval closure="open" id="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.1.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.2.2"><ci id="A4.Ex2.m1.1.1.cmml" xref="A4.Ex2.m1.1.1">𝑎</ci><ci id="A4.Ex2.m1.2.2.cmml" xref="A4.Ex2.m1.2.2">𝑏</ci></interval><cn type="integer" id="A4.Ex2.m1.5.5.1.1.1.1.1.3.3.cmml" xref="A4.Ex2.m1.5.5.1.1.1.1.1.3.3">2</cn></apply></apply></apply></apply><apply id="A4.Ex2.m1.6.6.2.2.cmml" xref="A4.Ex2.m1.6.6.2.2"><times id="A4.Ex2.m1.6.6.2.2.2.cmml" xref="A4.Ex2.m1.6.6.2.2.2"></times><ci id="A4.Ex2.m1.6.6.2.2.3.cmml" xref="A4.Ex2.m1.6.6.2.2.3">𝐸</ci><apply id="A4.Ex2.m1.6.6.2.2.1.cmml" xref="A4.Ex2.m1.6.6.2.2.1"><csymbol cd="ambiguous" id="A4.Ex2.m1.6.6.2.2.1.2.cmml" xref="A4.Ex2.m1.6.6.2.2.1">superscript</csymbol><apply id="A4.Ex2.m1.6.6.2.2.1.1.2.cmml" xref="A4.Ex2.m1.6.6.2.2.1.1.1"><csymbol cd="latexml" id="A4.Ex2.m1.6.6.2.2.1.1.2.1.cmml" xref="A4.Ex2.m1.6.6.2.2.1.1.1.2">delimited-[]</csymbol><apply id="A4.Ex2.m1.6.6.2.2.1.1.1.1.cmml" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1"><times id="A4.Ex2.m1.6.6.2.2.1.1.1.1.1.cmml" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.1"></times><ci id="A4.Ex2.m1.6.6.2.2.1.1.1.1.2.cmml" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.2">𝜀</ci><interval closure="open" id="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.1.cmml" xref="A4.Ex2.m1.6.6.2.2.1.1.1.1.3.2"><ci id="A4.Ex2.m1.3.3.cmml" xref="A4.Ex2.m1.3.3">𝑥</ci><ci id="A4.Ex2.m1.4.4.cmml" xref="A4.Ex2.m1.4.4">𝑦</ci></interval></apply></apply><cn type="integer" id="A4.Ex2.m1.6.6.2.2.1.3.cmml" xref="A4.Ex2.m1.6.6.2.2.1.3">2</cn></apply></apply></apply><apply id="A4.Ex2.m1.6.6.4.cmml" xref="A4.Ex2.m1.6.6.4"><divide id="A4.Ex2.m1.6.6.4.1.cmml" xref="A4.Ex2.m1.6.6.4"></divide><cn type="integer" id="A4.Ex2.m1.6.6.4.2.cmml" xref="A4.Ex2.m1.6.6.4.2">1</cn><cn type="integer" id="A4.Ex2.m1.6.6.4.3.cmml" xref="A4.Ex2.m1.6.6.4.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Ex2.m1.6c">E[\varepsilon(a,b)^{2}]-E[\varepsilon(x,y)]^{2}=\frac{1}{4}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A4.2" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A4.2.p1" class="ltx_para">
<p id="A4.2.p1.7" class="ltx_p">Using Lemma <a href="#A4.Thmlemma1" title="Lemma D.1 (Bias when averaging one pair). ‣ Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a>, the above can be rewritten as:</p>
<table id="A4.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A4.Ex3.m1.3" class="ltx_Math" alttext="E[\varepsilon(a,b)^{2}]=\frac{1}{2}" display="block"><semantics id="A4.Ex3.m1.3a"><mrow id="A4.Ex3.m1.3.3" xref="A4.Ex3.m1.3.3.cmml"><mrow id="A4.Ex3.m1.3.3.1" xref="A4.Ex3.m1.3.3.1.cmml"><mi id="A4.Ex3.m1.3.3.1.3" xref="A4.Ex3.m1.3.3.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.Ex3.m1.3.3.1.2" xref="A4.Ex3.m1.3.3.1.2.cmml">​</mo><mrow id="A4.Ex3.m1.3.3.1.1.1" xref="A4.Ex3.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="A4.Ex3.m1.3.3.1.1.1.2" xref="A4.Ex3.m1.3.3.1.1.2.1.cmml">[</mo><mrow id="A4.Ex3.m1.3.3.1.1.1.1" xref="A4.Ex3.m1.3.3.1.1.1.1.cmml"><mi id="A4.Ex3.m1.3.3.1.1.1.1.2" xref="A4.Ex3.m1.3.3.1.1.1.1.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="A4.Ex3.m1.3.3.1.1.1.1.1" xref="A4.Ex3.m1.3.3.1.1.1.1.1.cmml">​</mo><msup id="A4.Ex3.m1.3.3.1.1.1.1.3" xref="A4.Ex3.m1.3.3.1.1.1.1.3.cmml"><mrow id="A4.Ex3.m1.3.3.1.1.1.1.3.2.2" xref="A4.Ex3.m1.3.3.1.1.1.1.3.2.1.cmml"><mo stretchy="false" id="A4.Ex3.m1.3.3.1.1.1.1.3.2.2.1" xref="A4.Ex3.m1.3.3.1.1.1.1.3.2.1.cmml">(</mo><mi id="A4.Ex3.m1.1.1" xref="A4.Ex3.m1.1.1.cmml">a</mi><mo id="A4.Ex3.m1.3.3.1.1.1.1.3.2.2.2" xref="A4.Ex3.m1.3.3.1.1.1.1.3.2.1.cmml">,</mo><mi id="A4.Ex3.m1.2.2" xref="A4.Ex3.m1.2.2.cmml">b</mi><mo stretchy="false" id="A4.Ex3.m1.3.3.1.1.1.1.3.2.2.3" xref="A4.Ex3.m1.3.3.1.1.1.1.3.2.1.cmml">)</mo></mrow><mn id="A4.Ex3.m1.3.3.1.1.1.1.3.3" xref="A4.Ex3.m1.3.3.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="A4.Ex3.m1.3.3.1.1.1.3" xref="A4.Ex3.m1.3.3.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A4.Ex3.m1.3.3.2" xref="A4.Ex3.m1.3.3.2.cmml">=</mo><mfrac id="A4.Ex3.m1.3.3.3" xref="A4.Ex3.m1.3.3.3.cmml"><mn id="A4.Ex3.m1.3.3.3.2" xref="A4.Ex3.m1.3.3.3.2.cmml">1</mn><mn id="A4.Ex3.m1.3.3.3.3" xref="A4.Ex3.m1.3.3.3.3.cmml">2</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A4.Ex3.m1.3b"><apply id="A4.Ex3.m1.3.3.cmml" xref="A4.Ex3.m1.3.3"><eq id="A4.Ex3.m1.3.3.2.cmml" xref="A4.Ex3.m1.3.3.2"></eq><apply id="A4.Ex3.m1.3.3.1.cmml" xref="A4.Ex3.m1.3.3.1"><times id="A4.Ex3.m1.3.3.1.2.cmml" xref="A4.Ex3.m1.3.3.1.2"></times><ci id="A4.Ex3.m1.3.3.1.3.cmml" xref="A4.Ex3.m1.3.3.1.3">𝐸</ci><apply id="A4.Ex3.m1.3.3.1.1.2.cmml" xref="A4.Ex3.m1.3.3.1.1.1"><csymbol cd="latexml" id="A4.Ex3.m1.3.3.1.1.2.1.cmml" xref="A4.Ex3.m1.3.3.1.1.1.2">delimited-[]</csymbol><apply id="A4.Ex3.m1.3.3.1.1.1.1.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1"><times id="A4.Ex3.m1.3.3.1.1.1.1.1.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1.1"></times><ci id="A4.Ex3.m1.3.3.1.1.1.1.2.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1.2">𝜀</ci><apply id="A4.Ex3.m1.3.3.1.1.1.1.3.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="A4.Ex3.m1.3.3.1.1.1.1.3.1.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1.3">superscript</csymbol><interval closure="open" id="A4.Ex3.m1.3.3.1.1.1.1.3.2.1.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1.3.2.2"><ci id="A4.Ex3.m1.1.1.cmml" xref="A4.Ex3.m1.1.1">𝑎</ci><ci id="A4.Ex3.m1.2.2.cmml" xref="A4.Ex3.m1.2.2">𝑏</ci></interval><cn type="integer" id="A4.Ex3.m1.3.3.1.1.1.1.3.3.cmml" xref="A4.Ex3.m1.3.3.1.1.1.1.3.3">2</cn></apply></apply></apply></apply><apply id="A4.Ex3.m1.3.3.3.cmml" xref="A4.Ex3.m1.3.3.3"><divide id="A4.Ex3.m1.3.3.3.1.cmml" xref="A4.Ex3.m1.3.3.3"></divide><cn type="integer" id="A4.Ex3.m1.3.3.3.2.cmml" xref="A4.Ex3.m1.3.3.3.2">1</cn><cn type="integer" id="A4.Ex3.m1.3.3.3.3.cmml" xref="A4.Ex3.m1.3.3.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Ex3.m1.3c">E[\varepsilon(a,b)^{2}]=\frac{1}{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A4.2.p1.6" class="ltx_p">The proof then follows by again considering the four equiprobable cases as in Lemma <a href="#A4.Thmlemma1" title="Lemma D.1 (Bias when averaging one pair). ‣ Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a>. In the cases <math id="A4.2.p1.1.m1.2" class="ltx_Math" alttext="(0,0)" display="inline"><semantics id="A4.2.p1.1.m1.2a"><mrow id="A4.2.p1.1.m1.2.3.2" xref="A4.2.p1.1.m1.2.3.1.cmml"><mo stretchy="false" id="A4.2.p1.1.m1.2.3.2.1" xref="A4.2.p1.1.m1.2.3.1.cmml">(</mo><mn id="A4.2.p1.1.m1.1.1" xref="A4.2.p1.1.m1.1.1.cmml">0</mn><mo id="A4.2.p1.1.m1.2.3.2.2" xref="A4.2.p1.1.m1.2.3.1.cmml">,</mo><mn id="A4.2.p1.1.m1.2.2" xref="A4.2.p1.1.m1.2.2.cmml">0</mn><mo stretchy="false" id="A4.2.p1.1.m1.2.3.2.3" xref="A4.2.p1.1.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.2.p1.1.m1.2b"><interval closure="open" id="A4.2.p1.1.m1.2.3.1.cmml" xref="A4.2.p1.1.m1.2.3.2"><cn type="integer" id="A4.2.p1.1.m1.1.1.cmml" xref="A4.2.p1.1.m1.1.1">0</cn><cn type="integer" id="A4.2.p1.1.m1.2.2.cmml" xref="A4.2.p1.1.m1.2.2">0</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.2.p1.1.m1.2c">(0,0)</annotation></semantics></math> and <math id="A4.2.p1.2.m2.2" class="ltx_Math" alttext="(1,1)" display="inline"><semantics id="A4.2.p1.2.m2.2a"><mrow id="A4.2.p1.2.m2.2.3.2" xref="A4.2.p1.2.m2.2.3.1.cmml"><mo stretchy="false" id="A4.2.p1.2.m2.2.3.2.1" xref="A4.2.p1.2.m2.2.3.1.cmml">(</mo><mn id="A4.2.p1.2.m2.1.1" xref="A4.2.p1.2.m2.1.1.cmml">1</mn><mo id="A4.2.p1.2.m2.2.3.2.2" xref="A4.2.p1.2.m2.2.3.1.cmml">,</mo><mn id="A4.2.p1.2.m2.2.2" xref="A4.2.p1.2.m2.2.2.cmml">1</mn><mo stretchy="false" id="A4.2.p1.2.m2.2.3.2.3" xref="A4.2.p1.2.m2.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.2.p1.2.m2.2b"><interval closure="open" id="A4.2.p1.2.m2.2.3.1.cmml" xref="A4.2.p1.2.m2.2.3.2"><cn type="integer" id="A4.2.p1.2.m2.1.1.cmml" xref="A4.2.p1.2.m2.1.1">1</cn><cn type="integer" id="A4.2.p1.2.m2.2.2.cmml" xref="A4.2.p1.2.m2.2.2">1</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.2.p1.2.m2.2c">(1,1)</annotation></semantics></math>, <math id="A4.2.p1.3.m3.2" class="ltx_Math" alttext="\varepsilon(a,b)^{2}=0" display="inline"><semantics id="A4.2.p1.3.m3.2a"><mrow id="A4.2.p1.3.m3.2.3" xref="A4.2.p1.3.m3.2.3.cmml"><mrow id="A4.2.p1.3.m3.2.3.2" xref="A4.2.p1.3.m3.2.3.2.cmml"><mi id="A4.2.p1.3.m3.2.3.2.2" xref="A4.2.p1.3.m3.2.3.2.2.cmml">ε</mi><mo lspace="0em" rspace="0em" id="A4.2.p1.3.m3.2.3.2.1" xref="A4.2.p1.3.m3.2.3.2.1.cmml">​</mo><msup id="A4.2.p1.3.m3.2.3.2.3" xref="A4.2.p1.3.m3.2.3.2.3.cmml"><mrow id="A4.2.p1.3.m3.2.3.2.3.2.2" xref="A4.2.p1.3.m3.2.3.2.3.2.1.cmml"><mo stretchy="false" id="A4.2.p1.3.m3.2.3.2.3.2.2.1" xref="A4.2.p1.3.m3.2.3.2.3.2.1.cmml">(</mo><mi id="A4.2.p1.3.m3.1.1" xref="A4.2.p1.3.m3.1.1.cmml">a</mi><mo id="A4.2.p1.3.m3.2.3.2.3.2.2.2" xref="A4.2.p1.3.m3.2.3.2.3.2.1.cmml">,</mo><mi id="A4.2.p1.3.m3.2.2" xref="A4.2.p1.3.m3.2.2.cmml">b</mi><mo stretchy="false" id="A4.2.p1.3.m3.2.3.2.3.2.2.3" xref="A4.2.p1.3.m3.2.3.2.3.2.1.cmml">)</mo></mrow><mn id="A4.2.p1.3.m3.2.3.2.3.3" xref="A4.2.p1.3.m3.2.3.2.3.3.cmml">2</mn></msup></mrow><mo id="A4.2.p1.3.m3.2.3.1" xref="A4.2.p1.3.m3.2.3.1.cmml">=</mo><mn id="A4.2.p1.3.m3.2.3.3" xref="A4.2.p1.3.m3.2.3.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.2.p1.3.m3.2b"><apply id="A4.2.p1.3.m3.2.3.cmml" xref="A4.2.p1.3.m3.2.3"><eq id="A4.2.p1.3.m3.2.3.1.cmml" xref="A4.2.p1.3.m3.2.3.1"></eq><apply id="A4.2.p1.3.m3.2.3.2.cmml" xref="A4.2.p1.3.m3.2.3.2"><times id="A4.2.p1.3.m3.2.3.2.1.cmml" xref="A4.2.p1.3.m3.2.3.2.1"></times><ci id="A4.2.p1.3.m3.2.3.2.2.cmml" xref="A4.2.p1.3.m3.2.3.2.2">𝜀</ci><apply id="A4.2.p1.3.m3.2.3.2.3.cmml" xref="A4.2.p1.3.m3.2.3.2.3"><csymbol cd="ambiguous" id="A4.2.p1.3.m3.2.3.2.3.1.cmml" xref="A4.2.p1.3.m3.2.3.2.3">superscript</csymbol><interval closure="open" id="A4.2.p1.3.m3.2.3.2.3.2.1.cmml" xref="A4.2.p1.3.m3.2.3.2.3.2.2"><ci id="A4.2.p1.3.m3.1.1.cmml" xref="A4.2.p1.3.m3.1.1">𝑎</ci><ci id="A4.2.p1.3.m3.2.2.cmml" xref="A4.2.p1.3.m3.2.2">𝑏</ci></interval><cn type="integer" id="A4.2.p1.3.m3.2.3.2.3.3.cmml" xref="A4.2.p1.3.m3.2.3.2.3.3">2</cn></apply></apply><cn type="integer" id="A4.2.p1.3.m3.2.3.3.cmml" xref="A4.2.p1.3.m3.2.3.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.2.p1.3.m3.2c">\varepsilon(a,b)^{2}=0</annotation></semantics></math>. In the cases <math id="A4.2.p1.4.m4.2" class="ltx_Math" alttext="(0,1)" display="inline"><semantics id="A4.2.p1.4.m4.2a"><mrow id="A4.2.p1.4.m4.2.3.2" xref="A4.2.p1.4.m4.2.3.1.cmml"><mo stretchy="false" id="A4.2.p1.4.m4.2.3.2.1" xref="A4.2.p1.4.m4.2.3.1.cmml">(</mo><mn id="A4.2.p1.4.m4.1.1" xref="A4.2.p1.4.m4.1.1.cmml">0</mn><mo id="A4.2.p1.4.m4.2.3.2.2" xref="A4.2.p1.4.m4.2.3.1.cmml">,</mo><mn id="A4.2.p1.4.m4.2.2" xref="A4.2.p1.4.m4.2.2.cmml">1</mn><mo stretchy="false" id="A4.2.p1.4.m4.2.3.2.3" xref="A4.2.p1.4.m4.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.2.p1.4.m4.2b"><interval closure="open" id="A4.2.p1.4.m4.2.3.1.cmml" xref="A4.2.p1.4.m4.2.3.2"><cn type="integer" id="A4.2.p1.4.m4.1.1.cmml" xref="A4.2.p1.4.m4.1.1">0</cn><cn type="integer" id="A4.2.p1.4.m4.2.2.cmml" xref="A4.2.p1.4.m4.2.2">1</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.2.p1.4.m4.2c">(0,1)</annotation></semantics></math> and <math id="A4.2.p1.5.m5.2" class="ltx_Math" alttext="(1,0)" display="inline"><semantics id="A4.2.p1.5.m5.2a"><mrow id="A4.2.p1.5.m5.2.3.2" xref="A4.2.p1.5.m5.2.3.1.cmml"><mo stretchy="false" id="A4.2.p1.5.m5.2.3.2.1" xref="A4.2.p1.5.m5.2.3.1.cmml">(</mo><mn id="A4.2.p1.5.m5.1.1" xref="A4.2.p1.5.m5.1.1.cmml">1</mn><mo id="A4.2.p1.5.m5.2.3.2.2" xref="A4.2.p1.5.m5.2.3.1.cmml">,</mo><mn id="A4.2.p1.5.m5.2.2" xref="A4.2.p1.5.m5.2.2.cmml">0</mn><mo stretchy="false" id="A4.2.p1.5.m5.2.3.2.3" xref="A4.2.p1.5.m5.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.2.p1.5.m5.2b"><interval closure="open" id="A4.2.p1.5.m5.2.3.1.cmml" xref="A4.2.p1.5.m5.2.3.2"><cn type="integer" id="A4.2.p1.5.m5.1.1.cmml" xref="A4.2.p1.5.m5.1.1">1</cn><cn type="integer" id="A4.2.p1.5.m5.2.2.cmml" xref="A4.2.p1.5.m5.2.2">0</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.2.p1.5.m5.2c">(1,0)</annotation></semantics></math>, <math id="A4.2.p1.6.m6.6" class="ltx_Math" alttext="(2\hat{s}(a,b)-s(a,b))^{2}=(2-1)^{2}=1" display="inline"><semantics id="A4.2.p1.6.m6.6a"><mrow id="A4.2.p1.6.m6.6.6" xref="A4.2.p1.6.m6.6.6.cmml"><msup id="A4.2.p1.6.m6.5.5.1" xref="A4.2.p1.6.m6.5.5.1.cmml"><mrow id="A4.2.p1.6.m6.5.5.1.1.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.cmml"><mo stretchy="false" id="A4.2.p1.6.m6.5.5.1.1.1.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.cmml">(</mo><mrow id="A4.2.p1.6.m6.5.5.1.1.1.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.cmml"><mrow id="A4.2.p1.6.m6.5.5.1.1.1.1.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.cmml"><mn id="A4.2.p1.6.m6.5.5.1.1.1.1.2.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.1.cmml">​</mo><mover accent="true" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.3" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.cmml"><mi id="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.2.cmml">s</mi><mo id="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.1a" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.1.cmml">​</mo><mrow id="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.1.cmml"><mo stretchy="false" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.2.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.1.cmml">(</mo><mi id="A4.2.p1.6.m6.1.1" xref="A4.2.p1.6.m6.1.1.cmml">a</mi><mo id="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.2.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.1.cmml">,</mo><mi id="A4.2.p1.6.m6.2.2" xref="A4.2.p1.6.m6.2.2.cmml">b</mi><mo stretchy="false" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.2.3" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.1.cmml">)</mo></mrow></mrow><mo id="A4.2.p1.6.m6.5.5.1.1.1.1.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.1.cmml">−</mo><mrow id="A4.2.p1.6.m6.5.5.1.1.1.1.3" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.cmml"><mi id="A4.2.p1.6.m6.5.5.1.1.1.1.3.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A4.2.p1.6.m6.5.5.1.1.1.1.3.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.1.cmml">​</mo><mrow id="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.1.cmml"><mo stretchy="false" id="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.2.1" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.1.cmml">(</mo><mi id="A4.2.p1.6.m6.3.3" xref="A4.2.p1.6.m6.3.3.cmml">a</mi><mo id="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.2.2" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.1.cmml">,</mo><mi id="A4.2.p1.6.m6.4.4" xref="A4.2.p1.6.m6.4.4.cmml">b</mi><mo stretchy="false" id="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.2.3" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A4.2.p1.6.m6.5.5.1.1.1.3" xref="A4.2.p1.6.m6.5.5.1.1.1.1.cmml">)</mo></mrow><mn id="A4.2.p1.6.m6.5.5.1.3" xref="A4.2.p1.6.m6.5.5.1.3.cmml">2</mn></msup><mo id="A4.2.p1.6.m6.6.6.4" xref="A4.2.p1.6.m6.6.6.4.cmml">=</mo><msup id="A4.2.p1.6.m6.6.6.2" xref="A4.2.p1.6.m6.6.6.2.cmml"><mrow id="A4.2.p1.6.m6.6.6.2.1.1" xref="A4.2.p1.6.m6.6.6.2.1.1.1.cmml"><mo stretchy="false" id="A4.2.p1.6.m6.6.6.2.1.1.2" xref="A4.2.p1.6.m6.6.6.2.1.1.1.cmml">(</mo><mrow id="A4.2.p1.6.m6.6.6.2.1.1.1" xref="A4.2.p1.6.m6.6.6.2.1.1.1.cmml"><mn id="A4.2.p1.6.m6.6.6.2.1.1.1.2" xref="A4.2.p1.6.m6.6.6.2.1.1.1.2.cmml">2</mn><mo id="A4.2.p1.6.m6.6.6.2.1.1.1.1" xref="A4.2.p1.6.m6.6.6.2.1.1.1.1.cmml">−</mo><mn id="A4.2.p1.6.m6.6.6.2.1.1.1.3" xref="A4.2.p1.6.m6.6.6.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A4.2.p1.6.m6.6.6.2.1.1.3" xref="A4.2.p1.6.m6.6.6.2.1.1.1.cmml">)</mo></mrow><mn id="A4.2.p1.6.m6.6.6.2.3" xref="A4.2.p1.6.m6.6.6.2.3.cmml">2</mn></msup><mo id="A4.2.p1.6.m6.6.6.5" xref="A4.2.p1.6.m6.6.6.5.cmml">=</mo><mn id="A4.2.p1.6.m6.6.6.6" xref="A4.2.p1.6.m6.6.6.6.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.2.p1.6.m6.6b"><apply id="A4.2.p1.6.m6.6.6.cmml" xref="A4.2.p1.6.m6.6.6"><and id="A4.2.p1.6.m6.6.6a.cmml" xref="A4.2.p1.6.m6.6.6"></and><apply id="A4.2.p1.6.m6.6.6b.cmml" xref="A4.2.p1.6.m6.6.6"><eq id="A4.2.p1.6.m6.6.6.4.cmml" xref="A4.2.p1.6.m6.6.6.4"></eq><apply id="A4.2.p1.6.m6.5.5.1.cmml" xref="A4.2.p1.6.m6.5.5.1"><csymbol cd="ambiguous" id="A4.2.p1.6.m6.5.5.1.2.cmml" xref="A4.2.p1.6.m6.5.5.1">superscript</csymbol><apply id="A4.2.p1.6.m6.5.5.1.1.1.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1"><minus id="A4.2.p1.6.m6.5.5.1.1.1.1.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.1"></minus><apply id="A4.2.p1.6.m6.5.5.1.1.1.1.2.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2"><times id="A4.2.p1.6.m6.5.5.1.1.1.1.2.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.1"></times><cn type="integer" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.2.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.2">2</cn><apply id="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.3"><ci id="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.1">^</ci><ci id="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.2.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.3.2">𝑠</ci></apply><interval closure="open" id="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.2.4.2"><ci id="A4.2.p1.6.m6.1.1.cmml" xref="A4.2.p1.6.m6.1.1">𝑎</ci><ci id="A4.2.p1.6.m6.2.2.cmml" xref="A4.2.p1.6.m6.2.2">𝑏</ci></interval></apply><apply id="A4.2.p1.6.m6.5.5.1.1.1.1.3.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3"><times id="A4.2.p1.6.m6.5.5.1.1.1.1.3.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.1"></times><ci id="A4.2.p1.6.m6.5.5.1.1.1.1.3.2.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.2">𝑠</ci><interval closure="open" id="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.1.cmml" xref="A4.2.p1.6.m6.5.5.1.1.1.1.3.3.2"><ci id="A4.2.p1.6.m6.3.3.cmml" xref="A4.2.p1.6.m6.3.3">𝑎</ci><ci id="A4.2.p1.6.m6.4.4.cmml" xref="A4.2.p1.6.m6.4.4">𝑏</ci></interval></apply></apply><cn type="integer" id="A4.2.p1.6.m6.5.5.1.3.cmml" xref="A4.2.p1.6.m6.5.5.1.3">2</cn></apply><apply id="A4.2.p1.6.m6.6.6.2.cmml" xref="A4.2.p1.6.m6.6.6.2"><csymbol cd="ambiguous" id="A4.2.p1.6.m6.6.6.2.2.cmml" xref="A4.2.p1.6.m6.6.6.2">superscript</csymbol><apply id="A4.2.p1.6.m6.6.6.2.1.1.1.cmml" xref="A4.2.p1.6.m6.6.6.2.1.1"><minus id="A4.2.p1.6.m6.6.6.2.1.1.1.1.cmml" xref="A4.2.p1.6.m6.6.6.2.1.1.1.1"></minus><cn type="integer" id="A4.2.p1.6.m6.6.6.2.1.1.1.2.cmml" xref="A4.2.p1.6.m6.6.6.2.1.1.1.2">2</cn><cn type="integer" id="A4.2.p1.6.m6.6.6.2.1.1.1.3.cmml" xref="A4.2.p1.6.m6.6.6.2.1.1.1.3">1</cn></apply><cn type="integer" id="A4.2.p1.6.m6.6.6.2.3.cmml" xref="A4.2.p1.6.m6.6.6.2.3">2</cn></apply></apply><apply id="A4.2.p1.6.m6.6.6c.cmml" xref="A4.2.p1.6.m6.6.6"><eq id="A4.2.p1.6.m6.6.6.5.cmml" xref="A4.2.p1.6.m6.6.6.5"></eq><share href="#A4.2.p1.6.m6.6.6.2.cmml" id="A4.2.p1.6.m6.6.6d.cmml" xref="A4.2.p1.6.m6.6.6"></share><cn type="integer" id="A4.2.p1.6.m6.6.6.6.cmml" xref="A4.2.p1.6.m6.6.6.6">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.2.p1.6.m6.6c">(2\hat{s}(a,b)-s(a,b))^{2}=(2-1)^{2}=1</annotation></semantics></math>.
∎</p>
</div>
</div>
<div id="A4.Thmlemma3" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A4.Thmlemma3.1.1.1" class="ltx_text ltx_font_bold">Lemma D.3</span></span><span id="A4.Thmlemma3.2.2" class="ltx_text ltx_font_bold"> </span>(Bias of AISE within a subspace)<span id="A4.Thmlemma3.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A4.Thmlemma3.p1" class="ltx_para">
<p id="A4.Thmlemma3.p1.2" class="ltx_p"><span id="A4.Thmlemma3.p1.2.2" class="ltx_text ltx_font_italic">Suppose that the scalar elements <math id="A4.Thmlemma3.p1.1.1.m1.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="A4.Thmlemma3.p1.1.1.m1.1a"><msub id="A4.Thmlemma3.p1.1.1.m1.1.1" xref="A4.Thmlemma3.p1.1.1.m1.1.1.cmml"><mi id="A4.Thmlemma3.p1.1.1.m1.1.1.2" xref="A4.Thmlemma3.p1.1.1.m1.1.1.2.cmml">x</mi><mi id="A4.Thmlemma3.p1.1.1.m1.1.1.3" xref="A4.Thmlemma3.p1.1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A4.Thmlemma3.p1.1.1.m1.1b"><apply id="A4.Thmlemma3.p1.1.1.m1.1.1.cmml" xref="A4.Thmlemma3.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A4.Thmlemma3.p1.1.1.m1.1.1.1.cmml" xref="A4.Thmlemma3.p1.1.1.m1.1.1">subscript</csymbol><ci id="A4.Thmlemma3.p1.1.1.m1.1.1.2.cmml" xref="A4.Thmlemma3.p1.1.1.m1.1.1.2">𝑥</ci><ci id="A4.Thmlemma3.p1.1.1.m1.1.1.3.cmml" xref="A4.Thmlemma3.p1.1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma3.p1.1.1.m1.1c">x_{i}</annotation></semantics></math> of <math id="A4.Thmlemma3.p1.2.2.m2.1" class="ltx_Math" alttext="\bm{x}" display="inline"><semantics id="A4.Thmlemma3.p1.2.2.m2.1a"><mi id="A4.Thmlemma3.p1.2.2.m2.1.1" xref="A4.Thmlemma3.p1.2.2.m2.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="A4.Thmlemma3.p1.2.2.m2.1b"><ci id="A4.Thmlemma3.p1.2.2.m2.1.1.cmml" xref="A4.Thmlemma3.p1.2.2.m2.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmlemma3.p1.2.2.m2.1c">\bm{x}</annotation></semantics></math> are drawn from independent Bernoulli(.5) distributions. Then</span></p>
<table id="A6.EGx15" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A4.E21"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E21.m1.3" class="ltx_Math" alttext="\displaystyle E[s_{U}(\bm{x})-\hat{s}_{U}(\bm{x})]" display="inline"><semantics id="A4.E21.m1.3a"><mrow id="A4.E21.m1.3.3" xref="A4.E21.m1.3.3.cmml"><mi id="A4.E21.m1.3.3.3" xref="A4.E21.m1.3.3.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.E21.m1.3.3.2" xref="A4.E21.m1.3.3.2.cmml">​</mo><mrow id="A4.E21.m1.3.3.1.1" xref="A4.E21.m1.3.3.1.2.cmml"><mo stretchy="false" id="A4.E21.m1.3.3.1.1.2" xref="A4.E21.m1.3.3.1.2.1.cmml">[</mo><mrow id="A4.E21.m1.3.3.1.1.1" xref="A4.E21.m1.3.3.1.1.1.cmml"><mrow id="A4.E21.m1.3.3.1.1.1.2" xref="A4.E21.m1.3.3.1.1.1.2.cmml"><msub id="A4.E21.m1.3.3.1.1.1.2.2" xref="A4.E21.m1.3.3.1.1.1.2.2.cmml"><mi id="A4.E21.m1.3.3.1.1.1.2.2.2" xref="A4.E21.m1.3.3.1.1.1.2.2.2.cmml">s</mi><mi id="A4.E21.m1.3.3.1.1.1.2.2.3" xref="A4.E21.m1.3.3.1.1.1.2.2.3.cmml">U</mi></msub><mo lspace="0em" rspace="0em" id="A4.E21.m1.3.3.1.1.1.2.1" xref="A4.E21.m1.3.3.1.1.1.2.1.cmml">​</mo><mrow id="A4.E21.m1.3.3.1.1.1.2.3.2" xref="A4.E21.m1.3.3.1.1.1.2.cmml"><mo stretchy="false" id="A4.E21.m1.3.3.1.1.1.2.3.2.1" xref="A4.E21.m1.3.3.1.1.1.2.cmml">(</mo><mi id="A4.E21.m1.1.1" xref="A4.E21.m1.1.1.cmml">𝒙</mi><mo stretchy="false" id="A4.E21.m1.3.3.1.1.1.2.3.2.2" xref="A4.E21.m1.3.3.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A4.E21.m1.3.3.1.1.1.1" xref="A4.E21.m1.3.3.1.1.1.1.cmml">−</mo><mrow id="A4.E21.m1.3.3.1.1.1.3" xref="A4.E21.m1.3.3.1.1.1.3.cmml"><msub id="A4.E21.m1.3.3.1.1.1.3.2" xref="A4.E21.m1.3.3.1.1.1.3.2.cmml"><mover accent="true" id="A4.E21.m1.3.3.1.1.1.3.2.2" xref="A4.E21.m1.3.3.1.1.1.3.2.2.cmml"><mi id="A4.E21.m1.3.3.1.1.1.3.2.2.2" xref="A4.E21.m1.3.3.1.1.1.3.2.2.2.cmml">s</mi><mo id="A4.E21.m1.3.3.1.1.1.3.2.2.1" xref="A4.E21.m1.3.3.1.1.1.3.2.2.1.cmml">^</mo></mover><mi id="A4.E21.m1.3.3.1.1.1.3.2.3" xref="A4.E21.m1.3.3.1.1.1.3.2.3.cmml">U</mi></msub><mo lspace="0em" rspace="0em" id="A4.E21.m1.3.3.1.1.1.3.1" xref="A4.E21.m1.3.3.1.1.1.3.1.cmml">​</mo><mrow id="A4.E21.m1.3.3.1.1.1.3.3.2" xref="A4.E21.m1.3.3.1.1.1.3.cmml"><mo stretchy="false" id="A4.E21.m1.3.3.1.1.1.3.3.2.1" xref="A4.E21.m1.3.3.1.1.1.3.cmml">(</mo><mi id="A4.E21.m1.2.2" xref="A4.E21.m1.2.2.cmml">𝒙</mi><mo stretchy="false" id="A4.E21.m1.3.3.1.1.1.3.3.2.2" xref="A4.E21.m1.3.3.1.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A4.E21.m1.3.3.1.1.3" xref="A4.E21.m1.3.3.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E21.m1.3b"><apply id="A4.E21.m1.3.3.cmml" xref="A4.E21.m1.3.3"><times id="A4.E21.m1.3.3.2.cmml" xref="A4.E21.m1.3.3.2"></times><ci id="A4.E21.m1.3.3.3.cmml" xref="A4.E21.m1.3.3.3">𝐸</ci><apply id="A4.E21.m1.3.3.1.2.cmml" xref="A4.E21.m1.3.3.1.1"><csymbol cd="latexml" id="A4.E21.m1.3.3.1.2.1.cmml" xref="A4.E21.m1.3.3.1.1.2">delimited-[]</csymbol><apply id="A4.E21.m1.3.3.1.1.1.cmml" xref="A4.E21.m1.3.3.1.1.1"><minus id="A4.E21.m1.3.3.1.1.1.1.cmml" xref="A4.E21.m1.3.3.1.1.1.1"></minus><apply id="A4.E21.m1.3.3.1.1.1.2.cmml" xref="A4.E21.m1.3.3.1.1.1.2"><times id="A4.E21.m1.3.3.1.1.1.2.1.cmml" xref="A4.E21.m1.3.3.1.1.1.2.1"></times><apply id="A4.E21.m1.3.3.1.1.1.2.2.cmml" xref="A4.E21.m1.3.3.1.1.1.2.2"><csymbol cd="ambiguous" id="A4.E21.m1.3.3.1.1.1.2.2.1.cmml" xref="A4.E21.m1.3.3.1.1.1.2.2">subscript</csymbol><ci id="A4.E21.m1.3.3.1.1.1.2.2.2.cmml" xref="A4.E21.m1.3.3.1.1.1.2.2.2">𝑠</ci><ci id="A4.E21.m1.3.3.1.1.1.2.2.3.cmml" xref="A4.E21.m1.3.3.1.1.1.2.2.3">𝑈</ci></apply><ci id="A4.E21.m1.1.1.cmml" xref="A4.E21.m1.1.1">𝒙</ci></apply><apply id="A4.E21.m1.3.3.1.1.1.3.cmml" xref="A4.E21.m1.3.3.1.1.1.3"><times id="A4.E21.m1.3.3.1.1.1.3.1.cmml" xref="A4.E21.m1.3.3.1.1.1.3.1"></times><apply id="A4.E21.m1.3.3.1.1.1.3.2.cmml" xref="A4.E21.m1.3.3.1.1.1.3.2"><csymbol cd="ambiguous" id="A4.E21.m1.3.3.1.1.1.3.2.1.cmml" xref="A4.E21.m1.3.3.1.1.1.3.2">subscript</csymbol><apply id="A4.E21.m1.3.3.1.1.1.3.2.2.cmml" xref="A4.E21.m1.3.3.1.1.1.3.2.2"><ci id="A4.E21.m1.3.3.1.1.1.3.2.2.1.cmml" xref="A4.E21.m1.3.3.1.1.1.3.2.2.1">^</ci><ci id="A4.E21.m1.3.3.1.1.1.3.2.2.2.cmml" xref="A4.E21.m1.3.3.1.1.1.3.2.2.2">𝑠</ci></apply><ci id="A4.E21.m1.3.3.1.1.1.3.2.3.cmml" xref="A4.E21.m1.3.3.1.1.1.3.2.3">𝑈</ci></apply><ci id="A4.E21.m1.2.2.cmml" xref="A4.E21.m1.2.2">𝒙</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E21.m1.3c">\displaystyle E[s_{U}(\bm{x})-\hat{s}_{U}(\bm{x})]</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A4.E21.m2.2" class="ltx_Math" alttext="\displaystyle=U\log_{2}(U)/4" display="inline"><semantics id="A4.E21.m2.2a"><mrow id="A4.E21.m2.2.2" xref="A4.E21.m2.2.2.cmml"><mi id="A4.E21.m2.2.2.3" xref="A4.E21.m2.2.2.3.cmml"></mi><mo id="A4.E21.m2.2.2.2" xref="A4.E21.m2.2.2.2.cmml">=</mo><mrow id="A4.E21.m2.2.2.1" xref="A4.E21.m2.2.2.1.cmml"><mrow id="A4.E21.m2.2.2.1.1" xref="A4.E21.m2.2.2.1.1.cmml"><mi id="A4.E21.m2.2.2.1.1.3" xref="A4.E21.m2.2.2.1.1.3.cmml">U</mi><mo lspace="0.167em" rspace="0em" id="A4.E21.m2.2.2.1.1.2" xref="A4.E21.m2.2.2.1.1.2.cmml">​</mo><mrow id="A4.E21.m2.2.2.1.1.1.1" xref="A4.E21.m2.2.2.1.1.1.2.cmml"><msub id="A4.E21.m2.2.2.1.1.1.1.1" xref="A4.E21.m2.2.2.1.1.1.1.1.cmml"><mi id="A4.E21.m2.2.2.1.1.1.1.1.2" xref="A4.E21.m2.2.2.1.1.1.1.1.2.cmml">log</mi><mn id="A4.E21.m2.2.2.1.1.1.1.1.3" xref="A4.E21.m2.2.2.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A4.E21.m2.2.2.1.1.1.1a" xref="A4.E21.m2.2.2.1.1.1.2.cmml">⁡</mo><mrow id="A4.E21.m2.2.2.1.1.1.1.2" xref="A4.E21.m2.2.2.1.1.1.2.cmml"><mo stretchy="false" id="A4.E21.m2.2.2.1.1.1.1.2.1" xref="A4.E21.m2.2.2.1.1.1.2.cmml">(</mo><mi id="A4.E21.m2.1.1" xref="A4.E21.m2.1.1.cmml">U</mi><mo stretchy="false" id="A4.E21.m2.2.2.1.1.1.1.2.2" xref="A4.E21.m2.2.2.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="A4.E21.m2.2.2.1.2" xref="A4.E21.m2.2.2.1.2.cmml">/</mo><mn id="A4.E21.m2.2.2.1.3" xref="A4.E21.m2.2.2.1.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E21.m2.2b"><apply id="A4.E21.m2.2.2.cmml" xref="A4.E21.m2.2.2"><eq id="A4.E21.m2.2.2.2.cmml" xref="A4.E21.m2.2.2.2"></eq><csymbol cd="latexml" id="A4.E21.m2.2.2.3.cmml" xref="A4.E21.m2.2.2.3">absent</csymbol><apply id="A4.E21.m2.2.2.1.cmml" xref="A4.E21.m2.2.2.1"><divide id="A4.E21.m2.2.2.1.2.cmml" xref="A4.E21.m2.2.2.1.2"></divide><apply id="A4.E21.m2.2.2.1.1.cmml" xref="A4.E21.m2.2.2.1.1"><times id="A4.E21.m2.2.2.1.1.2.cmml" xref="A4.E21.m2.2.2.1.1.2"></times><ci id="A4.E21.m2.2.2.1.1.3.cmml" xref="A4.E21.m2.2.2.1.1.3">𝑈</ci><apply id="A4.E21.m2.2.2.1.1.1.2.cmml" xref="A4.E21.m2.2.2.1.1.1.1"><apply id="A4.E21.m2.2.2.1.1.1.1.1.cmml" xref="A4.E21.m2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A4.E21.m2.2.2.1.1.1.1.1.1.cmml" xref="A4.E21.m2.2.2.1.1.1.1.1">subscript</csymbol><log id="A4.E21.m2.2.2.1.1.1.1.1.2.cmml" xref="A4.E21.m2.2.2.1.1.1.1.1.2"></log><cn type="integer" id="A4.E21.m2.2.2.1.1.1.1.1.3.cmml" xref="A4.E21.m2.2.2.1.1.1.1.1.3">2</cn></apply><ci id="A4.E21.m2.1.1.cmml" xref="A4.E21.m2.1.1">𝑈</ci></apply></apply><cn type="integer" id="A4.E21.m2.2.2.1.3.cmml" xref="A4.E21.m2.2.2.1.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E21.m2.2c">\displaystyle=U\log_{2}(U)/4</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(21)</span></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A4.4" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A4.3.p1" class="ltx_para">
<p id="A4.3.p1.7" class="ltx_p">Observe that the computation graph can be cast as a balanced binary tree with <math id="A4.3.p1.1.m1.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A4.3.p1.1.m1.1a"><mi id="A4.3.p1.1.m1.1.1" xref="A4.3.p1.1.m1.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A4.3.p1.1.m1.1b"><ci id="A4.3.p1.1.m1.1.1.cmml" xref="A4.3.p1.1.m1.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.1.m1.1c">U</annotation></semantics></math> leaves and each parent equal to the integer average of its children. Consider the bias introduced at each level <math id="A4.3.p1.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="A4.3.p1.2.m2.1a"><mi id="A4.3.p1.2.m2.1.1" xref="A4.3.p1.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="A4.3.p1.2.m2.1b"><ci id="A4.3.p1.2.m2.1.1.cmml" xref="A4.3.p1.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.2.m2.1c">t</annotation></semantics></math> of the tree, where <math id="A4.3.p1.3.m3.1" class="ltx_Math" alttext="t=0" display="inline"><semantics id="A4.3.p1.3.m3.1a"><mrow id="A4.3.p1.3.m3.1.1" xref="A4.3.p1.3.m3.1.1.cmml"><mi id="A4.3.p1.3.m3.1.1.2" xref="A4.3.p1.3.m3.1.1.2.cmml">t</mi><mo id="A4.3.p1.3.m3.1.1.1" xref="A4.3.p1.3.m3.1.1.1.cmml">=</mo><mn id="A4.3.p1.3.m3.1.1.3" xref="A4.3.p1.3.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.3.m3.1b"><apply id="A4.3.p1.3.m3.1.1.cmml" xref="A4.3.p1.3.m3.1.1"><eq id="A4.3.p1.3.m3.1.1.1.cmml" xref="A4.3.p1.3.m3.1.1.1"></eq><ci id="A4.3.p1.3.m3.1.1.2.cmml" xref="A4.3.p1.3.m3.1.1.2">𝑡</ci><cn type="integer" id="A4.3.p1.3.m3.1.1.3.cmml" xref="A4.3.p1.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.3.m3.1c">t=0</annotation></semantics></math> corresponds to the leaves and <math id="A4.3.p1.4.m4.2" class="ltx_Math" alttext="t=\log_{2}(U)" display="inline"><semantics id="A4.3.p1.4.m4.2a"><mrow id="A4.3.p1.4.m4.2.2" xref="A4.3.p1.4.m4.2.2.cmml"><mi id="A4.3.p1.4.m4.2.2.3" xref="A4.3.p1.4.m4.2.2.3.cmml">t</mi><mo id="A4.3.p1.4.m4.2.2.2" xref="A4.3.p1.4.m4.2.2.2.cmml">=</mo><mrow id="A4.3.p1.4.m4.2.2.1.1" xref="A4.3.p1.4.m4.2.2.1.2.cmml"><msub id="A4.3.p1.4.m4.2.2.1.1.1" xref="A4.3.p1.4.m4.2.2.1.1.1.cmml"><mi id="A4.3.p1.4.m4.2.2.1.1.1.2" xref="A4.3.p1.4.m4.2.2.1.1.1.2.cmml">log</mi><mn id="A4.3.p1.4.m4.2.2.1.1.1.3" xref="A4.3.p1.4.m4.2.2.1.1.1.3.cmml">2</mn></msub><mo id="A4.3.p1.4.m4.2.2.1.1a" xref="A4.3.p1.4.m4.2.2.1.2.cmml">⁡</mo><mrow id="A4.3.p1.4.m4.2.2.1.1.2" xref="A4.3.p1.4.m4.2.2.1.2.cmml"><mo stretchy="false" id="A4.3.p1.4.m4.2.2.1.1.2.1" xref="A4.3.p1.4.m4.2.2.1.2.cmml">(</mo><mi id="A4.3.p1.4.m4.1.1" xref="A4.3.p1.4.m4.1.1.cmml">U</mi><mo stretchy="false" id="A4.3.p1.4.m4.2.2.1.1.2.2" xref="A4.3.p1.4.m4.2.2.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.4.m4.2b"><apply id="A4.3.p1.4.m4.2.2.cmml" xref="A4.3.p1.4.m4.2.2"><eq id="A4.3.p1.4.m4.2.2.2.cmml" xref="A4.3.p1.4.m4.2.2.2"></eq><ci id="A4.3.p1.4.m4.2.2.3.cmml" xref="A4.3.p1.4.m4.2.2.3">𝑡</ci><apply id="A4.3.p1.4.m4.2.2.1.2.cmml" xref="A4.3.p1.4.m4.2.2.1.1"><apply id="A4.3.p1.4.m4.2.2.1.1.1.cmml" xref="A4.3.p1.4.m4.2.2.1.1.1"><csymbol cd="ambiguous" id="A4.3.p1.4.m4.2.2.1.1.1.1.cmml" xref="A4.3.p1.4.m4.2.2.1.1.1">subscript</csymbol><log id="A4.3.p1.4.m4.2.2.1.1.1.2.cmml" xref="A4.3.p1.4.m4.2.2.1.1.1.2"></log><cn type="integer" id="A4.3.p1.4.m4.2.2.1.1.1.3.cmml" xref="A4.3.p1.4.m4.2.2.1.1.1.3">2</cn></apply><ci id="A4.3.p1.4.m4.1.1.cmml" xref="A4.3.p1.4.m4.1.1">𝑈</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.4.m4.2c">t=\log_{2}(U)</annotation></semantics></math> corresponds to the root. The expected error <math id="A4.3.p1.5.m5.3" class="ltx_Math" alttext="E[\xi(t,n)]" display="inline"><semantics id="A4.3.p1.5.m5.3a"><mrow id="A4.3.p1.5.m5.3.3" xref="A4.3.p1.5.m5.3.3.cmml"><mi id="A4.3.p1.5.m5.3.3.3" xref="A4.3.p1.5.m5.3.3.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.3.p1.5.m5.3.3.2" xref="A4.3.p1.5.m5.3.3.2.cmml">​</mo><mrow id="A4.3.p1.5.m5.3.3.1.1" xref="A4.3.p1.5.m5.3.3.1.2.cmml"><mo stretchy="false" id="A4.3.p1.5.m5.3.3.1.1.2" xref="A4.3.p1.5.m5.3.3.1.2.1.cmml">[</mo><mrow id="A4.3.p1.5.m5.3.3.1.1.1" xref="A4.3.p1.5.m5.3.3.1.1.1.cmml"><mi id="A4.3.p1.5.m5.3.3.1.1.1.2" xref="A4.3.p1.5.m5.3.3.1.1.1.2.cmml">ξ</mi><mo lspace="0em" rspace="0em" id="A4.3.p1.5.m5.3.3.1.1.1.1" xref="A4.3.p1.5.m5.3.3.1.1.1.1.cmml">​</mo><mrow id="A4.3.p1.5.m5.3.3.1.1.1.3.2" xref="A4.3.p1.5.m5.3.3.1.1.1.3.1.cmml"><mo stretchy="false" id="A4.3.p1.5.m5.3.3.1.1.1.3.2.1" xref="A4.3.p1.5.m5.3.3.1.1.1.3.1.cmml">(</mo><mi id="A4.3.p1.5.m5.1.1" xref="A4.3.p1.5.m5.1.1.cmml">t</mi><mo id="A4.3.p1.5.m5.3.3.1.1.1.3.2.2" xref="A4.3.p1.5.m5.3.3.1.1.1.3.1.cmml">,</mo><mi id="A4.3.p1.5.m5.2.2" xref="A4.3.p1.5.m5.2.2.cmml">n</mi><mo stretchy="false" id="A4.3.p1.5.m5.3.3.1.1.1.3.2.3" xref="A4.3.p1.5.m5.3.3.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.3.p1.5.m5.3.3.1.1.3" xref="A4.3.p1.5.m5.3.3.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.5.m5.3b"><apply id="A4.3.p1.5.m5.3.3.cmml" xref="A4.3.p1.5.m5.3.3"><times id="A4.3.p1.5.m5.3.3.2.cmml" xref="A4.3.p1.5.m5.3.3.2"></times><ci id="A4.3.p1.5.m5.3.3.3.cmml" xref="A4.3.p1.5.m5.3.3.3">𝐸</ci><apply id="A4.3.p1.5.m5.3.3.1.2.cmml" xref="A4.3.p1.5.m5.3.3.1.1"><csymbol cd="latexml" id="A4.3.p1.5.m5.3.3.1.2.1.cmml" xref="A4.3.p1.5.m5.3.3.1.1.2">delimited-[]</csymbol><apply id="A4.3.p1.5.m5.3.3.1.1.1.cmml" xref="A4.3.p1.5.m5.3.3.1.1.1"><times id="A4.3.p1.5.m5.3.3.1.1.1.1.cmml" xref="A4.3.p1.5.m5.3.3.1.1.1.1"></times><ci id="A4.3.p1.5.m5.3.3.1.1.1.2.cmml" xref="A4.3.p1.5.m5.3.3.1.1.1.2">𝜉</ci><interval closure="open" id="A4.3.p1.5.m5.3.3.1.1.1.3.1.cmml" xref="A4.3.p1.5.m5.3.3.1.1.1.3.2"><ci id="A4.3.p1.5.m5.1.1.cmml" xref="A4.3.p1.5.m5.1.1">𝑡</ci><ci id="A4.3.p1.5.m5.2.2.cmml" xref="A4.3.p1.5.m5.2.2">𝑛</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.5.m5.3c">E[\xi(t,n)]</annotation></semantics></math> introduced at a node <math id="A4.3.p1.6.m6.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.3.p1.6.m6.1a"><mi id="A4.3.p1.6.m6.1.1" xref="A4.3.p1.6.m6.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.3.p1.6.m6.1b"><ci id="A4.3.p1.6.m6.1.1.cmml" xref="A4.3.p1.6.m6.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.6.m6.1c">n</annotation></semantics></math> in level <math id="A4.3.p1.7.m7.1" class="ltx_Math" alttext="t&gt;0" display="inline"><semantics id="A4.3.p1.7.m7.1a"><mrow id="A4.3.p1.7.m7.1.1" xref="A4.3.p1.7.m7.1.1.cmml"><mi id="A4.3.p1.7.m7.1.1.2" xref="A4.3.p1.7.m7.1.1.2.cmml">t</mi><mo id="A4.3.p1.7.m7.1.1.1" xref="A4.3.p1.7.m7.1.1.1.cmml">&gt;</mo><mn id="A4.3.p1.7.m7.1.1.3" xref="A4.3.p1.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.7.m7.1b"><apply id="A4.3.p1.7.m7.1.1.cmml" xref="A4.3.p1.7.m7.1.1"><gt id="A4.3.p1.7.m7.1.1.1.cmml" xref="A4.3.p1.7.m7.1.1.1"></gt><ci id="A4.3.p1.7.m7.1.1.2.cmml" xref="A4.3.p1.7.m7.1.1.2">𝑡</ci><cn type="integer" id="A4.3.p1.7.m7.1.1.3.cmml" xref="A4.3.p1.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.7.m7.1c">t&gt;0</annotation></semantics></math> is given by:</p>
<table id="A6.EGx16" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A4.E22"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E22.m1.3" class="ltx_Math" alttext="\displaystyle E[\xi(t,n)]=\frac{1}{2}\cdot 2^{t-1}" display="inline"><semantics id="A4.E22.m1.3a"><mrow id="A4.E22.m1.3.3" xref="A4.E22.m1.3.3.cmml"><mrow id="A4.E22.m1.3.3.1" xref="A4.E22.m1.3.3.1.cmml"><mi id="A4.E22.m1.3.3.1.3" xref="A4.E22.m1.3.3.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.E22.m1.3.3.1.2" xref="A4.E22.m1.3.3.1.2.cmml">​</mo><mrow id="A4.E22.m1.3.3.1.1.1" xref="A4.E22.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="A4.E22.m1.3.3.1.1.1.2" xref="A4.E22.m1.3.3.1.1.2.1.cmml">[</mo><mrow id="A4.E22.m1.3.3.1.1.1.1" xref="A4.E22.m1.3.3.1.1.1.1.cmml"><mi id="A4.E22.m1.3.3.1.1.1.1.2" xref="A4.E22.m1.3.3.1.1.1.1.2.cmml">ξ</mi><mo lspace="0em" rspace="0em" id="A4.E22.m1.3.3.1.1.1.1.1" xref="A4.E22.m1.3.3.1.1.1.1.1.cmml">​</mo><mrow id="A4.E22.m1.3.3.1.1.1.1.3.2" xref="A4.E22.m1.3.3.1.1.1.1.3.1.cmml"><mo stretchy="false" id="A4.E22.m1.3.3.1.1.1.1.3.2.1" xref="A4.E22.m1.3.3.1.1.1.1.3.1.cmml">(</mo><mi id="A4.E22.m1.1.1" xref="A4.E22.m1.1.1.cmml">t</mi><mo id="A4.E22.m1.3.3.1.1.1.1.3.2.2" xref="A4.E22.m1.3.3.1.1.1.1.3.1.cmml">,</mo><mi id="A4.E22.m1.2.2" xref="A4.E22.m1.2.2.cmml">n</mi><mo stretchy="false" id="A4.E22.m1.3.3.1.1.1.1.3.2.3" xref="A4.E22.m1.3.3.1.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.E22.m1.3.3.1.1.1.3" xref="A4.E22.m1.3.3.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A4.E22.m1.3.3.2" xref="A4.E22.m1.3.3.2.cmml">=</mo><mrow id="A4.E22.m1.3.3.3" xref="A4.E22.m1.3.3.3.cmml"><mstyle displaystyle="true" id="A4.E22.m1.3.3.3.2" xref="A4.E22.m1.3.3.3.2.cmml"><mfrac id="A4.E22.m1.3.3.3.2a" xref="A4.E22.m1.3.3.3.2.cmml"><mn id="A4.E22.m1.3.3.3.2.2" xref="A4.E22.m1.3.3.3.2.2.cmml">1</mn><mn id="A4.E22.m1.3.3.3.2.3" xref="A4.E22.m1.3.3.3.2.3.cmml">2</mn></mfrac></mstyle><mo lspace="0.222em" rspace="0.222em" id="A4.E22.m1.3.3.3.1" xref="A4.E22.m1.3.3.3.1.cmml">⋅</mo><msup id="A4.E22.m1.3.3.3.3" xref="A4.E22.m1.3.3.3.3.cmml"><mn id="A4.E22.m1.3.3.3.3.2" xref="A4.E22.m1.3.3.3.3.2.cmml">2</mn><mrow id="A4.E22.m1.3.3.3.3.3" xref="A4.E22.m1.3.3.3.3.3.cmml"><mi id="A4.E22.m1.3.3.3.3.3.2" xref="A4.E22.m1.3.3.3.3.3.2.cmml">t</mi><mo id="A4.E22.m1.3.3.3.3.3.1" xref="A4.E22.m1.3.3.3.3.3.1.cmml">−</mo><mn id="A4.E22.m1.3.3.3.3.3.3" xref="A4.E22.m1.3.3.3.3.3.3.cmml">1</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E22.m1.3b"><apply id="A4.E22.m1.3.3.cmml" xref="A4.E22.m1.3.3"><eq id="A4.E22.m1.3.3.2.cmml" xref="A4.E22.m1.3.3.2"></eq><apply id="A4.E22.m1.3.3.1.cmml" xref="A4.E22.m1.3.3.1"><times id="A4.E22.m1.3.3.1.2.cmml" xref="A4.E22.m1.3.3.1.2"></times><ci id="A4.E22.m1.3.3.1.3.cmml" xref="A4.E22.m1.3.3.1.3">𝐸</ci><apply id="A4.E22.m1.3.3.1.1.2.cmml" xref="A4.E22.m1.3.3.1.1.1"><csymbol cd="latexml" id="A4.E22.m1.3.3.1.1.2.1.cmml" xref="A4.E22.m1.3.3.1.1.1.2">delimited-[]</csymbol><apply id="A4.E22.m1.3.3.1.1.1.1.cmml" xref="A4.E22.m1.3.3.1.1.1.1"><times id="A4.E22.m1.3.3.1.1.1.1.1.cmml" xref="A4.E22.m1.3.3.1.1.1.1.1"></times><ci id="A4.E22.m1.3.3.1.1.1.1.2.cmml" xref="A4.E22.m1.3.3.1.1.1.1.2">𝜉</ci><interval closure="open" id="A4.E22.m1.3.3.1.1.1.1.3.1.cmml" xref="A4.E22.m1.3.3.1.1.1.1.3.2"><ci id="A4.E22.m1.1.1.cmml" xref="A4.E22.m1.1.1">𝑡</ci><ci id="A4.E22.m1.2.2.cmml" xref="A4.E22.m1.2.2">𝑛</ci></interval></apply></apply></apply><apply id="A4.E22.m1.3.3.3.cmml" xref="A4.E22.m1.3.3.3"><ci id="A4.E22.m1.3.3.3.1.cmml" xref="A4.E22.m1.3.3.3.1">⋅</ci><apply id="A4.E22.m1.3.3.3.2.cmml" xref="A4.E22.m1.3.3.3.2"><divide id="A4.E22.m1.3.3.3.2.1.cmml" xref="A4.E22.m1.3.3.3.2"></divide><cn type="integer" id="A4.E22.m1.3.3.3.2.2.cmml" xref="A4.E22.m1.3.3.3.2.2">1</cn><cn type="integer" id="A4.E22.m1.3.3.3.2.3.cmml" xref="A4.E22.m1.3.3.3.2.3">2</cn></apply><apply id="A4.E22.m1.3.3.3.3.cmml" xref="A4.E22.m1.3.3.3.3"><csymbol cd="ambiguous" id="A4.E22.m1.3.3.3.3.1.cmml" xref="A4.E22.m1.3.3.3.3">superscript</csymbol><cn type="integer" id="A4.E22.m1.3.3.3.3.2.cmml" xref="A4.E22.m1.3.3.3.3.2">2</cn><apply id="A4.E22.m1.3.3.3.3.3.cmml" xref="A4.E22.m1.3.3.3.3.3"><minus id="A4.E22.m1.3.3.3.3.3.1.cmml" xref="A4.E22.m1.3.3.3.3.3.1"></minus><ci id="A4.E22.m1.3.3.3.3.3.2.cmml" xref="A4.E22.m1.3.3.3.3.3.2">𝑡</ci><cn type="integer" id="A4.E22.m1.3.3.3.3.3.3.cmml" xref="A4.E22.m1.3.3.3.3.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E22.m1.3c">\displaystyle E[\xi(t,n)]=\frac{1}{2}\cdot 2^{t-1}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(22)</span></td>
</tr></tbody>
</table>
<p id="A4.3.p1.15" class="ltx_p">where the <math id="A4.3.p1.8.m1.1" class="ltx_Math" alttext="\frac{1}{2}" display="inline"><semantics id="A4.3.p1.8.m1.1a"><mfrac id="A4.3.p1.8.m1.1.1" xref="A4.3.p1.8.m1.1.1.cmml"><mn id="A4.3.p1.8.m1.1.1.2" xref="A4.3.p1.8.m1.1.1.2.cmml">1</mn><mn id="A4.3.p1.8.m1.1.1.3" xref="A4.3.p1.8.m1.1.1.3.cmml">2</mn></mfrac><annotation-xml encoding="MathML-Content" id="A4.3.p1.8.m1.1b"><apply id="A4.3.p1.8.m1.1.1.cmml" xref="A4.3.p1.8.m1.1.1"><divide id="A4.3.p1.8.m1.1.1.1.cmml" xref="A4.3.p1.8.m1.1.1"></divide><cn type="integer" id="A4.3.p1.8.m1.1.1.2.cmml" xref="A4.3.p1.8.m1.1.1.2">1</cn><cn type="integer" id="A4.3.p1.8.m1.1.1.3.cmml" xref="A4.3.p1.8.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.8.m1.1c">\frac{1}{2}</annotation></semantics></math> follows from Lemma <a href="#A4.Thmlemma1" title="Lemma D.1 (Bias when averaging one pair). ‣ Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a> and the scale <math id="A4.3.p1.9.m2.1" class="ltx_Math" alttext="2^{t-1}" display="inline"><semantics id="A4.3.p1.9.m2.1a"><msup id="A4.3.p1.9.m2.1.1" xref="A4.3.p1.9.m2.1.1.cmml"><mn id="A4.3.p1.9.m2.1.1.2" xref="A4.3.p1.9.m2.1.1.2.cmml">2</mn><mrow id="A4.3.p1.9.m2.1.1.3" xref="A4.3.p1.9.m2.1.1.3.cmml"><mi id="A4.3.p1.9.m2.1.1.3.2" xref="A4.3.p1.9.m2.1.1.3.2.cmml">t</mi><mo id="A4.3.p1.9.m2.1.1.3.1" xref="A4.3.p1.9.m2.1.1.3.1.cmml">−</mo><mn id="A4.3.p1.9.m2.1.1.3.3" xref="A4.3.p1.9.m2.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A4.3.p1.9.m2.1b"><apply id="A4.3.p1.9.m2.1.1.cmml" xref="A4.3.p1.9.m2.1.1"><csymbol cd="ambiguous" id="A4.3.p1.9.m2.1.1.1.cmml" xref="A4.3.p1.9.m2.1.1">superscript</csymbol><cn type="integer" id="A4.3.p1.9.m2.1.1.2.cmml" xref="A4.3.p1.9.m2.1.1.2">2</cn><apply id="A4.3.p1.9.m2.1.1.3.cmml" xref="A4.3.p1.9.m2.1.1.3"><minus id="A4.3.p1.9.m2.1.1.3.1.cmml" xref="A4.3.p1.9.m2.1.1.3.1"></minus><ci id="A4.3.p1.9.m2.1.1.3.2.cmml" xref="A4.3.p1.9.m2.1.1.3.2">𝑡</ci><cn type="integer" id="A4.3.p1.9.m2.1.1.3.3.cmml" xref="A4.3.p1.9.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.9.m2.1c">2^{t-1}</annotation></semantics></math> is the number of leaf nodes to which the bias is effectively applied. E.g., adding one to the estimated average of four leaf nodes would increase the estimated sum by four. Since there are <math id="A4.3.p1.10.m3.1" class="ltx_Math" alttext="U\cdot 2^{-t}" display="inline"><semantics id="A4.3.p1.10.m3.1a"><mrow id="A4.3.p1.10.m3.1.1" xref="A4.3.p1.10.m3.1.1.cmml"><mi id="A4.3.p1.10.m3.1.1.2" xref="A4.3.p1.10.m3.1.1.2.cmml">U</mi><mo lspace="0.222em" rspace="0.222em" id="A4.3.p1.10.m3.1.1.1" xref="A4.3.p1.10.m3.1.1.1.cmml">⋅</mo><msup id="A4.3.p1.10.m3.1.1.3" xref="A4.3.p1.10.m3.1.1.3.cmml"><mn id="A4.3.p1.10.m3.1.1.3.2" xref="A4.3.p1.10.m3.1.1.3.2.cmml">2</mn><mrow id="A4.3.p1.10.m3.1.1.3.3" xref="A4.3.p1.10.m3.1.1.3.3.cmml"><mo id="A4.3.p1.10.m3.1.1.3.3a" xref="A4.3.p1.10.m3.1.1.3.3.cmml">−</mo><mi id="A4.3.p1.10.m3.1.1.3.3.2" xref="A4.3.p1.10.m3.1.1.3.3.2.cmml">t</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.10.m3.1b"><apply id="A4.3.p1.10.m3.1.1.cmml" xref="A4.3.p1.10.m3.1.1"><ci id="A4.3.p1.10.m3.1.1.1.cmml" xref="A4.3.p1.10.m3.1.1.1">⋅</ci><ci id="A4.3.p1.10.m3.1.1.2.cmml" xref="A4.3.p1.10.m3.1.1.2">𝑈</ci><apply id="A4.3.p1.10.m3.1.1.3.cmml" xref="A4.3.p1.10.m3.1.1.3"><csymbol cd="ambiguous" id="A4.3.p1.10.m3.1.1.3.1.cmml" xref="A4.3.p1.10.m3.1.1.3">superscript</csymbol><cn type="integer" id="A4.3.p1.10.m3.1.1.3.2.cmml" xref="A4.3.p1.10.m3.1.1.3.2">2</cn><apply id="A4.3.p1.10.m3.1.1.3.3.cmml" xref="A4.3.p1.10.m3.1.1.3.3"><minus id="A4.3.p1.10.m3.1.1.3.3.1.cmml" xref="A4.3.p1.10.m3.1.1.3.3"></minus><ci id="A4.3.p1.10.m3.1.1.3.3.2.cmml" xref="A4.3.p1.10.m3.1.1.3.3.2">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.10.m3.1c">U\cdot 2^{-t}</annotation></semantics></math> nodes per level, this means that the total expected error introduced at level <math id="A4.3.p1.11.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="A4.3.p1.11.m4.1a"><mi id="A4.3.p1.11.m4.1.1" xref="A4.3.p1.11.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="A4.3.p1.11.m4.1b"><ci id="A4.3.p1.11.m4.1.1.cmml" xref="A4.3.p1.11.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.11.m4.1c">t</annotation></semantics></math> is <math id="A4.3.p1.12.m5.1" class="ltx_Math" alttext="\frac{1}{2}\cdot 2^{t-1}\cdot 2^{-t}=\frac{1}{4}" display="inline"><semantics id="A4.3.p1.12.m5.1a"><mrow id="A4.3.p1.12.m5.1.1" xref="A4.3.p1.12.m5.1.1.cmml"><mrow id="A4.3.p1.12.m5.1.1.2" xref="A4.3.p1.12.m5.1.1.2.cmml"><mfrac id="A4.3.p1.12.m5.1.1.2.2" xref="A4.3.p1.12.m5.1.1.2.2.cmml"><mn id="A4.3.p1.12.m5.1.1.2.2.2" xref="A4.3.p1.12.m5.1.1.2.2.2.cmml">1</mn><mn id="A4.3.p1.12.m5.1.1.2.2.3" xref="A4.3.p1.12.m5.1.1.2.2.3.cmml">2</mn></mfrac><mo lspace="0.222em" rspace="0.222em" id="A4.3.p1.12.m5.1.1.2.1" xref="A4.3.p1.12.m5.1.1.2.1.cmml">⋅</mo><msup id="A4.3.p1.12.m5.1.1.2.3" xref="A4.3.p1.12.m5.1.1.2.3.cmml"><mn id="A4.3.p1.12.m5.1.1.2.3.2" xref="A4.3.p1.12.m5.1.1.2.3.2.cmml">2</mn><mrow id="A4.3.p1.12.m5.1.1.2.3.3" xref="A4.3.p1.12.m5.1.1.2.3.3.cmml"><mi id="A4.3.p1.12.m5.1.1.2.3.3.2" xref="A4.3.p1.12.m5.1.1.2.3.3.2.cmml">t</mi><mo id="A4.3.p1.12.m5.1.1.2.3.3.1" xref="A4.3.p1.12.m5.1.1.2.3.3.1.cmml">−</mo><mn id="A4.3.p1.12.m5.1.1.2.3.3.3" xref="A4.3.p1.12.m5.1.1.2.3.3.3.cmml">1</mn></mrow></msup><mo lspace="0.222em" rspace="0.222em" id="A4.3.p1.12.m5.1.1.2.1a" xref="A4.3.p1.12.m5.1.1.2.1.cmml">⋅</mo><msup id="A4.3.p1.12.m5.1.1.2.4" xref="A4.3.p1.12.m5.1.1.2.4.cmml"><mn id="A4.3.p1.12.m5.1.1.2.4.2" xref="A4.3.p1.12.m5.1.1.2.4.2.cmml">2</mn><mrow id="A4.3.p1.12.m5.1.1.2.4.3" xref="A4.3.p1.12.m5.1.1.2.4.3.cmml"><mo id="A4.3.p1.12.m5.1.1.2.4.3a" xref="A4.3.p1.12.m5.1.1.2.4.3.cmml">−</mo><mi id="A4.3.p1.12.m5.1.1.2.4.3.2" xref="A4.3.p1.12.m5.1.1.2.4.3.2.cmml">t</mi></mrow></msup></mrow><mo id="A4.3.p1.12.m5.1.1.1" xref="A4.3.p1.12.m5.1.1.1.cmml">=</mo><mfrac id="A4.3.p1.12.m5.1.1.3" xref="A4.3.p1.12.m5.1.1.3.cmml"><mn id="A4.3.p1.12.m5.1.1.3.2" xref="A4.3.p1.12.m5.1.1.3.2.cmml">1</mn><mn id="A4.3.p1.12.m5.1.1.3.3" xref="A4.3.p1.12.m5.1.1.3.3.cmml">4</mn></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.12.m5.1b"><apply id="A4.3.p1.12.m5.1.1.cmml" xref="A4.3.p1.12.m5.1.1"><eq id="A4.3.p1.12.m5.1.1.1.cmml" xref="A4.3.p1.12.m5.1.1.1"></eq><apply id="A4.3.p1.12.m5.1.1.2.cmml" xref="A4.3.p1.12.m5.1.1.2"><ci id="A4.3.p1.12.m5.1.1.2.1.cmml" xref="A4.3.p1.12.m5.1.1.2.1">⋅</ci><apply id="A4.3.p1.12.m5.1.1.2.2.cmml" xref="A4.3.p1.12.m5.1.1.2.2"><divide id="A4.3.p1.12.m5.1.1.2.2.1.cmml" xref="A4.3.p1.12.m5.1.1.2.2"></divide><cn type="integer" id="A4.3.p1.12.m5.1.1.2.2.2.cmml" xref="A4.3.p1.12.m5.1.1.2.2.2">1</cn><cn type="integer" id="A4.3.p1.12.m5.1.1.2.2.3.cmml" xref="A4.3.p1.12.m5.1.1.2.2.3">2</cn></apply><apply id="A4.3.p1.12.m5.1.1.2.3.cmml" xref="A4.3.p1.12.m5.1.1.2.3"><csymbol cd="ambiguous" id="A4.3.p1.12.m5.1.1.2.3.1.cmml" xref="A4.3.p1.12.m5.1.1.2.3">superscript</csymbol><cn type="integer" id="A4.3.p1.12.m5.1.1.2.3.2.cmml" xref="A4.3.p1.12.m5.1.1.2.3.2">2</cn><apply id="A4.3.p1.12.m5.1.1.2.3.3.cmml" xref="A4.3.p1.12.m5.1.1.2.3.3"><minus id="A4.3.p1.12.m5.1.1.2.3.3.1.cmml" xref="A4.3.p1.12.m5.1.1.2.3.3.1"></minus><ci id="A4.3.p1.12.m5.1.1.2.3.3.2.cmml" xref="A4.3.p1.12.m5.1.1.2.3.3.2">𝑡</ci><cn type="integer" id="A4.3.p1.12.m5.1.1.2.3.3.3.cmml" xref="A4.3.p1.12.m5.1.1.2.3.3.3">1</cn></apply></apply><apply id="A4.3.p1.12.m5.1.1.2.4.cmml" xref="A4.3.p1.12.m5.1.1.2.4"><csymbol cd="ambiguous" id="A4.3.p1.12.m5.1.1.2.4.1.cmml" xref="A4.3.p1.12.m5.1.1.2.4">superscript</csymbol><cn type="integer" id="A4.3.p1.12.m5.1.1.2.4.2.cmml" xref="A4.3.p1.12.m5.1.1.2.4.2">2</cn><apply id="A4.3.p1.12.m5.1.1.2.4.3.cmml" xref="A4.3.p1.12.m5.1.1.2.4.3"><minus id="A4.3.p1.12.m5.1.1.2.4.3.1.cmml" xref="A4.3.p1.12.m5.1.1.2.4.3"></minus><ci id="A4.3.p1.12.m5.1.1.2.4.3.2.cmml" xref="A4.3.p1.12.m5.1.1.2.4.3.2">𝑡</ci></apply></apply></apply><apply id="A4.3.p1.12.m5.1.1.3.cmml" xref="A4.3.p1.12.m5.1.1.3"><divide id="A4.3.p1.12.m5.1.1.3.1.cmml" xref="A4.3.p1.12.m5.1.1.3"></divide><cn type="integer" id="A4.3.p1.12.m5.1.1.3.2.cmml" xref="A4.3.p1.12.m5.1.1.3.2">1</cn><cn type="integer" id="A4.3.p1.12.m5.1.1.3.3.cmml" xref="A4.3.p1.12.m5.1.1.3.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.12.m5.1c">\frac{1}{2}\cdot 2^{t-1}\cdot 2^{-t}=\frac{1}{4}</annotation></semantics></math>. Summing from <math id="A4.3.p1.13.m6.1" class="ltx_Math" alttext="t=1" display="inline"><semantics id="A4.3.p1.13.m6.1a"><mrow id="A4.3.p1.13.m6.1.1" xref="A4.3.p1.13.m6.1.1.cmml"><mi id="A4.3.p1.13.m6.1.1.2" xref="A4.3.p1.13.m6.1.1.2.cmml">t</mi><mo id="A4.3.p1.13.m6.1.1.1" xref="A4.3.p1.13.m6.1.1.1.cmml">=</mo><mn id="A4.3.p1.13.m6.1.1.3" xref="A4.3.p1.13.m6.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.13.m6.1b"><apply id="A4.3.p1.13.m6.1.1.cmml" xref="A4.3.p1.13.m6.1.1"><eq id="A4.3.p1.13.m6.1.1.1.cmml" xref="A4.3.p1.13.m6.1.1.1"></eq><ci id="A4.3.p1.13.m6.1.1.2.cmml" xref="A4.3.p1.13.m6.1.1.2">𝑡</ci><cn type="integer" id="A4.3.p1.13.m6.1.1.3.cmml" xref="A4.3.p1.13.m6.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.13.m6.1c">t=1</annotation></semantics></math> to <math id="A4.3.p1.14.m7.2" class="ltx_Math" alttext="t=\log_{2}(U)" display="inline"><semantics id="A4.3.p1.14.m7.2a"><mrow id="A4.3.p1.14.m7.2.2" xref="A4.3.p1.14.m7.2.2.cmml"><mi id="A4.3.p1.14.m7.2.2.3" xref="A4.3.p1.14.m7.2.2.3.cmml">t</mi><mo id="A4.3.p1.14.m7.2.2.2" xref="A4.3.p1.14.m7.2.2.2.cmml">=</mo><mrow id="A4.3.p1.14.m7.2.2.1.1" xref="A4.3.p1.14.m7.2.2.1.2.cmml"><msub id="A4.3.p1.14.m7.2.2.1.1.1" xref="A4.3.p1.14.m7.2.2.1.1.1.cmml"><mi id="A4.3.p1.14.m7.2.2.1.1.1.2" xref="A4.3.p1.14.m7.2.2.1.1.1.2.cmml">log</mi><mn id="A4.3.p1.14.m7.2.2.1.1.1.3" xref="A4.3.p1.14.m7.2.2.1.1.1.3.cmml">2</mn></msub><mo id="A4.3.p1.14.m7.2.2.1.1a" xref="A4.3.p1.14.m7.2.2.1.2.cmml">⁡</mo><mrow id="A4.3.p1.14.m7.2.2.1.1.2" xref="A4.3.p1.14.m7.2.2.1.2.cmml"><mo stretchy="false" id="A4.3.p1.14.m7.2.2.1.1.2.1" xref="A4.3.p1.14.m7.2.2.1.2.cmml">(</mo><mi id="A4.3.p1.14.m7.1.1" xref="A4.3.p1.14.m7.1.1.cmml">U</mi><mo stretchy="false" id="A4.3.p1.14.m7.2.2.1.1.2.2" xref="A4.3.p1.14.m7.2.2.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.14.m7.2b"><apply id="A4.3.p1.14.m7.2.2.cmml" xref="A4.3.p1.14.m7.2.2"><eq id="A4.3.p1.14.m7.2.2.2.cmml" xref="A4.3.p1.14.m7.2.2.2"></eq><ci id="A4.3.p1.14.m7.2.2.3.cmml" xref="A4.3.p1.14.m7.2.2.3">𝑡</ci><apply id="A4.3.p1.14.m7.2.2.1.2.cmml" xref="A4.3.p1.14.m7.2.2.1.1"><apply id="A4.3.p1.14.m7.2.2.1.1.1.cmml" xref="A4.3.p1.14.m7.2.2.1.1.1"><csymbol cd="ambiguous" id="A4.3.p1.14.m7.2.2.1.1.1.1.cmml" xref="A4.3.p1.14.m7.2.2.1.1.1">subscript</csymbol><log id="A4.3.p1.14.m7.2.2.1.1.1.2.cmml" xref="A4.3.p1.14.m7.2.2.1.1.1.2"></log><cn type="integer" id="A4.3.p1.14.m7.2.2.1.1.1.3.cmml" xref="A4.3.p1.14.m7.2.2.1.1.1.3">2</cn></apply><ci id="A4.3.p1.14.m7.1.1.cmml" xref="A4.3.p1.14.m7.1.1">𝑈</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.14.m7.2c">t=\log_{2}(U)</annotation></semantics></math> completes the proof of the expected error. Note that <math id="A4.3.p1.15.m8.1" class="ltx_Math" alttext="t=0" display="inline"><semantics id="A4.3.p1.15.m8.1a"><mrow id="A4.3.p1.15.m8.1.1" xref="A4.3.p1.15.m8.1.1.cmml"><mi id="A4.3.p1.15.m8.1.1.2" xref="A4.3.p1.15.m8.1.1.2.cmml">t</mi><mo id="A4.3.p1.15.m8.1.1.1" xref="A4.3.p1.15.m8.1.1.1.cmml">=</mo><mn id="A4.3.p1.15.m8.1.1.3" xref="A4.3.p1.15.m8.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.3.p1.15.m8.1b"><apply id="A4.3.p1.15.m8.1.1.cmml" xref="A4.3.p1.15.m8.1.1"><eq id="A4.3.p1.15.m8.1.1.1.cmml" xref="A4.3.p1.15.m8.1.1.1"></eq><ci id="A4.3.p1.15.m8.1.1.2.cmml" xref="A4.3.p1.15.m8.1.1.2">𝑡</ci><cn type="integer" id="A4.3.p1.15.m8.1.1.3.cmml" xref="A4.3.p1.15.m8.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.3.p1.15.m8.1c">t=0</annotation></semantics></math> is omitted since the leaf nodes are not the result of averaging operations and so introduce no error.</p>
</div>
<div id="A4.4.p2" class="ltx_para">
<p id="A4.4.p2.1" class="ltx_p">∎</p>
</div>
</div>
<div id="A4.Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A4.Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem D.1</span></span><span id="A4.Thmtheorem1.2.2" class="ltx_text ltx_font_bold"> </span>(Bias of AISE)<span id="A4.Thmtheorem1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A4.Thmtheorem1.p1" class="ltx_para">
<p id="A4.Thmtheorem1.p1.2" class="ltx_p"><span id="A4.Thmtheorem1.p1.2.2" class="ltx_text ltx_font_italic">Suppose that the scalar elements <math id="A4.Thmtheorem1.p1.1.1.m1.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="A4.Thmtheorem1.p1.1.1.m1.1a"><msub id="A4.Thmtheorem1.p1.1.1.m1.1.1" xref="A4.Thmtheorem1.p1.1.1.m1.1.1.cmml"><mi id="A4.Thmtheorem1.p1.1.1.m1.1.1.2" xref="A4.Thmtheorem1.p1.1.1.m1.1.1.2.cmml">x</mi><mi id="A4.Thmtheorem1.p1.1.1.m1.1.1.3" xref="A4.Thmtheorem1.p1.1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A4.Thmtheorem1.p1.1.1.m1.1b"><apply id="A4.Thmtheorem1.p1.1.1.m1.1.1.cmml" xref="A4.Thmtheorem1.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A4.Thmtheorem1.p1.1.1.m1.1.1.1.cmml" xref="A4.Thmtheorem1.p1.1.1.m1.1.1">subscript</csymbol><ci id="A4.Thmtheorem1.p1.1.1.m1.1.1.2.cmml" xref="A4.Thmtheorem1.p1.1.1.m1.1.1.2">𝑥</ci><ci id="A4.Thmtheorem1.p1.1.1.m1.1.1.3.cmml" xref="A4.Thmtheorem1.p1.1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmtheorem1.p1.1.1.m1.1c">x_{i}</annotation></semantics></math> of <math id="A4.Thmtheorem1.p1.2.2.m2.1" class="ltx_Math" alttext="\bm{x}" display="inline"><semantics id="A4.Thmtheorem1.p1.2.2.m2.1a"><mi id="A4.Thmtheorem1.p1.2.2.m2.1.1" xref="A4.Thmtheorem1.p1.2.2.m2.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="A4.Thmtheorem1.p1.2.2.m2.1b"><ci id="A4.Thmtheorem1.p1.2.2.m2.1.1.cmml" xref="A4.Thmtheorem1.p1.2.2.m2.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.Thmtheorem1.p1.2.2.m2.1c">\bm{x}</annotation></semantics></math> are drawn from independent Bernoulli(.5) distributions. Then</span></p>
<table id="A6.EGx17" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A4.E23"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E23.m1.3" class="ltx_Math" alttext="\displaystyle E[s(\bm{x})-\hat{s}(\bm{x})]" display="inline"><semantics id="A4.E23.m1.3a"><mrow id="A4.E23.m1.3.3" xref="A4.E23.m1.3.3.cmml"><mi id="A4.E23.m1.3.3.3" xref="A4.E23.m1.3.3.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A4.E23.m1.3.3.2" xref="A4.E23.m1.3.3.2.cmml">​</mo><mrow id="A4.E23.m1.3.3.1.1" xref="A4.E23.m1.3.3.1.2.cmml"><mo stretchy="false" id="A4.E23.m1.3.3.1.1.2" xref="A4.E23.m1.3.3.1.2.1.cmml">[</mo><mrow id="A4.E23.m1.3.3.1.1.1" xref="A4.E23.m1.3.3.1.1.1.cmml"><mrow id="A4.E23.m1.3.3.1.1.1.2" xref="A4.E23.m1.3.3.1.1.1.2.cmml"><mi id="A4.E23.m1.3.3.1.1.1.2.2" xref="A4.E23.m1.3.3.1.1.1.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A4.E23.m1.3.3.1.1.1.2.1" xref="A4.E23.m1.3.3.1.1.1.2.1.cmml">​</mo><mrow id="A4.E23.m1.3.3.1.1.1.2.3.2" xref="A4.E23.m1.3.3.1.1.1.2.cmml"><mo stretchy="false" id="A4.E23.m1.3.3.1.1.1.2.3.2.1" xref="A4.E23.m1.3.3.1.1.1.2.cmml">(</mo><mi id="A4.E23.m1.1.1" xref="A4.E23.m1.1.1.cmml">𝒙</mi><mo stretchy="false" id="A4.E23.m1.3.3.1.1.1.2.3.2.2" xref="A4.E23.m1.3.3.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A4.E23.m1.3.3.1.1.1.1" xref="A4.E23.m1.3.3.1.1.1.1.cmml">−</mo><mrow id="A4.E23.m1.3.3.1.1.1.3" xref="A4.E23.m1.3.3.1.1.1.3.cmml"><mover accent="true" id="A4.E23.m1.3.3.1.1.1.3.2" xref="A4.E23.m1.3.3.1.1.1.3.2.cmml"><mi id="A4.E23.m1.3.3.1.1.1.3.2.2" xref="A4.E23.m1.3.3.1.1.1.3.2.2.cmml">s</mi><mo id="A4.E23.m1.3.3.1.1.1.3.2.1" xref="A4.E23.m1.3.3.1.1.1.3.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A4.E23.m1.3.3.1.1.1.3.1" xref="A4.E23.m1.3.3.1.1.1.3.1.cmml">​</mo><mrow id="A4.E23.m1.3.3.1.1.1.3.3.2" xref="A4.E23.m1.3.3.1.1.1.3.cmml"><mo stretchy="false" id="A4.E23.m1.3.3.1.1.1.3.3.2.1" xref="A4.E23.m1.3.3.1.1.1.3.cmml">(</mo><mi id="A4.E23.m1.2.2" xref="A4.E23.m1.2.2.cmml">𝒙</mi><mo stretchy="false" id="A4.E23.m1.3.3.1.1.1.3.3.2.2" xref="A4.E23.m1.3.3.1.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A4.E23.m1.3.3.1.1.3" xref="A4.E23.m1.3.3.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E23.m1.3b"><apply id="A4.E23.m1.3.3.cmml" xref="A4.E23.m1.3.3"><times id="A4.E23.m1.3.3.2.cmml" xref="A4.E23.m1.3.3.2"></times><ci id="A4.E23.m1.3.3.3.cmml" xref="A4.E23.m1.3.3.3">𝐸</ci><apply id="A4.E23.m1.3.3.1.2.cmml" xref="A4.E23.m1.3.3.1.1"><csymbol cd="latexml" id="A4.E23.m1.3.3.1.2.1.cmml" xref="A4.E23.m1.3.3.1.1.2">delimited-[]</csymbol><apply id="A4.E23.m1.3.3.1.1.1.cmml" xref="A4.E23.m1.3.3.1.1.1"><minus id="A4.E23.m1.3.3.1.1.1.1.cmml" xref="A4.E23.m1.3.3.1.1.1.1"></minus><apply id="A4.E23.m1.3.3.1.1.1.2.cmml" xref="A4.E23.m1.3.3.1.1.1.2"><times id="A4.E23.m1.3.3.1.1.1.2.1.cmml" xref="A4.E23.m1.3.3.1.1.1.2.1"></times><ci id="A4.E23.m1.3.3.1.1.1.2.2.cmml" xref="A4.E23.m1.3.3.1.1.1.2.2">𝑠</ci><ci id="A4.E23.m1.1.1.cmml" xref="A4.E23.m1.1.1">𝒙</ci></apply><apply id="A4.E23.m1.3.3.1.1.1.3.cmml" xref="A4.E23.m1.3.3.1.1.1.3"><times id="A4.E23.m1.3.3.1.1.1.3.1.cmml" xref="A4.E23.m1.3.3.1.1.1.3.1"></times><apply id="A4.E23.m1.3.3.1.1.1.3.2.cmml" xref="A4.E23.m1.3.3.1.1.1.3.2"><ci id="A4.E23.m1.3.3.1.1.1.3.2.1.cmml" xref="A4.E23.m1.3.3.1.1.1.3.2.1">^</ci><ci id="A4.E23.m1.3.3.1.1.1.3.2.2.cmml" xref="A4.E23.m1.3.3.1.1.1.3.2.2">𝑠</ci></apply><ci id="A4.E23.m1.2.2.cmml" xref="A4.E23.m1.2.2">𝒙</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E23.m1.3c">\displaystyle E[s(\bm{x})-\hat{s}(\bm{x})]</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A4.E23.m2.2" class="ltx_Math" alttext="\displaystyle=C\log_{2}(U)/4" display="inline"><semantics id="A4.E23.m2.2a"><mrow id="A4.E23.m2.2.2" xref="A4.E23.m2.2.2.cmml"><mi id="A4.E23.m2.2.2.3" xref="A4.E23.m2.2.2.3.cmml"></mi><mo id="A4.E23.m2.2.2.2" xref="A4.E23.m2.2.2.2.cmml">=</mo><mrow id="A4.E23.m2.2.2.1" xref="A4.E23.m2.2.2.1.cmml"><mrow id="A4.E23.m2.2.2.1.1" xref="A4.E23.m2.2.2.1.1.cmml"><mi id="A4.E23.m2.2.2.1.1.3" xref="A4.E23.m2.2.2.1.1.3.cmml">C</mi><mo lspace="0.167em" rspace="0em" id="A4.E23.m2.2.2.1.1.2" xref="A4.E23.m2.2.2.1.1.2.cmml">​</mo><mrow id="A4.E23.m2.2.2.1.1.1.1" xref="A4.E23.m2.2.2.1.1.1.2.cmml"><msub id="A4.E23.m2.2.2.1.1.1.1.1" xref="A4.E23.m2.2.2.1.1.1.1.1.cmml"><mi id="A4.E23.m2.2.2.1.1.1.1.1.2" xref="A4.E23.m2.2.2.1.1.1.1.1.2.cmml">log</mi><mn id="A4.E23.m2.2.2.1.1.1.1.1.3" xref="A4.E23.m2.2.2.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A4.E23.m2.2.2.1.1.1.1a" xref="A4.E23.m2.2.2.1.1.1.2.cmml">⁡</mo><mrow id="A4.E23.m2.2.2.1.1.1.1.2" xref="A4.E23.m2.2.2.1.1.1.2.cmml"><mo stretchy="false" id="A4.E23.m2.2.2.1.1.1.1.2.1" xref="A4.E23.m2.2.2.1.1.1.2.cmml">(</mo><mi id="A4.E23.m2.1.1" xref="A4.E23.m2.1.1.cmml">U</mi><mo stretchy="false" id="A4.E23.m2.2.2.1.1.1.1.2.2" xref="A4.E23.m2.2.2.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="A4.E23.m2.2.2.1.2" xref="A4.E23.m2.2.2.1.2.cmml">/</mo><mn id="A4.E23.m2.2.2.1.3" xref="A4.E23.m2.2.2.1.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E23.m2.2b"><apply id="A4.E23.m2.2.2.cmml" xref="A4.E23.m2.2.2"><eq id="A4.E23.m2.2.2.2.cmml" xref="A4.E23.m2.2.2.2"></eq><csymbol cd="latexml" id="A4.E23.m2.2.2.3.cmml" xref="A4.E23.m2.2.2.3">absent</csymbol><apply id="A4.E23.m2.2.2.1.cmml" xref="A4.E23.m2.2.2.1"><divide id="A4.E23.m2.2.2.1.2.cmml" xref="A4.E23.m2.2.2.1.2"></divide><apply id="A4.E23.m2.2.2.1.1.cmml" xref="A4.E23.m2.2.2.1.1"><times id="A4.E23.m2.2.2.1.1.2.cmml" xref="A4.E23.m2.2.2.1.1.2"></times><ci id="A4.E23.m2.2.2.1.1.3.cmml" xref="A4.E23.m2.2.2.1.1.3">𝐶</ci><apply id="A4.E23.m2.2.2.1.1.1.2.cmml" xref="A4.E23.m2.2.2.1.1.1.1"><apply id="A4.E23.m2.2.2.1.1.1.1.1.cmml" xref="A4.E23.m2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A4.E23.m2.2.2.1.1.1.1.1.1.cmml" xref="A4.E23.m2.2.2.1.1.1.1.1">subscript</csymbol><log id="A4.E23.m2.2.2.1.1.1.1.1.2.cmml" xref="A4.E23.m2.2.2.1.1.1.1.1.2"></log><cn type="integer" id="A4.E23.m2.2.2.1.1.1.1.1.3.cmml" xref="A4.E23.m2.2.2.1.1.1.1.1.3">2</cn></apply><ci id="A4.E23.m2.1.1.cmml" xref="A4.E23.m2.1.1">𝑈</ci></apply></apply><cn type="integer" id="A4.E23.m2.2.2.1.3.cmml" xref="A4.E23.m2.2.2.1.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E23.m2.2c">\displaystyle=C\log_{2}(U)/4</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(23)</span></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A4.5" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A4.5.p1" class="ltx_para">
<p id="A4.5.p1.2" class="ltx_p">This follows immediately from Lemma <a href="#A4.Thmlemma3" title="Lemma D.3 (Bias of AISE within a subspace). ‣ Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.3</span></a>, the fact that the overall sum is estimated within each of <math id="A4.5.p1.1.m1.1" class="ltx_Math" alttext="C/U" display="inline"><semantics id="A4.5.p1.1.m1.1a"><mrow id="A4.5.p1.1.m1.1.1" xref="A4.5.p1.1.m1.1.1.cmml"><mi id="A4.5.p1.1.m1.1.1.2" xref="A4.5.p1.1.m1.1.1.2.cmml">C</mi><mo id="A4.5.p1.1.m1.1.1.1" xref="A4.5.p1.1.m1.1.1.1.cmml">/</mo><mi id="A4.5.p1.1.m1.1.1.3" xref="A4.5.p1.1.m1.1.1.3.cmml">U</mi></mrow><annotation-xml encoding="MathML-Content" id="A4.5.p1.1.m1.1b"><apply id="A4.5.p1.1.m1.1.1.cmml" xref="A4.5.p1.1.m1.1.1"><divide id="A4.5.p1.1.m1.1.1.1.cmml" xref="A4.5.p1.1.m1.1.1.1"></divide><ci id="A4.5.p1.1.m1.1.1.2.cmml" xref="A4.5.p1.1.m1.1.1.2">𝐶</ci><ci id="A4.5.p1.1.m1.1.1.3.cmml" xref="A4.5.p1.1.m1.1.1.3">𝑈</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.5.p1.1.m1.1c">C/U</annotation></semantics></math> subspaces of size <math id="A4.5.p1.2.m2.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A4.5.p1.2.m2.1a"><mi id="A4.5.p1.2.m2.1.1" xref="A4.5.p1.2.m2.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A4.5.p1.2.m2.1b"><ci id="A4.5.p1.2.m2.1.1.cmml" xref="A4.5.p1.2.m2.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.5.p1.2.m2.1c">U</annotation></semantics></math>, and the assumption that the errors in each subspace are independent.
∎</p>
</div>
</div>
<div id="A4.p4" class="ltx_para">
<p id="A4.p4.1" class="ltx_p">We also verified Theorem <a href="#A4.Thmtheorem1" title="Theorem D.1 (Bias of AISE). ‣ Appendix D Aggregation Using Pairwise Averages ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a> numerically by summing large numbers of integers drawn uniformly from the interval <math id="A4.p4.1.m1.3" class="ltx_Math" alttext="0,\ldots,255" display="inline"><semantics id="A4.p4.1.m1.3a"><mrow id="A4.p4.1.m1.3.4.2" xref="A4.p4.1.m1.3.4.1.cmml"><mn id="A4.p4.1.m1.1.1" xref="A4.p4.1.m1.1.1.cmml">0</mn><mo id="A4.p4.1.m1.3.4.2.1" xref="A4.p4.1.m1.3.4.1.cmml">,</mo><mi mathvariant="normal" id="A4.p4.1.m1.2.2" xref="A4.p4.1.m1.2.2.cmml">…</mi><mo id="A4.p4.1.m1.3.4.2.2" xref="A4.p4.1.m1.3.4.1.cmml">,</mo><mn id="A4.p4.1.m1.3.3" xref="A4.p4.1.m1.3.3.cmml">255</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.p4.1.m1.3b"><list id="A4.p4.1.m1.3.4.1.cmml" xref="A4.p4.1.m1.3.4.2"><cn type="integer" id="A4.p4.1.m1.1.1.cmml" xref="A4.p4.1.m1.1.1">0</cn><ci id="A4.p4.1.m1.2.2.cmml" xref="A4.p4.1.m1.2.2">…</ci><cn type="integer" id="A4.p4.1.m1.3.3.cmml" xref="A4.p4.1.m1.3.3">255</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.1.m1.3c">0,\ldots,255</annotation></semantics></math>.</p>
</div>
<div id="A4.p5" class="ltx_para">
<p id="A4.p5.1" class="ltx_p">Note that the assumption that the elements are independent is not especially strong in reality. This is because this section focuses on the effects on the least significant bits (which are the ones affected by each averaging operation), and the least significant bit does tend to be nearly uniformly random in a great deal of real-world data.</p>
</div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Additional Experimental Details</h2>

<section id="A5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.1 </span>Choice of Matrix Multiplication Tasks</h3>

<div id="A5.SS1.p1" class="ltx_para">
<p id="A5.SS1.p1.1" class="ltx_p">Because nearly all existing work on approximate matrix multiplication either focuses on special cases that do not satisfy our problem definition <cite class="ltx_cite ltx_citemacro_citep">(André et al., <a href="#bib.bib5" title="" class="ltx_ref">2019</a>; Jegou et al., <a href="#bib.bib41" title="" class="ltx_ref">2011</a>; Ge et al., <a href="#bib.bib29" title="" class="ltx_ref">2014</a>)</cite> or synthetic matrices, there is not a clear set of benchmark matrix multiply tasks to use. We therefore propose a collection of tasks that we believe are both reproducible and representative of many real-world matrices. To the best of our knowledge, our experiments use over an order of magnitude more matrices than any previous study.</p>
</div>
</section>
<section id="A5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.2 </span>Choice of Single-Threaded Benchmarks</h3>

<div id="A5.SS2.p1" class="ltx_para">
<p id="A5.SS2.p1.1" class="ltx_p">Given the ubiquity of GPUs and multicore CPUs, it may not be obvious why single-threaded experiments are desirable. There are a number of reasons we restrict our focus to CPUs and the single-threaded case:</p>
<ul id="A5.I1" class="ltx_itemize">
<li id="A5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i1.p1" class="ltx_para">
<p id="A5.I1.i1.p1.1" class="ltx_p">To enable fair comparisons to existing work, particularly the nearest rival, Bolt <cite class="ltx_cite ltx_citemacro_citep">(Blalock &amp; Guttag, <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>.</p>
</div>
</li>
<li id="A5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i2.p1" class="ltx_para">
<p id="A5.I1.i2.p1.1" class="ltx_p">To facilitate fair comparisons to our work by future authors—single-threaded experiments are much easier to reproduce and extend than multithreaded ones.</p>
</div>
</li>
<li id="A5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i3.p1" class="ltx_para">
<p id="A5.I1.i3.p1.1" class="ltx_p">Matrix multiplication is embarrassingly parallel with respect to rows of the input and columns of the output. There is therefore nothing “interesting” about how our method parallelizes relative to any other; all methods reduce to a single-threaded kernel that can easily be applied to disjoint submatrices. While we certainly could spend the considerable time required to construct and debug multicore benchmarks, this would be unlikely to yield any useful insights.</p>
</div>
</li>
<li id="A5.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i4.p1" class="ltx_para">
<p id="A5.I1.i4.p1.1" class="ltx_p">Parallelization in modern numerical libraries is often managed at a higher level than the lowest-level subroutines. For example, the authors of FBGEMM <cite class="ltx_cite ltx_citemacro_citep">(Khudia et al., <a href="#bib.bib45" title="" class="ltx_ref">2018</a>)</cite> state: <span id="A5.I1.i4.p1.1.1" class="ltx_text ltx_font_italic">“Internally, FBGEMM is intentionally designed not to create any threads. Usually, such a library is intended to be used as a backend by deep learning frameworks, such as PyTorch and Caffe2, that create and manage their own threads.”<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note"><span id="footnote3.1.1.1" class="ltx_text ltx_font_upright">3</span></span><span id="footnote3.5" class="ltx_text ltx_font_upright">https://engineering.fb.com/ml-applications/fbgemm/</span></span></span></span></span> I.e., a multithreaded library calls into single-threaded subroutines (such as a matrix multiplication function); it is this single-threaded subroutine where we make contributions, and therefore where we focus our experimental efforts. Independent of common practices in modern libraries, this pattern is also the only sensible strategy for small matrices, like many of those we consider—the overhead of launching and joining threads is extremely unlikely to be worth it for sufficiently small matrices. We could perhaps characterize where this breakpoint is, but this is a hardware-specific result that has little to do with our contributions.</p>
</div>
</li>
<li id="A5.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i5.p1" class="ltx_para">
<p id="A5.I1.i5.p1.1" class="ltx_p">While training of deep neural networks is typically done on GPUs or other accelerators, trained models (including, but not limited to, neural networks) are commonly deployed on smartphones with just CPUs and/or graphics acceleration that is no better than the CPU <cite class="ltx_cite ltx_citemacro_citep">(Wu et al., <a href="#bib.bib68" title="" class="ltx_ref">2019</a>)</cite>. Since most of the billions of smartphones in the world tend to be low-end or old, the need to deploy models on CPUs (including those with few cores) is unlikely to change for many years.</p>
</div>
</li>
<li id="A5.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i6.p1" class="ltx_para">
<p id="A5.I1.i6.p1.1" class="ltx_p">Creating, benchmarking, and analyzing a performant implementation of our method for GPUs would require a great deal of engineering work. We plan to create such an implementation in the future, but believe that the many empirical and theoretical results we currently have are more than adequate proof
of concept and already worth sharing with the community. </p>
</div>
</li>
</ul>
</div>
</section>
<section id="A5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.3 </span>SparsePCA Details</h3>

<div id="A5.SS3.p1" class="ltx_para">
<p id="A5.SS3.p1.3" class="ltx_p">We took steps to ensure that SparsePCA’s results were not hampered by insufficient hyperparameter tuning. First, for each matrix product, we tried a range of <math id="A5.SS3.p1.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A5.SS3.p1.1.m1.1a"><mi id="A5.SS3.p1.1.m1.1.1" xref="A5.SS3.p1.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A5.SS3.p1.1.m1.1b"><ci id="A5.SS3.p1.1.m1.1.1.cmml" xref="A5.SS3.p1.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS3.p1.1.m1.1c">\lambda</annotation></semantics></math> values which we found to encompass the full gamut of nearly 0% to nearly 100% sparsity: <math id="A5.SS3.p1.2.m2.6" class="ltx_Math" alttext="\lambda\in 2^{i},i\in\{-5,-4,-3,-2,-1,0,1,2,3\}" display="inline"><semantics id="A5.SS3.p1.2.m2.6a"><mrow id="A5.SS3.p1.2.m2.6.6.2" xref="A5.SS3.p1.2.m2.6.6.3.cmml"><mrow id="A5.SS3.p1.2.m2.5.5.1.1" xref="A5.SS3.p1.2.m2.5.5.1.1.cmml"><mi id="A5.SS3.p1.2.m2.5.5.1.1.2" xref="A5.SS3.p1.2.m2.5.5.1.1.2.cmml">λ</mi><mo id="A5.SS3.p1.2.m2.5.5.1.1.1" xref="A5.SS3.p1.2.m2.5.5.1.1.1.cmml">∈</mo><msup id="A5.SS3.p1.2.m2.5.5.1.1.3" xref="A5.SS3.p1.2.m2.5.5.1.1.3.cmml"><mn id="A5.SS3.p1.2.m2.5.5.1.1.3.2" xref="A5.SS3.p1.2.m2.5.5.1.1.3.2.cmml">2</mn><mi id="A5.SS3.p1.2.m2.5.5.1.1.3.3" xref="A5.SS3.p1.2.m2.5.5.1.1.3.3.cmml">i</mi></msup></mrow><mo id="A5.SS3.p1.2.m2.6.6.2.3" xref="A5.SS3.p1.2.m2.6.6.3a.cmml">,</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2" xref="A5.SS3.p1.2.m2.6.6.2.2.cmml"><mi id="A5.SS3.p1.2.m2.6.6.2.2.7" xref="A5.SS3.p1.2.m2.6.6.2.2.7.cmml">i</mi><mo id="A5.SS3.p1.2.m2.6.6.2.2.6" xref="A5.SS3.p1.2.m2.6.6.2.2.6.cmml">∈</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2.5.5" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml"><mo stretchy="false" id="A5.SS3.p1.2.m2.6.6.2.2.5.5.6" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">{</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2.1.1.1" xref="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.cmml"><mo id="A5.SS3.p1.2.m2.6.6.2.2.1.1.1a" xref="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.cmml">−</mo><mn id="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.2" xref="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.2.cmml">5</mn></mrow><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.7" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2.2.2.2" xref="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.cmml"><mo id="A5.SS3.p1.2.m2.6.6.2.2.2.2.2a" xref="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.cmml">−</mo><mn id="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.2" xref="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.2.cmml">4</mn></mrow><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.8" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2.3.3.3" xref="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.cmml"><mo id="A5.SS3.p1.2.m2.6.6.2.2.3.3.3a" xref="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.cmml">−</mo><mn id="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.2" xref="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.2.cmml">3</mn></mrow><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.9" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2.4.4.4" xref="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.cmml"><mo id="A5.SS3.p1.2.m2.6.6.2.2.4.4.4a" xref="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.cmml">−</mo><mn id="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.2" xref="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.2.cmml">2</mn></mrow><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.10" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mrow id="A5.SS3.p1.2.m2.6.6.2.2.5.5.5" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.cmml"><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.5a" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.cmml">−</mo><mn id="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.2" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.2.cmml">1</mn></mrow><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.11" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mn id="A5.SS3.p1.2.m2.1.1" xref="A5.SS3.p1.2.m2.1.1.cmml">0</mn><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.12" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mn id="A5.SS3.p1.2.m2.2.2" xref="A5.SS3.p1.2.m2.2.2.cmml">1</mn><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.13" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mn id="A5.SS3.p1.2.m2.3.3" xref="A5.SS3.p1.2.m2.3.3.cmml">2</mn><mo id="A5.SS3.p1.2.m2.6.6.2.2.5.5.14" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">,</mo><mn id="A5.SS3.p1.2.m2.4.4" xref="A5.SS3.p1.2.m2.4.4.cmml">3</mn><mo stretchy="false" id="A5.SS3.p1.2.m2.6.6.2.2.5.5.15" xref="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS3.p1.2.m2.6b"><apply id="A5.SS3.p1.2.m2.6.6.3.cmml" xref="A5.SS3.p1.2.m2.6.6.2"><csymbol cd="ambiguous" id="A5.SS3.p1.2.m2.6.6.3a.cmml" xref="A5.SS3.p1.2.m2.6.6.2.3">formulae-sequence</csymbol><apply id="A5.SS3.p1.2.m2.5.5.1.1.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1"><in id="A5.SS3.p1.2.m2.5.5.1.1.1.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1.1"></in><ci id="A5.SS3.p1.2.m2.5.5.1.1.2.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1.2">𝜆</ci><apply id="A5.SS3.p1.2.m2.5.5.1.1.3.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1.3"><csymbol cd="ambiguous" id="A5.SS3.p1.2.m2.5.5.1.1.3.1.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1.3">superscript</csymbol><cn type="integer" id="A5.SS3.p1.2.m2.5.5.1.1.3.2.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1.3.2">2</cn><ci id="A5.SS3.p1.2.m2.5.5.1.1.3.3.cmml" xref="A5.SS3.p1.2.m2.5.5.1.1.3.3">𝑖</ci></apply></apply><apply id="A5.SS3.p1.2.m2.6.6.2.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2"><in id="A5.SS3.p1.2.m2.6.6.2.2.6.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.6"></in><ci id="A5.SS3.p1.2.m2.6.6.2.2.7.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.7">𝑖</ci><set id="A5.SS3.p1.2.m2.6.6.2.2.5.6.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5"><apply id="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.1.1.1"><minus id="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.1.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.1.1.1"></minus><cn type="integer" id="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.1.1.1.2">5</cn></apply><apply id="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.2.2.2"><minus id="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.1.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.2.2.2"></minus><cn type="integer" id="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.2.2.2.2">4</cn></apply><apply id="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.3.3.3"><minus id="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.1.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.3.3.3"></minus><cn type="integer" id="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.3.3.3.2">3</cn></apply><apply id="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.4.4.4"><minus id="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.1.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.4.4.4"></minus><cn type="integer" id="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.4.4.4.2">2</cn></apply><apply id="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5.5"><minus id="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.1.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5.5"></minus><cn type="integer" id="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.2.cmml" xref="A5.SS3.p1.2.m2.6.6.2.2.5.5.5.2">1</cn></apply><cn type="integer" id="A5.SS3.p1.2.m2.1.1.cmml" xref="A5.SS3.p1.2.m2.1.1">0</cn><cn type="integer" id="A5.SS3.p1.2.m2.2.2.cmml" xref="A5.SS3.p1.2.m2.2.2">1</cn><cn type="integer" id="A5.SS3.p1.2.m2.3.3.cmml" xref="A5.SS3.p1.2.m2.3.3">2</cn><cn type="integer" id="A5.SS3.p1.2.m2.4.4.cmml" xref="A5.SS3.p1.2.m2.4.4">3</cn></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS3.p1.2.m2.6c">\lambda\in 2^{i},i\in\{-5,-4,-3,-2,-1,0,1,2,3\}</annotation></semantics></math>. Second, because different sparsity patterns may yield different execution times, we report not times from the single matrix SparsePCA produces for a given (<math id="A5.SS3.p1.3.m3.2" class="ltx_Math" alttext="d,\lambda" display="inline"><semantics id="A5.SS3.p1.3.m3.2a"><mrow id="A5.SS3.p1.3.m3.2.3.2" xref="A5.SS3.p1.3.m3.2.3.1.cmml"><mi id="A5.SS3.p1.3.m3.1.1" xref="A5.SS3.p1.3.m3.1.1.cmml">d</mi><mo id="A5.SS3.p1.3.m3.2.3.2.1" xref="A5.SS3.p1.3.m3.2.3.1.cmml">,</mo><mi id="A5.SS3.p1.3.m3.2.2" xref="A5.SS3.p1.3.m3.2.2.cmml">λ</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS3.p1.3.m3.2b"><list id="A5.SS3.p1.3.m3.2.3.1.cmml" xref="A5.SS3.p1.3.m3.2.3.2"><ci id="A5.SS3.p1.3.m3.1.1.cmml" xref="A5.SS3.p1.3.m3.1.1">𝑑</ci><ci id="A5.SS3.p1.3.m3.2.2.cmml" xref="A5.SS3.p1.3.m3.2.2">𝜆</ci></list></annotation-xml><annotation encoding="application/x-tex" id="A5.SS3.p1.3.m3.2c">d,\lambda</annotation></semantics></math>) pair, but the best times from any of 10 random matrices of the same size and at most the same sparsity. Finally and most importantly, we plot only the Pareto frontier of (speed, quality) pairs produced for a given matrix multiply. I.e., we let SparsePCA cherry-pick its best results on each individual matrix multiply.</p>
</div>
</section>
<section id="A5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.4 </span>Exact Matrix Multiplication</h3>

<div id="A5.SS4.p1" class="ltx_para">
<p id="A5.SS4.p1.1" class="ltx_p">We also implemented our own matrix product function specialized for tall, skinny matrices. In all cases, we report the timings based on the faster of this function and Eigen’s <cite class="ltx_cite ltx_citemacro_citep">(Guennebaud et al., <a href="#bib.bib32" title="" class="ltx_ref">2010</a>)</cite> matrix multiply function for a given matrix product.</p>
</div>
</section>
<section id="A5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.5 </span>Additional Baselines</h3>

<div id="A5.SS5.p1" class="ltx_para">
<p id="A5.SS5.p1.1" class="ltx_p">We also tested Frequent Directions / Fast Frequent Directions <cite class="ltx_cite ltx_citemacro_citep">(Liberty, <a href="#bib.bib49" title="" class="ltx_ref">2012</a>; Ghashami et al., <a href="#bib.bib31" title="" class="ltx_ref">2016</a>; Desai et al., <a href="#bib.bib20" title="" class="ltx_ref">2016</a>)</cite>, many variations of the sampling method of <cite class="ltx_cite ltx_citemacro_citet">Drineas et al. (<a href="#bib.bib21" title="" class="ltx_ref">2006a</a>)</cite>, projection using orthogonalized Gaussian random matrices <cite class="ltx_cite ltx_citemacro_citep">(Ji et al., <a href="#bib.bib42" title="" class="ltx_ref">2012</a>)</cite>, projection using matrices of scaled i.i.d. Rademacher random variables <cite class="ltx_cite ltx_citemacro_citep">(Achlioptas, <a href="#bib.bib2" title="" class="ltx_ref">2001</a>)</cite>, projection using orthonormalized matrices of Rademacher random variables, the co-occurring directions sketch <cite class="ltx_cite ltx_citemacro_citep">(Mroueh et al., <a href="#bib.bib56" title="" class="ltx_ref">2016</a>)</cite>, OSNAP <cite class="ltx_cite ltx_citemacro_citep">(Nelson &amp; Nguyên, <a href="#bib.bib57" title="" class="ltx_ref">2013</a>)</cite>, Product Quantization <cite class="ltx_cite ltx_citemacro_citep">(Jegou et al., <a href="#bib.bib41" title="" class="ltx_ref">2011</a>)</cite>, and Optimized Product Quantization <cite class="ltx_cite ltx_citemacro_citep">(Ge et al., <a href="#bib.bib29" title="" class="ltx_ref">2014</a>)</cite>.</p>
</div>
<div id="A5.SS5.p2" class="ltx_para">
<p id="A5.SS5.p2.1" class="ltx_p">The poor performance of many of these methods is unsurprising in our setting. Given that we have access to a training set on which to learn the true principal components, the Eckart-Young-Mirsky theorem <cite class="ltx_cite ltx_citemacro_citep">(Eckart &amp; Young, <a href="#bib.bib25" title="" class="ltx_ref">1936</a>)</cite> indicates that PCA should outperform any other individual matrix sketching method employing dense projection matrices, at least in the limit of infinite training data. Also, since PQ and OPQ use 256 dense centroids (except in the Bolt / QuickerADC variations), it is also impossible for them to perform well when <math id="A5.SS5.p2.1.m1.3" class="ltx_Math" alttext="\min(D,M)" display="inline"><semantics id="A5.SS5.p2.1.m1.3a"><mrow id="A5.SS5.p2.1.m1.3.4.2" xref="A5.SS5.p2.1.m1.3.4.1.cmml"><mi id="A5.SS5.p2.1.m1.1.1" xref="A5.SS5.p2.1.m1.1.1.cmml">min</mi><mo id="A5.SS5.p2.1.m1.3.4.2a" xref="A5.SS5.p2.1.m1.3.4.1.cmml">⁡</mo><mrow id="A5.SS5.p2.1.m1.3.4.2.1" xref="A5.SS5.p2.1.m1.3.4.1.cmml"><mo stretchy="false" id="A5.SS5.p2.1.m1.3.4.2.1.1" xref="A5.SS5.p2.1.m1.3.4.1.cmml">(</mo><mi id="A5.SS5.p2.1.m1.2.2" xref="A5.SS5.p2.1.m1.2.2.cmml">D</mi><mo id="A5.SS5.p2.1.m1.3.4.2.1.2" xref="A5.SS5.p2.1.m1.3.4.1.cmml">,</mo><mi id="A5.SS5.p2.1.m1.3.3" xref="A5.SS5.p2.1.m1.3.3.cmml">M</mi><mo stretchy="false" id="A5.SS5.p2.1.m1.3.4.2.1.3" xref="A5.SS5.p2.1.m1.3.4.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.1.m1.3b"><apply id="A5.SS5.p2.1.m1.3.4.1.cmml" xref="A5.SS5.p2.1.m1.3.4.2"><min id="A5.SS5.p2.1.m1.1.1.cmml" xref="A5.SS5.p2.1.m1.1.1"></min><ci id="A5.SS5.p2.1.m1.2.2.cmml" xref="A5.SS5.p2.1.m1.2.2">𝐷</ci><ci id="A5.SS5.p2.1.m1.3.3.cmml" xref="A5.SS5.p2.1.m1.3.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.1.m1.3c">\min(D,M)</annotation></semantics></math> is not significantly larger than 256.</p>
</div>
</section>
<section id="A5.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.6 </span>UCR Time Series Archive</h3>

<div id="A5.SS6.p1" class="ltx_para">
<p id="A5.SS6.p1.1" class="ltx_p">We set the number of returned neighbors to 128 (results with 64 and 256 were similar). We omitted datasets with fewer than 128 training examples, since it is not possible for Stochastic Neighbor Compression to draw 128 samples without replacement in this case.</p>
</div>
<div id="A5.SS6.p2" class="ltx_para">
<p id="A5.SS6.p2.6" class="ltx_p">In addition to being a large, public corpus of over a hundred datasets from a huge variety of different domains, the UCR Time Series Archive also has the advantage that it can be used to produce matrix multiplication tasks of a fixed size. This is necessary for meaningful comparison of speed versus accuracy tradeoffs across datasets. We constructed training and test matrices <math id="A5.SS6.p2.1.m1.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="A5.SS6.p2.1.m1.1a"><mover accent="true" id="A5.SS6.p2.1.m1.1.1" xref="A5.SS6.p2.1.m1.1.1.cmml"><mi id="A5.SS6.p2.1.m1.1.1.2" xref="A5.SS6.p2.1.m1.1.1.2.cmml">𝑨</mi><mo id="A5.SS6.p2.1.m1.1.1.1" xref="A5.SS6.p2.1.m1.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.1.m1.1b"><apply id="A5.SS6.p2.1.m1.1.1.cmml" xref="A5.SS6.p2.1.m1.1.1"><ci id="A5.SS6.p2.1.m1.1.1.1.cmml" xref="A5.SS6.p2.1.m1.1.1.1">~</ci><ci id="A5.SS6.p2.1.m1.1.1.2.cmml" xref="A5.SS6.p2.1.m1.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.1.m1.1c">\tilde{\bm{A}}</annotation></semantics></math> and <math id="A5.SS6.p2.2.m2.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="A5.SS6.p2.2.m2.1a"><mi id="A5.SS6.p2.2.m2.1.1" xref="A5.SS6.p2.2.m2.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.2.m2.1b"><ci id="A5.SS6.p2.2.m2.1.1.cmml" xref="A5.SS6.p2.2.m2.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.2.m2.1c">\bm{A}</annotation></semantics></math> by resampling each time series in each dataset’s train and test set to a length of <math id="A5.SS6.p2.3.m3.1" class="ltx_Math" alttext="320" display="inline"><semantics id="A5.SS6.p2.3.m3.1a"><mn id="A5.SS6.p2.3.m3.1.1" xref="A5.SS6.p2.3.m3.1.1.cmml">320</mn><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.3.m3.1b"><cn type="integer" id="A5.SS6.p2.3.m3.1.1.cmml" xref="A5.SS6.p2.3.m3.1.1">320</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.3.m3.1c">320</annotation></semantics></math> (the closest multiple of 32 to the median length of 310). We obtained the matrix <math id="A5.SS6.p2.4.m4.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A5.SS6.p2.4.m4.1a"><mi id="A5.SS6.p2.4.m4.1.1" xref="A5.SS6.p2.4.m4.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.4.m4.1b"><ci id="A5.SS6.p2.4.m4.1.1.cmml" xref="A5.SS6.p2.4.m4.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.4.m4.1c">\bm{B}</annotation></semantics></math> for each dataset by running Stochastic Neighbor Compression <cite class="ltx_cite ltx_citemacro_citep">(Kusner et al., <a href="#bib.bib47" title="" class="ltx_ref">2014</a>)</cite> on the training set with an RBF kernel of bandwidth one. We set the number of returned neighbors to 128 (results with 64 and 256 were similar), yielding a <math id="A5.SS6.p2.5.m5.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A5.SS6.p2.5.m5.1a"><mi id="A5.SS6.p2.5.m5.1.1" xref="A5.SS6.p2.5.m5.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.5.m5.1b"><ci id="A5.SS6.p2.5.m5.1.1.cmml" xref="A5.SS6.p2.5.m5.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.5.m5.1c">\bm{B}</annotation></semantics></math> matrix of size <math id="A5.SS6.p2.6.m6.1" class="ltx_Math" alttext="320\times 128" display="inline"><semantics id="A5.SS6.p2.6.m6.1a"><mrow id="A5.SS6.p2.6.m6.1.1" xref="A5.SS6.p2.6.m6.1.1.cmml"><mn id="A5.SS6.p2.6.m6.1.1.2" xref="A5.SS6.p2.6.m6.1.1.2.cmml">320</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS6.p2.6.m6.1.1.1" xref="A5.SS6.p2.6.m6.1.1.1.cmml">×</mo><mn id="A5.SS6.p2.6.m6.1.1.3" xref="A5.SS6.p2.6.m6.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.6.m6.1b"><apply id="A5.SS6.p2.6.m6.1.1.cmml" xref="A5.SS6.p2.6.m6.1.1"><times id="A5.SS6.p2.6.m6.1.1.1.cmml" xref="A5.SS6.p2.6.m6.1.1.1"></times><cn type="integer" id="A5.SS6.p2.6.m6.1.1.2.cmml" xref="A5.SS6.p2.6.m6.1.1.2">320</cn><cn type="integer" id="A5.SS6.p2.6.m6.1.1.3.cmml" xref="A5.SS6.p2.6.m6.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.6.m6.1c">320\times 128</annotation></semantics></math>. Since different datasets have different test set sizes, all results are for a standardized test set size of 1000 rows. We wanted the length to be a multiple of 32 since existing methods operate best with sizes that are either powers of two or, failing that, multiples of large powers of two.</p>
</div>
<div id="A5.SS6.p3" class="ltx_para">
<p id="A5.SS6.p3.6" class="ltx_p">We approximate Euclidean distances using the identity <math id="A5.SS6.p3.1.m1.3" class="ltx_Math" alttext="\lVert\bm{x}-\bm{y}\rVert_{2}^{2}=\lVert\bm{x}\rVert_{2}^{2}-2\bm{x}^{\top}\bm{y}+\lVert\bm{y}\rVert_{2}^{2}" display="inline"><semantics id="A5.SS6.p3.1.m1.3a"><mrow id="A5.SS6.p3.1.m1.3.3" xref="A5.SS6.p3.1.m1.3.3.cmml"><msubsup id="A5.SS6.p3.1.m1.3.3.1" xref="A5.SS6.p3.1.m1.3.3.1.cmml"><mrow id="A5.SS6.p3.1.m1.3.3.1.1.1.1" xref="A5.SS6.p3.1.m1.3.3.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="A5.SS6.p3.1.m1.3.3.1.1.1.1.2" xref="A5.SS6.p3.1.m1.3.3.1.1.1.2.1.cmml">∥</mo><mrow id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.cmml"><mi id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.2" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.2.cmml">𝒙</mi><mo id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.1" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.1.cmml">−</mo><mi id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.3" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.3.cmml">𝒚</mi></mrow><mo fence="true" lspace="0em" id="A5.SS6.p3.1.m1.3.3.1.1.1.1.3" xref="A5.SS6.p3.1.m1.3.3.1.1.1.2.1.cmml">∥</mo></mrow><mn id="A5.SS6.p3.1.m1.3.3.1.1.3" xref="A5.SS6.p3.1.m1.3.3.1.1.3.cmml">2</mn><mn id="A5.SS6.p3.1.m1.3.3.1.3" xref="A5.SS6.p3.1.m1.3.3.1.3.cmml">2</mn></msubsup><mo rspace="0.1389em" id="A5.SS6.p3.1.m1.3.3.2" xref="A5.SS6.p3.1.m1.3.3.2.cmml">=</mo><mrow id="A5.SS6.p3.1.m1.3.3.3" xref="A5.SS6.p3.1.m1.3.3.3.cmml"><mrow id="A5.SS6.p3.1.m1.3.3.3.2" xref="A5.SS6.p3.1.m1.3.3.3.2.cmml"><msubsup id="A5.SS6.p3.1.m1.3.3.3.2.2" xref="A5.SS6.p3.1.m1.3.3.3.2.2.cmml"><mrow id="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.2" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.1.cmml"><mo fence="true" lspace="0.1389em" rspace="0em" id="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.2.1" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.1.1.cmml">∥</mo><mi id="A5.SS6.p3.1.m1.1.1" xref="A5.SS6.p3.1.m1.1.1.cmml">𝒙</mi><mo fence="true" lspace="0em" rspace="0em" id="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.2.2" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.1.1.cmml">∥</mo></mrow><mn id="A5.SS6.p3.1.m1.3.3.3.2.2.2.3" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.3.cmml">2</mn><mn id="A5.SS6.p3.1.m1.3.3.3.2.2.3" xref="A5.SS6.p3.1.m1.3.3.3.2.2.3.cmml">2</mn></msubsup><mo id="A5.SS6.p3.1.m1.3.3.3.2.1" xref="A5.SS6.p3.1.m1.3.3.3.2.1.cmml">−</mo><mrow id="A5.SS6.p3.1.m1.3.3.3.2.3" xref="A5.SS6.p3.1.m1.3.3.3.2.3.cmml"><mn id="A5.SS6.p3.1.m1.3.3.3.2.3.2" xref="A5.SS6.p3.1.m1.3.3.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.SS6.p3.1.m1.3.3.3.2.3.1" xref="A5.SS6.p3.1.m1.3.3.3.2.3.1.cmml">​</mo><msup id="A5.SS6.p3.1.m1.3.3.3.2.3.3" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3.cmml"><mi id="A5.SS6.p3.1.m1.3.3.3.2.3.3.2" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3.2.cmml">𝒙</mi><mo id="A5.SS6.p3.1.m1.3.3.3.2.3.3.3" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS6.p3.1.m1.3.3.3.2.3.1a" xref="A5.SS6.p3.1.m1.3.3.3.2.3.1.cmml">​</mo><mi id="A5.SS6.p3.1.m1.3.3.3.2.3.4" xref="A5.SS6.p3.1.m1.3.3.3.2.3.4.cmml">𝒚</mi></mrow></mrow><mo id="A5.SS6.p3.1.m1.3.3.3.1" xref="A5.SS6.p3.1.m1.3.3.3.1.cmml">+</mo><msubsup id="A5.SS6.p3.1.m1.3.3.3.3" xref="A5.SS6.p3.1.m1.3.3.3.3.cmml"><mrow id="A5.SS6.p3.1.m1.3.3.3.3.2.2.2" xref="A5.SS6.p3.1.m1.3.3.3.3.2.2.1.cmml"><mo fence="true" lspace="0em" rspace="0em" id="A5.SS6.p3.1.m1.3.3.3.3.2.2.2.1" xref="A5.SS6.p3.1.m1.3.3.3.3.2.2.1.1.cmml">∥</mo><mi id="A5.SS6.p3.1.m1.2.2" xref="A5.SS6.p3.1.m1.2.2.cmml">𝒚</mi><mo fence="true" lspace="0em" id="A5.SS6.p3.1.m1.3.3.3.3.2.2.2.2" xref="A5.SS6.p3.1.m1.3.3.3.3.2.2.1.1.cmml">∥</mo></mrow><mn id="A5.SS6.p3.1.m1.3.3.3.3.2.3" xref="A5.SS6.p3.1.m1.3.3.3.3.2.3.cmml">2</mn><mn id="A5.SS6.p3.1.m1.3.3.3.3.3" xref="A5.SS6.p3.1.m1.3.3.3.3.3.cmml">2</mn></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.1.m1.3b"><apply id="A5.SS6.p3.1.m1.3.3.cmml" xref="A5.SS6.p3.1.m1.3.3"><eq id="A5.SS6.p3.1.m1.3.3.2.cmml" xref="A5.SS6.p3.1.m1.3.3.2"></eq><apply id="A5.SS6.p3.1.m1.3.3.1.cmml" xref="A5.SS6.p3.1.m1.3.3.1"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.1.2.cmml" xref="A5.SS6.p3.1.m1.3.3.1">superscript</csymbol><apply id="A5.SS6.p3.1.m1.3.3.1.1.cmml" xref="A5.SS6.p3.1.m1.3.3.1"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.1.1.2.cmml" xref="A5.SS6.p3.1.m1.3.3.1">subscript</csymbol><apply id="A5.SS6.p3.1.m1.3.3.1.1.1.2.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1"><csymbol cd="latexml" id="A5.SS6.p3.1.m1.3.3.1.1.1.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.2">delimited-∥∥</csymbol><apply id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1"><minus id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.1.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.1"></minus><ci id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.2.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.2">𝒙</ci><ci id="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.3.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.1.1.1.3">𝒚</ci></apply></apply><cn type="integer" id="A5.SS6.p3.1.m1.3.3.1.1.3.cmml" xref="A5.SS6.p3.1.m1.3.3.1.1.3">2</cn></apply><cn type="integer" id="A5.SS6.p3.1.m1.3.3.1.3.cmml" xref="A5.SS6.p3.1.m1.3.3.1.3">2</cn></apply><apply id="A5.SS6.p3.1.m1.3.3.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3"><plus id="A5.SS6.p3.1.m1.3.3.3.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.1"></plus><apply id="A5.SS6.p3.1.m1.3.3.3.2.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2"><minus id="A5.SS6.p3.1.m1.3.3.3.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.1"></minus><apply id="A5.SS6.p3.1.m1.3.3.3.2.2.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.3.2.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2">superscript</csymbol><apply id="A5.SS6.p3.1.m1.3.3.3.2.2.2.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.3.2.2.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2">subscript</csymbol><apply id="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.2"><csymbol cd="latexml" id="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.1.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.2.2.1">delimited-∥∥</csymbol><ci id="A5.SS6.p3.1.m1.1.1.cmml" xref="A5.SS6.p3.1.m1.1.1">𝒙</ci></apply><cn type="integer" id="A5.SS6.p3.1.m1.3.3.3.2.2.2.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2.2.3">2</cn></apply><cn type="integer" id="A5.SS6.p3.1.m1.3.3.3.2.2.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.2.3">2</cn></apply><apply id="A5.SS6.p3.1.m1.3.3.3.2.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3"><times id="A5.SS6.p3.1.m1.3.3.3.2.3.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.1"></times><cn type="integer" id="A5.SS6.p3.1.m1.3.3.3.2.3.2.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.2">2</cn><apply id="A5.SS6.p3.1.m1.3.3.3.2.3.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.3.2.3.3.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3">superscript</csymbol><ci id="A5.SS6.p3.1.m1.3.3.3.2.3.3.2.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3.2">𝒙</ci><csymbol cd="latexml" id="A5.SS6.p3.1.m1.3.3.3.2.3.3.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.3.3">top</csymbol></apply><ci id="A5.SS6.p3.1.m1.3.3.3.2.3.4.cmml" xref="A5.SS6.p3.1.m1.3.3.3.2.3.4">𝒚</ci></apply></apply><apply id="A5.SS6.p3.1.m1.3.3.3.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.3.3.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3">superscript</csymbol><apply id="A5.SS6.p3.1.m1.3.3.3.3.2.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="A5.SS6.p3.1.m1.3.3.3.3.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3">subscript</csymbol><apply id="A5.SS6.p3.1.m1.3.3.3.3.2.2.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3.2.2.2"><csymbol cd="latexml" id="A5.SS6.p3.1.m1.3.3.3.3.2.2.1.1.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3.2.2.2.1">delimited-∥∥</csymbol><ci id="A5.SS6.p3.1.m1.2.2.cmml" xref="A5.SS6.p3.1.m1.2.2">𝒚</ci></apply><cn type="integer" id="A5.SS6.p3.1.m1.3.3.3.3.2.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3.2.3">2</cn></apply><cn type="integer" id="A5.SS6.p3.1.m1.3.3.3.3.3.cmml" xref="A5.SS6.p3.1.m1.3.3.3.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.1.m1.3c">\lVert\bm{x}-\bm{y}\rVert_{2}^{2}=\lVert\bm{x}\rVert_{2}^{2}-2\bm{x}^{\top}\bm{y}+\lVert\bm{y}\rVert_{2}^{2}</annotation></semantics></math>. We approximate only the inner products <math id="A5.SS6.p3.2.m2.1" class="ltx_Math" alttext="\bm{x}^{\top}\bm{y}" display="inline"><semantics id="A5.SS6.p3.2.m2.1a"><mrow id="A5.SS6.p3.2.m2.1.1" xref="A5.SS6.p3.2.m2.1.1.cmml"><msup id="A5.SS6.p3.2.m2.1.1.2" xref="A5.SS6.p3.2.m2.1.1.2.cmml"><mi id="A5.SS6.p3.2.m2.1.1.2.2" xref="A5.SS6.p3.2.m2.1.1.2.2.cmml">𝒙</mi><mo id="A5.SS6.p3.2.m2.1.1.2.3" xref="A5.SS6.p3.2.m2.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS6.p3.2.m2.1.1.1" xref="A5.SS6.p3.2.m2.1.1.1.cmml">​</mo><mi id="A5.SS6.p3.2.m2.1.1.3" xref="A5.SS6.p3.2.m2.1.1.3.cmml">𝒚</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.2.m2.1b"><apply id="A5.SS6.p3.2.m2.1.1.cmml" xref="A5.SS6.p3.2.m2.1.1"><times id="A5.SS6.p3.2.m2.1.1.1.cmml" xref="A5.SS6.p3.2.m2.1.1.1"></times><apply id="A5.SS6.p3.2.m2.1.1.2.cmml" xref="A5.SS6.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="A5.SS6.p3.2.m2.1.1.2.1.cmml" xref="A5.SS6.p3.2.m2.1.1.2">superscript</csymbol><ci id="A5.SS6.p3.2.m2.1.1.2.2.cmml" xref="A5.SS6.p3.2.m2.1.1.2.2">𝒙</ci><csymbol cd="latexml" id="A5.SS6.p3.2.m2.1.1.2.3.cmml" xref="A5.SS6.p3.2.m2.1.1.2.3">top</csymbol></apply><ci id="A5.SS6.p3.2.m2.1.1.3.cmml" xref="A5.SS6.p3.2.m2.1.1.3">𝒚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.2.m2.1c">\bm{x}^{\top}\bm{y}</annotation></semantics></math>, since <math id="A5.SS6.p3.3.m3.1" class="ltx_Math" alttext="\lVert\bm{y}\rVert_{2}^{2}" display="inline"><semantics id="A5.SS6.p3.3.m3.1a"><msubsup id="A5.SS6.p3.3.m3.1.2" xref="A5.SS6.p3.3.m3.1.2.cmml"><mrow id="A5.SS6.p3.3.m3.1.2.2.2.2" xref="A5.SS6.p3.3.m3.1.2.2.2.1.cmml"><mo fence="true" rspace="0em" id="A5.SS6.p3.3.m3.1.2.2.2.2.1" xref="A5.SS6.p3.3.m3.1.2.2.2.1.1.cmml">∥</mo><mi id="A5.SS6.p3.3.m3.1.1" xref="A5.SS6.p3.3.m3.1.1.cmml">𝒚</mi><mo fence="true" lspace="0em" id="A5.SS6.p3.3.m3.1.2.2.2.2.2" xref="A5.SS6.p3.3.m3.1.2.2.2.1.1.cmml">∥</mo></mrow><mn id="A5.SS6.p3.3.m3.1.2.2.3" xref="A5.SS6.p3.3.m3.1.2.2.3.cmml">2</mn><mn id="A5.SS6.p3.3.m3.1.2.3" xref="A5.SS6.p3.3.m3.1.2.3.cmml">2</mn></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.3.m3.1b"><apply id="A5.SS6.p3.3.m3.1.2.cmml" xref="A5.SS6.p3.3.m3.1.2"><csymbol cd="ambiguous" id="A5.SS6.p3.3.m3.1.2.1.cmml" xref="A5.SS6.p3.3.m3.1.2">superscript</csymbol><apply id="A5.SS6.p3.3.m3.1.2.2.cmml" xref="A5.SS6.p3.3.m3.1.2"><csymbol cd="ambiguous" id="A5.SS6.p3.3.m3.1.2.2.1.cmml" xref="A5.SS6.p3.3.m3.1.2">subscript</csymbol><apply id="A5.SS6.p3.3.m3.1.2.2.2.1.cmml" xref="A5.SS6.p3.3.m3.1.2.2.2.2"><csymbol cd="latexml" id="A5.SS6.p3.3.m3.1.2.2.2.1.1.cmml" xref="A5.SS6.p3.3.m3.1.2.2.2.2.1">delimited-∥∥</csymbol><ci id="A5.SS6.p3.3.m3.1.1.cmml" xref="A5.SS6.p3.3.m3.1.1">𝒚</ci></apply><cn type="integer" id="A5.SS6.p3.3.m3.1.2.2.3.cmml" xref="A5.SS6.p3.3.m3.1.2.2.3">2</cn></apply><cn type="integer" id="A5.SS6.p3.3.m3.1.2.3.cmml" xref="A5.SS6.p3.3.m3.1.2.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.3.m3.1c">\lVert\bm{y}\rVert_{2}^{2}</annotation></semantics></math> can be precomputed for fixed exemplars <math id="A5.SS6.p3.4.m4.1" class="ltx_Math" alttext="\bm{y}" display="inline"><semantics id="A5.SS6.p3.4.m4.1a"><mi id="A5.SS6.p3.4.m4.1.1" xref="A5.SS6.p3.4.m4.1.1.cmml">𝒚</mi><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.4.m4.1b"><ci id="A5.SS6.p3.4.m4.1.1.cmml" xref="A5.SS6.p3.4.m4.1.1">𝒚</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.4.m4.1c">\bm{y}</annotation></semantics></math> and <math id="A5.SS6.p3.5.m5.1" class="ltx_Math" alttext="\lVert\bm{x}\rVert_{2}^{2}" display="inline"><semantics id="A5.SS6.p3.5.m5.1a"><msubsup id="A5.SS6.p3.5.m5.1.2" xref="A5.SS6.p3.5.m5.1.2.cmml"><mrow id="A5.SS6.p3.5.m5.1.2.2.2.2" xref="A5.SS6.p3.5.m5.1.2.2.2.1.cmml"><mo fence="true" rspace="0em" id="A5.SS6.p3.5.m5.1.2.2.2.2.1" xref="A5.SS6.p3.5.m5.1.2.2.2.1.1.cmml">∥</mo><mi id="A5.SS6.p3.5.m5.1.1" xref="A5.SS6.p3.5.m5.1.1.cmml">𝒙</mi><mo fence="true" lspace="0em" id="A5.SS6.p3.5.m5.1.2.2.2.2.2" xref="A5.SS6.p3.5.m5.1.2.2.2.1.1.cmml">∥</mo></mrow><mn id="A5.SS6.p3.5.m5.1.2.2.3" xref="A5.SS6.p3.5.m5.1.2.2.3.cmml">2</mn><mn id="A5.SS6.p3.5.m5.1.2.3" xref="A5.SS6.p3.5.m5.1.2.3.cmml">2</mn></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.5.m5.1b"><apply id="A5.SS6.p3.5.m5.1.2.cmml" xref="A5.SS6.p3.5.m5.1.2"><csymbol cd="ambiguous" id="A5.SS6.p3.5.m5.1.2.1.cmml" xref="A5.SS6.p3.5.m5.1.2">superscript</csymbol><apply id="A5.SS6.p3.5.m5.1.2.2.cmml" xref="A5.SS6.p3.5.m5.1.2"><csymbol cd="ambiguous" id="A5.SS6.p3.5.m5.1.2.2.1.cmml" xref="A5.SS6.p3.5.m5.1.2">subscript</csymbol><apply id="A5.SS6.p3.5.m5.1.2.2.2.1.cmml" xref="A5.SS6.p3.5.m5.1.2.2.2.2"><csymbol cd="latexml" id="A5.SS6.p3.5.m5.1.2.2.2.1.1.cmml" xref="A5.SS6.p3.5.m5.1.2.2.2.2.1">delimited-∥∥</csymbol><ci id="A5.SS6.p3.5.m5.1.1.cmml" xref="A5.SS6.p3.5.m5.1.1">𝒙</ci></apply><cn type="integer" id="A5.SS6.p3.5.m5.1.2.2.3.cmml" xref="A5.SS6.p3.5.m5.1.2.2.3">2</cn></apply><cn type="integer" id="A5.SS6.p3.5.m5.1.2.3.cmml" xref="A5.SS6.p3.5.m5.1.2.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.5.m5.1c">\lVert\bm{x}\rVert_{2}^{2}</annotation></semantics></math> doesn’t affect the class prediction since it is constant across all exemplars for a given input <math id="A5.SS6.p3.6.m6.1" class="ltx_Math" alttext="\bm{x}" display="inline"><semantics id="A5.SS6.p3.6.m6.1a"><mi id="A5.SS6.p3.6.m6.1.1" xref="A5.SS6.p3.6.m6.1.1.cmml">𝒙</mi><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.6.m6.1b"><ci id="A5.SS6.p3.6.m6.1.1.cmml" xref="A5.SS6.p3.6.m6.1.1">𝒙</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.6.m6.1c">\bm{x}</annotation></semantics></math>.</p>
</div>
</section>
<section id="A5.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.7 </span>Caltech101</h3>

<div id="A5.SS7.p1" class="ltx_para">
<p id="A5.SS7.p1.1" class="ltx_p">We only extracted valid windows—i.e., never past the edge of an image. We extracted the windows in CHW order, meaning that scalars from the same color channel were placed at contiguous indices. The “first” images are based on filename in lexicographic order.</p>
</div>
<div id="A5.SS7.p2" class="ltx_para">
<p id="A5.SS7.p2.1" class="ltx_p">We used pairs of filters because using a single filter would mean timing a matrix-vector product instead of a matrix-matrix product.</p>
</div>
<div id="A5.SS7.p3" class="ltx_para">
<p id="A5.SS7.p3.5" class="ltx_p">To allow meaningful speed comparisons across images, we resized and center-cropped each image to <math id="A5.SS7.p3.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="A5.SS7.p3.1.m1.1a"><mrow id="A5.SS7.p3.1.m1.1.1" xref="A5.SS7.p3.1.m1.1.1.cmml"><mn id="A5.SS7.p3.1.m1.1.1.2" xref="A5.SS7.p3.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS7.p3.1.m1.1.1.1" xref="A5.SS7.p3.1.m1.1.1.1.cmml">×</mo><mn id="A5.SS7.p3.1.m1.1.1.3" xref="A5.SS7.p3.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p3.1.m1.1b"><apply id="A5.SS7.p3.1.m1.1.1.cmml" xref="A5.SS7.p3.1.m1.1.1"><times id="A5.SS7.p3.1.m1.1.1.1.cmml" xref="A5.SS7.p3.1.m1.1.1.1"></times><cn type="integer" id="A5.SS7.p3.1.m1.1.1.2.cmml" xref="A5.SS7.p3.1.m1.1.1.2">224</cn><cn type="integer" id="A5.SS7.p3.1.m1.1.1.3.cmml" xref="A5.SS7.p3.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p3.1.m1.1c">224\times 224</annotation></semantics></math> as commonly done in image classification pipelines <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib35" title="" class="ltx_ref">2016a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>; Huang et al., <a href="#bib.bib37" title="" class="ltx_ref">2017</a>)</cite>. We then extracted sliding windows of the appropriate size and used each (flattened) window as one row of <math id="A5.SS7.p3.2.m2.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="A5.SS7.p3.2.m2.1a"><mover accent="true" id="A5.SS7.p3.2.m2.1.1" xref="A5.SS7.p3.2.m2.1.1.cmml"><mi id="A5.SS7.p3.2.m2.1.1.2" xref="A5.SS7.p3.2.m2.1.1.2.cmml">𝑨</mi><mo id="A5.SS7.p3.2.m2.1.1.1" xref="A5.SS7.p3.2.m2.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="A5.SS7.p3.2.m2.1b"><apply id="A5.SS7.p3.2.m2.1.1.cmml" xref="A5.SS7.p3.2.m2.1.1"><ci id="A5.SS7.p3.2.m2.1.1.1.cmml" xref="A5.SS7.p3.2.m2.1.1.1">~</ci><ci id="A5.SS7.p3.2.m2.1.1.2.cmml" xref="A5.SS7.p3.2.m2.1.1.2">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p3.2.m2.1c">\tilde{\bm{A}}</annotation></semantics></math> or <math id="A5.SS7.p3.3.m3.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="A5.SS7.p3.3.m3.1a"><mi id="A5.SS7.p3.3.m3.1.1" xref="A5.SS7.p3.3.m3.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p3.3.m3.1b"><ci id="A5.SS7.p3.3.m3.1.1.cmml" xref="A5.SS7.p3.3.m3.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p3.3.m3.1c">\bm{A}</annotation></semantics></math>. We similarly flattened the filters, with each set of coefficients forming one column of <math id="A5.SS7.p3.4.m4.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A5.SS7.p3.4.m4.1a"><mi id="A5.SS7.p3.4.m4.1.1" xref="A5.SS7.p3.4.m4.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p3.4.m4.1b"><ci id="A5.SS7.p3.4.m4.1.1.cmml" xref="A5.SS7.p3.4.m4.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p3.4.m4.1c">\bm{B}</annotation></semantics></math>. In both cases, <math id="A5.SS7.p3.5.m5.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A5.SS7.p3.5.m5.1a"><mi id="A5.SS7.p3.5.m5.1.1" xref="A5.SS7.p3.5.m5.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p3.5.m5.1b"><ci id="A5.SS7.p3.5.m5.1.1.cmml" xref="A5.SS7.p3.5.m5.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p3.5.m5.1c">\bm{B}</annotation></semantics></math> has two columns—this is because using a single filter would mean timing a matrix-vector product instead of a matrix-matrix product. Two columns also made sense since Sobel filters are often used in horizontal and vertical pairings, and Gaussian filters are often used together to perform difference-of-Gaussians transforms.</p>
</div>
<div id="A5.SS7.p4" class="ltx_para">
<p id="A5.SS7.p4.1" class="ltx_p">Even though the RGB values at each position are naturally unsigned 8-bit integers, we allowed all rival methods to operate on them as 32-bit floating point, without including the conversion when timing them. Because it only requires checking whether values are above a threshold, <span id="A5.SS7.p4.1.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A5.SS7.p4.1.1.1" class="ltx_text ltx_font_upright"> </span></span>can operate on 8-bit data directly.</p>
</div>
</section>
<section id="A5.SS8" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.8 </span>Why Not Speed Up Whole Neural Nets?</h3>

<div id="A5.SS8.p1" class="ltx_para">
<p id="A5.SS8.p1.1" class="ltx_p">Using our ideas to accelerate overall neural networks and other layer types would be a valuable contribution. In fact, we are actively working on this problem. However, as we state in the introduction and problem statement, our focus in this paper is <span id="A5.SS8.p1.1.1" class="ltx_text ltx_font_italic">approximate matrix multiplication</span> (AMM) and we deliberately make no claim about accelerating entire neural networks or convolutional layers. We limit our scope in this way for several reasons:</p>
<ol id="A5.I2" class="ltx_enumerate">
<li id="A5.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="A5.I2.i1.p1" class="ltx_para">
<p id="A5.I2.i1.p1.1" class="ltx_p">Approximate matrix multiplication is an established research problem of general interest independent of deep learning.</p>
</div>
</li>
<li id="A5.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="A5.I2.i2.p1" class="ltx_para">
<p id="A5.I2.i2.p1.1" class="ltx_p">Lifting a method from accelerating a single layer to an overall network is challenging. Just as scalar quantization of network parameters is simple for a single layer but an active area of research for an entire network, so too is using our method on multiple layers at once an open research problem. For example, it is not clear how to deal with the fact that distributions of activations change throughout training, or how to efficiently incorporate our non-differentiable hash function.
We could show how to accelerate one internal FC layer in a network, but we don’t want to risk misleading the reader—it would be unclear what conclusions to draw from such results, particularly given the difficulty of retraining / fine-tuning without introducing many lurking variables (c.f., <cite class="ltx_cite ltx_citemacro_citep">(Blalock et al., <a href="#bib.bib12" title="" class="ltx_ref">2020</a>)</cite>).</p>
</div>
</li>
<li id="A5.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="A5.I2.i3.p1" class="ltx_para">
<p id="A5.I2.i3.p1.1" class="ltx_p">It is correct that convolution can be reduced to GEMM using im2col, and that accelerating convolution using our ideas would be a valuable contribution. However, state-of-the-art algorithms for convolution exploit structure that is not available to general matrix multiply algorithms. To match the performance of specialized Winograd, direct, FFT-based, and hybrid convolution schemes that do exploit this additional structure, we would have to make modifications to our approach that would make it less general. For example, the individual spatial positions should be encoded only once, and then reused at multiple filter positions. Regarding Section 5.5: while we do test our method on matrices of flattened image patches, we do not claim that the overall pipeline of flattening + matrix multiply constitutes a state-of-the-art convolution method—we only claim that using our method in this pipeline outperforms using other AMM methods there.</p>
</div>
</li>
</ol>
</div>
<div id="A5.SS8.p2" class="ltx_para">
<p id="A5.SS8.p2.1" class="ltx_p">In short, while we believe that our ideas show great promise for accelerating full neural networks and more layer types, making this happen requires much more research.</p>
</div>
</section>
<section id="A5.SS9" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.9 </span>Additional Results</h3>

<div id="A5.SS9.p1" class="ltx_para">
<p id="A5.SS9.p1.1" class="ltx_p">In Section <a href="#S5" title="5 Experiments ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we showed the classification accuracy as a function of wall time for the CIFAR-10 and CIFAR-100 softmax classifiers, as well as on the UCR datasets. In Figure <a href="#A5.F8" title="Figure 8 ‣ E.9 Additional Results ‣ Appendix E Additional Experimental Details ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> and Figure <a href="#A5.F9" title="Figure 9 ‣ E.9 Additional Results ‣ Appendix E Additional Experimental Details ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, we instead show normalized mean squared error versus time. In Figure <a href="#A5.F10" title="Figure 10 ‣ E.9 Additional Results ‣ Appendix E Additional Experimental Details ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> and Figure <a href="#A5.F11" title="Figure 11 ‣ E.9 Additional Results ‣ Appendix E Additional Experimental Details ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>, we show accuracy or NMSE versus number of operations performed, where one operation is either one multiply-add or one table lookup, depending on the method. The first two figures illustrate that NMSE is closely related to classification accuracy, but with imperfect NMSE still yielding excellent accuracy in many cases. The second two figures show that our method’s superior results are not merely caused by the use of faster CPU instructions, but also by the use of fewer basic operations at the algorithm level.</p>
</div>
<figure id="A5.F8" class="ltx_figure"><img src="/html/2106.10860/assets/x8.png" id="A5.F8.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="367" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span><span id="A5.F8.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A5.F8.3.1.1" class="ltx_text ltx_font_upright"> </span></span>achieves a far better speed versus squared error tradeoff than any existing method when approximating two softmax classifiers. These results parallel the speed versus classification accuracy results, except that the addition of our ridge regression is much more beneficial on CIFAR-100.</figcaption>
</figure>
<figure id="A5.F9" class="ltx_figure"><img src="/html/2106.10860/assets/x9.png" id="A5.F9.1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="473" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span><span id="A5.F9.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A5.F9.3.1.1" class="ltx_text ltx_font_upright"> </span></span>achieves the lowest squared error at high speedups on the UCR datasets. These results parallel the speed versus classification accuracy results.</figcaption>
</figure>
<figure id="A5.F10" class="ltx_figure"><img src="/html/2106.10860/assets/x10.png" id="A5.F10.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="368" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span><span id="A5.F10.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A5.F10.3.1.1" class="ltx_text ltx_font_upright"> </span></span>achieves the best speed versus accuracy tradeoff on the CIFAR datasets of any method even when speed is measured as number of operations instead of wall time. Note that fewer operations with a high accuracy (up and to the left) is better.</figcaption>
</figure>
<figure id="A5.F11" class="ltx_figure"><img src="/html/2106.10860/assets/x11.png" id="A5.F11.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="308" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span><span id="A5.F11.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A5.F11.3.1.1" class="ltx_text ltx_font_upright"> </span></span>still achieves the best speed versus squared error tradeoff on the image processing tasks when speed is measured as number of operations instead of wall time.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Theoretical Analysis of <span id="A6.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>
</h2>

<section id="A6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.1 </span>Complexity</h3>

<div id="A6.SS1.p1" class="ltx_para">
<p id="A6.SS1.p1.21" class="ltx_p">Our encoding function <math id="A6.SS1.p1.1.m1.3" class="ltx_Math" alttext="g(\bm{A}),\bm{A}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="A6.SS1.p1.1.m1.3a"><mrow id="A6.SS1.p1.1.m1.3.3" xref="A6.SS1.p1.1.m1.3.3.cmml"><mrow id="A6.SS1.p1.1.m1.3.3.1.1" xref="A6.SS1.p1.1.m1.3.3.1.2.cmml"><mrow id="A6.SS1.p1.1.m1.3.3.1.1.1" xref="A6.SS1.p1.1.m1.3.3.1.1.1.cmml"><mi id="A6.SS1.p1.1.m1.3.3.1.1.1.2" xref="A6.SS1.p1.1.m1.3.3.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.1.m1.3.3.1.1.1.1" xref="A6.SS1.p1.1.m1.3.3.1.1.1.1.cmml">​</mo><mrow id="A6.SS1.p1.1.m1.3.3.1.1.1.3.2" xref="A6.SS1.p1.1.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.1.m1.3.3.1.1.1.3.2.1" xref="A6.SS1.p1.1.m1.3.3.1.1.1.cmml">(</mo><mi id="A6.SS1.p1.1.m1.1.1" xref="A6.SS1.p1.1.m1.1.1.cmml">𝑨</mi><mo stretchy="false" id="A6.SS1.p1.1.m1.3.3.1.1.1.3.2.2" xref="A6.SS1.p1.1.m1.3.3.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.SS1.p1.1.m1.3.3.1.1.2" xref="A6.SS1.p1.1.m1.3.3.1.2.cmml">,</mo><mi id="A6.SS1.p1.1.m1.2.2" xref="A6.SS1.p1.1.m1.2.2.cmml">𝑨</mi></mrow><mo id="A6.SS1.p1.1.m1.3.3.2" xref="A6.SS1.p1.1.m1.3.3.2.cmml">∈</mo><msup id="A6.SS1.p1.1.m1.3.3.3" xref="A6.SS1.p1.1.m1.3.3.3.cmml"><mi id="A6.SS1.p1.1.m1.3.3.3.2" xref="A6.SS1.p1.1.m1.3.3.3.2.cmml">ℝ</mi><mrow id="A6.SS1.p1.1.m1.3.3.3.3" xref="A6.SS1.p1.1.m1.3.3.3.3.cmml"><mi id="A6.SS1.p1.1.m1.3.3.3.3.2" xref="A6.SS1.p1.1.m1.3.3.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="A6.SS1.p1.1.m1.3.3.3.3.1" xref="A6.SS1.p1.1.m1.3.3.3.3.1.cmml">×</mo><mi id="A6.SS1.p1.1.m1.3.3.3.3.3" xref="A6.SS1.p1.1.m1.3.3.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.1.m1.3b"><apply id="A6.SS1.p1.1.m1.3.3.cmml" xref="A6.SS1.p1.1.m1.3.3"><in id="A6.SS1.p1.1.m1.3.3.2.cmml" xref="A6.SS1.p1.1.m1.3.3.2"></in><list id="A6.SS1.p1.1.m1.3.3.1.2.cmml" xref="A6.SS1.p1.1.m1.3.3.1.1"><apply id="A6.SS1.p1.1.m1.3.3.1.1.1.cmml" xref="A6.SS1.p1.1.m1.3.3.1.1.1"><times id="A6.SS1.p1.1.m1.3.3.1.1.1.1.cmml" xref="A6.SS1.p1.1.m1.3.3.1.1.1.1"></times><ci id="A6.SS1.p1.1.m1.3.3.1.1.1.2.cmml" xref="A6.SS1.p1.1.m1.3.3.1.1.1.2">𝑔</ci><ci id="A6.SS1.p1.1.m1.1.1.cmml" xref="A6.SS1.p1.1.m1.1.1">𝑨</ci></apply><ci id="A6.SS1.p1.1.m1.2.2.cmml" xref="A6.SS1.p1.1.m1.2.2">𝑨</ci></list><apply id="A6.SS1.p1.1.m1.3.3.3.cmml" xref="A6.SS1.p1.1.m1.3.3.3"><csymbol cd="ambiguous" id="A6.SS1.p1.1.m1.3.3.3.1.cmml" xref="A6.SS1.p1.1.m1.3.3.3">superscript</csymbol><ci id="A6.SS1.p1.1.m1.3.3.3.2.cmml" xref="A6.SS1.p1.1.m1.3.3.3.2">ℝ</ci><apply id="A6.SS1.p1.1.m1.3.3.3.3.cmml" xref="A6.SS1.p1.1.m1.3.3.3.3"><times id="A6.SS1.p1.1.m1.3.3.3.3.1.cmml" xref="A6.SS1.p1.1.m1.3.3.3.3.1"></times><ci id="A6.SS1.p1.1.m1.3.3.3.3.2.cmml" xref="A6.SS1.p1.1.m1.3.3.3.3.2">𝑁</ci><ci id="A6.SS1.p1.1.m1.3.3.3.3.3.cmml" xref="A6.SS1.p1.1.m1.3.3.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.1.m1.3c">g(\bm{A}),\bm{A}\in\mathbb{R}^{N\times D}</annotation></semantics></math> has complexity <math id="A6.SS1.p1.2.m2.1" class="ltx_Math" alttext="\Theta(NC)" display="inline"><semantics id="A6.SS1.p1.2.m2.1a"><mrow id="A6.SS1.p1.2.m2.1.1" xref="A6.SS1.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="A6.SS1.p1.2.m2.1.1.3" xref="A6.SS1.p1.2.m2.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.2.m2.1.1.2" xref="A6.SS1.p1.2.m2.1.1.2.cmml">​</mo><mrow id="A6.SS1.p1.2.m2.1.1.1.1" xref="A6.SS1.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.2.m2.1.1.1.1.2" xref="A6.SS1.p1.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS1.p1.2.m2.1.1.1.1.1" xref="A6.SS1.p1.2.m2.1.1.1.1.1.cmml"><mi id="A6.SS1.p1.2.m2.1.1.1.1.1.2" xref="A6.SS1.p1.2.m2.1.1.1.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.2.m2.1.1.1.1.1.1" xref="A6.SS1.p1.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.2.m2.1.1.1.1.1.3" xref="A6.SS1.p1.2.m2.1.1.1.1.1.3.cmml">C</mi></mrow><mo stretchy="false" id="A6.SS1.p1.2.m2.1.1.1.1.3" xref="A6.SS1.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.2.m2.1b"><apply id="A6.SS1.p1.2.m2.1.1.cmml" xref="A6.SS1.p1.2.m2.1.1"><times id="A6.SS1.p1.2.m2.1.1.2.cmml" xref="A6.SS1.p1.2.m2.1.1.2"></times><ci id="A6.SS1.p1.2.m2.1.1.3.cmml" xref="A6.SS1.p1.2.m2.1.1.3">Θ</ci><apply id="A6.SS1.p1.2.m2.1.1.1.1.1.cmml" xref="A6.SS1.p1.2.m2.1.1.1.1"><times id="A6.SS1.p1.2.m2.1.1.1.1.1.1.cmml" xref="A6.SS1.p1.2.m2.1.1.1.1.1.1"></times><ci id="A6.SS1.p1.2.m2.1.1.1.1.1.2.cmml" xref="A6.SS1.p1.2.m2.1.1.1.1.1.2">𝑁</ci><ci id="A6.SS1.p1.2.m2.1.1.1.1.1.3.cmml" xref="A6.SS1.p1.2.m2.1.1.1.1.1.3">𝐶</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.2.m2.1c">\Theta(NC)</annotation></semantics></math>, since it does a constant amount of work per row per codebook. Our table creation function <math id="A6.SS1.p1.3.m3.3" class="ltx_Math" alttext="h(\bm{B}),\bm{B}\in\mathbb{R}^{D\times M}" display="inline"><semantics id="A6.SS1.p1.3.m3.3a"><mrow id="A6.SS1.p1.3.m3.3.3" xref="A6.SS1.p1.3.m3.3.3.cmml"><mrow id="A6.SS1.p1.3.m3.3.3.1.1" xref="A6.SS1.p1.3.m3.3.3.1.2.cmml"><mrow id="A6.SS1.p1.3.m3.3.3.1.1.1" xref="A6.SS1.p1.3.m3.3.3.1.1.1.cmml"><mi id="A6.SS1.p1.3.m3.3.3.1.1.1.2" xref="A6.SS1.p1.3.m3.3.3.1.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.3.m3.3.3.1.1.1.1" xref="A6.SS1.p1.3.m3.3.3.1.1.1.1.cmml">​</mo><mrow id="A6.SS1.p1.3.m3.3.3.1.1.1.3.2" xref="A6.SS1.p1.3.m3.3.3.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.3.m3.3.3.1.1.1.3.2.1" xref="A6.SS1.p1.3.m3.3.3.1.1.1.cmml">(</mo><mi id="A6.SS1.p1.3.m3.1.1" xref="A6.SS1.p1.3.m3.1.1.cmml">𝑩</mi><mo stretchy="false" id="A6.SS1.p1.3.m3.3.3.1.1.1.3.2.2" xref="A6.SS1.p1.3.m3.3.3.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.SS1.p1.3.m3.3.3.1.1.2" xref="A6.SS1.p1.3.m3.3.3.1.2.cmml">,</mo><mi id="A6.SS1.p1.3.m3.2.2" xref="A6.SS1.p1.3.m3.2.2.cmml">𝑩</mi></mrow><mo id="A6.SS1.p1.3.m3.3.3.2" xref="A6.SS1.p1.3.m3.3.3.2.cmml">∈</mo><msup id="A6.SS1.p1.3.m3.3.3.3" xref="A6.SS1.p1.3.m3.3.3.3.cmml"><mi id="A6.SS1.p1.3.m3.3.3.3.2" xref="A6.SS1.p1.3.m3.3.3.3.2.cmml">ℝ</mi><mrow id="A6.SS1.p1.3.m3.3.3.3.3" xref="A6.SS1.p1.3.m3.3.3.3.3.cmml"><mi id="A6.SS1.p1.3.m3.3.3.3.3.2" xref="A6.SS1.p1.3.m3.3.3.3.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="A6.SS1.p1.3.m3.3.3.3.3.1" xref="A6.SS1.p1.3.m3.3.3.3.3.1.cmml">×</mo><mi id="A6.SS1.p1.3.m3.3.3.3.3.3" xref="A6.SS1.p1.3.m3.3.3.3.3.3.cmml">M</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.3.m3.3b"><apply id="A6.SS1.p1.3.m3.3.3.cmml" xref="A6.SS1.p1.3.m3.3.3"><in id="A6.SS1.p1.3.m3.3.3.2.cmml" xref="A6.SS1.p1.3.m3.3.3.2"></in><list id="A6.SS1.p1.3.m3.3.3.1.2.cmml" xref="A6.SS1.p1.3.m3.3.3.1.1"><apply id="A6.SS1.p1.3.m3.3.3.1.1.1.cmml" xref="A6.SS1.p1.3.m3.3.3.1.1.1"><times id="A6.SS1.p1.3.m3.3.3.1.1.1.1.cmml" xref="A6.SS1.p1.3.m3.3.3.1.1.1.1"></times><ci id="A6.SS1.p1.3.m3.3.3.1.1.1.2.cmml" xref="A6.SS1.p1.3.m3.3.3.1.1.1.2">ℎ</ci><ci id="A6.SS1.p1.3.m3.1.1.cmml" xref="A6.SS1.p1.3.m3.1.1">𝑩</ci></apply><ci id="A6.SS1.p1.3.m3.2.2.cmml" xref="A6.SS1.p1.3.m3.2.2">𝑩</ci></list><apply id="A6.SS1.p1.3.m3.3.3.3.cmml" xref="A6.SS1.p1.3.m3.3.3.3"><csymbol cd="ambiguous" id="A6.SS1.p1.3.m3.3.3.3.1.cmml" xref="A6.SS1.p1.3.m3.3.3.3">superscript</csymbol><ci id="A6.SS1.p1.3.m3.3.3.3.2.cmml" xref="A6.SS1.p1.3.m3.3.3.3.2">ℝ</ci><apply id="A6.SS1.p1.3.m3.3.3.3.3.cmml" xref="A6.SS1.p1.3.m3.3.3.3.3"><times id="A6.SS1.p1.3.m3.3.3.3.3.1.cmml" xref="A6.SS1.p1.3.m3.3.3.3.3.1"></times><ci id="A6.SS1.p1.3.m3.3.3.3.3.2.cmml" xref="A6.SS1.p1.3.m3.3.3.3.3.2">𝐷</ci><ci id="A6.SS1.p1.3.m3.3.3.3.3.3.cmml" xref="A6.SS1.p1.3.m3.3.3.3.3.3">𝑀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.3.m3.3c">h(\bm{B}),\bm{B}\in\mathbb{R}^{D\times M}</annotation></semantics></math> has complexity <math id="A6.SS1.p1.4.m4.1" class="ltx_Math" alttext="\Theta(MKCD)" display="inline"><semantics id="A6.SS1.p1.4.m4.1a"><mrow id="A6.SS1.p1.4.m4.1.1" xref="A6.SS1.p1.4.m4.1.1.cmml"><mi mathvariant="normal" id="A6.SS1.p1.4.m4.1.1.3" xref="A6.SS1.p1.4.m4.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.4.m4.1.1.2" xref="A6.SS1.p1.4.m4.1.1.2.cmml">​</mo><mrow id="A6.SS1.p1.4.m4.1.1.1.1" xref="A6.SS1.p1.4.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.4.m4.1.1.1.1.2" xref="A6.SS1.p1.4.m4.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS1.p1.4.m4.1.1.1.1.1" xref="A6.SS1.p1.4.m4.1.1.1.1.1.cmml"><mi id="A6.SS1.p1.4.m4.1.1.1.1.1.2" xref="A6.SS1.p1.4.m4.1.1.1.1.1.2.cmml">M</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.4.m4.1.1.1.1.1.1" xref="A6.SS1.p1.4.m4.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.4.m4.1.1.1.1.1.3" xref="A6.SS1.p1.4.m4.1.1.1.1.1.3.cmml">K</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.4.m4.1.1.1.1.1.1a" xref="A6.SS1.p1.4.m4.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.4.m4.1.1.1.1.1.4" xref="A6.SS1.p1.4.m4.1.1.1.1.1.4.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.4.m4.1.1.1.1.1.1b" xref="A6.SS1.p1.4.m4.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.4.m4.1.1.1.1.1.5" xref="A6.SS1.p1.4.m4.1.1.1.1.1.5.cmml">D</mi></mrow><mo stretchy="false" id="A6.SS1.p1.4.m4.1.1.1.1.3" xref="A6.SS1.p1.4.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.4.m4.1b"><apply id="A6.SS1.p1.4.m4.1.1.cmml" xref="A6.SS1.p1.4.m4.1.1"><times id="A6.SS1.p1.4.m4.1.1.2.cmml" xref="A6.SS1.p1.4.m4.1.1.2"></times><ci id="A6.SS1.p1.4.m4.1.1.3.cmml" xref="A6.SS1.p1.4.m4.1.1.3">Θ</ci><apply id="A6.SS1.p1.4.m4.1.1.1.1.1.cmml" xref="A6.SS1.p1.4.m4.1.1.1.1"><times id="A6.SS1.p1.4.m4.1.1.1.1.1.1.cmml" xref="A6.SS1.p1.4.m4.1.1.1.1.1.1"></times><ci id="A6.SS1.p1.4.m4.1.1.1.1.1.2.cmml" xref="A6.SS1.p1.4.m4.1.1.1.1.1.2">𝑀</ci><ci id="A6.SS1.p1.4.m4.1.1.1.1.1.3.cmml" xref="A6.SS1.p1.4.m4.1.1.1.1.1.3">𝐾</ci><ci id="A6.SS1.p1.4.m4.1.1.1.1.1.4.cmml" xref="A6.SS1.p1.4.m4.1.1.1.1.1.4">𝐶</ci><ci id="A6.SS1.p1.4.m4.1.1.1.1.1.5.cmml" xref="A6.SS1.p1.4.m4.1.1.1.1.1.5">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.4.m4.1c">\Theta(MKCD)</annotation></semantics></math>, since it must compute the inner product between each column of <math id="A6.SS1.p1.5.m5.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A6.SS1.p1.5.m5.1a"><mi id="A6.SS1.p1.5.m5.1.1" xref="A6.SS1.p1.5.m5.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.5.m5.1b"><ci id="A6.SS1.p1.5.m5.1.1.cmml" xref="A6.SS1.p1.5.m5.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.5.m5.1c">\bm{B}</annotation></semantics></math> and <math id="A6.SS1.p1.6.m6.1" class="ltx_Math" alttext="KC" display="inline"><semantics id="A6.SS1.p1.6.m6.1a"><mrow id="A6.SS1.p1.6.m6.1.1" xref="A6.SS1.p1.6.m6.1.1.cmml"><mi id="A6.SS1.p1.6.m6.1.1.2" xref="A6.SS1.p1.6.m6.1.1.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.6.m6.1.1.1" xref="A6.SS1.p1.6.m6.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.6.m6.1.1.3" xref="A6.SS1.p1.6.m6.1.1.3.cmml">C</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.6.m6.1b"><apply id="A6.SS1.p1.6.m6.1.1.cmml" xref="A6.SS1.p1.6.m6.1.1"><times id="A6.SS1.p1.6.m6.1.1.1.cmml" xref="A6.SS1.p1.6.m6.1.1.1"></times><ci id="A6.SS1.p1.6.m6.1.1.2.cmml" xref="A6.SS1.p1.6.m6.1.1.2">𝐾</ci><ci id="A6.SS1.p1.6.m6.1.1.3.cmml" xref="A6.SS1.p1.6.m6.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.6.m6.1c">KC</annotation></semantics></math> prototypes of length <math id="A6.SS1.p1.7.m7.1" class="ltx_Math" alttext="D" display="inline"><semantics id="A6.SS1.p1.7.m7.1a"><mi id="A6.SS1.p1.7.m7.1.1" xref="A6.SS1.p1.7.m7.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.7.m7.1b"><ci id="A6.SS1.p1.7.m7.1.1.cmml" xref="A6.SS1.p1.7.m7.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.7.m7.1c">D</annotation></semantics></math>. This is a factor of <math id="A6.SS1.p1.8.m8.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.SS1.p1.8.m8.1a"><mi id="A6.SS1.p1.8.m8.1.1" xref="A6.SS1.p1.8.m8.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.8.m8.1b"><ci id="A6.SS1.p1.8.m8.1.1.cmml" xref="A6.SS1.p1.8.m8.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.8.m8.1c">C</annotation></semantics></math> worse than PQ since we do not require the prototypes for different codebooks to have disjoint nonzero indices. However, this reduction in the speed of <math id="A6.SS1.p1.9.m9.1" class="ltx_Math" alttext="h(\cdot)" display="inline"><semantics id="A6.SS1.p1.9.m9.1a"><mrow id="A6.SS1.p1.9.m9.1.2" xref="A6.SS1.p1.9.m9.1.2.cmml"><mi id="A6.SS1.p1.9.m9.1.2.2" xref="A6.SS1.p1.9.m9.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.9.m9.1.2.1" xref="A6.SS1.p1.9.m9.1.2.1.cmml">​</mo><mrow id="A6.SS1.p1.9.m9.1.2.3.2" xref="A6.SS1.p1.9.m9.1.2.cmml"><mo stretchy="false" id="A6.SS1.p1.9.m9.1.2.3.2.1" xref="A6.SS1.p1.9.m9.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="A6.SS1.p1.9.m9.1.1" xref="A6.SS1.p1.9.m9.1.1.cmml">⋅</mo><mo stretchy="false" id="A6.SS1.p1.9.m9.1.2.3.2.2" xref="A6.SS1.p1.9.m9.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.9.m9.1b"><apply id="A6.SS1.p1.9.m9.1.2.cmml" xref="A6.SS1.p1.9.m9.1.2"><times id="A6.SS1.p1.9.m9.1.2.1.cmml" xref="A6.SS1.p1.9.m9.1.2.1"></times><ci id="A6.SS1.p1.9.m9.1.2.2.cmml" xref="A6.SS1.p1.9.m9.1.2.2">ℎ</ci><ci id="A6.SS1.p1.9.m9.1.1.cmml" xref="A6.SS1.p1.9.m9.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.9.m9.1c">h(\cdot)</annotation></semantics></math> is not a concern because <math id="A6.SS1.p1.10.m10.2" class="ltx_Math" alttext="N\gg M,D" display="inline"><semantics id="A6.SS1.p1.10.m10.2a"><mrow id="A6.SS1.p1.10.m10.2.3" xref="A6.SS1.p1.10.m10.2.3.cmml"><mi id="A6.SS1.p1.10.m10.2.3.2" xref="A6.SS1.p1.10.m10.2.3.2.cmml">N</mi><mo id="A6.SS1.p1.10.m10.2.3.1" xref="A6.SS1.p1.10.m10.2.3.1.cmml">≫</mo><mrow id="A6.SS1.p1.10.m10.2.3.3.2" xref="A6.SS1.p1.10.m10.2.3.3.1.cmml"><mi id="A6.SS1.p1.10.m10.1.1" xref="A6.SS1.p1.10.m10.1.1.cmml">M</mi><mo id="A6.SS1.p1.10.m10.2.3.3.2.1" xref="A6.SS1.p1.10.m10.2.3.3.1.cmml">,</mo><mi id="A6.SS1.p1.10.m10.2.2" xref="A6.SS1.p1.10.m10.2.2.cmml">D</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.10.m10.2b"><apply id="A6.SS1.p1.10.m10.2.3.cmml" xref="A6.SS1.p1.10.m10.2.3"><csymbol cd="latexml" id="A6.SS1.p1.10.m10.2.3.1.cmml" xref="A6.SS1.p1.10.m10.2.3.1">much-greater-than</csymbol><ci id="A6.SS1.p1.10.m10.2.3.2.cmml" xref="A6.SS1.p1.10.m10.2.3.2">𝑁</ci><list id="A6.SS1.p1.10.m10.2.3.3.1.cmml" xref="A6.SS1.p1.10.m10.2.3.3.2"><ci id="A6.SS1.p1.10.m10.1.1.cmml" xref="A6.SS1.p1.10.m10.1.1">𝑀</ci><ci id="A6.SS1.p1.10.m10.2.2.cmml" xref="A6.SS1.p1.10.m10.2.2">𝐷</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.10.m10.2c">N\gg M,D</annotation></semantics></math>; moreover, the matrix <math id="A6.SS1.p1.11.m11.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="A6.SS1.p1.11.m11.1a"><mi id="A6.SS1.p1.11.m11.1.1" xref="A6.SS1.p1.11.m11.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.11.m11.1b"><ci id="A6.SS1.p1.11.m11.1.1.cmml" xref="A6.SS1.p1.11.m11.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.11.m11.1c">\bm{B}</annotation></semantics></math> is often known ahead of time in realistic settings, allowing <math id="A6.SS1.p1.12.m12.1" class="ltx_Math" alttext="h(\bm{B})" display="inline"><semantics id="A6.SS1.p1.12.m12.1a"><mrow id="A6.SS1.p1.12.m12.1.2" xref="A6.SS1.p1.12.m12.1.2.cmml"><mi id="A6.SS1.p1.12.m12.1.2.2" xref="A6.SS1.p1.12.m12.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.12.m12.1.2.1" xref="A6.SS1.p1.12.m12.1.2.1.cmml">​</mo><mrow id="A6.SS1.p1.12.m12.1.2.3.2" xref="A6.SS1.p1.12.m12.1.2.cmml"><mo stretchy="false" id="A6.SS1.p1.12.m12.1.2.3.2.1" xref="A6.SS1.p1.12.m12.1.2.cmml">(</mo><mi id="A6.SS1.p1.12.m12.1.1" xref="A6.SS1.p1.12.m12.1.1.cmml">𝑩</mi><mo stretchy="false" id="A6.SS1.p1.12.m12.1.2.3.2.2" xref="A6.SS1.p1.12.m12.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.12.m12.1b"><apply id="A6.SS1.p1.12.m12.1.2.cmml" xref="A6.SS1.p1.12.m12.1.2"><times id="A6.SS1.p1.12.m12.1.2.1.cmml" xref="A6.SS1.p1.12.m12.1.2.1"></times><ci id="A6.SS1.p1.12.m12.1.2.2.cmml" xref="A6.SS1.p1.12.m12.1.2.2">ℎ</ci><ci id="A6.SS1.p1.12.m12.1.1.cmml" xref="A6.SS1.p1.12.m12.1.1">𝑩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.12.m12.1c">h(\bm{B})</annotation></semantics></math> to be computed offline. Finally, the complexity of our aggregation function <math id="A6.SS1.p1.13.m13.1" class="ltx_Math" alttext="f(\cdot)" display="inline"><semantics id="A6.SS1.p1.13.m13.1a"><mrow id="A6.SS1.p1.13.m13.1.2" xref="A6.SS1.p1.13.m13.1.2.cmml"><mi id="A6.SS1.p1.13.m13.1.2.2" xref="A6.SS1.p1.13.m13.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.13.m13.1.2.1" xref="A6.SS1.p1.13.m13.1.2.1.cmml">​</mo><mrow id="A6.SS1.p1.13.m13.1.2.3.2" xref="A6.SS1.p1.13.m13.1.2.cmml"><mo stretchy="false" id="A6.SS1.p1.13.m13.1.2.3.2.1" xref="A6.SS1.p1.13.m13.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="A6.SS1.p1.13.m13.1.1" xref="A6.SS1.p1.13.m13.1.1.cmml">⋅</mo><mo stretchy="false" id="A6.SS1.p1.13.m13.1.2.3.2.2" xref="A6.SS1.p1.13.m13.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.13.m13.1b"><apply id="A6.SS1.p1.13.m13.1.2.cmml" xref="A6.SS1.p1.13.m13.1.2"><times id="A6.SS1.p1.13.m13.1.2.1.cmml" xref="A6.SS1.p1.13.m13.1.2.1"></times><ci id="A6.SS1.p1.13.m13.1.2.2.cmml" xref="A6.SS1.p1.13.m13.1.2.2">𝑓</ci><ci id="A6.SS1.p1.13.m13.1.1.cmml" xref="A6.SS1.p1.13.m13.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.13.m13.1c">f(\cdot)</annotation></semantics></math> is <math id="A6.SS1.p1.14.m14.1" class="ltx_Math" alttext="\Theta(NCM)" display="inline"><semantics id="A6.SS1.p1.14.m14.1a"><mrow id="A6.SS1.p1.14.m14.1.1" xref="A6.SS1.p1.14.m14.1.1.cmml"><mi mathvariant="normal" id="A6.SS1.p1.14.m14.1.1.3" xref="A6.SS1.p1.14.m14.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.14.m14.1.1.2" xref="A6.SS1.p1.14.m14.1.1.2.cmml">​</mo><mrow id="A6.SS1.p1.14.m14.1.1.1.1" xref="A6.SS1.p1.14.m14.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.14.m14.1.1.1.1.2" xref="A6.SS1.p1.14.m14.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS1.p1.14.m14.1.1.1.1.1" xref="A6.SS1.p1.14.m14.1.1.1.1.1.cmml"><mi id="A6.SS1.p1.14.m14.1.1.1.1.1.2" xref="A6.SS1.p1.14.m14.1.1.1.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.14.m14.1.1.1.1.1.1" xref="A6.SS1.p1.14.m14.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.14.m14.1.1.1.1.1.3" xref="A6.SS1.p1.14.m14.1.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.14.m14.1.1.1.1.1.1a" xref="A6.SS1.p1.14.m14.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.14.m14.1.1.1.1.1.4" xref="A6.SS1.p1.14.m14.1.1.1.1.1.4.cmml">M</mi></mrow><mo stretchy="false" id="A6.SS1.p1.14.m14.1.1.1.1.3" xref="A6.SS1.p1.14.m14.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.14.m14.1b"><apply id="A6.SS1.p1.14.m14.1.1.cmml" xref="A6.SS1.p1.14.m14.1.1"><times id="A6.SS1.p1.14.m14.1.1.2.cmml" xref="A6.SS1.p1.14.m14.1.1.2"></times><ci id="A6.SS1.p1.14.m14.1.1.3.cmml" xref="A6.SS1.p1.14.m14.1.1.3">Θ</ci><apply id="A6.SS1.p1.14.m14.1.1.1.1.1.cmml" xref="A6.SS1.p1.14.m14.1.1.1.1"><times id="A6.SS1.p1.14.m14.1.1.1.1.1.1.cmml" xref="A6.SS1.p1.14.m14.1.1.1.1.1.1"></times><ci id="A6.SS1.p1.14.m14.1.1.1.1.1.2.cmml" xref="A6.SS1.p1.14.m14.1.1.1.1.1.2">𝑁</ci><ci id="A6.SS1.p1.14.m14.1.1.1.1.1.3.cmml" xref="A6.SS1.p1.14.m14.1.1.1.1.1.3">𝐶</ci><ci id="A6.SS1.p1.14.m14.1.1.1.1.1.4.cmml" xref="A6.SS1.p1.14.m14.1.1.1.1.1.4">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.14.m14.1c">\Theta(NCM)</annotation></semantics></math>, since it performs <math id="A6.SS1.p1.15.m15.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.SS1.p1.15.m15.1a"><mi id="A6.SS1.p1.15.m15.1.1" xref="A6.SS1.p1.15.m15.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.15.m15.1b"><ci id="A6.SS1.p1.15.m15.1.1.cmml" xref="A6.SS1.p1.15.m15.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.15.m15.1c">C</annotation></semantics></math> table lookups for each of <math id="A6.SS1.p1.16.m16.1" class="ltx_Math" alttext="M" display="inline"><semantics id="A6.SS1.p1.16.m16.1a"><mi id="A6.SS1.p1.16.m16.1.1" xref="A6.SS1.p1.16.m16.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.16.m16.1b"><ci id="A6.SS1.p1.16.m16.1.1.cmml" xref="A6.SS1.p1.16.m16.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.16.m16.1c">M</annotation></semantics></math> output columns and <math id="A6.SS1.p1.17.m17.1" class="ltx_Math" alttext="N" display="inline"><semantics id="A6.SS1.p1.17.m17.1a"><mi id="A6.SS1.p1.17.m17.1.1" xref="A6.SS1.p1.17.m17.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.17.m17.1b"><ci id="A6.SS1.p1.17.m17.1.1.cmml" xref="A6.SS1.p1.17.m17.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.17.m17.1c">N</annotation></semantics></math> output rows. This means our overall algorithm has complexity <math id="A6.SS1.p1.18.m18.1" class="ltx_Math" alttext="\Theta(MC(KD+N))" display="inline"><semantics id="A6.SS1.p1.18.m18.1a"><mrow id="A6.SS1.p1.18.m18.1.1" xref="A6.SS1.p1.18.m18.1.1.cmml"><mi mathvariant="normal" id="A6.SS1.p1.18.m18.1.1.3" xref="A6.SS1.p1.18.m18.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.18.m18.1.1.2" xref="A6.SS1.p1.18.m18.1.1.2.cmml">​</mo><mrow id="A6.SS1.p1.18.m18.1.1.1.1" xref="A6.SS1.p1.18.m18.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.18.m18.1.1.1.1.2" xref="A6.SS1.p1.18.m18.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS1.p1.18.m18.1.1.1.1.1" xref="A6.SS1.p1.18.m18.1.1.1.1.1.cmml"><mi id="A6.SS1.p1.18.m18.1.1.1.1.1.3" xref="A6.SS1.p1.18.m18.1.1.1.1.1.3.cmml">M</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.18.m18.1.1.1.1.1.2" xref="A6.SS1.p1.18.m18.1.1.1.1.1.2.cmml">​</mo><mi id="A6.SS1.p1.18.m18.1.1.1.1.1.4" xref="A6.SS1.p1.18.m18.1.1.1.1.1.4.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.18.m18.1.1.1.1.1.2a" xref="A6.SS1.p1.18.m18.1.1.1.1.1.2.cmml">​</mo><mrow id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.2" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.cmml"><mrow id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.cmml"><mi id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.2" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.1" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.3" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.3.cmml">D</mi></mrow><mo id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.1" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.3" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.3.cmml">N</mi></mrow><mo stretchy="false" id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.3" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A6.SS1.p1.18.m18.1.1.1.1.3" xref="A6.SS1.p1.18.m18.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.18.m18.1b"><apply id="A6.SS1.p1.18.m18.1.1.cmml" xref="A6.SS1.p1.18.m18.1.1"><times id="A6.SS1.p1.18.m18.1.1.2.cmml" xref="A6.SS1.p1.18.m18.1.1.2"></times><ci id="A6.SS1.p1.18.m18.1.1.3.cmml" xref="A6.SS1.p1.18.m18.1.1.3">Θ</ci><apply id="A6.SS1.p1.18.m18.1.1.1.1.1.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1"><times id="A6.SS1.p1.18.m18.1.1.1.1.1.2.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.2"></times><ci id="A6.SS1.p1.18.m18.1.1.1.1.1.3.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.3">𝑀</ci><ci id="A6.SS1.p1.18.m18.1.1.1.1.1.4.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.4">𝐶</ci><apply id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1"><plus id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.1.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.1"></plus><apply id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2"><times id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.1.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.1"></times><ci id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.2.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.2">𝐾</ci><ci id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.3.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.2.3">𝐷</ci></apply><ci id="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.3.cmml" xref="A6.SS1.p1.18.m18.1.1.1.1.1.1.1.1.3">𝑁</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.18.m18.1c">\Theta(MC(KD+N))</annotation></semantics></math>, which reduces to <math id="A6.SS1.p1.19.m19.1" class="ltx_Math" alttext="\Theta(NCM)" display="inline"><semantics id="A6.SS1.p1.19.m19.1a"><mrow id="A6.SS1.p1.19.m19.1.1" xref="A6.SS1.p1.19.m19.1.1.cmml"><mi mathvariant="normal" id="A6.SS1.p1.19.m19.1.1.3" xref="A6.SS1.p1.19.m19.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.19.m19.1.1.2" xref="A6.SS1.p1.19.m19.1.1.2.cmml">​</mo><mrow id="A6.SS1.p1.19.m19.1.1.1.1" xref="A6.SS1.p1.19.m19.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS1.p1.19.m19.1.1.1.1.2" xref="A6.SS1.p1.19.m19.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS1.p1.19.m19.1.1.1.1.1" xref="A6.SS1.p1.19.m19.1.1.1.1.1.cmml"><mi id="A6.SS1.p1.19.m19.1.1.1.1.1.2" xref="A6.SS1.p1.19.m19.1.1.1.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.19.m19.1.1.1.1.1.1" xref="A6.SS1.p1.19.m19.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.19.m19.1.1.1.1.1.3" xref="A6.SS1.p1.19.m19.1.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS1.p1.19.m19.1.1.1.1.1.1a" xref="A6.SS1.p1.19.m19.1.1.1.1.1.1.cmml">​</mo><mi id="A6.SS1.p1.19.m19.1.1.1.1.1.4" xref="A6.SS1.p1.19.m19.1.1.1.1.1.4.cmml">M</mi></mrow><mo stretchy="false" id="A6.SS1.p1.19.m19.1.1.1.1.3" xref="A6.SS1.p1.19.m19.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.19.m19.1b"><apply id="A6.SS1.p1.19.m19.1.1.cmml" xref="A6.SS1.p1.19.m19.1.1"><times id="A6.SS1.p1.19.m19.1.1.2.cmml" xref="A6.SS1.p1.19.m19.1.1.2"></times><ci id="A6.SS1.p1.19.m19.1.1.3.cmml" xref="A6.SS1.p1.19.m19.1.1.3">Θ</ci><apply id="A6.SS1.p1.19.m19.1.1.1.1.1.cmml" xref="A6.SS1.p1.19.m19.1.1.1.1"><times id="A6.SS1.p1.19.m19.1.1.1.1.1.1.cmml" xref="A6.SS1.p1.19.m19.1.1.1.1.1.1"></times><ci id="A6.SS1.p1.19.m19.1.1.1.1.1.2.cmml" xref="A6.SS1.p1.19.m19.1.1.1.1.1.2">𝑁</ci><ci id="A6.SS1.p1.19.m19.1.1.1.1.1.3.cmml" xref="A6.SS1.p1.19.m19.1.1.1.1.1.3">𝐶</ci><ci id="A6.SS1.p1.19.m19.1.1.1.1.1.4.cmml" xref="A6.SS1.p1.19.m19.1.1.1.1.1.4">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.19.m19.1c">\Theta(NCM)</annotation></semantics></math> since we fix <math id="A6.SS1.p1.20.m20.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="A6.SS1.p1.20.m20.1a"><mrow id="A6.SS1.p1.20.m20.1.1" xref="A6.SS1.p1.20.m20.1.1.cmml"><mi id="A6.SS1.p1.20.m20.1.1.2" xref="A6.SS1.p1.20.m20.1.1.2.cmml">K</mi><mo id="A6.SS1.p1.20.m20.1.1.1" xref="A6.SS1.p1.20.m20.1.1.1.cmml">=</mo><mn id="A6.SS1.p1.20.m20.1.1.3" xref="A6.SS1.p1.20.m20.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.20.m20.1b"><apply id="A6.SS1.p1.20.m20.1.1.cmml" xref="A6.SS1.p1.20.m20.1.1"><eq id="A6.SS1.p1.20.m20.1.1.1.cmml" xref="A6.SS1.p1.20.m20.1.1.1"></eq><ci id="A6.SS1.p1.20.m20.1.1.2.cmml" xref="A6.SS1.p1.20.m20.1.1.2">𝐾</ci><cn type="integer" id="A6.SS1.p1.20.m20.1.1.3.cmml" xref="A6.SS1.p1.20.m20.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.20.m20.1c">K=16</annotation></semantics></math> and our problem statement requires <math id="A6.SS1.p1.21.m21.1" class="ltx_Math" alttext="N\gg D" display="inline"><semantics id="A6.SS1.p1.21.m21.1a"><mrow id="A6.SS1.p1.21.m21.1.1" xref="A6.SS1.p1.21.m21.1.1.cmml"><mi id="A6.SS1.p1.21.m21.1.1.2" xref="A6.SS1.p1.21.m21.1.1.2.cmml">N</mi><mo id="A6.SS1.p1.21.m21.1.1.1" xref="A6.SS1.p1.21.m21.1.1.1.cmml">≫</mo><mi id="A6.SS1.p1.21.m21.1.1.3" xref="A6.SS1.p1.21.m21.1.1.3.cmml">D</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.21.m21.1b"><apply id="A6.SS1.p1.21.m21.1.1.cmml" xref="A6.SS1.p1.21.m21.1.1"><csymbol cd="latexml" id="A6.SS1.p1.21.m21.1.1.1.cmml" xref="A6.SS1.p1.21.m21.1.1.1">much-greater-than</csymbol><ci id="A6.SS1.p1.21.m21.1.1.2.cmml" xref="A6.SS1.p1.21.m21.1.1.2">𝑁</ci><ci id="A6.SS1.p1.21.m21.1.1.3.cmml" xref="A6.SS1.p1.21.m21.1.1.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.21.m21.1c">N\gg D</annotation></semantics></math>.</p>
</div>
</section>
<section id="A6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.2 </span>Proof of Generalization Guarantee</h3>

<div id="A6.SS2.p1" class="ltx_para">
<p id="A6.SS2.p1.1" class="ltx_p">In this section, we prove Theorem <a href="#S4.Thmtheorem1" title="Theorem 4.1 (Generalization Error of Maddness). ‣ 4.5 Theoretical Guarantees ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>, restated below for convenience.</p>
</div>
<div id="Thmtheoremx1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmtheoremx1.1.1.1" class="ltx_text ltx_font_bold">Theorem</span></span><span id="Thmtheoremx1.2.2" class="ltx_text ltx_font_bold"> </span>(Generalization Error of <span id="Thmtheoremx1.3.3" class="ltx_text ltx_font_smallcaps">Maddness</span>)<span id="Thmtheoremx1.4.4" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmtheoremx1.p1" class="ltx_para">
<p id="Thmtheoremx1.p1.11" class="ltx_p"><span id="Thmtheoremx1.p1.11.11" class="ltx_text ltx_font_italic">Let <math id="Thmtheoremx1.p1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="Thmtheoremx1.p1.1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmtheoremx1.p1.1.1.m1.1.1" xref="Thmtheoremx1.p1.1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.1.1.m1.1b"><ci id="Thmtheoremx1.p1.1.1.m1.1.1.cmml" xref="Thmtheoremx1.p1.1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.1.1.m1.1c">\mathcal{D}</annotation></semantics></math> be a probability distribution over <math id="Thmtheoremx1.p1.2.2.m2.1" class="ltx_Math" alttext="\mathbb{R}^{D}" display="inline"><semantics id="Thmtheoremx1.p1.2.2.m2.1a"><msup id="Thmtheoremx1.p1.2.2.m2.1.1" xref="Thmtheoremx1.p1.2.2.m2.1.1.cmml"><mi id="Thmtheoremx1.p1.2.2.m2.1.1.2" xref="Thmtheoremx1.p1.2.2.m2.1.1.2.cmml">ℝ</mi><mi id="Thmtheoremx1.p1.2.2.m2.1.1.3" xref="Thmtheoremx1.p1.2.2.m2.1.1.3.cmml">D</mi></msup><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.2.2.m2.1b"><apply id="Thmtheoremx1.p1.2.2.m2.1.1.cmml" xref="Thmtheoremx1.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="Thmtheoremx1.p1.2.2.m2.1.1.1.cmml" xref="Thmtheoremx1.p1.2.2.m2.1.1">superscript</csymbol><ci id="Thmtheoremx1.p1.2.2.m2.1.1.2.cmml" xref="Thmtheoremx1.p1.2.2.m2.1.1.2">ℝ</ci><ci id="Thmtheoremx1.p1.2.2.m2.1.1.3.cmml" xref="Thmtheoremx1.p1.2.2.m2.1.1.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.2.2.m2.1c">\mathbb{R}^{D}</annotation></semantics></math> and suppose that <span id="Thmtheoremx1.p1.11.11.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="Thmtheoremx1.p1.11.11.1.1" class="ltx_text ltx_font_italic"> </span></span>is trained on a matrix <math id="Thmtheoremx1.p1.3.3.m3.1" class="ltx_Math" alttext="\tilde{\bm{A}}\in R^{N\times D}" display="inline"><semantics id="Thmtheoremx1.p1.3.3.m3.1a"><mrow id="Thmtheoremx1.p1.3.3.m3.1.1" xref="Thmtheoremx1.p1.3.3.m3.1.1.cmml"><mover accent="true" id="Thmtheoremx1.p1.3.3.m3.1.1.2" xref="Thmtheoremx1.p1.3.3.m3.1.1.2.cmml"><mi id="Thmtheoremx1.p1.3.3.m3.1.1.2.2" xref="Thmtheoremx1.p1.3.3.m3.1.1.2.2.cmml">𝐀</mi><mo id="Thmtheoremx1.p1.3.3.m3.1.1.2.1" xref="Thmtheoremx1.p1.3.3.m3.1.1.2.1.cmml">~</mo></mover><mo id="Thmtheoremx1.p1.3.3.m3.1.1.1" xref="Thmtheoremx1.p1.3.3.m3.1.1.1.cmml">∈</mo><msup id="Thmtheoremx1.p1.3.3.m3.1.1.3" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.cmml"><mi id="Thmtheoremx1.p1.3.3.m3.1.1.3.2" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.2.cmml">R</mi><mrow id="Thmtheoremx1.p1.3.3.m3.1.1.3.3" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.cmml"><mi id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.2" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.1" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.1.cmml">×</mo><mi id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.3" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.3.3.m3.1b"><apply id="Thmtheoremx1.p1.3.3.m3.1.1.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1"><in id="Thmtheoremx1.p1.3.3.m3.1.1.1.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.1"></in><apply id="Thmtheoremx1.p1.3.3.m3.1.1.2.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.2"><ci id="Thmtheoremx1.p1.3.3.m3.1.1.2.1.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.2.1">~</ci><ci id="Thmtheoremx1.p1.3.3.m3.1.1.2.2.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.2.2">𝐀</ci></apply><apply id="Thmtheoremx1.p1.3.3.m3.1.1.3.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3"><csymbol cd="ambiguous" id="Thmtheoremx1.p1.3.3.m3.1.1.3.1.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3">superscript</csymbol><ci id="Thmtheoremx1.p1.3.3.m3.1.1.3.2.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.2">𝑅</ci><apply id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3"><times id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.1.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.1"></times><ci id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.2.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.2">𝑁</ci><ci id="Thmtheoremx1.p1.3.3.m3.1.1.3.3.3.cmml" xref="Thmtheoremx1.p1.3.3.m3.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.3.3.m3.1c">\tilde{\bm{A}}\in R^{N\times D}</annotation></semantics></math> whose rows are drawn independently from <math id="Thmtheoremx1.p1.4.4.m4.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="Thmtheoremx1.p1.4.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="Thmtheoremx1.p1.4.4.m4.1.1" xref="Thmtheoremx1.p1.4.4.m4.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.4.4.m4.1b"><ci id="Thmtheoremx1.p1.4.4.m4.1.1.cmml" xref="Thmtheoremx1.p1.4.4.m4.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.4.4.m4.1c">\mathcal{D}</annotation></semantics></math> and with maximum singular value bounded by <math id="Thmtheoremx1.p1.5.5.m5.1" class="ltx_Math" alttext="\sigma_{A}" display="inline"><semantics id="Thmtheoremx1.p1.5.5.m5.1a"><msub id="Thmtheoremx1.p1.5.5.m5.1.1" xref="Thmtheoremx1.p1.5.5.m5.1.1.cmml"><mi id="Thmtheoremx1.p1.5.5.m5.1.1.2" xref="Thmtheoremx1.p1.5.5.m5.1.1.2.cmml">σ</mi><mi id="Thmtheoremx1.p1.5.5.m5.1.1.3" xref="Thmtheoremx1.p1.5.5.m5.1.1.3.cmml">A</mi></msub><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.5.5.m5.1b"><apply id="Thmtheoremx1.p1.5.5.m5.1.1.cmml" xref="Thmtheoremx1.p1.5.5.m5.1.1"><csymbol cd="ambiguous" id="Thmtheoremx1.p1.5.5.m5.1.1.1.cmml" xref="Thmtheoremx1.p1.5.5.m5.1.1">subscript</csymbol><ci id="Thmtheoremx1.p1.5.5.m5.1.1.2.cmml" xref="Thmtheoremx1.p1.5.5.m5.1.1.2">𝜎</ci><ci id="Thmtheoremx1.p1.5.5.m5.1.1.3.cmml" xref="Thmtheoremx1.p1.5.5.m5.1.1.3">𝐴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.5.5.m5.1c">\sigma_{A}</annotation></semantics></math>. Let <math id="Thmtheoremx1.p1.6.6.m6.1" class="ltx_Math" alttext="C" display="inline"><semantics id="Thmtheoremx1.p1.6.6.m6.1a"><mi id="Thmtheoremx1.p1.6.6.m6.1.1" xref="Thmtheoremx1.p1.6.6.m6.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.6.6.m6.1b"><ci id="Thmtheoremx1.p1.6.6.m6.1.1.cmml" xref="Thmtheoremx1.p1.6.6.m6.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.6.6.m6.1c">C</annotation></semantics></math> be the number of codebooks used by <span id="Thmtheoremx1.p1.11.11.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="Thmtheoremx1.p1.11.11.2.1" class="ltx_text ltx_font_italic"> </span></span>and <math id="Thmtheoremx1.p1.7.7.m7.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="Thmtheoremx1.p1.7.7.m7.1a"><mrow id="Thmtheoremx1.p1.7.7.m7.1.1" xref="Thmtheoremx1.p1.7.7.m7.1.1.cmml"><mi id="Thmtheoremx1.p1.7.7.m7.1.1.2" xref="Thmtheoremx1.p1.7.7.m7.1.1.2.cmml">λ</mi><mo id="Thmtheoremx1.p1.7.7.m7.1.1.1" xref="Thmtheoremx1.p1.7.7.m7.1.1.1.cmml">&gt;</mo><mn id="Thmtheoremx1.p1.7.7.m7.1.1.3" xref="Thmtheoremx1.p1.7.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.7.7.m7.1b"><apply id="Thmtheoremx1.p1.7.7.m7.1.1.cmml" xref="Thmtheoremx1.p1.7.7.m7.1.1"><gt id="Thmtheoremx1.p1.7.7.m7.1.1.1.cmml" xref="Thmtheoremx1.p1.7.7.m7.1.1.1"></gt><ci id="Thmtheoremx1.p1.7.7.m7.1.1.2.cmml" xref="Thmtheoremx1.p1.7.7.m7.1.1.2">𝜆</ci><cn type="integer" id="Thmtheoremx1.p1.7.7.m7.1.1.3.cmml" xref="Thmtheoremx1.p1.7.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.7.7.m7.1c">\lambda&gt;0</annotation></semantics></math> be the regularization parameter used in the ridge regression step. Then for any <math id="Thmtheoremx1.p1.8.8.m8.1" class="ltx_Math" alttext="\bm{b}\in\mathbb{R}^{D}" display="inline"><semantics id="Thmtheoremx1.p1.8.8.m8.1a"><mrow id="Thmtheoremx1.p1.8.8.m8.1.1" xref="Thmtheoremx1.p1.8.8.m8.1.1.cmml"><mi id="Thmtheoremx1.p1.8.8.m8.1.1.2" xref="Thmtheoremx1.p1.8.8.m8.1.1.2.cmml">𝐛</mi><mo id="Thmtheoremx1.p1.8.8.m8.1.1.1" xref="Thmtheoremx1.p1.8.8.m8.1.1.1.cmml">∈</mo><msup id="Thmtheoremx1.p1.8.8.m8.1.1.3" xref="Thmtheoremx1.p1.8.8.m8.1.1.3.cmml"><mi id="Thmtheoremx1.p1.8.8.m8.1.1.3.2" xref="Thmtheoremx1.p1.8.8.m8.1.1.3.2.cmml">ℝ</mi><mi id="Thmtheoremx1.p1.8.8.m8.1.1.3.3" xref="Thmtheoremx1.p1.8.8.m8.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.8.8.m8.1b"><apply id="Thmtheoremx1.p1.8.8.m8.1.1.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1"><in id="Thmtheoremx1.p1.8.8.m8.1.1.1.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1.1"></in><ci id="Thmtheoremx1.p1.8.8.m8.1.1.2.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1.2">𝐛</ci><apply id="Thmtheoremx1.p1.8.8.m8.1.1.3.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1.3"><csymbol cd="ambiguous" id="Thmtheoremx1.p1.8.8.m8.1.1.3.1.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1.3">superscript</csymbol><ci id="Thmtheoremx1.p1.8.8.m8.1.1.3.2.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1.3.2">ℝ</ci><ci id="Thmtheoremx1.p1.8.8.m8.1.1.3.3.cmml" xref="Thmtheoremx1.p1.8.8.m8.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.8.8.m8.1c">\bm{b}\in\mathbb{R}^{D}</annotation></semantics></math>, any <math id="Thmtheoremx1.p1.9.9.m9.1" class="ltx_Math" alttext="\bm{a}\sim\mathcal{D}" display="inline"><semantics id="Thmtheoremx1.p1.9.9.m9.1a"><mrow id="Thmtheoremx1.p1.9.9.m9.1.1" xref="Thmtheoremx1.p1.9.9.m9.1.1.cmml"><mi id="Thmtheoremx1.p1.9.9.m9.1.1.2" xref="Thmtheoremx1.p1.9.9.m9.1.1.2.cmml">𝐚</mi><mo id="Thmtheoremx1.p1.9.9.m9.1.1.1" xref="Thmtheoremx1.p1.9.9.m9.1.1.1.cmml">∼</mo><mi class="ltx_font_mathcaligraphic" id="Thmtheoremx1.p1.9.9.m9.1.1.3" xref="Thmtheoremx1.p1.9.9.m9.1.1.3.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.9.9.m9.1b"><apply id="Thmtheoremx1.p1.9.9.m9.1.1.cmml" xref="Thmtheoremx1.p1.9.9.m9.1.1"><csymbol cd="latexml" id="Thmtheoremx1.p1.9.9.m9.1.1.1.cmml" xref="Thmtheoremx1.p1.9.9.m9.1.1.1">similar-to</csymbol><ci id="Thmtheoremx1.p1.9.9.m9.1.1.2.cmml" xref="Thmtheoremx1.p1.9.9.m9.1.1.2">𝐚</ci><ci id="Thmtheoremx1.p1.9.9.m9.1.1.3.cmml" xref="Thmtheoremx1.p1.9.9.m9.1.1.3">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.9.9.m9.1c">\bm{a}\sim\mathcal{D}</annotation></semantics></math>, and any <math id="Thmtheoremx1.p1.10.10.m10.1" class="ltx_Math" alttext="0&lt;\delta&lt;1" display="inline"><semantics id="Thmtheoremx1.p1.10.10.m10.1a"><mrow id="Thmtheoremx1.p1.10.10.m10.1.1" xref="Thmtheoremx1.p1.10.10.m10.1.1.cmml"><mn id="Thmtheoremx1.p1.10.10.m10.1.1.2" xref="Thmtheoremx1.p1.10.10.m10.1.1.2.cmml">0</mn><mo id="Thmtheoremx1.p1.10.10.m10.1.1.3" xref="Thmtheoremx1.p1.10.10.m10.1.1.3.cmml">&lt;</mo><mi id="Thmtheoremx1.p1.10.10.m10.1.1.4" xref="Thmtheoremx1.p1.10.10.m10.1.1.4.cmml">δ</mi><mo id="Thmtheoremx1.p1.10.10.m10.1.1.5" xref="Thmtheoremx1.p1.10.10.m10.1.1.5.cmml">&lt;</mo><mn id="Thmtheoremx1.p1.10.10.m10.1.1.6" xref="Thmtheoremx1.p1.10.10.m10.1.1.6.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.10.10.m10.1b"><apply id="Thmtheoremx1.p1.10.10.m10.1.1.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1"><and id="Thmtheoremx1.p1.10.10.m10.1.1a.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1"></and><apply id="Thmtheoremx1.p1.10.10.m10.1.1b.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1"><lt id="Thmtheoremx1.p1.10.10.m10.1.1.3.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1.3"></lt><cn type="integer" id="Thmtheoremx1.p1.10.10.m10.1.1.2.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1.2">0</cn><ci id="Thmtheoremx1.p1.10.10.m10.1.1.4.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1.4">𝛿</ci></apply><apply id="Thmtheoremx1.p1.10.10.m10.1.1c.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1"><lt id="Thmtheoremx1.p1.10.10.m10.1.1.5.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1.5"></lt><share href="#Thmtheoremx1.p1.10.10.m10.1.1.4.cmml" id="Thmtheoremx1.p1.10.10.m10.1.1d.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1"></share><cn type="integer" id="Thmtheoremx1.p1.10.10.m10.1.1.6.cmml" xref="Thmtheoremx1.p1.10.10.m10.1.1.6">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.10.10.m10.1c">0&lt;\delta&lt;1</annotation></semantics></math>, we have with probability at least <math id="Thmtheoremx1.p1.11.11.m11.1" class="ltx_Math" alttext="1-\delta" display="inline"><semantics id="Thmtheoremx1.p1.11.11.m11.1a"><mrow id="Thmtheoremx1.p1.11.11.m11.1.1" xref="Thmtheoremx1.p1.11.11.m11.1.1.cmml"><mn id="Thmtheoremx1.p1.11.11.m11.1.1.2" xref="Thmtheoremx1.p1.11.11.m11.1.1.2.cmml">1</mn><mo id="Thmtheoremx1.p1.11.11.m11.1.1.1" xref="Thmtheoremx1.p1.11.11.m11.1.1.1.cmml">−</mo><mi id="Thmtheoremx1.p1.11.11.m11.1.1.3" xref="Thmtheoremx1.p1.11.11.m11.1.1.3.cmml">δ</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.11.11.m11.1b"><apply id="Thmtheoremx1.p1.11.11.m11.1.1.cmml" xref="Thmtheoremx1.p1.11.11.m11.1.1"><minus id="Thmtheoremx1.p1.11.11.m11.1.1.1.cmml" xref="Thmtheoremx1.p1.11.11.m11.1.1.1"></minus><cn type="integer" id="Thmtheoremx1.p1.11.11.m11.1.1.2.cmml" xref="Thmtheoremx1.p1.11.11.m11.1.1.2">1</cn><ci id="Thmtheoremx1.p1.11.11.m11.1.1.3.cmml" xref="Thmtheoremx1.p1.11.11.m11.1.1.3">𝛿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.11.11.m11.1c">1-\delta</annotation></semantics></math> that</span></p>
<table id="A6.EGx18" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td colspan="2" class="ltx_td ltx_align_center ltx_eqn_cell"><math id="A6.Ex4.m1.32" class="ltx_Math" alttext="\displaystyle\begin{split}\mathbb{E}_{\mathcal{D}}[&amp;\mathcal{L}(\bm{a},\bm{b})]\leq\mathbb{E}_{\tilde{\bm{A}}}[\mathcal{L}(\bm{a},\bm{b})]+\\
&amp;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}\left(\frac{1}{256}+\frac{8+\sqrt{\nu(C,D,\delta)}}{\sqrt{2n}}\right)\end{split}" display="inline"><semantics id="A6.Ex4.m1.32a"><mtable columnspacing="0pt" rowspacing="0pt" id="A6.Ex4.m1.32.32.4" xref="A6.Ex4.m1.31.31.3.cmml"><mtr id="A6.Ex4.m1.32.32.4a" xref="A6.Ex4.m1.31.31.3.cmml"><mtd class="ltx_align_right" columnalign="right" id="A6.Ex4.m1.32.32.4b" xref="A6.Ex4.m1.31.31.3.cmml"><mrow id="A6.Ex4.m1.3.3.3.3.3a" xref="A6.Ex4.m1.31.31.3.cmml"><msub id="A6.Ex4.m1.3.3.3.3.3a.4" xref="A6.Ex4.m1.31.31.3.cmml"><mi id="A6.Ex4.m1.1.1.1.1.1.1" xref="A6.Ex4.m1.1.1.1.1.1.1.cmml">𝔼</mi><mi class="ltx_font_mathcaligraphic" id="A6.Ex4.m1.2.2.2.2.2.2.1" xref="A6.Ex4.m1.2.2.2.2.2.2.1.cmml">𝒟</mi></msub><mo stretchy="false" id="A6.Ex4.m1.3.3.3.3.3.3" xref="A6.Ex4.m1.31.31.3.cmml">[</mo></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="A6.Ex4.m1.32.32.4c" xref="A6.Ex4.m1.31.31.3.cmml"><mrow id="A6.Ex4.m1.22.22.22.22.19a" xref="A6.Ex4.m1.31.31.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.Ex4.m1.4.4.4.4.1.1" xref="A6.Ex4.m1.4.4.4.4.1.1.cmml">ℒ</mi><mrow id="A6.Ex4.m1.22.22.22.22.19a.20" xref="A6.Ex4.m1.31.31.3.cmml"><mo stretchy="false" id="A6.Ex4.m1.5.5.5.5.2.2" xref="A6.Ex4.m1.31.31.3.cmml">(</mo><mi id="A6.Ex4.m1.6.6.6.6.3.3" xref="A6.Ex4.m1.6.6.6.6.3.3.cmml">𝒂</mi><mo id="A6.Ex4.m1.7.7.7.7.4.4" xref="A6.Ex4.m1.31.31.3.cmml">,</mo><mi id="A6.Ex4.m1.8.8.8.8.5.5" xref="A6.Ex4.m1.8.8.8.8.5.5.cmml">𝒃</mi><mo stretchy="false" id="A6.Ex4.m1.9.9.9.9.6.6" xref="A6.Ex4.m1.31.31.3.cmml">)</mo></mrow><mo stretchy="false" id="A6.Ex4.m1.10.10.10.10.7.7" xref="A6.Ex4.m1.31.31.3.cmml">]</mo><mo id="A6.Ex4.m1.11.11.11.11.8.8" xref="A6.Ex4.m1.11.11.11.11.8.8.cmml">≤</mo><mi id="A6.Ex4.m1.12.12.12.12.9.9" xref="A6.Ex4.m1.12.12.12.12.9.9.cmml">𝔼</mi><msub id="A6.Ex4.m1.22.22.22.22.19a.21" xref="A6.Ex4.m1.31.31.3.cmml"><mi id="A6.Ex4.m1.22.22.22.22.19a.21a" xref="A6.Ex4.m1.31.31.3.cmml"></mi><mover accent="true" id="A6.Ex4.m1.13.13.13.13.10.10.1" xref="A6.Ex4.m1.13.13.13.13.10.10.1.cmml"><mi id="A6.Ex4.m1.13.13.13.13.10.10.1.2" xref="A6.Ex4.m1.13.13.13.13.10.10.1.2.cmml">𝑨</mi><mo id="A6.Ex4.m1.13.13.13.13.10.10.1.1" xref="A6.Ex4.m1.13.13.13.13.10.10.1.1.cmml">~</mo></mover></msub><mo stretchy="false" id="A6.Ex4.m1.14.14.14.14.11.11" xref="A6.Ex4.m1.31.31.3.cmml">[</mo><mi class="ltx_font_mathcaligraphic" id="A6.Ex4.m1.15.15.15.15.12.12" xref="A6.Ex4.m1.15.15.15.15.12.12.cmml">ℒ</mi><mrow id="A6.Ex4.m1.22.22.22.22.19a.22" xref="A6.Ex4.m1.31.31.3.cmml"><mo stretchy="false" id="A6.Ex4.m1.16.16.16.16.13.13" xref="A6.Ex4.m1.31.31.3.cmml">(</mo><mi id="A6.Ex4.m1.17.17.17.17.14.14" xref="A6.Ex4.m1.17.17.17.17.14.14.cmml">𝒂</mi><mo id="A6.Ex4.m1.18.18.18.18.15.15" xref="A6.Ex4.m1.31.31.3.cmml">,</mo><mi id="A6.Ex4.m1.19.19.19.19.16.16" xref="A6.Ex4.m1.19.19.19.19.16.16.cmml">𝒃</mi><mo stretchy="false" id="A6.Ex4.m1.20.20.20.20.17.17" xref="A6.Ex4.m1.31.31.3.cmml">)</mo></mrow><mo stretchy="false" id="A6.Ex4.m1.21.21.21.21.18.18" xref="A6.Ex4.m1.31.31.3.cmml">]</mo><mo id="A6.Ex4.m1.22.22.22.22.19.19" xref="A6.Ex4.m1.22.22.22.22.19.19.cmml">+</mo></mrow></mtd></mtr><mtr id="A6.Ex4.m1.32.32.4d" xref="A6.Ex4.m1.31.31.3.cmml"><mtd id="A6.Ex4.m1.32.32.4e" xref="A6.Ex4.m1.31.31.3.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="A6.Ex4.m1.32.32.4f" xref="A6.Ex4.m1.31.31.3.cmml"><mrow id="A6.Ex4.m1.32.32.4.29.7.7" xref="A6.Ex4.m1.31.31.3.cmml"><mstyle displaystyle="true" id="A6.Ex4.m1.23.23.23.1.1.1" xref="A6.Ex4.m1.23.23.23.1.1.1.cmml"><mfrac id="A6.Ex4.m1.23.23.23.1.1.1a" xref="A6.Ex4.m1.23.23.23.1.1.1.cmml"><mrow id="A6.Ex4.m1.23.23.23.1.1.1.1" xref="A6.Ex4.m1.23.23.23.1.1.1.1.cmml"><mi id="A6.Ex4.m1.23.23.23.1.1.1.1.3" xref="A6.Ex4.m1.23.23.23.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.Ex4.m1.23.23.23.1.1.1.1.2" xref="A6.Ex4.m1.23.23.23.1.1.1.1.2.cmml">​</mo><msub id="A6.Ex4.m1.23.23.23.1.1.1.1.4" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4.cmml"><mi id="A6.Ex4.m1.23.23.23.1.1.1.1.4.2" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4.2.cmml">σ</mi><mi id="A6.Ex4.m1.23.23.23.1.1.1.1.4.3" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.Ex4.m1.23.23.23.1.1.1.1.2a" xref="A6.Ex4.m1.23.23.23.1.1.1.1.2.cmml">​</mo><msub id="A6.Ex4.m1.23.23.23.1.1.1.1.5" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.cmml"><mrow id="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.2" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.2.1" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.1.1.cmml">∥</mo><mi id="A6.Ex4.m1.23.23.23.1.1.1.1.1" xref="A6.Ex4.m1.23.23.23.1.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.2.2" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.Ex4.m1.23.23.23.1.1.1.1.5.3" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.Ex4.m1.23.23.23.1.1.1.3" xref="A6.Ex4.m1.23.23.23.1.1.1.3.cmml"><mn id="A6.Ex4.m1.23.23.23.1.1.1.3.2" xref="A6.Ex4.m1.23.23.23.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.Ex4.m1.23.23.23.1.1.1.3.1" xref="A6.Ex4.m1.23.23.23.1.1.1.3.1.cmml">​</mo><msqrt id="A6.Ex4.m1.23.23.23.1.1.1.3.3" xref="A6.Ex4.m1.23.23.23.1.1.1.3.3.cmml"><mi id="A6.Ex4.m1.23.23.23.1.1.1.3.3.2" xref="A6.Ex4.m1.23.23.23.1.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A6.Ex4.m1.32.32.4.29.7.7.8" xref="A6.Ex4.m1.31.31.3.cmml">​</mo><mrow id="A6.Ex4.m1.32.32.4.29.7.7.7.1" xref="A6.Ex4.m1.31.31.3.cmml"><mo id="A6.Ex4.m1.24.24.24.2.2.2" xref="A6.Ex4.m1.31.31.3.cmml">(</mo><mrow id="A6.Ex4.m1.32.32.4.29.7.7.7.1.1" xref="A6.Ex4.m1.31.31.3.cmml"><mstyle displaystyle="true" id="A6.Ex4.m1.25.25.25.3.3.3" xref="A6.Ex4.m1.25.25.25.3.3.3.cmml"><mfrac id="A6.Ex4.m1.25.25.25.3.3.3a" xref="A6.Ex4.m1.25.25.25.3.3.3.cmml"><mn id="A6.Ex4.m1.25.25.25.3.3.3.2" xref="A6.Ex4.m1.25.25.25.3.3.3.2.cmml">1</mn><mn id="A6.Ex4.m1.25.25.25.3.3.3.3" xref="A6.Ex4.m1.25.25.25.3.3.3.3.cmml">256</mn></mfrac></mstyle><mo id="A6.Ex4.m1.26.26.26.4.4.4" xref="A6.Ex4.m1.26.26.26.4.4.4.cmml">+</mo><mstyle displaystyle="true" id="A6.Ex4.m1.27.27.27.5.5.5" xref="A6.Ex4.m1.27.27.27.5.5.5.cmml"><mfrac id="A6.Ex4.m1.27.27.27.5.5.5a" xref="A6.Ex4.m1.27.27.27.5.5.5.cmml"><mrow id="A6.Ex4.m1.27.27.27.5.5.5.3" xref="A6.Ex4.m1.27.27.27.5.5.5.3.cmml"><mn id="A6.Ex4.m1.27.27.27.5.5.5.3.5" xref="A6.Ex4.m1.27.27.27.5.5.5.3.5.cmml">8</mn><mo id="A6.Ex4.m1.27.27.27.5.5.5.3.4" xref="A6.Ex4.m1.27.27.27.5.5.5.3.4.cmml">+</mo><msqrt id="A6.Ex4.m1.27.27.27.5.5.5.3.3" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.cmml"><mrow id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.cmml"><mi id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.5" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.5.cmml">ν</mi><mo lspace="0em" rspace="0em" id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.4" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.4.cmml">​</mo><mrow id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.2" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.1.cmml"><mo stretchy="false" id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.2.1" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">(</mo><mi id="A6.Ex4.m1.27.27.27.5.5.5.1.1.1.1" xref="A6.Ex4.m1.27.27.27.5.5.5.1.1.1.1.cmml">C</mi><mo id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.2.2" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">,</mo><mi id="A6.Ex4.m1.27.27.27.5.5.5.2.2.2.2" xref="A6.Ex4.m1.27.27.27.5.5.5.2.2.2.2.cmml">D</mi><mo id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.2.3" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">,</mo><mi id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.3" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.3.cmml">δ</mi><mo stretchy="false" id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.2.4" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.1.cmml">)</mo></mrow></mrow></msqrt></mrow><msqrt id="A6.Ex4.m1.27.27.27.5.5.5.5" xref="A6.Ex4.m1.27.27.27.5.5.5.5.cmml"><mrow id="A6.Ex4.m1.27.27.27.5.5.5.5.2" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.cmml"><mn id="A6.Ex4.m1.27.27.27.5.5.5.5.2.2" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.Ex4.m1.27.27.27.5.5.5.5.2.1" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.1.cmml">​</mo><mi id="A6.Ex4.m1.27.27.27.5.5.5.5.2.3" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.3.cmml">n</mi></mrow></msqrt></mfrac></mstyle></mrow><mo id="A6.Ex4.m1.28.28.28.6.6.6" xref="A6.Ex4.m1.31.31.3.cmml">)</mo></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="A6.Ex4.m1.32b"><apply id="A6.Ex4.m1.31.31.3.cmml" xref="A6.Ex4.m1.32.32.4"><leq id="A6.Ex4.m1.11.11.11.11.8.8.cmml" xref="A6.Ex4.m1.11.11.11.11.8.8"></leq><apply id="A6.Ex4.m1.29.29.1.1.cmml" xref="A6.Ex4.m1.32.32.4"><times id="A6.Ex4.m1.29.29.1.1.2.cmml" xref="A6.Ex4.m1.32.32.4"></times><apply id="A6.Ex4.m1.29.29.1.1.3.cmml" xref="A6.Ex4.m1.32.32.4"><csymbol cd="ambiguous" id="A6.Ex4.m1.29.29.1.1.3.1.cmml" xref="A6.Ex4.m1.32.32.4">subscript</csymbol><ci id="A6.Ex4.m1.1.1.1.1.1.1.cmml" xref="A6.Ex4.m1.1.1.1.1.1.1">𝔼</ci><ci id="A6.Ex4.m1.2.2.2.2.2.2.1.cmml" xref="A6.Ex4.m1.2.2.2.2.2.2.1">𝒟</ci></apply><apply id="A6.Ex4.m1.29.29.1.1.1.2.cmml" xref="A6.Ex4.m1.32.32.4"><csymbol cd="latexml" id="A6.Ex4.m1.29.29.1.1.1.2.1.cmml" xref="A6.Ex4.m1.32.32.4">delimited-[]</csymbol><apply id="A6.Ex4.m1.29.29.1.1.1.1.1.cmml" xref="A6.Ex4.m1.32.32.4"><times id="A6.Ex4.m1.29.29.1.1.1.1.1.1.cmml" xref="A6.Ex4.m1.32.32.4"></times><ci id="A6.Ex4.m1.4.4.4.4.1.1.cmml" xref="A6.Ex4.m1.4.4.4.4.1.1">ℒ</ci><interval closure="open" id="A6.Ex4.m1.29.29.1.1.1.1.1.3.cmml" xref="A6.Ex4.m1.32.32.4"><ci id="A6.Ex4.m1.6.6.6.6.3.3.cmml" xref="A6.Ex4.m1.6.6.6.6.3.3">𝒂</ci><ci id="A6.Ex4.m1.8.8.8.8.5.5.cmml" xref="A6.Ex4.m1.8.8.8.8.5.5">𝒃</ci></interval></apply></apply></apply><apply id="A6.Ex4.m1.31.31.3.3.cmml" xref="A6.Ex4.m1.32.32.4"><plus id="A6.Ex4.m1.22.22.22.22.19.19.cmml" xref="A6.Ex4.m1.22.22.22.22.19.19"></plus><apply id="A6.Ex4.m1.30.30.2.2.1.cmml" xref="A6.Ex4.m1.32.32.4"><times id="A6.Ex4.m1.30.30.2.2.1.2.cmml" xref="A6.Ex4.m1.32.32.4"></times><apply id="A6.Ex4.m1.30.30.2.2.1.3.cmml" xref="A6.Ex4.m1.32.32.4"><csymbol cd="ambiguous" id="A6.Ex4.m1.30.30.2.2.1.3.1.cmml" xref="A6.Ex4.m1.32.32.4">subscript</csymbol><ci id="A6.Ex4.m1.12.12.12.12.9.9.cmml" xref="A6.Ex4.m1.12.12.12.12.9.9">𝔼</ci><apply id="A6.Ex4.m1.13.13.13.13.10.10.1.cmml" xref="A6.Ex4.m1.13.13.13.13.10.10.1"><ci id="A6.Ex4.m1.13.13.13.13.10.10.1.1.cmml" xref="A6.Ex4.m1.13.13.13.13.10.10.1.1">~</ci><ci id="A6.Ex4.m1.13.13.13.13.10.10.1.2.cmml" xref="A6.Ex4.m1.13.13.13.13.10.10.1.2">𝑨</ci></apply></apply><apply id="A6.Ex4.m1.30.30.2.2.1.1.2.cmml" xref="A6.Ex4.m1.32.32.4"><csymbol cd="latexml" id="A6.Ex4.m1.30.30.2.2.1.1.2.1.cmml" xref="A6.Ex4.m1.32.32.4">delimited-[]</csymbol><apply id="A6.Ex4.m1.30.30.2.2.1.1.1.1.cmml" xref="A6.Ex4.m1.32.32.4"><times id="A6.Ex4.m1.30.30.2.2.1.1.1.1.1.cmml" xref="A6.Ex4.m1.32.32.4"></times><ci id="A6.Ex4.m1.15.15.15.15.12.12.cmml" xref="A6.Ex4.m1.15.15.15.15.12.12">ℒ</ci><interval closure="open" id="A6.Ex4.m1.30.30.2.2.1.1.1.1.3.cmml" xref="A6.Ex4.m1.32.32.4"><ci id="A6.Ex4.m1.17.17.17.17.14.14.cmml" xref="A6.Ex4.m1.17.17.17.17.14.14">𝒂</ci><ci id="A6.Ex4.m1.19.19.19.19.16.16.cmml" xref="A6.Ex4.m1.19.19.19.19.16.16">𝒃</ci></interval></apply></apply></apply><apply id="A6.Ex4.m1.31.31.3.3.2.cmml" xref="A6.Ex4.m1.32.32.4"><times id="A6.Ex4.m1.31.31.3.3.2.2.cmml" xref="A6.Ex4.m1.32.32.4"></times><apply id="A6.Ex4.m1.23.23.23.1.1.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1"><divide id="A6.Ex4.m1.23.23.23.1.1.1.2.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1"></divide><apply id="A6.Ex4.m1.23.23.23.1.1.1.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1"><times id="A6.Ex4.m1.23.23.23.1.1.1.1.2.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.2"></times><ci id="A6.Ex4.m1.23.23.23.1.1.1.1.3.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.3">𝐶</ci><apply id="A6.Ex4.m1.23.23.23.1.1.1.1.4.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4"><csymbol cd="ambiguous" id="A6.Ex4.m1.23.23.23.1.1.1.1.4.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4">subscript</csymbol><ci id="A6.Ex4.m1.23.23.23.1.1.1.1.4.2.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4.2">𝜎</ci><ci id="A6.Ex4.m1.23.23.23.1.1.1.1.4.3.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.4.3">𝐴</ci></apply><apply id="A6.Ex4.m1.23.23.23.1.1.1.1.5.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5"><csymbol cd="ambiguous" id="A6.Ex4.m1.23.23.23.1.1.1.1.5.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5">subscript</csymbol><apply id="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.2"><csymbol cd="latexml" id="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.1.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.Ex4.m1.23.23.23.1.1.1.1.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.Ex4.m1.23.23.23.1.1.1.1.5.3.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.1.5.3">2</cn></apply></apply><apply id="A6.Ex4.m1.23.23.23.1.1.1.3.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.3"><times id="A6.Ex4.m1.23.23.23.1.1.1.3.1.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.3.1"></times><cn type="integer" id="A6.Ex4.m1.23.23.23.1.1.1.3.2.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.3.2">2</cn><apply id="A6.Ex4.m1.23.23.23.1.1.1.3.3.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.3.3"><root id="A6.Ex4.m1.23.23.23.1.1.1.3.3a.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.3.3"></root><ci id="A6.Ex4.m1.23.23.23.1.1.1.3.3.2.cmml" xref="A6.Ex4.m1.23.23.23.1.1.1.3.3.2">𝜆</ci></apply></apply></apply><apply id="A6.Ex4.m1.31.31.3.3.2.1.1.1.cmml" xref="A6.Ex4.m1.32.32.4"><plus id="A6.Ex4.m1.26.26.26.4.4.4.cmml" xref="A6.Ex4.m1.26.26.26.4.4.4"></plus><apply id="A6.Ex4.m1.25.25.25.3.3.3.cmml" xref="A6.Ex4.m1.25.25.25.3.3.3"><divide id="A6.Ex4.m1.25.25.25.3.3.3.1.cmml" xref="A6.Ex4.m1.25.25.25.3.3.3"></divide><cn type="integer" id="A6.Ex4.m1.25.25.25.3.3.3.2.cmml" xref="A6.Ex4.m1.25.25.25.3.3.3.2">1</cn><cn type="integer" id="A6.Ex4.m1.25.25.25.3.3.3.3.cmml" xref="A6.Ex4.m1.25.25.25.3.3.3.3">256</cn></apply><apply id="A6.Ex4.m1.27.27.27.5.5.5.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5"><divide id="A6.Ex4.m1.27.27.27.5.5.5.4.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5"></divide><apply id="A6.Ex4.m1.27.27.27.5.5.5.3.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3"><plus id="A6.Ex4.m1.27.27.27.5.5.5.3.4.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.4"></plus><cn type="integer" id="A6.Ex4.m1.27.27.27.5.5.5.3.5.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.5">8</cn><apply id="A6.Ex4.m1.27.27.27.5.5.5.3.3.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3"><root id="A6.Ex4.m1.27.27.27.5.5.5.3.3a.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3"></root><apply id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3"><times id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.4.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.4"></times><ci id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.5.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.5">𝜈</ci><vector id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.1.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.6.2"><ci id="A6.Ex4.m1.27.27.27.5.5.5.1.1.1.1.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.1.1.1.1">𝐶</ci><ci id="A6.Ex4.m1.27.27.27.5.5.5.2.2.2.2.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.2.2.2.2">𝐷</ci><ci id="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.3.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.3.3.3.3">𝛿</ci></vector></apply></apply></apply><apply id="A6.Ex4.m1.27.27.27.5.5.5.5.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.5"><root id="A6.Ex4.m1.27.27.27.5.5.5.5a.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.5"></root><apply id="A6.Ex4.m1.27.27.27.5.5.5.5.2.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2"><times id="A6.Ex4.m1.27.27.27.5.5.5.5.2.1.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.1"></times><cn type="integer" id="A6.Ex4.m1.27.27.27.5.5.5.5.2.2.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.2">2</cn><ci id="A6.Ex4.m1.27.27.27.5.5.5.5.2.3.cmml" xref="A6.Ex4.m1.27.27.27.5.5.5.5.2.3">𝑛</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Ex4.m1.32c">\displaystyle\begin{split}\mathbb{E}_{\mathcal{D}}[&amp;\mathcal{L}(\bm{a},\bm{b})]\leq\mathbb{E}_{\tilde{\bm{A}}}[\mathcal{L}(\bm{a},\bm{b})]+\\
&amp;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}\left(\frac{1}{256}+\frac{8+\sqrt{\nu(C,D,\delta)}}{\sqrt{2n}}\right)\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="Thmtheoremx1.p1.14" class="ltx_p"><span id="Thmtheoremx1.p1.14.3" class="ltx_text ltx_font_italic">where <math id="Thmtheoremx1.p1.12.1.m1.5" class="ltx_Math" alttext="\mathcal{L}(\bm{a},\bm{b})\triangleq|\bm{a}^{\top}\bm{b}-\alpha f(g(\bm{a}),h(\bm{b}))-\bm{\beta}|" display="inline"><semantics id="Thmtheoremx1.p1.12.1.m1.5a"><mrow id="Thmtheoremx1.p1.12.1.m1.5.5" xref="Thmtheoremx1.p1.12.1.m1.5.5.cmml"><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.cmml"><mi class="ltx_font_mathcaligraphic" id="Thmtheoremx1.p1.12.1.m1.5.5.3.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="Thmtheoremx1.p1.12.1.m1.5.5.3.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.1.cmml">​</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.3.3.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.3.1.cmml"><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.3.3.2.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.3.1.cmml">(</mo><mi id="Thmtheoremx1.p1.12.1.m1.1.1" xref="Thmtheoremx1.p1.12.1.m1.1.1.cmml">𝐚</mi><mo id="Thmtheoremx1.p1.12.1.m1.5.5.3.3.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.3.1.cmml">,</mo><mi id="Thmtheoremx1.p1.12.1.m1.2.2" xref="Thmtheoremx1.p1.12.1.m1.2.2.cmml">𝐛</mi><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.3.3.2.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.3.1.cmml">)</mo></mrow></mrow><mo id="Thmtheoremx1.p1.12.1.m1.5.5.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.2.cmml">≜</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.2.cmml"><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.2.1.cmml">|</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.cmml"><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.cmml"><msup id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.cmml"><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.2.cmml">𝐚</mi><mo id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.1.cmml">​</mo><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.3.cmml">𝐛</mi></mrow><mo id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.3.cmml">−</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.cmml"><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.4" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.4.cmml">α</mi><mo lspace="0em" rspace="0em" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.3.cmml">​</mo><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.5" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.3a" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.3.cmml">​</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml"><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml">(</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml"><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.3.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.3.2.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mi id="Thmtheoremx1.p1.12.1.m1.3.3" xref="Thmtheoremx1.p1.12.1.m1.3.3.cmml">𝐚</mi><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.3.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.4" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml">,</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml"><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1.cmml">​</mo><mrow id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.3.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml"><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.3.2.1" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml">(</mo><mi id="Thmtheoremx1.p1.12.1.m1.4.4" xref="Thmtheoremx1.p1.12.1.m1.4.4.cmml">𝐛</mi><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.3.2.2" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.5" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.3a" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.3.cmml">−</mo><mi id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.5" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.5.cmml">𝛃</mi></mrow><mo stretchy="false" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.3" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.12.1.m1.5b"><apply id="Thmtheoremx1.p1.12.1.m1.5.5.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5"><ci id="Thmtheoremx1.p1.12.1.m1.5.5.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.2">≜</ci><apply id="Thmtheoremx1.p1.12.1.m1.5.5.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.3"><times id="Thmtheoremx1.p1.12.1.m1.5.5.3.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.1"></times><ci id="Thmtheoremx1.p1.12.1.m1.5.5.3.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.2">ℒ</ci><interval closure="open" id="Thmtheoremx1.p1.12.1.m1.5.5.3.3.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.3.3.2"><ci id="Thmtheoremx1.p1.12.1.m1.1.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.1.1">𝐚</ci><ci id="Thmtheoremx1.p1.12.1.m1.2.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.2.2">𝐛</ci></interval></apply><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1"><abs id="Thmtheoremx1.p1.12.1.m1.5.5.1.2.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.2"></abs><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1"><minus id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.3"></minus><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4"><times id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.1"></times><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2"><csymbol cd="ambiguous" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2">superscript</csymbol><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.2">𝐚</ci><csymbol cd="latexml" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.2.3">top</csymbol></apply><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.4.3">𝐛</ci></apply><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2"><times id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.3"></times><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.4.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.4">𝛼</ci><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.5.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.5">𝑓</ci><interval closure="open" id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2"><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1"><times id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.1"></times><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.1.1.1.1.2">𝑔</ci><ci id="Thmtheoremx1.p1.12.1.m1.3.3.cmml" xref="Thmtheoremx1.p1.12.1.m1.3.3">𝐚</ci></apply><apply id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2"><times id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.1"></times><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.2.2.2.2.2">ℎ</ci><ci id="Thmtheoremx1.p1.12.1.m1.4.4.cmml" xref="Thmtheoremx1.p1.12.1.m1.4.4">𝐛</ci></apply></interval></apply><ci id="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.5.cmml" xref="Thmtheoremx1.p1.12.1.m1.5.5.1.1.1.5">𝛃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.12.1.m1.5c">\mathcal{L}(\bm{a},\bm{b})\triangleq|\bm{a}^{\top}\bm{b}-\alpha f(g(\bm{a}),h(\bm{b}))-\bm{\beta}|</annotation></semantics></math>, <math id="Thmtheoremx1.p1.13.2.m2.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="Thmtheoremx1.p1.13.2.m2.1a"><mi id="Thmtheoremx1.p1.13.2.m2.1.1" xref="Thmtheoremx1.p1.13.2.m2.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.13.2.m2.1b"><ci id="Thmtheoremx1.p1.13.2.m2.1.1.cmml" xref="Thmtheoremx1.p1.13.2.m2.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.13.2.m2.1c">\alpha</annotation></semantics></math> is the scale used for quantizing the lookup tables, <math id="Thmtheoremx1.p1.14.3.m3.1" class="ltx_Math" alttext="\bm{\beta}" display="inline"><semantics id="Thmtheoremx1.p1.14.3.m3.1a"><mi id="Thmtheoremx1.p1.14.3.m3.1.1" xref="Thmtheoremx1.p1.14.3.m3.1.1.cmml">𝛃</mi><annotation-xml encoding="MathML-Content" id="Thmtheoremx1.p1.14.3.m3.1b"><ci id="Thmtheoremx1.p1.14.3.m3.1.1.cmml" xref="Thmtheoremx1.p1.14.3.m3.1.1">𝛃</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmtheoremx1.p1.14.3.m3.1c">\bm{\beta}</annotation></semantics></math> is the constants used in quantizing the lookup tables plus the debiasing constant of Section <a href="#S4.SS4" title="4.4 Fast 8-bit Aggregation, 𝒇⁢(⋅,⋅) ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>, and</span></p>
<table id="A6.EGx19" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.Ex5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.Ex5.m1.5" class="ltx_Math" alttext="\displaystyle\nu(C,D,\delta)\triangleq C(4\left\lceil\log_{2}(D)\right\rceil+256)\log{2}-\log{\delta}." display="inline"><semantics id="A6.Ex5.m1.5a"><mrow id="A6.Ex5.m1.5.5.1" xref="A6.Ex5.m1.5.5.1.1.cmml"><mrow id="A6.Ex5.m1.5.5.1.1" xref="A6.Ex5.m1.5.5.1.1.cmml"><mrow id="A6.Ex5.m1.5.5.1.1.3" xref="A6.Ex5.m1.5.5.1.1.3.cmml"><mi id="A6.Ex5.m1.5.5.1.1.3.2" xref="A6.Ex5.m1.5.5.1.1.3.2.cmml">ν</mi><mo lspace="0em" rspace="0em" id="A6.Ex5.m1.5.5.1.1.3.1" xref="A6.Ex5.m1.5.5.1.1.3.1.cmml">​</mo><mrow id="A6.Ex5.m1.5.5.1.1.3.3.2" xref="A6.Ex5.m1.5.5.1.1.3.3.1.cmml"><mo stretchy="false" id="A6.Ex5.m1.5.5.1.1.3.3.2.1" xref="A6.Ex5.m1.5.5.1.1.3.3.1.cmml">(</mo><mi id="A6.Ex5.m1.1.1" xref="A6.Ex5.m1.1.1.cmml">C</mi><mo id="A6.Ex5.m1.5.5.1.1.3.3.2.2" xref="A6.Ex5.m1.5.5.1.1.3.3.1.cmml">,</mo><mi id="A6.Ex5.m1.2.2" xref="A6.Ex5.m1.2.2.cmml">D</mi><mo id="A6.Ex5.m1.5.5.1.1.3.3.2.3" xref="A6.Ex5.m1.5.5.1.1.3.3.1.cmml">,</mo><mi id="A6.Ex5.m1.3.3" xref="A6.Ex5.m1.3.3.cmml">δ</mi><mo stretchy="false" id="A6.Ex5.m1.5.5.1.1.3.3.2.4" xref="A6.Ex5.m1.5.5.1.1.3.3.1.cmml">)</mo></mrow></mrow><mo id="A6.Ex5.m1.5.5.1.1.2" xref="A6.Ex5.m1.5.5.1.1.2.cmml">≜</mo><mrow id="A6.Ex5.m1.5.5.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.cmml"><mrow id="A6.Ex5.m1.5.5.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.cmml"><mi id="A6.Ex5.m1.5.5.1.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.Ex5.m1.5.5.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.cmml"><mrow id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.cmml"><mn id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml"><mo id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml">⌈</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><msub id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">log</mi><mn id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1a" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml">(</mo><mi id="A6.Ex5.m1.4.4" xref="A6.Ex5.m1.4.4.cmml">D</mi><mo stretchy="false" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml">⌉</mo></mrow></mrow><mo id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.2.cmml">+</mo><mn id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.3.cmml">256</mn></mrow><mo stretchy="false" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="A6.Ex5.m1.5.5.1.1.1.1.2a" xref="A6.Ex5.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.1.4" xref="A6.Ex5.m1.5.5.1.1.1.1.4.cmml"><mi id="A6.Ex5.m1.5.5.1.1.1.1.4.1" xref="A6.Ex5.m1.5.5.1.1.1.1.4.1.cmml">log</mi><mo lspace="0.167em" id="A6.Ex5.m1.5.5.1.1.1.1.4a" xref="A6.Ex5.m1.5.5.1.1.1.1.4.cmml">⁡</mo><mn id="A6.Ex5.m1.5.5.1.1.1.1.4.2" xref="A6.Ex5.m1.5.5.1.1.1.1.4.2.cmml">2</mn></mrow></mrow><mo id="A6.Ex5.m1.5.5.1.1.1.2" xref="A6.Ex5.m1.5.5.1.1.1.2.cmml">−</mo><mrow id="A6.Ex5.m1.5.5.1.1.1.3" xref="A6.Ex5.m1.5.5.1.1.1.3.cmml"><mi id="A6.Ex5.m1.5.5.1.1.1.3.1" xref="A6.Ex5.m1.5.5.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="A6.Ex5.m1.5.5.1.1.1.3a" xref="A6.Ex5.m1.5.5.1.1.1.3.cmml">⁡</mo><mi id="A6.Ex5.m1.5.5.1.1.1.3.2" xref="A6.Ex5.m1.5.5.1.1.1.3.2.cmml">δ</mi></mrow></mrow></mrow><mo lspace="0em" id="A6.Ex5.m1.5.5.1.2" xref="A6.Ex5.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.Ex5.m1.5b"><apply id="A6.Ex5.m1.5.5.1.1.cmml" xref="A6.Ex5.m1.5.5.1"><ci id="A6.Ex5.m1.5.5.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.2">≜</ci><apply id="A6.Ex5.m1.5.5.1.1.3.cmml" xref="A6.Ex5.m1.5.5.1.1.3"><times id="A6.Ex5.m1.5.5.1.1.3.1.cmml" xref="A6.Ex5.m1.5.5.1.1.3.1"></times><ci id="A6.Ex5.m1.5.5.1.1.3.2.cmml" xref="A6.Ex5.m1.5.5.1.1.3.2">𝜈</ci><vector id="A6.Ex5.m1.5.5.1.1.3.3.1.cmml" xref="A6.Ex5.m1.5.5.1.1.3.3.2"><ci id="A6.Ex5.m1.1.1.cmml" xref="A6.Ex5.m1.1.1">𝐶</ci><ci id="A6.Ex5.m1.2.2.cmml" xref="A6.Ex5.m1.2.2">𝐷</ci><ci id="A6.Ex5.m1.3.3.cmml" xref="A6.Ex5.m1.3.3">𝛿</ci></vector></apply><apply id="A6.Ex5.m1.5.5.1.1.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1"><minus id="A6.Ex5.m1.5.5.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.2"></minus><apply id="A6.Ex5.m1.5.5.1.1.1.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1"><times id="A6.Ex5.m1.5.5.1.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.2"></times><ci id="A6.Ex5.m1.5.5.1.1.1.1.3.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.3">𝐶</ci><apply id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1"><plus id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.2"></plus><apply id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1"><times id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.2"></times><cn type="integer" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.3.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.3">4</cn><apply id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1"><ceiling id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.2"></ceiling><apply id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1"><apply id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><log id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.2"></log><cn type="integer" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.3">2</cn></apply><ci id="A6.Ex5.m1.4.4.cmml" xref="A6.Ex5.m1.4.4">𝐷</ci></apply></apply></apply><cn type="integer" id="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.3.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.1.1.1.3">256</cn></apply><apply id="A6.Ex5.m1.5.5.1.1.1.1.4.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.4"><log id="A6.Ex5.m1.5.5.1.1.1.1.4.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.4.1"></log><cn type="integer" id="A6.Ex5.m1.5.5.1.1.1.1.4.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.1.4.2">2</cn></apply></apply><apply id="A6.Ex5.m1.5.5.1.1.1.3.cmml" xref="A6.Ex5.m1.5.5.1.1.1.3"><log id="A6.Ex5.m1.5.5.1.1.1.3.1.cmml" xref="A6.Ex5.m1.5.5.1.1.1.3.1"></log><ci id="A6.Ex5.m1.5.5.1.1.1.3.2.cmml" xref="A6.Ex5.m1.5.5.1.1.1.3.2">𝛿</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Ex5.m1.5c">\displaystyle\nu(C,D,\delta)\triangleq C(4\left\lceil\log_{2}(D)\right\rceil+256)\log{2}-\log{\delta}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A6.SS2.p2" class="ltx_para">
<p id="A6.SS2.p2.5" class="ltx_p">The proof relies on the observation that <span id="A6.SS2.p2.5.1" class="ltx_text ltx_font_smallcaps">Maddness</span>’s training procedure can be decomposed into two sequential subroutines: <span id="A6.SS2.p2.5.2" class="ltx_text ltx_font_typewriter">Maddness-Build-Tree</span>, which learns the function <math id="A6.SS2.p2.1.m1.1" class="ltx_Math" alttext="g(\bm{a})" display="inline"><semantics id="A6.SS2.p2.1.m1.1a"><mrow id="A6.SS2.p2.1.m1.1.2" xref="A6.SS2.p2.1.m1.1.2.cmml"><mi id="A6.SS2.p2.1.m1.1.2.2" xref="A6.SS2.p2.1.m1.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.SS2.p2.1.m1.1.2.1" xref="A6.SS2.p2.1.m1.1.2.1.cmml">​</mo><mrow id="A6.SS2.p2.1.m1.1.2.3.2" xref="A6.SS2.p2.1.m1.1.2.cmml"><mo stretchy="false" id="A6.SS2.p2.1.m1.1.2.3.2.1" xref="A6.SS2.p2.1.m1.1.2.cmml">(</mo><mi id="A6.SS2.p2.1.m1.1.1" xref="A6.SS2.p2.1.m1.1.1.cmml">𝒂</mi><mo stretchy="false" id="A6.SS2.p2.1.m1.1.2.3.2.2" xref="A6.SS2.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.p2.1.m1.1b"><apply id="A6.SS2.p2.1.m1.1.2.cmml" xref="A6.SS2.p2.1.m1.1.2"><times id="A6.SS2.p2.1.m1.1.2.1.cmml" xref="A6.SS2.p2.1.m1.1.2.1"></times><ci id="A6.SS2.p2.1.m1.1.2.2.cmml" xref="A6.SS2.p2.1.m1.1.2.2">𝑔</ci><ci id="A6.SS2.p2.1.m1.1.1.cmml" xref="A6.SS2.p2.1.m1.1.1">𝒂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p2.1.m1.1c">g(\bm{a})</annotation></semantics></math> by constructing a binary decision tree, and <span id="A6.SS2.p2.5.3" class="ltx_text ltx_font_typewriter">Maddness-Regress</span>, which learns the function <math id="A6.SS2.p2.2.m2.1" class="ltx_Math" alttext="h(\bm{b})" display="inline"><semantics id="A6.SS2.p2.2.m2.1a"><mrow id="A6.SS2.p2.2.m2.1.2" xref="A6.SS2.p2.2.m2.1.2.cmml"><mi id="A6.SS2.p2.2.m2.1.2.2" xref="A6.SS2.p2.2.m2.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A6.SS2.p2.2.m2.1.2.1" xref="A6.SS2.p2.2.m2.1.2.1.cmml">​</mo><mrow id="A6.SS2.p2.2.m2.1.2.3.2" xref="A6.SS2.p2.2.m2.1.2.cmml"><mo stretchy="false" id="A6.SS2.p2.2.m2.1.2.3.2.1" xref="A6.SS2.p2.2.m2.1.2.cmml">(</mo><mi id="A6.SS2.p2.2.m2.1.1" xref="A6.SS2.p2.2.m2.1.1.cmml">𝒃</mi><mo stretchy="false" id="A6.SS2.p2.2.m2.1.2.3.2.2" xref="A6.SS2.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.p2.2.m2.1b"><apply id="A6.SS2.p2.2.m2.1.2.cmml" xref="A6.SS2.p2.2.m2.1.2"><times id="A6.SS2.p2.2.m2.1.2.1.cmml" xref="A6.SS2.p2.2.m2.1.2.1"></times><ci id="A6.SS2.p2.2.m2.1.2.2.cmml" xref="A6.SS2.p2.2.m2.1.2.2">ℎ</ci><ci id="A6.SS2.p2.2.m2.1.1.cmml" xref="A6.SS2.p2.2.m2.1.1">𝒃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p2.2.m2.1c">h(\bm{b})</annotation></semantics></math> by optimizing a prototype matrix <math id="A6.SS2.p2.3.m3.1" class="ltx_Math" alttext="\bm{P}" display="inline"><semantics id="A6.SS2.p2.3.m3.1a"><mi id="A6.SS2.p2.3.m3.1.1" xref="A6.SS2.p2.3.m3.1.1.cmml">𝑷</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.p2.3.m3.1b"><ci id="A6.SS2.p2.3.m3.1.1.cmml" xref="A6.SS2.p2.3.m3.1.1">𝑷</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p2.3.m3.1c">\bm{P}</annotation></semantics></math> such that <math id="A6.SS2.p2.4.m4.1" class="ltx_Math" alttext="g(\tilde{\bm{A}})\bm{P}\approx\tilde{\bm{A}}" display="inline"><semantics id="A6.SS2.p2.4.m4.1a"><mrow id="A6.SS2.p2.4.m4.1.2" xref="A6.SS2.p2.4.m4.1.2.cmml"><mrow id="A6.SS2.p2.4.m4.1.2.2" xref="A6.SS2.p2.4.m4.1.2.2.cmml"><mi id="A6.SS2.p2.4.m4.1.2.2.2" xref="A6.SS2.p2.4.m4.1.2.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.SS2.p2.4.m4.1.2.2.1" xref="A6.SS2.p2.4.m4.1.2.2.1.cmml">​</mo><mrow id="A6.SS2.p2.4.m4.1.2.2.3.2" xref="A6.SS2.p2.4.m4.1.1.cmml"><mo stretchy="false" id="A6.SS2.p2.4.m4.1.2.2.3.2.1" xref="A6.SS2.p2.4.m4.1.1.cmml">(</mo><mover accent="true" id="A6.SS2.p2.4.m4.1.1" xref="A6.SS2.p2.4.m4.1.1.cmml"><mi id="A6.SS2.p2.4.m4.1.1.2" xref="A6.SS2.p2.4.m4.1.1.2.cmml">𝑨</mi><mo id="A6.SS2.p2.4.m4.1.1.1" xref="A6.SS2.p2.4.m4.1.1.1.cmml">~</mo></mover><mo stretchy="false" id="A6.SS2.p2.4.m4.1.2.2.3.2.2" xref="A6.SS2.p2.4.m4.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A6.SS2.p2.4.m4.1.2.2.1a" xref="A6.SS2.p2.4.m4.1.2.2.1.cmml">​</mo><mi id="A6.SS2.p2.4.m4.1.2.2.4" xref="A6.SS2.p2.4.m4.1.2.2.4.cmml">𝑷</mi></mrow><mo id="A6.SS2.p2.4.m4.1.2.1" xref="A6.SS2.p2.4.m4.1.2.1.cmml">≈</mo><mover accent="true" id="A6.SS2.p2.4.m4.1.2.3" xref="A6.SS2.p2.4.m4.1.2.3.cmml"><mi id="A6.SS2.p2.4.m4.1.2.3.2" xref="A6.SS2.p2.4.m4.1.2.3.2.cmml">𝑨</mi><mo id="A6.SS2.p2.4.m4.1.2.3.1" xref="A6.SS2.p2.4.m4.1.2.3.1.cmml">~</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.p2.4.m4.1b"><apply id="A6.SS2.p2.4.m4.1.2.cmml" xref="A6.SS2.p2.4.m4.1.2"><approx id="A6.SS2.p2.4.m4.1.2.1.cmml" xref="A6.SS2.p2.4.m4.1.2.1"></approx><apply id="A6.SS2.p2.4.m4.1.2.2.cmml" xref="A6.SS2.p2.4.m4.1.2.2"><times id="A6.SS2.p2.4.m4.1.2.2.1.cmml" xref="A6.SS2.p2.4.m4.1.2.2.1"></times><ci id="A6.SS2.p2.4.m4.1.2.2.2.cmml" xref="A6.SS2.p2.4.m4.1.2.2.2">𝑔</ci><apply id="A6.SS2.p2.4.m4.1.1.cmml" xref="A6.SS2.p2.4.m4.1.2.2.3.2"><ci id="A6.SS2.p2.4.m4.1.1.1.cmml" xref="A6.SS2.p2.4.m4.1.1.1">~</ci><ci id="A6.SS2.p2.4.m4.1.1.2.cmml" xref="A6.SS2.p2.4.m4.1.1.2">𝑨</ci></apply><ci id="A6.SS2.p2.4.m4.1.2.2.4.cmml" xref="A6.SS2.p2.4.m4.1.2.2.4">𝑷</ci></apply><apply id="A6.SS2.p2.4.m4.1.2.3.cmml" xref="A6.SS2.p2.4.m4.1.2.3"><ci id="A6.SS2.p2.4.m4.1.2.3.1.cmml" xref="A6.SS2.p2.4.m4.1.2.3.1">~</ci><ci id="A6.SS2.p2.4.m4.1.2.3.2.cmml" xref="A6.SS2.p2.4.m4.1.2.3.2">𝑨</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p2.4.m4.1c">g(\tilde{\bm{A}})\bm{P}\approx\tilde{\bm{A}}</annotation></semantics></math>.
This observation allows us to prove <a href="#S4.Thmtheorem1" title="Theorem 4.1 (Generalization Error of Maddness). ‣ 4.5 Theoretical Guarantees ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a> by first providing a guarantee for 
<br class="ltx_break"><span id="A6.SS2.p2.5.4" class="ltx_text ltx_font_typewriter">Maddness-Regress</span> for a fixed <span id="A6.SS2.p2.5.5" class="ltx_text ltx_font_typewriter">Maddness-Build-Tree</span> hypothesis, and then union bounding over the hypothesis space for <span id="A6.SS2.p2.5.6" class="ltx_text ltx_font_typewriter">Maddness-Build-Tree</span>. Bounding the size of the hypothesis space is straightforward (Lemma <a href="#A6.Thmlemma1" title="Lemma F.1 (Number of Hypotheses for Maddness-Build-Tree). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.1</span></a>), so the bulk of this section focuses on providing a guarantee for <span id="A6.SS2.p2.5.7" class="ltx_text ltx_font_typewriter">Maddness-Regress</span>. We must also prove a bound on the loss contributed by quantizing the lookup tables array <math id="A6.SS2.p2.5.m5.1" class="ltx_Math" alttext="\bm{P}^{\top}\bm{b}" display="inline"><semantics id="A6.SS2.p2.5.m5.1a"><mrow id="A6.SS2.p2.5.m5.1.1" xref="A6.SS2.p2.5.m5.1.1.cmml"><msup id="A6.SS2.p2.5.m5.1.1.2" xref="A6.SS2.p2.5.m5.1.1.2.cmml"><mi id="A6.SS2.p2.5.m5.1.1.2.2" xref="A6.SS2.p2.5.m5.1.1.2.2.cmml">𝑷</mi><mo id="A6.SS2.p2.5.m5.1.1.2.3" xref="A6.SS2.p2.5.m5.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.p2.5.m5.1.1.1" xref="A6.SS2.p2.5.m5.1.1.1.cmml">​</mo><mi id="A6.SS2.p2.5.m5.1.1.3" xref="A6.SS2.p2.5.m5.1.1.3.cmml">𝒃</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.p2.5.m5.1b"><apply id="A6.SS2.p2.5.m5.1.1.cmml" xref="A6.SS2.p2.5.m5.1.1"><times id="A6.SS2.p2.5.m5.1.1.1.cmml" xref="A6.SS2.p2.5.m5.1.1.1"></times><apply id="A6.SS2.p2.5.m5.1.1.2.cmml" xref="A6.SS2.p2.5.m5.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.p2.5.m5.1.1.2.1.cmml" xref="A6.SS2.p2.5.m5.1.1.2">superscript</csymbol><ci id="A6.SS2.p2.5.m5.1.1.2.2.cmml" xref="A6.SS2.p2.5.m5.1.1.2.2">𝑷</ci><csymbol cd="latexml" id="A6.SS2.p2.5.m5.1.1.2.3.cmml" xref="A6.SS2.p2.5.m5.1.1.2.3">top</csymbol></apply><ci id="A6.SS2.p2.5.m5.1.1.3.cmml" xref="A6.SS2.p2.5.m5.1.1.3">𝒃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p2.5.m5.1c">\bm{P}^{\top}\bm{b}</annotation></semantics></math>.</p>
</div>
<div id="A6.Thmlemma1" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmlemma1.1.1.1" class="ltx_text ltx_font_bold">Lemma F.1</span></span><span id="A6.Thmlemma1.2.2" class="ltx_text ltx_font_bold"> </span>(Number of Hypotheses for <span id="A6.Thmlemma1.3.3" class="ltx_text ltx_font_typewriter">Maddness-Build-Tree</span>)<span id="A6.Thmlemma1.4.4" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmlemma1.p1" class="ltx_para">
<p id="A6.Thmlemma1.p1.4" class="ltx_p"><span id="A6.Thmlemma1.p1.4.4" class="ltx_text ltx_font_italic">Let <math id="A6.Thmlemma1.p1.1.1.m1.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.Thmlemma1.p1.1.1.m1.1a"><mi id="A6.Thmlemma1.p1.1.1.m1.1.1" xref="A6.Thmlemma1.p1.1.1.m1.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma1.p1.1.1.m1.1b"><ci id="A6.Thmlemma1.p1.1.1.m1.1.1.cmml" xref="A6.Thmlemma1.p1.1.1.m1.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma1.p1.1.1.m1.1c">C</annotation></semantics></math> be the number of codebooks used by <span id="A6.Thmlemma1.p1.4.4.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.Thmlemma1.p1.4.4.1.1" class="ltx_text ltx_font_italic"> </span></span>and let <math id="A6.Thmlemma1.p1.2.2.m2.1" class="ltx_Math" alttext="D" display="inline"><semantics id="A6.Thmlemma1.p1.2.2.m2.1a"><mi id="A6.Thmlemma1.p1.2.2.m2.1.1" xref="A6.Thmlemma1.p1.2.2.m2.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma1.p1.2.2.m2.1b"><ci id="A6.Thmlemma1.p1.2.2.m2.1.1.cmml" xref="A6.Thmlemma1.p1.2.2.m2.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma1.p1.2.2.m2.1c">D</annotation></semantics></math> be the number of columns in the matrix <math id="A6.Thmlemma1.p1.3.3.m3.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="A6.Thmlemma1.p1.3.3.m3.1a"><mover accent="true" id="A6.Thmlemma1.p1.3.3.m3.1.1" xref="A6.Thmlemma1.p1.3.3.m3.1.1.cmml"><mi id="A6.Thmlemma1.p1.3.3.m3.1.1.2" xref="A6.Thmlemma1.p1.3.3.m3.1.1.2.cmml">𝐀</mi><mo id="A6.Thmlemma1.p1.3.3.m3.1.1.1" xref="A6.Thmlemma1.p1.3.3.m3.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="A6.Thmlemma1.p1.3.3.m3.1b"><apply id="A6.Thmlemma1.p1.3.3.m3.1.1.cmml" xref="A6.Thmlemma1.p1.3.3.m3.1.1"><ci id="A6.Thmlemma1.p1.3.3.m3.1.1.1.cmml" xref="A6.Thmlemma1.p1.3.3.m3.1.1.1">~</ci><ci id="A6.Thmlemma1.p1.3.3.m3.1.1.2.cmml" xref="A6.Thmlemma1.p1.3.3.m3.1.1.2">𝐀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma1.p1.3.3.m3.1c">\tilde{\bm{A}}</annotation></semantics></math> on which <span id="A6.Thmlemma1.p1.4.4.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.Thmlemma1.p1.4.4.2.1" class="ltx_text ltx_font_italic"> </span></span>is trained. Then there are at most <math id="A6.Thmlemma1.p1.4.4.m4.2" class="ltx_Math" alttext="2^{C(4\left\lceil\log_{2}(D)\right\rceil+256)}" display="inline"><semantics id="A6.Thmlemma1.p1.4.4.m4.2a"><msup id="A6.Thmlemma1.p1.4.4.m4.2.3" xref="A6.Thmlemma1.p1.4.4.m4.2.3.cmml"><mn id="A6.Thmlemma1.p1.4.4.m4.2.3.2" xref="A6.Thmlemma1.p1.4.4.m4.2.3.2.cmml">2</mn><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.cmml"><mi id="A6.Thmlemma1.p1.4.4.m4.2.2.2.4" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.4.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.3" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.3.cmml">​</mo><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.cmml"><mo stretchy="false" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.cmml">(</mo><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.cmml"><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.cmml"><mn id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.3" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.2.cmml">​</mo><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.2.cmml"><mo id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.2.1.cmml">⌈</mo><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.2.cmml"><msub id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.2.cmml">log</mi><mn id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.3" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1a" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.2.1" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.2.cmml">(</mo><mi id="A6.Thmlemma1.p1.4.4.m4.1.1.1.1" xref="A6.Thmlemma1.p1.4.4.m4.1.1.1.1.cmml">D</mi><mo stretchy="false" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.2.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.3" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.2.1.cmml">⌉</mo></mrow></mrow><mo id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.2" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.2.cmml">+</mo><mn id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.3" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.3.cmml">256</mn></mrow><mo stretchy="false" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.3" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.cmml">)</mo></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="A6.Thmlemma1.p1.4.4.m4.2b"><apply id="A6.Thmlemma1.p1.4.4.m4.2.3.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.3"><csymbol cd="ambiguous" id="A6.Thmlemma1.p1.4.4.m4.2.3.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.3">superscript</csymbol><cn type="integer" id="A6.Thmlemma1.p1.4.4.m4.2.3.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.3.2">2</cn><apply id="A6.Thmlemma1.p1.4.4.m4.2.2.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2"><times id="A6.Thmlemma1.p1.4.4.m4.2.2.2.3.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.3"></times><ci id="A6.Thmlemma1.p1.4.4.m4.2.2.2.4.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.4">𝐶</ci><apply id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1"><plus id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.2"></plus><apply id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1"><times id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.2"></times><cn type="integer" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.3.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.3">4</cn><apply id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1"><ceiling id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.2.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.2"></ceiling><apply id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1"><apply id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1">subscript</csymbol><log id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.2"></log><cn type="integer" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.1.1.1.1.1.1.3">2</cn></apply><ci id="A6.Thmlemma1.p1.4.4.m4.1.1.1.1.cmml" xref="A6.Thmlemma1.p1.4.4.m4.1.1.1.1">𝐷</ci></apply></apply></apply><cn type="integer" id="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.3.cmml" xref="A6.Thmlemma1.p1.4.4.m4.2.2.2.2.1.1.3">256</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma1.p1.4.4.m4.2c">2^{C(4\left\lceil\log_{2}(D)\right\rceil+256)}</annotation></semantics></math> unique trees that <span id="A6.Thmlemma1.p1.4.4.3" class="ltx_text ltx_font_typewriter">Maddness-Build-Tree</span> can generate.</span></p>
</div>
</div>
<div id="A6.SS2.4" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A6.SS2.1.p1" class="ltx_para">
<p id="A6.SS2.1.p1.1" class="ltx_p"><span id="A6.SS2.1.p1.1.1" class="ltx_text ltx_font_typewriter">Maddness-Build-Tree</span> learns four sets of parameters for each of the <math id="A6.SS2.1.p1.1.m1.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.SS2.1.p1.1.m1.1a"><mi id="A6.SS2.1.p1.1.m1.1.1" xref="A6.SS2.1.p1.1.m1.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.1.p1.1.m1.1b"><ci id="A6.SS2.1.p1.1.m1.1.1.cmml" xref="A6.SS2.1.p1.1.m1.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.1.p1.1.m1.1c">C</annotation></semantics></math> trees it produces: split indices, split offsets, split scales, and split values.</p>
</div>
<div id="A6.SS2.2.p2" class="ltx_para">
<p id="A6.SS2.2.p2.3" class="ltx_p">There are four split indices per tree because there is one for each of the tree’s four levels. Each index requires <math id="A6.SS2.2.p2.1.m1.2" class="ltx_Math" alttext="\left\lceil\log_{2}(D)\right\rceil" display="inline"><semantics id="A6.SS2.2.p2.1.m1.2a"><mrow id="A6.SS2.2.p2.1.m1.2.2.1" xref="A6.SS2.2.p2.1.m1.2.2.2.cmml"><mo id="A6.SS2.2.p2.1.m1.2.2.1.2" xref="A6.SS2.2.p2.1.m1.2.2.2.1.cmml">⌈</mo><mrow id="A6.SS2.2.p2.1.m1.2.2.1.1.1" xref="A6.SS2.2.p2.1.m1.2.2.1.1.2.cmml"><msub id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.cmml"><mi id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.2" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.2.cmml">log</mi><mn id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.3" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.3.cmml">2</mn></msub><mo id="A6.SS2.2.p2.1.m1.2.2.1.1.1a" xref="A6.SS2.2.p2.1.m1.2.2.1.1.2.cmml">⁡</mo><mrow id="A6.SS2.2.p2.1.m1.2.2.1.1.1.2" xref="A6.SS2.2.p2.1.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="A6.SS2.2.p2.1.m1.2.2.1.1.1.2.1" xref="A6.SS2.2.p2.1.m1.2.2.1.1.2.cmml">(</mo><mi id="A6.SS2.2.p2.1.m1.1.1" xref="A6.SS2.2.p2.1.m1.1.1.cmml">D</mi><mo stretchy="false" id="A6.SS2.2.p2.1.m1.2.2.1.1.1.2.2" xref="A6.SS2.2.p2.1.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow><mo id="A6.SS2.2.p2.1.m1.2.2.1.3" xref="A6.SS2.2.p2.1.m1.2.2.2.1.cmml">⌉</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.2.p2.1.m1.2b"><apply id="A6.SS2.2.p2.1.m1.2.2.2.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1"><ceiling id="A6.SS2.2.p2.1.m1.2.2.2.1.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1.2"></ceiling><apply id="A6.SS2.2.p2.1.m1.2.2.1.1.2.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1"><apply id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.1.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1">subscript</csymbol><log id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.2.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.2"></log><cn type="integer" id="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.3.cmml" xref="A6.SS2.2.p2.1.m1.2.2.1.1.1.1.3">2</cn></apply><ci id="A6.SS2.2.p2.1.m1.1.1.cmml" xref="A6.SS2.2.p2.1.m1.1.1">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.2.p2.1.m1.2c">\left\lceil\log_{2}(D)\right\rceil</annotation></semantics></math> bits to store, so the split indices require a total of <math id="A6.SS2.2.p2.2.m2.2" class="ltx_Math" alttext="4\left\lceil\log_{2}(D)\right\rceil" display="inline"><semantics id="A6.SS2.2.p2.2.m2.2a"><mrow id="A6.SS2.2.p2.2.m2.2.2" xref="A6.SS2.2.p2.2.m2.2.2.cmml"><mn id="A6.SS2.2.p2.2.m2.2.2.3" xref="A6.SS2.2.p2.2.m2.2.2.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A6.SS2.2.p2.2.m2.2.2.2" xref="A6.SS2.2.p2.2.m2.2.2.2.cmml">​</mo><mrow id="A6.SS2.2.p2.2.m2.2.2.1.1" xref="A6.SS2.2.p2.2.m2.2.2.1.2.cmml"><mo id="A6.SS2.2.p2.2.m2.2.2.1.1.2" xref="A6.SS2.2.p2.2.m2.2.2.1.2.1.cmml">⌈</mo><mrow id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.2.cmml"><msub id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.cmml"><mi id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.2" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.2.cmml">log</mi><mn id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.3" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1a" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.2.cmml">⁡</mo><mrow id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.2" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.2.cmml"><mo stretchy="false" id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.2.1" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.2.cmml">(</mo><mi id="A6.SS2.2.p2.2.m2.1.1" xref="A6.SS2.2.p2.2.m2.1.1.cmml">D</mi><mo stretchy="false" id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.2.2" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A6.SS2.2.p2.2.m2.2.2.1.1.3" xref="A6.SS2.2.p2.2.m2.2.2.1.2.1.cmml">⌉</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.2.p2.2.m2.2b"><apply id="A6.SS2.2.p2.2.m2.2.2.cmml" xref="A6.SS2.2.p2.2.m2.2.2"><times id="A6.SS2.2.p2.2.m2.2.2.2.cmml" xref="A6.SS2.2.p2.2.m2.2.2.2"></times><cn type="integer" id="A6.SS2.2.p2.2.m2.2.2.3.cmml" xref="A6.SS2.2.p2.2.m2.2.2.3">4</cn><apply id="A6.SS2.2.p2.2.m2.2.2.1.2.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1"><ceiling id="A6.SS2.2.p2.2.m2.2.2.1.2.1.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1.2"></ceiling><apply id="A6.SS2.2.p2.2.m2.2.2.1.1.1.2.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1"><apply id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.1.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1">subscript</csymbol><log id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.2.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.2"></log><cn type="integer" id="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.3.cmml" xref="A6.SS2.2.p2.2.m2.2.2.1.1.1.1.1.3">2</cn></apply><ci id="A6.SS2.2.p2.2.m2.1.1.cmml" xref="A6.SS2.2.p2.2.m2.1.1">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.2.p2.2.m2.2c">4\left\lceil\log_{2}(D)\right\rceil</annotation></semantics></math> bits per tree. For each split index, there is one split offset and scale, used to map floating point data in an arbitrary range to the interval <math id="A6.SS2.2.p2.3.m3.2" class="ltx_Math" alttext="[0,255]" display="inline"><semantics id="A6.SS2.2.p2.3.m3.2a"><mrow id="A6.SS2.2.p2.3.m3.2.3.2" xref="A6.SS2.2.p2.3.m3.2.3.1.cmml"><mo stretchy="false" id="A6.SS2.2.p2.3.m3.2.3.2.1" xref="A6.SS2.2.p2.3.m3.2.3.1.cmml">[</mo><mn id="A6.SS2.2.p2.3.m3.1.1" xref="A6.SS2.2.p2.3.m3.1.1.cmml">0</mn><mo id="A6.SS2.2.p2.3.m3.2.3.2.2" xref="A6.SS2.2.p2.3.m3.2.3.1.cmml">,</mo><mn id="A6.SS2.2.p2.3.m3.2.2" xref="A6.SS2.2.p2.3.m3.2.2.cmml">255</mn><mo stretchy="false" id="A6.SS2.2.p2.3.m3.2.3.2.3" xref="A6.SS2.2.p2.3.m3.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.2.p2.3.m3.2b"><interval closure="closed" id="A6.SS2.2.p2.3.m3.2.3.1.cmml" xref="A6.SS2.2.p2.3.m3.2.3.2"><cn type="integer" id="A6.SS2.2.p2.3.m3.1.1.cmml" xref="A6.SS2.2.p2.3.m3.1.1">0</cn><cn type="integer" id="A6.SS2.2.p2.3.m3.2.2.cmml" xref="A6.SS2.2.p2.3.m3.2.2">255</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.2.p2.3.m3.2c">[0,255]</annotation></semantics></math> to match up with the 8-bit split values.</p>
</div>
<div id="A6.SS2.3.p3" class="ltx_para">
<p id="A6.SS2.3.p3.1" class="ltx_p">The offsets require at most 25 bits each for 32-bit floating point data, since the low seven bits can be dropped without affecting the post-scaling quantized output. The scales are constrained to be powers of two, and so require at most nine bits for non-subnormal 32-bit floating point inputs (which have one sign bit and eight exponent bits). The offsets and scales together therefore contribute <math id="A6.SS2.3.p3.1.m1.1" class="ltx_Math" alttext="4(25+9)=136" display="inline"><semantics id="A6.SS2.3.p3.1.m1.1a"><mrow id="A6.SS2.3.p3.1.m1.1.1" xref="A6.SS2.3.p3.1.m1.1.1.cmml"><mrow id="A6.SS2.3.p3.1.m1.1.1.1" xref="A6.SS2.3.p3.1.m1.1.1.1.cmml"><mn id="A6.SS2.3.p3.1.m1.1.1.1.3" xref="A6.SS2.3.p3.1.m1.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A6.SS2.3.p3.1.m1.1.1.1.2" xref="A6.SS2.3.p3.1.m1.1.1.1.2.cmml">​</mo><mrow id="A6.SS2.3.p3.1.m1.1.1.1.1.1" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS2.3.p3.1.m1.1.1.1.1.1.2" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.cmml"><mn id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.2" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.2.cmml">25</mn><mo id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.1" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.1.cmml">+</mo><mn id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.3" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.3.cmml">9</mn></mrow><mo stretchy="false" id="A6.SS2.3.p3.1.m1.1.1.1.1.1.3" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.SS2.3.p3.1.m1.1.1.2" xref="A6.SS2.3.p3.1.m1.1.1.2.cmml">=</mo><mn id="A6.SS2.3.p3.1.m1.1.1.3" xref="A6.SS2.3.p3.1.m1.1.1.3.cmml">136</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.3.p3.1.m1.1b"><apply id="A6.SS2.3.p3.1.m1.1.1.cmml" xref="A6.SS2.3.p3.1.m1.1.1"><eq id="A6.SS2.3.p3.1.m1.1.1.2.cmml" xref="A6.SS2.3.p3.1.m1.1.1.2"></eq><apply id="A6.SS2.3.p3.1.m1.1.1.1.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1"><times id="A6.SS2.3.p3.1.m1.1.1.1.2.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1.2"></times><cn type="integer" id="A6.SS2.3.p3.1.m1.1.1.1.3.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1.3">4</cn><apply id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1"><plus id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.1"></plus><cn type="integer" id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.2.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.2">25</cn><cn type="integer" id="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.3.cmml" xref="A6.SS2.3.p3.1.m1.1.1.1.1.1.1.3">9</cn></apply></apply><cn type="integer" id="A6.SS2.3.p3.1.m1.1.1.3.cmml" xref="A6.SS2.3.p3.1.m1.1.1.3">136</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.3.p3.1.m1.1c">4(25+9)=136</annotation></semantics></math> bits per tree.</p>
</div>
<div id="A6.SS2.4.p4" class="ltx_para">
<p id="A6.SS2.4.p4.2" class="ltx_p">There are 15 split values because there is one for the root of each tree, then two for the second level, four for the third, and eight for the fourth. Each split value is stored using eight bits, so each tree requires <math id="A6.SS2.4.p4.1.m1.1" class="ltx_Math" alttext="15\cdot 8=120" display="inline"><semantics id="A6.SS2.4.p4.1.m1.1a"><mrow id="A6.SS2.4.p4.1.m1.1.1" xref="A6.SS2.4.p4.1.m1.1.1.cmml"><mrow id="A6.SS2.4.p4.1.m1.1.1.2" xref="A6.SS2.4.p4.1.m1.1.1.2.cmml"><mn id="A6.SS2.4.p4.1.m1.1.1.2.2" xref="A6.SS2.4.p4.1.m1.1.1.2.2.cmml">15</mn><mo lspace="0.222em" rspace="0.222em" id="A6.SS2.4.p4.1.m1.1.1.2.1" xref="A6.SS2.4.p4.1.m1.1.1.2.1.cmml">⋅</mo><mn id="A6.SS2.4.p4.1.m1.1.1.2.3" xref="A6.SS2.4.p4.1.m1.1.1.2.3.cmml">8</mn></mrow><mo id="A6.SS2.4.p4.1.m1.1.1.1" xref="A6.SS2.4.p4.1.m1.1.1.1.cmml">=</mo><mn id="A6.SS2.4.p4.1.m1.1.1.3" xref="A6.SS2.4.p4.1.m1.1.1.3.cmml">120</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.4.p4.1.m1.1b"><apply id="A6.SS2.4.p4.1.m1.1.1.cmml" xref="A6.SS2.4.p4.1.m1.1.1"><eq id="A6.SS2.4.p4.1.m1.1.1.1.cmml" xref="A6.SS2.4.p4.1.m1.1.1.1"></eq><apply id="A6.SS2.4.p4.1.m1.1.1.2.cmml" xref="A6.SS2.4.p4.1.m1.1.1.2"><ci id="A6.SS2.4.p4.1.m1.1.1.2.1.cmml" xref="A6.SS2.4.p4.1.m1.1.1.2.1">⋅</ci><cn type="integer" id="A6.SS2.4.p4.1.m1.1.1.2.2.cmml" xref="A6.SS2.4.p4.1.m1.1.1.2.2">15</cn><cn type="integer" id="A6.SS2.4.p4.1.m1.1.1.2.3.cmml" xref="A6.SS2.4.p4.1.m1.1.1.2.3">8</cn></apply><cn type="integer" id="A6.SS2.4.p4.1.m1.1.1.3.cmml" xref="A6.SS2.4.p4.1.m1.1.1.3">120</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.4.p4.1.m1.1c">15\cdot 8=120</annotation></semantics></math> bits for split values. The total number of bits used for all trees is therefore <math id="A6.SS2.4.p4.2.m2.2" class="ltx_Math" alttext="C(4\left\lceil\log_{2}(D)\right\rceil+256)" display="inline"><semantics id="A6.SS2.4.p4.2.m2.2a"><mrow id="A6.SS2.4.p4.2.m2.2.2" xref="A6.SS2.4.p4.2.m2.2.2.cmml"><mi id="A6.SS2.4.p4.2.m2.2.2.3" xref="A6.SS2.4.p4.2.m2.2.2.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS2.4.p4.2.m2.2.2.2" xref="A6.SS2.4.p4.2.m2.2.2.2.cmml">​</mo><mrow id="A6.SS2.4.p4.2.m2.2.2.1.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.cmml"><mo stretchy="false" id="A6.SS2.4.p4.2.m2.2.2.1.1.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.cmml">(</mo><mrow id="A6.SS2.4.p4.2.m2.2.2.1.1.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.cmml"><mrow id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.cmml"><mn id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.3" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.2.cmml">​</mo><mrow id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.2.cmml"><mo id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.2.1.cmml">⌈</mo><mrow id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.2.cmml"><msub id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.cmml"><mi id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.2.cmml">log</mi><mn id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.3" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1a" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.2.1" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.2.cmml">(</mo><mi id="A6.SS2.4.p4.2.m2.1.1" xref="A6.SS2.4.p4.2.m2.1.1.cmml">D</mi><mo stretchy="false" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.2.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.3" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.2.1.cmml">⌉</mo></mrow></mrow><mo id="A6.SS2.4.p4.2.m2.2.2.1.1.1.2" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.2.cmml">+</mo><mn id="A6.SS2.4.p4.2.m2.2.2.1.1.1.3" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.3.cmml">256</mn></mrow><mo stretchy="false" id="A6.SS2.4.p4.2.m2.2.2.1.1.3" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.4.p4.2.m2.2b"><apply id="A6.SS2.4.p4.2.m2.2.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2"><times id="A6.SS2.4.p4.2.m2.2.2.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2.2"></times><ci id="A6.SS2.4.p4.2.m2.2.2.3.cmml" xref="A6.SS2.4.p4.2.m2.2.2.3">𝐶</ci><apply id="A6.SS2.4.p4.2.m2.2.2.1.1.1.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1"><plus id="A6.SS2.4.p4.2.m2.2.2.1.1.1.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.2"></plus><apply id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1"><times id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.2"></times><cn type="integer" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.3.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.3">4</cn><apply id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1"><ceiling id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.2.1.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.2"></ceiling><apply id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1"><apply id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1">subscript</csymbol><log id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.2"></log><cn type="integer" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.1.1.1.1.1.1.3">2</cn></apply><ci id="A6.SS2.4.p4.2.m2.1.1.cmml" xref="A6.SS2.4.p4.2.m2.1.1">𝐷</ci></apply></apply></apply><cn type="integer" id="A6.SS2.4.p4.2.m2.2.2.1.1.1.3.cmml" xref="A6.SS2.4.p4.2.m2.2.2.1.1.1.3">256</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.4.p4.2.m2.2c">C(4\left\lceil\log_{2}(D)\right\rceil+256)</annotation></semantics></math>. Note that the constant 256 being a power of two is just an accident of floating point formats. The claimed hypothesis count follows from the number of expressible hypotheses being at most two to the power of the largest number of bits used to store any hypothesis.
∎</p>
</div>
</div>
<div id="A6.SS2.p3" class="ltx_para">
<p id="A6.SS2.p3.3" class="ltx_p">We now turn our attention to bounding the errors of the regression component of training. Our strategy for doing so is to bound the largest singular value of the learned matrix of prototypes <math id="A6.SS2.p3.1.m1.1" class="ltx_Math" alttext="\bm{P}" display="inline"><semantics id="A6.SS2.p3.1.m1.1a"><mi id="A6.SS2.p3.1.m1.1.1" xref="A6.SS2.p3.1.m1.1.1.cmml">𝑷</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.p3.1.m1.1b"><ci id="A6.SS2.p3.1.m1.1.1.cmml" xref="A6.SS2.p3.1.m1.1.1">𝑷</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p3.1.m1.1c">\bm{P}</annotation></semantics></math>. Given such a bound, the norms of both <math id="A6.SS2.p3.2.m2.1" class="ltx_Math" alttext="g(\bm{a})^{\top}\bm{P}" display="inline"><semantics id="A6.SS2.p3.2.m2.1a"><mrow id="A6.SS2.p3.2.m2.1.2" xref="A6.SS2.p3.2.m2.1.2.cmml"><mi id="A6.SS2.p3.2.m2.1.2.2" xref="A6.SS2.p3.2.m2.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.SS2.p3.2.m2.1.2.1" xref="A6.SS2.p3.2.m2.1.2.1.cmml">​</mo><msup id="A6.SS2.p3.2.m2.1.2.3" xref="A6.SS2.p3.2.m2.1.2.3.cmml"><mrow id="A6.SS2.p3.2.m2.1.2.3.2.2" xref="A6.SS2.p3.2.m2.1.2.3.cmml"><mo stretchy="false" id="A6.SS2.p3.2.m2.1.2.3.2.2.1" xref="A6.SS2.p3.2.m2.1.2.3.cmml">(</mo><mi id="A6.SS2.p3.2.m2.1.1" xref="A6.SS2.p3.2.m2.1.1.cmml">𝒂</mi><mo stretchy="false" id="A6.SS2.p3.2.m2.1.2.3.2.2.2" xref="A6.SS2.p3.2.m2.1.2.3.cmml">)</mo></mrow><mo id="A6.SS2.p3.2.m2.1.2.3.3" xref="A6.SS2.p3.2.m2.1.2.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.p3.2.m2.1.2.1a" xref="A6.SS2.p3.2.m2.1.2.1.cmml">​</mo><mi id="A6.SS2.p3.2.m2.1.2.4" xref="A6.SS2.p3.2.m2.1.2.4.cmml">𝑷</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.p3.2.m2.1b"><apply id="A6.SS2.p3.2.m2.1.2.cmml" xref="A6.SS2.p3.2.m2.1.2"><times id="A6.SS2.p3.2.m2.1.2.1.cmml" xref="A6.SS2.p3.2.m2.1.2.1"></times><ci id="A6.SS2.p3.2.m2.1.2.2.cmml" xref="A6.SS2.p3.2.m2.1.2.2">𝑔</ci><apply id="A6.SS2.p3.2.m2.1.2.3.cmml" xref="A6.SS2.p3.2.m2.1.2.3"><csymbol cd="ambiguous" id="A6.SS2.p3.2.m2.1.2.3.1.cmml" xref="A6.SS2.p3.2.m2.1.2.3">superscript</csymbol><ci id="A6.SS2.p3.2.m2.1.1.cmml" xref="A6.SS2.p3.2.m2.1.1">𝒂</ci><csymbol cd="latexml" id="A6.SS2.p3.2.m2.1.2.3.3.cmml" xref="A6.SS2.p3.2.m2.1.2.3.3">top</csymbol></apply><ci id="A6.SS2.p3.2.m2.1.2.4.cmml" xref="A6.SS2.p3.2.m2.1.2.4">𝑷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p3.2.m2.1c">g(\bm{a})^{\top}\bm{P}</annotation></semantics></math> and <math id="A6.SS2.p3.3.m3.1" class="ltx_Math" alttext="\bm{P}^{\top}\bm{b}" display="inline"><semantics id="A6.SS2.p3.3.m3.1a"><mrow id="A6.SS2.p3.3.m3.1.1" xref="A6.SS2.p3.3.m3.1.1.cmml"><msup id="A6.SS2.p3.3.m3.1.1.2" xref="A6.SS2.p3.3.m3.1.1.2.cmml"><mi id="A6.SS2.p3.3.m3.1.1.2.2" xref="A6.SS2.p3.3.m3.1.1.2.2.cmml">𝑷</mi><mo id="A6.SS2.p3.3.m3.1.1.2.3" xref="A6.SS2.p3.3.m3.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.p3.3.m3.1.1.1" xref="A6.SS2.p3.3.m3.1.1.1.cmml">​</mo><mi id="A6.SS2.p3.3.m3.1.1.3" xref="A6.SS2.p3.3.m3.1.1.3.cmml">𝒃</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.p3.3.m3.1b"><apply id="A6.SS2.p3.3.m3.1.1.cmml" xref="A6.SS2.p3.3.m3.1.1"><times id="A6.SS2.p3.3.m3.1.1.1.cmml" xref="A6.SS2.p3.3.m3.1.1.1"></times><apply id="A6.SS2.p3.3.m3.1.1.2.cmml" xref="A6.SS2.p3.3.m3.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.p3.3.m3.1.1.2.1.cmml" xref="A6.SS2.p3.3.m3.1.1.2">superscript</csymbol><ci id="A6.SS2.p3.3.m3.1.1.2.2.cmml" xref="A6.SS2.p3.3.m3.1.1.2.2">𝑷</ci><csymbol cd="latexml" id="A6.SS2.p3.3.m3.1.1.2.3.cmml" xref="A6.SS2.p3.3.m3.1.1.2.3">top</csymbol></apply><ci id="A6.SS2.p3.3.m3.1.1.3.cmml" xref="A6.SS2.p3.3.m3.1.1.3">𝒃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p3.3.m3.1c">\bm{P}^{\top}\bm{b}</annotation></semantics></math> can be bounded.</p>
</div>
<div id="A6.Thmlemma2" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmlemma2.1.1.1" class="ltx_text ltx_font_bold">Lemma F.2</span></span><span id="A6.Thmlemma2.2.2" class="ltx_text ltx_font_bold"> </span>(Regularized Pseudoinverse Operator Norm Bound)<span id="A6.Thmlemma2.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmlemma2.p1" class="ltx_para">
<p id="A6.Thmlemma2.p1.5" class="ltx_p"><span id="A6.Thmlemma2.p1.5.5" class="ltx_text ltx_font_italic">Let <math id="A6.Thmlemma2.p1.1.1.m1.1" class="ltx_Math" alttext="\bm{X}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="A6.Thmlemma2.p1.1.1.m1.1a"><mrow id="A6.Thmlemma2.p1.1.1.m1.1.1" xref="A6.Thmlemma2.p1.1.1.m1.1.1.cmml"><mi id="A6.Thmlemma2.p1.1.1.m1.1.1.2" xref="A6.Thmlemma2.p1.1.1.m1.1.1.2.cmml">𝐗</mi><mo id="A6.Thmlemma2.p1.1.1.m1.1.1.1" xref="A6.Thmlemma2.p1.1.1.m1.1.1.1.cmml">∈</mo><msup id="A6.Thmlemma2.p1.1.1.m1.1.1.3" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.cmml"><mi id="A6.Thmlemma2.p1.1.1.m1.1.1.3.2" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.cmml"><mi id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.2" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.1" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.3" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma2.p1.1.1.m1.1b"><apply id="A6.Thmlemma2.p1.1.1.m1.1.1.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1"><in id="A6.Thmlemma2.p1.1.1.m1.1.1.1.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.1"></in><ci id="A6.Thmlemma2.p1.1.1.m1.1.1.2.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.2">𝐗</ci><apply id="A6.Thmlemma2.p1.1.1.m1.1.1.3.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma2.p1.1.1.m1.1.1.3.1.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3">superscript</csymbol><ci id="A6.Thmlemma2.p1.1.1.m1.1.1.3.2.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.2">ℝ</ci><apply id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3"><times id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.1.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.1"></times><ci id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.2.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.2">𝑁</ci><ci id="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.3.cmml" xref="A6.Thmlemma2.p1.1.1.m1.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma2.p1.1.1.m1.1c">\bm{X}\in\mathbb{R}^{N\times D}</annotation></semantics></math>
be an arbitrary matrix with finite elements. Then every singular value <math id="A6.Thmlemma2.p1.2.2.m2.1" class="ltx_Math" alttext="\sigma_{i}" display="inline"><semantics id="A6.Thmlemma2.p1.2.2.m2.1a"><msub id="A6.Thmlemma2.p1.2.2.m2.1.1" xref="A6.Thmlemma2.p1.2.2.m2.1.1.cmml"><mi id="A6.Thmlemma2.p1.2.2.m2.1.1.2" xref="A6.Thmlemma2.p1.2.2.m2.1.1.2.cmml">σ</mi><mi id="A6.Thmlemma2.p1.2.2.m2.1.1.3" xref="A6.Thmlemma2.p1.2.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.Thmlemma2.p1.2.2.m2.1b"><apply id="A6.Thmlemma2.p1.2.2.m2.1.1.cmml" xref="A6.Thmlemma2.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma2.p1.2.2.m2.1.1.1.cmml" xref="A6.Thmlemma2.p1.2.2.m2.1.1">subscript</csymbol><ci id="A6.Thmlemma2.p1.2.2.m2.1.1.2.cmml" xref="A6.Thmlemma2.p1.2.2.m2.1.1.2">𝜎</ci><ci id="A6.Thmlemma2.p1.2.2.m2.1.1.3.cmml" xref="A6.Thmlemma2.p1.2.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma2.p1.2.2.m2.1c">\sigma_{i}</annotation></semantics></math> of the matrix <math id="A6.Thmlemma2.p1.3.3.m3.1" class="ltx_Math" alttext="\bm{Z}\triangleq(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}" display="inline"><semantics id="A6.Thmlemma2.p1.3.3.m3.1a"><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.cmml"><mi id="A6.Thmlemma2.p1.3.3.m3.1.1.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.3.cmml">𝐙</mi><mo id="A6.Thmlemma2.p1.3.3.m3.1.1.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.2.cmml">≜</mo><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.cmml"><msup id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.cmml"><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.cmml"><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.cmml"><msup id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.cmml"><mi id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2.cmml">𝐗</mi><mo id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.3.cmml">𝐗</mi></mrow><mo id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.cmml"><mi id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.1" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.3.cmml">𝐈</mi></mrow></mrow><mo stretchy="false" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.cmml"><mo id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3a" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.cmml">−</mo><mn id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.2.cmml">​</mo><msup id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.cmml"><mi id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.2" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.2.cmml">𝐗</mi><mo id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.3" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma2.p1.3.3.m3.1b"><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1"><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.2">≜</ci><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.3">𝐙</ci><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1"><times id="A6.Thmlemma2.p1.3.3.m3.1.1.1.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.2"></times><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1">superscript</csymbol><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1"><plus id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.1"></plus><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2"><times id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.1"></times><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2">𝐗</ci><csymbol cd="latexml" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3">top</csymbol></apply><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.2.3">𝐗</ci></apply><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3"><times id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.1"></times><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.1.1.1.3.3">𝐈</ci></apply></apply><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3"><minus id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3"></minus><cn type="integer" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.1.3.2">1</cn></apply></apply><apply id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.1.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3">superscript</csymbol><ci id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.2.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.2">𝐗</ci><csymbol cd="latexml" id="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.3.cmml" xref="A6.Thmlemma2.p1.3.3.m3.1.1.1.3.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma2.p1.3.3.m3.1c">\bm{Z}\triangleq(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}</annotation></semantics></math>, <math id="A6.Thmlemma2.p1.4.4.m4.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="A6.Thmlemma2.p1.4.4.m4.1a"><mrow id="A6.Thmlemma2.p1.4.4.m4.1.1" xref="A6.Thmlemma2.p1.4.4.m4.1.1.cmml"><mi id="A6.Thmlemma2.p1.4.4.m4.1.1.2" xref="A6.Thmlemma2.p1.4.4.m4.1.1.2.cmml">λ</mi><mo id="A6.Thmlemma2.p1.4.4.m4.1.1.1" xref="A6.Thmlemma2.p1.4.4.m4.1.1.1.cmml">&gt;</mo><mn id="A6.Thmlemma2.p1.4.4.m4.1.1.3" xref="A6.Thmlemma2.p1.4.4.m4.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma2.p1.4.4.m4.1b"><apply id="A6.Thmlemma2.p1.4.4.m4.1.1.cmml" xref="A6.Thmlemma2.p1.4.4.m4.1.1"><gt id="A6.Thmlemma2.p1.4.4.m4.1.1.1.cmml" xref="A6.Thmlemma2.p1.4.4.m4.1.1.1"></gt><ci id="A6.Thmlemma2.p1.4.4.m4.1.1.2.cmml" xref="A6.Thmlemma2.p1.4.4.m4.1.1.2">𝜆</ci><cn type="integer" id="A6.Thmlemma2.p1.4.4.m4.1.1.3.cmml" xref="A6.Thmlemma2.p1.4.4.m4.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma2.p1.4.4.m4.1c">\lambda&gt;0</annotation></semantics></math> is at most <math id="A6.Thmlemma2.p1.5.5.m5.1" class="ltx_Math" alttext="\frac{1}{2\sqrt{\lambda}}" display="inline"><semantics id="A6.Thmlemma2.p1.5.5.m5.1a"><mfrac id="A6.Thmlemma2.p1.5.5.m5.1.1" xref="A6.Thmlemma2.p1.5.5.m5.1.1.cmml"><mn id="A6.Thmlemma2.p1.5.5.m5.1.1.2" xref="A6.Thmlemma2.p1.5.5.m5.1.1.2.cmml">1</mn><mrow id="A6.Thmlemma2.p1.5.5.m5.1.1.3" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.cmml"><mn id="A6.Thmlemma2.p1.5.5.m5.1.1.3.2" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.Thmlemma2.p1.5.5.m5.1.1.3.1" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.1.cmml">​</mo><msqrt id="A6.Thmlemma2.p1.5.5.m5.1.1.3.3" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.3.cmml"><mi id="A6.Thmlemma2.p1.5.5.m5.1.1.3.3.2" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A6.Thmlemma2.p1.5.5.m5.1b"><apply id="A6.Thmlemma2.p1.5.5.m5.1.1.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1"><divide id="A6.Thmlemma2.p1.5.5.m5.1.1.1.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1"></divide><cn type="integer" id="A6.Thmlemma2.p1.5.5.m5.1.1.2.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.2">1</cn><apply id="A6.Thmlemma2.p1.5.5.m5.1.1.3.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3"><times id="A6.Thmlemma2.p1.5.5.m5.1.1.3.1.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.1"></times><cn type="integer" id="A6.Thmlemma2.p1.5.5.m5.1.1.3.2.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.2">2</cn><apply id="A6.Thmlemma2.p1.5.5.m5.1.1.3.3.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.3"><root id="A6.Thmlemma2.p1.5.5.m5.1.1.3.3a.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.3"></root><ci id="A6.Thmlemma2.p1.5.5.m5.1.1.3.3.2.cmml" xref="A6.Thmlemma2.p1.5.5.m5.1.1.3.3.2">𝜆</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma2.p1.5.5.m5.1c">\frac{1}{2\sqrt{\lambda}}</annotation></semantics></math>.</span></p>
</div>
</div>
<div id="A6.SS2.5" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A6.SS2.5.p1" class="ltx_para">
<p id="A6.SS2.5.p1.2" class="ltx_p">Let <math id="A6.SS2.5.p1.1.m1.1" class="ltx_Math" alttext="\bm{U}\bm{\Sigma}\bm{V}^{\top}" display="inline"><semantics id="A6.SS2.5.p1.1.m1.1a"><mrow id="A6.SS2.5.p1.1.m1.1.1" xref="A6.SS2.5.p1.1.m1.1.1.cmml"><mi id="A6.SS2.5.p1.1.m1.1.1.2" xref="A6.SS2.5.p1.1.m1.1.1.2.cmml">𝑼</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.1.m1.1.1.1" xref="A6.SS2.5.p1.1.m1.1.1.1.cmml">​</mo><mi id="A6.SS2.5.p1.1.m1.1.1.3" xref="A6.SS2.5.p1.1.m1.1.1.3.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.1.m1.1.1.1a" xref="A6.SS2.5.p1.1.m1.1.1.1.cmml">​</mo><msup id="A6.SS2.5.p1.1.m1.1.1.4" xref="A6.SS2.5.p1.1.m1.1.1.4.cmml"><mi id="A6.SS2.5.p1.1.m1.1.1.4.2" xref="A6.SS2.5.p1.1.m1.1.1.4.2.cmml">𝑽</mi><mo id="A6.SS2.5.p1.1.m1.1.1.4.3" xref="A6.SS2.5.p1.1.m1.1.1.4.3.cmml">⊤</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.1.m1.1b"><apply id="A6.SS2.5.p1.1.m1.1.1.cmml" xref="A6.SS2.5.p1.1.m1.1.1"><times id="A6.SS2.5.p1.1.m1.1.1.1.cmml" xref="A6.SS2.5.p1.1.m1.1.1.1"></times><ci id="A6.SS2.5.p1.1.m1.1.1.2.cmml" xref="A6.SS2.5.p1.1.m1.1.1.2">𝑼</ci><ci id="A6.SS2.5.p1.1.m1.1.1.3.cmml" xref="A6.SS2.5.p1.1.m1.1.1.3">𝚺</ci><apply id="A6.SS2.5.p1.1.m1.1.1.4.cmml" xref="A6.SS2.5.p1.1.m1.1.1.4"><csymbol cd="ambiguous" id="A6.SS2.5.p1.1.m1.1.1.4.1.cmml" xref="A6.SS2.5.p1.1.m1.1.1.4">superscript</csymbol><ci id="A6.SS2.5.p1.1.m1.1.1.4.2.cmml" xref="A6.SS2.5.p1.1.m1.1.1.4.2">𝑽</ci><csymbol cd="latexml" id="A6.SS2.5.p1.1.m1.1.1.4.3.cmml" xref="A6.SS2.5.p1.1.m1.1.1.4.3">top</csymbol></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.1.m1.1c">\bm{U}\bm{\Sigma}\bm{V}^{\top}</annotation></semantics></math> be the singular value decomposition of <math id="A6.SS2.5.p1.2.m2.1" class="ltx_Math" alttext="\bm{X}" display="inline"><semantics id="A6.SS2.5.p1.2.m2.1a"><mi id="A6.SS2.5.p1.2.m2.1.1" xref="A6.SS2.5.p1.2.m2.1.1.cmml">𝑿</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.2.m2.1b"><ci id="A6.SS2.5.p1.2.m2.1.1.cmml" xref="A6.SS2.5.p1.2.m2.1.1">𝑿</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.2.m2.1c">\bm{X}</annotation></semantics></math>. Then we have</p>
<table id="A6.EGx20" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E24"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E24.m1.1" class="ltx_Math" alttext="\displaystyle\bm{Z}" display="inline"><semantics id="A6.E24.m1.1a"><mi id="A6.E24.m1.1.1" xref="A6.E24.m1.1.1.cmml">𝒁</mi><annotation-xml encoding="MathML-Content" id="A6.E24.m1.1b"><ci id="A6.E24.m1.1.1.cmml" xref="A6.E24.m1.1.1">𝒁</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.E24.m1.1c">\displaystyle\bm{Z}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E24.m2.1" class="ltx_Math" alttext="\displaystyle=(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}" display="inline"><semantics id="A6.E24.m2.1a"><mrow id="A6.E24.m2.1.1" xref="A6.E24.m2.1.1.cmml"><mi id="A6.E24.m2.1.1.3" xref="A6.E24.m2.1.1.3.cmml"></mi><mo id="A6.E24.m2.1.1.2" xref="A6.E24.m2.1.1.2.cmml">=</mo><mrow id="A6.E24.m2.1.1.1" xref="A6.E24.m2.1.1.1.cmml"><msup id="A6.E24.m2.1.1.1.1" xref="A6.E24.m2.1.1.1.1.cmml"><mrow id="A6.E24.m2.1.1.1.1.1.1" xref="A6.E24.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.E24.m2.1.1.1.1.1.1.2" xref="A6.E24.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.E24.m2.1.1.1.1.1.1.1" xref="A6.E24.m2.1.1.1.1.1.1.1.cmml"><mrow id="A6.E24.m2.1.1.1.1.1.1.1.2" xref="A6.E24.m2.1.1.1.1.1.1.1.2.cmml"><msup id="A6.E24.m2.1.1.1.1.1.1.1.2.2" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2.cmml"><mi id="A6.E24.m2.1.1.1.1.1.1.1.2.2.2" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2.2.cmml">𝑿</mi><mo id="A6.E24.m2.1.1.1.1.1.1.1.2.2.3" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E24.m2.1.1.1.1.1.1.1.2.1" xref="A6.E24.m2.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.E24.m2.1.1.1.1.1.1.1.2.3" xref="A6.E24.m2.1.1.1.1.1.1.1.2.3.cmml">𝑿</mi></mrow><mo id="A6.E24.m2.1.1.1.1.1.1.1.1" xref="A6.E24.m2.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.E24.m2.1.1.1.1.1.1.1.3" xref="A6.E24.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A6.E24.m2.1.1.1.1.1.1.1.3.2" xref="A6.E24.m2.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.E24.m2.1.1.1.1.1.1.1.3.1" xref="A6.E24.m2.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.E24.m2.1.1.1.1.1.1.1.3.3" xref="A6.E24.m2.1.1.1.1.1.1.1.3.3.cmml">𝑰</mi></mrow></mrow><mo stretchy="false" id="A6.E24.m2.1.1.1.1.1.1.3" xref="A6.E24.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.E24.m2.1.1.1.1.3" xref="A6.E24.m2.1.1.1.1.3.cmml"><mo id="A6.E24.m2.1.1.1.1.3a" xref="A6.E24.m2.1.1.1.1.3.cmml">−</mo><mn id="A6.E24.m2.1.1.1.1.3.2" xref="A6.E24.m2.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.E24.m2.1.1.1.2" xref="A6.E24.m2.1.1.1.2.cmml">​</mo><msup id="A6.E24.m2.1.1.1.3" xref="A6.E24.m2.1.1.1.3.cmml"><mi id="A6.E24.m2.1.1.1.3.2" xref="A6.E24.m2.1.1.1.3.2.cmml">𝑿</mi><mo id="A6.E24.m2.1.1.1.3.3" xref="A6.E24.m2.1.1.1.3.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E24.m2.1b"><apply id="A6.E24.m2.1.1.cmml" xref="A6.E24.m2.1.1"><eq id="A6.E24.m2.1.1.2.cmml" xref="A6.E24.m2.1.1.2"></eq><csymbol cd="latexml" id="A6.E24.m2.1.1.3.cmml" xref="A6.E24.m2.1.1.3">absent</csymbol><apply id="A6.E24.m2.1.1.1.cmml" xref="A6.E24.m2.1.1.1"><times id="A6.E24.m2.1.1.1.2.cmml" xref="A6.E24.m2.1.1.1.2"></times><apply id="A6.E24.m2.1.1.1.1.cmml" xref="A6.E24.m2.1.1.1.1"><csymbol cd="ambiguous" id="A6.E24.m2.1.1.1.1.2.cmml" xref="A6.E24.m2.1.1.1.1">superscript</csymbol><apply id="A6.E24.m2.1.1.1.1.1.1.1.cmml" xref="A6.E24.m2.1.1.1.1.1.1"><plus id="A6.E24.m2.1.1.1.1.1.1.1.1.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.1"></plus><apply id="A6.E24.m2.1.1.1.1.1.1.1.2.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2"><times id="A6.E24.m2.1.1.1.1.1.1.1.2.1.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2.1"></times><apply id="A6.E24.m2.1.1.1.1.1.1.1.2.2.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A6.E24.m2.1.1.1.1.1.1.1.2.2.1.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="A6.E24.m2.1.1.1.1.1.1.1.2.2.2.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2.2">𝑿</ci><csymbol cd="latexml" id="A6.E24.m2.1.1.1.1.1.1.1.2.2.3.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2.2.3">top</csymbol></apply><ci id="A6.E24.m2.1.1.1.1.1.1.1.2.3.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.2.3">𝑿</ci></apply><apply id="A6.E24.m2.1.1.1.1.1.1.1.3.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.3"><times id="A6.E24.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.3.1"></times><ci id="A6.E24.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.E24.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A6.E24.m2.1.1.1.1.1.1.1.3.3">𝑰</ci></apply></apply><apply id="A6.E24.m2.1.1.1.1.3.cmml" xref="A6.E24.m2.1.1.1.1.3"><minus id="A6.E24.m2.1.1.1.1.3.1.cmml" xref="A6.E24.m2.1.1.1.1.3"></minus><cn type="integer" id="A6.E24.m2.1.1.1.1.3.2.cmml" xref="A6.E24.m2.1.1.1.1.3.2">1</cn></apply></apply><apply id="A6.E24.m2.1.1.1.3.cmml" xref="A6.E24.m2.1.1.1.3"><csymbol cd="ambiguous" id="A6.E24.m2.1.1.1.3.1.cmml" xref="A6.E24.m2.1.1.1.3">superscript</csymbol><ci id="A6.E24.m2.1.1.1.3.2.cmml" xref="A6.E24.m2.1.1.1.3.2">𝑿</ci><csymbol cd="latexml" id="A6.E24.m2.1.1.1.3.3.cmml" xref="A6.E24.m2.1.1.1.3.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E24.m2.1c">\displaystyle=(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(24)</span></td>
</tr></tbody>
<tbody id="A6.E25"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E25.m1.1" class="ltx_Math" alttext="\displaystyle=(\bm{V}\bm{\Sigma}\bm{U}^{\top}\bm{U}\bm{\Sigma}\bm{V}^{\top}+\lambda\bm{I})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}" display="inline"><semantics id="A6.E25.m1.1a"><mrow id="A6.E25.m1.1.1" xref="A6.E25.m1.1.1.cmml"><mi id="A6.E25.m1.1.1.3" xref="A6.E25.m1.1.1.3.cmml"></mi><mo id="A6.E25.m1.1.1.2" xref="A6.E25.m1.1.1.2.cmml">=</mo><mrow id="A6.E25.m1.1.1.1" xref="A6.E25.m1.1.1.1.cmml"><msup id="A6.E25.m1.1.1.1.1" xref="A6.E25.m1.1.1.1.1.cmml"><mrow id="A6.E25.m1.1.1.1.1.1.1" xref="A6.E25.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.E25.m1.1.1.1.1.1.1.2" xref="A6.E25.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.E25.m1.1.1.1.1.1.1.1" xref="A6.E25.m1.1.1.1.1.1.1.1.cmml"><mrow id="A6.E25.m1.1.1.1.1.1.1.1.2" xref="A6.E25.m1.1.1.1.1.1.1.1.2.cmml"><mi id="A6.E25.m1.1.1.1.1.1.1.1.2.2" xref="A6.E25.m1.1.1.1.1.1.1.1.2.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.1.1.1.1.2.1" xref="A6.E25.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.E25.m1.1.1.1.1.1.1.1.2.3" xref="A6.E25.m1.1.1.1.1.1.1.1.2.3.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.1.1.1.1.2.1a" xref="A6.E25.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="A6.E25.m1.1.1.1.1.1.1.1.2.4" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4.cmml"><mi id="A6.E25.m1.1.1.1.1.1.1.1.2.4.2" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4.2.cmml">𝑼</mi><mo id="A6.E25.m1.1.1.1.1.1.1.1.2.4.3" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.1.1.1.1.2.1b" xref="A6.E25.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.E25.m1.1.1.1.1.1.1.1.2.5" xref="A6.E25.m1.1.1.1.1.1.1.1.2.5.cmml">𝑼</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.1.1.1.1.2.1c" xref="A6.E25.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.E25.m1.1.1.1.1.1.1.1.2.6" xref="A6.E25.m1.1.1.1.1.1.1.1.2.6.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.1.1.1.1.2.1d" xref="A6.E25.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="A6.E25.m1.1.1.1.1.1.1.1.2.7" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7.cmml"><mi id="A6.E25.m1.1.1.1.1.1.1.1.2.7.2" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7.2.cmml">𝑽</mi><mo id="A6.E25.m1.1.1.1.1.1.1.1.2.7.3" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7.3.cmml">⊤</mo></msup></mrow><mo id="A6.E25.m1.1.1.1.1.1.1.1.1" xref="A6.E25.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.E25.m1.1.1.1.1.1.1.1.3" xref="A6.E25.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A6.E25.m1.1.1.1.1.1.1.1.3.2" xref="A6.E25.m1.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.1.1.1.1.3.1" xref="A6.E25.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.E25.m1.1.1.1.1.1.1.1.3.3" xref="A6.E25.m1.1.1.1.1.1.1.1.3.3.cmml">𝑰</mi></mrow></mrow><mo stretchy="false" id="A6.E25.m1.1.1.1.1.1.1.3" xref="A6.E25.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.E25.m1.1.1.1.1.3" xref="A6.E25.m1.1.1.1.1.3.cmml"><mo id="A6.E25.m1.1.1.1.1.3a" xref="A6.E25.m1.1.1.1.1.3.cmml">−</mo><mn id="A6.E25.m1.1.1.1.1.3.2" xref="A6.E25.m1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.2" xref="A6.E25.m1.1.1.1.2.cmml">​</mo><mi id="A6.E25.m1.1.1.1.3" xref="A6.E25.m1.1.1.1.3.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.2a" xref="A6.E25.m1.1.1.1.2.cmml">​</mo><mi id="A6.E25.m1.1.1.1.4" xref="A6.E25.m1.1.1.1.4.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E25.m1.1.1.1.2b" xref="A6.E25.m1.1.1.1.2.cmml">​</mo><msup id="A6.E25.m1.1.1.1.5" xref="A6.E25.m1.1.1.1.5.cmml"><mi id="A6.E25.m1.1.1.1.5.2" xref="A6.E25.m1.1.1.1.5.2.cmml">𝑼</mi><mo id="A6.E25.m1.1.1.1.5.3" xref="A6.E25.m1.1.1.1.5.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E25.m1.1b"><apply id="A6.E25.m1.1.1.cmml" xref="A6.E25.m1.1.1"><eq id="A6.E25.m1.1.1.2.cmml" xref="A6.E25.m1.1.1.2"></eq><csymbol cd="latexml" id="A6.E25.m1.1.1.3.cmml" xref="A6.E25.m1.1.1.3">absent</csymbol><apply id="A6.E25.m1.1.1.1.cmml" xref="A6.E25.m1.1.1.1"><times id="A6.E25.m1.1.1.1.2.cmml" xref="A6.E25.m1.1.1.1.2"></times><apply id="A6.E25.m1.1.1.1.1.cmml" xref="A6.E25.m1.1.1.1.1"><csymbol cd="ambiguous" id="A6.E25.m1.1.1.1.1.2.cmml" xref="A6.E25.m1.1.1.1.1">superscript</csymbol><apply id="A6.E25.m1.1.1.1.1.1.1.1.cmml" xref="A6.E25.m1.1.1.1.1.1.1"><plus id="A6.E25.m1.1.1.1.1.1.1.1.1.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.1"></plus><apply id="A6.E25.m1.1.1.1.1.1.1.1.2.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2"><times id="A6.E25.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.1"></times><ci id="A6.E25.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.2">𝑽</ci><ci id="A6.E25.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.3">𝚺</ci><apply id="A6.E25.m1.1.1.1.1.1.1.1.2.4.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A6.E25.m1.1.1.1.1.1.1.1.2.4.1.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4">superscript</csymbol><ci id="A6.E25.m1.1.1.1.1.1.1.1.2.4.2.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4.2">𝑼</ci><csymbol cd="latexml" id="A6.E25.m1.1.1.1.1.1.1.1.2.4.3.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.4.3">top</csymbol></apply><ci id="A6.E25.m1.1.1.1.1.1.1.1.2.5.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.5">𝑼</ci><ci id="A6.E25.m1.1.1.1.1.1.1.1.2.6.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.6">𝚺</ci><apply id="A6.E25.m1.1.1.1.1.1.1.1.2.7.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7"><csymbol cd="ambiguous" id="A6.E25.m1.1.1.1.1.1.1.1.2.7.1.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7">superscript</csymbol><ci id="A6.E25.m1.1.1.1.1.1.1.1.2.7.2.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7.2">𝑽</ci><csymbol cd="latexml" id="A6.E25.m1.1.1.1.1.1.1.1.2.7.3.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.2.7.3">top</csymbol></apply></apply><apply id="A6.E25.m1.1.1.1.1.1.1.1.3.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.3"><times id="A6.E25.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.3.1"></times><ci id="A6.E25.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.E25.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A6.E25.m1.1.1.1.1.1.1.1.3.3">𝑰</ci></apply></apply><apply id="A6.E25.m1.1.1.1.1.3.cmml" xref="A6.E25.m1.1.1.1.1.3"><minus id="A6.E25.m1.1.1.1.1.3.1.cmml" xref="A6.E25.m1.1.1.1.1.3"></minus><cn type="integer" id="A6.E25.m1.1.1.1.1.3.2.cmml" xref="A6.E25.m1.1.1.1.1.3.2">1</cn></apply></apply><ci id="A6.E25.m1.1.1.1.3.cmml" xref="A6.E25.m1.1.1.1.3">𝑽</ci><ci id="A6.E25.m1.1.1.1.4.cmml" xref="A6.E25.m1.1.1.1.4">𝚺</ci><apply id="A6.E25.m1.1.1.1.5.cmml" xref="A6.E25.m1.1.1.1.5"><csymbol cd="ambiguous" id="A6.E25.m1.1.1.1.5.1.cmml" xref="A6.E25.m1.1.1.1.5">superscript</csymbol><ci id="A6.E25.m1.1.1.1.5.2.cmml" xref="A6.E25.m1.1.1.1.5.2">𝑼</ci><csymbol cd="latexml" id="A6.E25.m1.1.1.1.5.3.cmml" xref="A6.E25.m1.1.1.1.5.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E25.m1.1c">\displaystyle=(\bm{V}\bm{\Sigma}\bm{U}^{\top}\bm{U}\bm{\Sigma}\bm{V}^{\top}+\lambda\bm{I})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(25)</span></td>
</tr></tbody>
<tbody id="A6.E26"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E26.m1.1" class="ltx_Math" alttext="\displaystyle=(\bm{V}\bm{\Sigma}^{2}\bm{V}^{\top}+\lambda\bm{I})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}" display="inline"><semantics id="A6.E26.m1.1a"><mrow id="A6.E26.m1.1.1" xref="A6.E26.m1.1.1.cmml"><mi id="A6.E26.m1.1.1.3" xref="A6.E26.m1.1.1.3.cmml"></mi><mo id="A6.E26.m1.1.1.2" xref="A6.E26.m1.1.1.2.cmml">=</mo><mrow id="A6.E26.m1.1.1.1" xref="A6.E26.m1.1.1.1.cmml"><msup id="A6.E26.m1.1.1.1.1" xref="A6.E26.m1.1.1.1.1.cmml"><mrow id="A6.E26.m1.1.1.1.1.1.1" xref="A6.E26.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.E26.m1.1.1.1.1.1.1.2" xref="A6.E26.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.E26.m1.1.1.1.1.1.1.1" xref="A6.E26.m1.1.1.1.1.1.1.1.cmml"><mrow id="A6.E26.m1.1.1.1.1.1.1.1.2" xref="A6.E26.m1.1.1.1.1.1.1.1.2.cmml"><mi id="A6.E26.m1.1.1.1.1.1.1.1.2.2" xref="A6.E26.m1.1.1.1.1.1.1.1.2.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E26.m1.1.1.1.1.1.1.1.2.1" xref="A6.E26.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="A6.E26.m1.1.1.1.1.1.1.1.2.3" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="A6.E26.m1.1.1.1.1.1.1.1.2.3.2" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3.2.cmml">𝚺</mi><mn id="A6.E26.m1.1.1.1.1.1.1.1.2.3.3" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3.3.cmml">2</mn></msup><mo lspace="0em" rspace="0em" id="A6.E26.m1.1.1.1.1.1.1.1.2.1a" xref="A6.E26.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="A6.E26.m1.1.1.1.1.1.1.1.2.4" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4.cmml"><mi id="A6.E26.m1.1.1.1.1.1.1.1.2.4.2" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4.2.cmml">𝑽</mi><mo id="A6.E26.m1.1.1.1.1.1.1.1.2.4.3" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4.3.cmml">⊤</mo></msup></mrow><mo id="A6.E26.m1.1.1.1.1.1.1.1.1" xref="A6.E26.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.E26.m1.1.1.1.1.1.1.1.3" xref="A6.E26.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A6.E26.m1.1.1.1.1.1.1.1.3.2" xref="A6.E26.m1.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.E26.m1.1.1.1.1.1.1.1.3.1" xref="A6.E26.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.E26.m1.1.1.1.1.1.1.1.3.3" xref="A6.E26.m1.1.1.1.1.1.1.1.3.3.cmml">𝑰</mi></mrow></mrow><mo stretchy="false" id="A6.E26.m1.1.1.1.1.1.1.3" xref="A6.E26.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.E26.m1.1.1.1.1.3" xref="A6.E26.m1.1.1.1.1.3.cmml"><mo id="A6.E26.m1.1.1.1.1.3a" xref="A6.E26.m1.1.1.1.1.3.cmml">−</mo><mn id="A6.E26.m1.1.1.1.1.3.2" xref="A6.E26.m1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.E26.m1.1.1.1.2" xref="A6.E26.m1.1.1.1.2.cmml">​</mo><mi id="A6.E26.m1.1.1.1.3" xref="A6.E26.m1.1.1.1.3.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E26.m1.1.1.1.2a" xref="A6.E26.m1.1.1.1.2.cmml">​</mo><mi id="A6.E26.m1.1.1.1.4" xref="A6.E26.m1.1.1.1.4.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E26.m1.1.1.1.2b" xref="A6.E26.m1.1.1.1.2.cmml">​</mo><msup id="A6.E26.m1.1.1.1.5" xref="A6.E26.m1.1.1.1.5.cmml"><mi id="A6.E26.m1.1.1.1.5.2" xref="A6.E26.m1.1.1.1.5.2.cmml">𝑼</mi><mo id="A6.E26.m1.1.1.1.5.3" xref="A6.E26.m1.1.1.1.5.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E26.m1.1b"><apply id="A6.E26.m1.1.1.cmml" xref="A6.E26.m1.1.1"><eq id="A6.E26.m1.1.1.2.cmml" xref="A6.E26.m1.1.1.2"></eq><csymbol cd="latexml" id="A6.E26.m1.1.1.3.cmml" xref="A6.E26.m1.1.1.3">absent</csymbol><apply id="A6.E26.m1.1.1.1.cmml" xref="A6.E26.m1.1.1.1"><times id="A6.E26.m1.1.1.1.2.cmml" xref="A6.E26.m1.1.1.1.2"></times><apply id="A6.E26.m1.1.1.1.1.cmml" xref="A6.E26.m1.1.1.1.1"><csymbol cd="ambiguous" id="A6.E26.m1.1.1.1.1.2.cmml" xref="A6.E26.m1.1.1.1.1">superscript</csymbol><apply id="A6.E26.m1.1.1.1.1.1.1.1.cmml" xref="A6.E26.m1.1.1.1.1.1.1"><plus id="A6.E26.m1.1.1.1.1.1.1.1.1.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.1"></plus><apply id="A6.E26.m1.1.1.1.1.1.1.1.2.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2"><times id="A6.E26.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.1"></times><ci id="A6.E26.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.2">𝑽</ci><apply id="A6.E26.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A6.E26.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="A6.E26.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3.2">𝚺</ci><cn type="integer" id="A6.E26.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.3.3">2</cn></apply><apply id="A6.E26.m1.1.1.1.1.1.1.1.2.4.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A6.E26.m1.1.1.1.1.1.1.1.2.4.1.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4">superscript</csymbol><ci id="A6.E26.m1.1.1.1.1.1.1.1.2.4.2.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4.2">𝑽</ci><csymbol cd="latexml" id="A6.E26.m1.1.1.1.1.1.1.1.2.4.3.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.2.4.3">top</csymbol></apply></apply><apply id="A6.E26.m1.1.1.1.1.1.1.1.3.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.3"><times id="A6.E26.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.3.1"></times><ci id="A6.E26.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.E26.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A6.E26.m1.1.1.1.1.1.1.1.3.3">𝑰</ci></apply></apply><apply id="A6.E26.m1.1.1.1.1.3.cmml" xref="A6.E26.m1.1.1.1.1.3"><minus id="A6.E26.m1.1.1.1.1.3.1.cmml" xref="A6.E26.m1.1.1.1.1.3"></minus><cn type="integer" id="A6.E26.m1.1.1.1.1.3.2.cmml" xref="A6.E26.m1.1.1.1.1.3.2">1</cn></apply></apply><ci id="A6.E26.m1.1.1.1.3.cmml" xref="A6.E26.m1.1.1.1.3">𝑽</ci><ci id="A6.E26.m1.1.1.1.4.cmml" xref="A6.E26.m1.1.1.1.4">𝚺</ci><apply id="A6.E26.m1.1.1.1.5.cmml" xref="A6.E26.m1.1.1.1.5"><csymbol cd="ambiguous" id="A6.E26.m1.1.1.1.5.1.cmml" xref="A6.E26.m1.1.1.1.5">superscript</csymbol><ci id="A6.E26.m1.1.1.1.5.2.cmml" xref="A6.E26.m1.1.1.1.5.2">𝑼</ci><csymbol cd="latexml" id="A6.E26.m1.1.1.1.5.3.cmml" xref="A6.E26.m1.1.1.1.5.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E26.m1.1c">\displaystyle=(\bm{V}\bm{\Sigma}^{2}\bm{V}^{\top}+\lambda\bm{I})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(26)</span></td>
</tr></tbody>
<tbody id="A6.E27"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E27.m1.1" class="ltx_Math" alttext="\displaystyle=(\bm{V}\bm{\Sigma}^{2}\bm{V}^{\top}+\bm{V}\lambda\bm{I}\bm{V}^{\top})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}" display="inline"><semantics id="A6.E27.m1.1a"><mrow id="A6.E27.m1.1.1" xref="A6.E27.m1.1.1.cmml"><mi id="A6.E27.m1.1.1.3" xref="A6.E27.m1.1.1.3.cmml"></mi><mo id="A6.E27.m1.1.1.2" xref="A6.E27.m1.1.1.2.cmml">=</mo><mrow id="A6.E27.m1.1.1.1" xref="A6.E27.m1.1.1.1.cmml"><msup id="A6.E27.m1.1.1.1.1" xref="A6.E27.m1.1.1.1.1.cmml"><mrow id="A6.E27.m1.1.1.1.1.1.1" xref="A6.E27.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.E27.m1.1.1.1.1.1.1.2" xref="A6.E27.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.E27.m1.1.1.1.1.1.1.1" xref="A6.E27.m1.1.1.1.1.1.1.1.cmml"><mrow id="A6.E27.m1.1.1.1.1.1.1.1.2" xref="A6.E27.m1.1.1.1.1.1.1.1.2.cmml"><mi id="A6.E27.m1.1.1.1.1.1.1.1.2.2" xref="A6.E27.m1.1.1.1.1.1.1.1.2.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.1.1.1.1.2.1" xref="A6.E27.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="A6.E27.m1.1.1.1.1.1.1.1.2.3" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="A6.E27.m1.1.1.1.1.1.1.1.2.3.2" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3.2.cmml">𝚺</mi><mn id="A6.E27.m1.1.1.1.1.1.1.1.2.3.3" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3.3.cmml">2</mn></msup><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.1.1.1.1.2.1a" xref="A6.E27.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="A6.E27.m1.1.1.1.1.1.1.1.2.4" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4.cmml"><mi id="A6.E27.m1.1.1.1.1.1.1.1.2.4.2" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4.2.cmml">𝑽</mi><mo id="A6.E27.m1.1.1.1.1.1.1.1.2.4.3" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4.3.cmml">⊤</mo></msup></mrow><mo id="A6.E27.m1.1.1.1.1.1.1.1.1" xref="A6.E27.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.E27.m1.1.1.1.1.1.1.1.3" xref="A6.E27.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A6.E27.m1.1.1.1.1.1.1.1.3.2" xref="A6.E27.m1.1.1.1.1.1.1.1.3.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.1.1.1.1.3.1" xref="A6.E27.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.E27.m1.1.1.1.1.1.1.1.3.3" xref="A6.E27.m1.1.1.1.1.1.1.1.3.3.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.1.1.1.1.3.1a" xref="A6.E27.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.E27.m1.1.1.1.1.1.1.1.3.4" xref="A6.E27.m1.1.1.1.1.1.1.1.3.4.cmml">𝑰</mi><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.1.1.1.1.3.1b" xref="A6.E27.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msup id="A6.E27.m1.1.1.1.1.1.1.1.3.5" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5.cmml"><mi id="A6.E27.m1.1.1.1.1.1.1.1.3.5.2" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5.2.cmml">𝑽</mi><mo id="A6.E27.m1.1.1.1.1.1.1.1.3.5.3" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5.3.cmml">⊤</mo></msup></mrow></mrow><mo stretchy="false" id="A6.E27.m1.1.1.1.1.1.1.3" xref="A6.E27.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.E27.m1.1.1.1.1.3" xref="A6.E27.m1.1.1.1.1.3.cmml"><mo id="A6.E27.m1.1.1.1.1.3a" xref="A6.E27.m1.1.1.1.1.3.cmml">−</mo><mn id="A6.E27.m1.1.1.1.1.3.2" xref="A6.E27.m1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.2" xref="A6.E27.m1.1.1.1.2.cmml">​</mo><mi id="A6.E27.m1.1.1.1.3" xref="A6.E27.m1.1.1.1.3.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.2a" xref="A6.E27.m1.1.1.1.2.cmml">​</mo><mi id="A6.E27.m1.1.1.1.4" xref="A6.E27.m1.1.1.1.4.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E27.m1.1.1.1.2b" xref="A6.E27.m1.1.1.1.2.cmml">​</mo><msup id="A6.E27.m1.1.1.1.5" xref="A6.E27.m1.1.1.1.5.cmml"><mi id="A6.E27.m1.1.1.1.5.2" xref="A6.E27.m1.1.1.1.5.2.cmml">𝑼</mi><mo id="A6.E27.m1.1.1.1.5.3" xref="A6.E27.m1.1.1.1.5.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E27.m1.1b"><apply id="A6.E27.m1.1.1.cmml" xref="A6.E27.m1.1.1"><eq id="A6.E27.m1.1.1.2.cmml" xref="A6.E27.m1.1.1.2"></eq><csymbol cd="latexml" id="A6.E27.m1.1.1.3.cmml" xref="A6.E27.m1.1.1.3">absent</csymbol><apply id="A6.E27.m1.1.1.1.cmml" xref="A6.E27.m1.1.1.1"><times id="A6.E27.m1.1.1.1.2.cmml" xref="A6.E27.m1.1.1.1.2"></times><apply id="A6.E27.m1.1.1.1.1.cmml" xref="A6.E27.m1.1.1.1.1"><csymbol cd="ambiguous" id="A6.E27.m1.1.1.1.1.2.cmml" xref="A6.E27.m1.1.1.1.1">superscript</csymbol><apply id="A6.E27.m1.1.1.1.1.1.1.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1"><plus id="A6.E27.m1.1.1.1.1.1.1.1.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.1"></plus><apply id="A6.E27.m1.1.1.1.1.1.1.1.2.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2"><times id="A6.E27.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.1"></times><ci id="A6.E27.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.2">𝑽</ci><apply id="A6.E27.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A6.E27.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="A6.E27.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3.2">𝚺</ci><cn type="integer" id="A6.E27.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.3.3">2</cn></apply><apply id="A6.E27.m1.1.1.1.1.1.1.1.2.4.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A6.E27.m1.1.1.1.1.1.1.1.2.4.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4">superscript</csymbol><ci id="A6.E27.m1.1.1.1.1.1.1.1.2.4.2.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4.2">𝑽</ci><csymbol cd="latexml" id="A6.E27.m1.1.1.1.1.1.1.1.2.4.3.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.2.4.3">top</csymbol></apply></apply><apply id="A6.E27.m1.1.1.1.1.1.1.1.3.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3"><times id="A6.E27.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.1"></times><ci id="A6.E27.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.2">𝑽</ci><ci id="A6.E27.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.3">𝜆</ci><ci id="A6.E27.m1.1.1.1.1.1.1.1.3.4.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.4">𝑰</ci><apply id="A6.E27.m1.1.1.1.1.1.1.1.3.5.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="A6.E27.m1.1.1.1.1.1.1.1.3.5.1.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5">superscript</csymbol><ci id="A6.E27.m1.1.1.1.1.1.1.1.3.5.2.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5.2">𝑽</ci><csymbol cd="latexml" id="A6.E27.m1.1.1.1.1.1.1.1.3.5.3.cmml" xref="A6.E27.m1.1.1.1.1.1.1.1.3.5.3">top</csymbol></apply></apply></apply><apply id="A6.E27.m1.1.1.1.1.3.cmml" xref="A6.E27.m1.1.1.1.1.3"><minus id="A6.E27.m1.1.1.1.1.3.1.cmml" xref="A6.E27.m1.1.1.1.1.3"></minus><cn type="integer" id="A6.E27.m1.1.1.1.1.3.2.cmml" xref="A6.E27.m1.1.1.1.1.3.2">1</cn></apply></apply><ci id="A6.E27.m1.1.1.1.3.cmml" xref="A6.E27.m1.1.1.1.3">𝑽</ci><ci id="A6.E27.m1.1.1.1.4.cmml" xref="A6.E27.m1.1.1.1.4">𝚺</ci><apply id="A6.E27.m1.1.1.1.5.cmml" xref="A6.E27.m1.1.1.1.5"><csymbol cd="ambiguous" id="A6.E27.m1.1.1.1.5.1.cmml" xref="A6.E27.m1.1.1.1.5">superscript</csymbol><ci id="A6.E27.m1.1.1.1.5.2.cmml" xref="A6.E27.m1.1.1.1.5.2">𝑼</ci><csymbol cd="latexml" id="A6.E27.m1.1.1.1.5.3.cmml" xref="A6.E27.m1.1.1.1.5.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E27.m1.1c">\displaystyle=(\bm{V}\bm{\Sigma}^{2}\bm{V}^{\top}+\bm{V}\lambda\bm{I}\bm{V}^{\top})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(27)</span></td>
</tr></tbody>
<tbody id="A6.E28"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E28.m1.1" class="ltx_Math" alttext="\displaystyle=(\bm{V}\bm{\Sigma}_{\lambda}\bm{V}^{\top})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}" display="inline"><semantics id="A6.E28.m1.1a"><mrow id="A6.E28.m1.1.1" xref="A6.E28.m1.1.1.cmml"><mi id="A6.E28.m1.1.1.3" xref="A6.E28.m1.1.1.3.cmml"></mi><mo id="A6.E28.m1.1.1.2" xref="A6.E28.m1.1.1.2.cmml">=</mo><mrow id="A6.E28.m1.1.1.1" xref="A6.E28.m1.1.1.1.cmml"><msup id="A6.E28.m1.1.1.1.1" xref="A6.E28.m1.1.1.1.1.cmml"><mrow id="A6.E28.m1.1.1.1.1.1.1" xref="A6.E28.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.E28.m1.1.1.1.1.1.1.2" xref="A6.E28.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.E28.m1.1.1.1.1.1.1.1" xref="A6.E28.m1.1.1.1.1.1.1.1.cmml"><mi id="A6.E28.m1.1.1.1.1.1.1.1.2" xref="A6.E28.m1.1.1.1.1.1.1.1.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E28.m1.1.1.1.1.1.1.1.1" xref="A6.E28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="A6.E28.m1.1.1.1.1.1.1.1.3" xref="A6.E28.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A6.E28.m1.1.1.1.1.1.1.1.3.2" xref="A6.E28.m1.1.1.1.1.1.1.1.3.2.cmml">𝚺</mi><mi id="A6.E28.m1.1.1.1.1.1.1.1.3.3" xref="A6.E28.m1.1.1.1.1.1.1.1.3.3.cmml">λ</mi></msub><mo lspace="0em" rspace="0em" id="A6.E28.m1.1.1.1.1.1.1.1.1a" xref="A6.E28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><msup id="A6.E28.m1.1.1.1.1.1.1.1.4" xref="A6.E28.m1.1.1.1.1.1.1.1.4.cmml"><mi id="A6.E28.m1.1.1.1.1.1.1.1.4.2" xref="A6.E28.m1.1.1.1.1.1.1.1.4.2.cmml">𝑽</mi><mo id="A6.E28.m1.1.1.1.1.1.1.1.4.3" xref="A6.E28.m1.1.1.1.1.1.1.1.4.3.cmml">⊤</mo></msup></mrow><mo stretchy="false" id="A6.E28.m1.1.1.1.1.1.1.3" xref="A6.E28.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.E28.m1.1.1.1.1.3" xref="A6.E28.m1.1.1.1.1.3.cmml"><mo id="A6.E28.m1.1.1.1.1.3a" xref="A6.E28.m1.1.1.1.1.3.cmml">−</mo><mn id="A6.E28.m1.1.1.1.1.3.2" xref="A6.E28.m1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.E28.m1.1.1.1.2" xref="A6.E28.m1.1.1.1.2.cmml">​</mo><mi id="A6.E28.m1.1.1.1.3" xref="A6.E28.m1.1.1.1.3.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E28.m1.1.1.1.2a" xref="A6.E28.m1.1.1.1.2.cmml">​</mo><mi id="A6.E28.m1.1.1.1.4" xref="A6.E28.m1.1.1.1.4.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E28.m1.1.1.1.2b" xref="A6.E28.m1.1.1.1.2.cmml">​</mo><msup id="A6.E28.m1.1.1.1.5" xref="A6.E28.m1.1.1.1.5.cmml"><mi id="A6.E28.m1.1.1.1.5.2" xref="A6.E28.m1.1.1.1.5.2.cmml">𝑼</mi><mo id="A6.E28.m1.1.1.1.5.3" xref="A6.E28.m1.1.1.1.5.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E28.m1.1b"><apply id="A6.E28.m1.1.1.cmml" xref="A6.E28.m1.1.1"><eq id="A6.E28.m1.1.1.2.cmml" xref="A6.E28.m1.1.1.2"></eq><csymbol cd="latexml" id="A6.E28.m1.1.1.3.cmml" xref="A6.E28.m1.1.1.3">absent</csymbol><apply id="A6.E28.m1.1.1.1.cmml" xref="A6.E28.m1.1.1.1"><times id="A6.E28.m1.1.1.1.2.cmml" xref="A6.E28.m1.1.1.1.2"></times><apply id="A6.E28.m1.1.1.1.1.cmml" xref="A6.E28.m1.1.1.1.1"><csymbol cd="ambiguous" id="A6.E28.m1.1.1.1.1.2.cmml" xref="A6.E28.m1.1.1.1.1">superscript</csymbol><apply id="A6.E28.m1.1.1.1.1.1.1.1.cmml" xref="A6.E28.m1.1.1.1.1.1.1"><times id="A6.E28.m1.1.1.1.1.1.1.1.1.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.1"></times><ci id="A6.E28.m1.1.1.1.1.1.1.1.2.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.2">𝑽</ci><apply id="A6.E28.m1.1.1.1.1.1.1.1.3.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A6.E28.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A6.E28.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.3.2">𝚺</ci><ci id="A6.E28.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.3.3">𝜆</ci></apply><apply id="A6.E28.m1.1.1.1.1.1.1.1.4.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="A6.E28.m1.1.1.1.1.1.1.1.4.1.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.4">superscript</csymbol><ci id="A6.E28.m1.1.1.1.1.1.1.1.4.2.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.4.2">𝑽</ci><csymbol cd="latexml" id="A6.E28.m1.1.1.1.1.1.1.1.4.3.cmml" xref="A6.E28.m1.1.1.1.1.1.1.1.4.3">top</csymbol></apply></apply><apply id="A6.E28.m1.1.1.1.1.3.cmml" xref="A6.E28.m1.1.1.1.1.3"><minus id="A6.E28.m1.1.1.1.1.3.1.cmml" xref="A6.E28.m1.1.1.1.1.3"></minus><cn type="integer" id="A6.E28.m1.1.1.1.1.3.2.cmml" xref="A6.E28.m1.1.1.1.1.3.2">1</cn></apply></apply><ci id="A6.E28.m1.1.1.1.3.cmml" xref="A6.E28.m1.1.1.1.3">𝑽</ci><ci id="A6.E28.m1.1.1.1.4.cmml" xref="A6.E28.m1.1.1.1.4">𝚺</ci><apply id="A6.E28.m1.1.1.1.5.cmml" xref="A6.E28.m1.1.1.1.5"><csymbol cd="ambiguous" id="A6.E28.m1.1.1.1.5.1.cmml" xref="A6.E28.m1.1.1.1.5">superscript</csymbol><ci id="A6.E28.m1.1.1.1.5.2.cmml" xref="A6.E28.m1.1.1.1.5.2">𝑼</ci><csymbol cd="latexml" id="A6.E28.m1.1.1.1.5.3.cmml" xref="A6.E28.m1.1.1.1.5.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E28.m1.1c">\displaystyle=(\bm{V}\bm{\Sigma}_{\lambda}\bm{V}^{\top})^{-1}\bm{V}\bm{\Sigma}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(28)</span></td>
</tr></tbody>
<tbody id="A6.E29"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E29.m1.1" class="ltx_Math" alttext="\displaystyle=\bm{V}\bm{\Sigma}_{\lambda}^{-1}\bm{V}^{\top}\bm{V}\bm{\Sigma}\bm{U}^{\top}" display="inline"><semantics id="A6.E29.m1.1a"><mrow id="A6.E29.m1.1.1" xref="A6.E29.m1.1.1.cmml"><mi id="A6.E29.m1.1.1.2" xref="A6.E29.m1.1.1.2.cmml"></mi><mo id="A6.E29.m1.1.1.1" xref="A6.E29.m1.1.1.1.cmml">=</mo><mrow id="A6.E29.m1.1.1.3" xref="A6.E29.m1.1.1.3.cmml"><mi id="A6.E29.m1.1.1.3.2" xref="A6.E29.m1.1.1.3.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E29.m1.1.1.3.1" xref="A6.E29.m1.1.1.3.1.cmml">​</mo><msubsup id="A6.E29.m1.1.1.3.3" xref="A6.E29.m1.1.1.3.3.cmml"><mi id="A6.E29.m1.1.1.3.3.2.2" xref="A6.E29.m1.1.1.3.3.2.2.cmml">𝚺</mi><mi id="A6.E29.m1.1.1.3.3.2.3" xref="A6.E29.m1.1.1.3.3.2.3.cmml">λ</mi><mrow id="A6.E29.m1.1.1.3.3.3" xref="A6.E29.m1.1.1.3.3.3.cmml"><mo id="A6.E29.m1.1.1.3.3.3a" xref="A6.E29.m1.1.1.3.3.3.cmml">−</mo><mn id="A6.E29.m1.1.1.3.3.3.2" xref="A6.E29.m1.1.1.3.3.3.2.cmml">1</mn></mrow></msubsup><mo lspace="0em" rspace="0em" id="A6.E29.m1.1.1.3.1a" xref="A6.E29.m1.1.1.3.1.cmml">​</mo><msup id="A6.E29.m1.1.1.3.4" xref="A6.E29.m1.1.1.3.4.cmml"><mi id="A6.E29.m1.1.1.3.4.2" xref="A6.E29.m1.1.1.3.4.2.cmml">𝑽</mi><mo id="A6.E29.m1.1.1.3.4.3" xref="A6.E29.m1.1.1.3.4.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E29.m1.1.1.3.1b" xref="A6.E29.m1.1.1.3.1.cmml">​</mo><mi id="A6.E29.m1.1.1.3.5" xref="A6.E29.m1.1.1.3.5.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E29.m1.1.1.3.1c" xref="A6.E29.m1.1.1.3.1.cmml">​</mo><mi id="A6.E29.m1.1.1.3.6" xref="A6.E29.m1.1.1.3.6.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E29.m1.1.1.3.1d" xref="A6.E29.m1.1.1.3.1.cmml">​</mo><msup id="A6.E29.m1.1.1.3.7" xref="A6.E29.m1.1.1.3.7.cmml"><mi id="A6.E29.m1.1.1.3.7.2" xref="A6.E29.m1.1.1.3.7.2.cmml">𝑼</mi><mo id="A6.E29.m1.1.1.3.7.3" xref="A6.E29.m1.1.1.3.7.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E29.m1.1b"><apply id="A6.E29.m1.1.1.cmml" xref="A6.E29.m1.1.1"><eq id="A6.E29.m1.1.1.1.cmml" xref="A6.E29.m1.1.1.1"></eq><csymbol cd="latexml" id="A6.E29.m1.1.1.2.cmml" xref="A6.E29.m1.1.1.2">absent</csymbol><apply id="A6.E29.m1.1.1.3.cmml" xref="A6.E29.m1.1.1.3"><times id="A6.E29.m1.1.1.3.1.cmml" xref="A6.E29.m1.1.1.3.1"></times><ci id="A6.E29.m1.1.1.3.2.cmml" xref="A6.E29.m1.1.1.3.2">𝑽</ci><apply id="A6.E29.m1.1.1.3.3.cmml" xref="A6.E29.m1.1.1.3.3"><csymbol cd="ambiguous" id="A6.E29.m1.1.1.3.3.1.cmml" xref="A6.E29.m1.1.1.3.3">superscript</csymbol><apply id="A6.E29.m1.1.1.3.3.2.cmml" xref="A6.E29.m1.1.1.3.3"><csymbol cd="ambiguous" id="A6.E29.m1.1.1.3.3.2.1.cmml" xref="A6.E29.m1.1.1.3.3">subscript</csymbol><ci id="A6.E29.m1.1.1.3.3.2.2.cmml" xref="A6.E29.m1.1.1.3.3.2.2">𝚺</ci><ci id="A6.E29.m1.1.1.3.3.2.3.cmml" xref="A6.E29.m1.1.1.3.3.2.3">𝜆</ci></apply><apply id="A6.E29.m1.1.1.3.3.3.cmml" xref="A6.E29.m1.1.1.3.3.3"><minus id="A6.E29.m1.1.1.3.3.3.1.cmml" xref="A6.E29.m1.1.1.3.3.3"></minus><cn type="integer" id="A6.E29.m1.1.1.3.3.3.2.cmml" xref="A6.E29.m1.1.1.3.3.3.2">1</cn></apply></apply><apply id="A6.E29.m1.1.1.3.4.cmml" xref="A6.E29.m1.1.1.3.4"><csymbol cd="ambiguous" id="A6.E29.m1.1.1.3.4.1.cmml" xref="A6.E29.m1.1.1.3.4">superscript</csymbol><ci id="A6.E29.m1.1.1.3.4.2.cmml" xref="A6.E29.m1.1.1.3.4.2">𝑽</ci><csymbol cd="latexml" id="A6.E29.m1.1.1.3.4.3.cmml" xref="A6.E29.m1.1.1.3.4.3">top</csymbol></apply><ci id="A6.E29.m1.1.1.3.5.cmml" xref="A6.E29.m1.1.1.3.5">𝑽</ci><ci id="A6.E29.m1.1.1.3.6.cmml" xref="A6.E29.m1.1.1.3.6">𝚺</ci><apply id="A6.E29.m1.1.1.3.7.cmml" xref="A6.E29.m1.1.1.3.7"><csymbol cd="ambiguous" id="A6.E29.m1.1.1.3.7.1.cmml" xref="A6.E29.m1.1.1.3.7">superscript</csymbol><ci id="A6.E29.m1.1.1.3.7.2.cmml" xref="A6.E29.m1.1.1.3.7.2">𝑼</ci><csymbol cd="latexml" id="A6.E29.m1.1.1.3.7.3.cmml" xref="A6.E29.m1.1.1.3.7.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E29.m1.1c">\displaystyle=\bm{V}\bm{\Sigma}_{\lambda}^{-1}\bm{V}^{\top}\bm{V}\bm{\Sigma}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(29)</span></td>
</tr></tbody>
<tbody id="A6.E30"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E30.m1.1" class="ltx_Math" alttext="\displaystyle=\bm{V}\bm{\Sigma}_{\lambda}^{-1}\bm{\Sigma}\bm{U}^{\top}" display="inline"><semantics id="A6.E30.m1.1a"><mrow id="A6.E30.m1.1.1" xref="A6.E30.m1.1.1.cmml"><mi id="A6.E30.m1.1.1.2" xref="A6.E30.m1.1.1.2.cmml"></mi><mo id="A6.E30.m1.1.1.1" xref="A6.E30.m1.1.1.1.cmml">=</mo><mrow id="A6.E30.m1.1.1.3" xref="A6.E30.m1.1.1.3.cmml"><mi id="A6.E30.m1.1.1.3.2" xref="A6.E30.m1.1.1.3.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E30.m1.1.1.3.1" xref="A6.E30.m1.1.1.3.1.cmml">​</mo><msubsup id="A6.E30.m1.1.1.3.3" xref="A6.E30.m1.1.1.3.3.cmml"><mi id="A6.E30.m1.1.1.3.3.2.2" xref="A6.E30.m1.1.1.3.3.2.2.cmml">𝚺</mi><mi id="A6.E30.m1.1.1.3.3.2.3" xref="A6.E30.m1.1.1.3.3.2.3.cmml">λ</mi><mrow id="A6.E30.m1.1.1.3.3.3" xref="A6.E30.m1.1.1.3.3.3.cmml"><mo id="A6.E30.m1.1.1.3.3.3a" xref="A6.E30.m1.1.1.3.3.3.cmml">−</mo><mn id="A6.E30.m1.1.1.3.3.3.2" xref="A6.E30.m1.1.1.3.3.3.2.cmml">1</mn></mrow></msubsup><mo lspace="0em" rspace="0em" id="A6.E30.m1.1.1.3.1a" xref="A6.E30.m1.1.1.3.1.cmml">​</mo><mi id="A6.E30.m1.1.1.3.4" xref="A6.E30.m1.1.1.3.4.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="A6.E30.m1.1.1.3.1b" xref="A6.E30.m1.1.1.3.1.cmml">​</mo><msup id="A6.E30.m1.1.1.3.5" xref="A6.E30.m1.1.1.3.5.cmml"><mi id="A6.E30.m1.1.1.3.5.2" xref="A6.E30.m1.1.1.3.5.2.cmml">𝑼</mi><mo id="A6.E30.m1.1.1.3.5.3" xref="A6.E30.m1.1.1.3.5.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E30.m1.1b"><apply id="A6.E30.m1.1.1.cmml" xref="A6.E30.m1.1.1"><eq id="A6.E30.m1.1.1.1.cmml" xref="A6.E30.m1.1.1.1"></eq><csymbol cd="latexml" id="A6.E30.m1.1.1.2.cmml" xref="A6.E30.m1.1.1.2">absent</csymbol><apply id="A6.E30.m1.1.1.3.cmml" xref="A6.E30.m1.1.1.3"><times id="A6.E30.m1.1.1.3.1.cmml" xref="A6.E30.m1.1.1.3.1"></times><ci id="A6.E30.m1.1.1.3.2.cmml" xref="A6.E30.m1.1.1.3.2">𝑽</ci><apply id="A6.E30.m1.1.1.3.3.cmml" xref="A6.E30.m1.1.1.3.3"><csymbol cd="ambiguous" id="A6.E30.m1.1.1.3.3.1.cmml" xref="A6.E30.m1.1.1.3.3">superscript</csymbol><apply id="A6.E30.m1.1.1.3.3.2.cmml" xref="A6.E30.m1.1.1.3.3"><csymbol cd="ambiguous" id="A6.E30.m1.1.1.3.3.2.1.cmml" xref="A6.E30.m1.1.1.3.3">subscript</csymbol><ci id="A6.E30.m1.1.1.3.3.2.2.cmml" xref="A6.E30.m1.1.1.3.3.2.2">𝚺</ci><ci id="A6.E30.m1.1.1.3.3.2.3.cmml" xref="A6.E30.m1.1.1.3.3.2.3">𝜆</ci></apply><apply id="A6.E30.m1.1.1.3.3.3.cmml" xref="A6.E30.m1.1.1.3.3.3"><minus id="A6.E30.m1.1.1.3.3.3.1.cmml" xref="A6.E30.m1.1.1.3.3.3"></minus><cn type="integer" id="A6.E30.m1.1.1.3.3.3.2.cmml" xref="A6.E30.m1.1.1.3.3.3.2">1</cn></apply></apply><ci id="A6.E30.m1.1.1.3.4.cmml" xref="A6.E30.m1.1.1.3.4">𝚺</ci><apply id="A6.E30.m1.1.1.3.5.cmml" xref="A6.E30.m1.1.1.3.5"><csymbol cd="ambiguous" id="A6.E30.m1.1.1.3.5.1.cmml" xref="A6.E30.m1.1.1.3.5">superscript</csymbol><ci id="A6.E30.m1.1.1.3.5.2.cmml" xref="A6.E30.m1.1.1.3.5.2">𝑼</ci><csymbol cd="latexml" id="A6.E30.m1.1.1.3.5.3.cmml" xref="A6.E30.m1.1.1.3.5.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E30.m1.1c">\displaystyle=\bm{V}\bm{\Sigma}_{\lambda}^{-1}\bm{\Sigma}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(30)</span></td>
</tr></tbody>
<tbody id="A6.E31"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E31.m1.1" class="ltx_Math" alttext="\displaystyle=\bm{V}\bm{\Sigma}^{\prime}\bm{U}^{\top}" display="inline"><semantics id="A6.E31.m1.1a"><mrow id="A6.E31.m1.1.1" xref="A6.E31.m1.1.1.cmml"><mi id="A6.E31.m1.1.1.2" xref="A6.E31.m1.1.1.2.cmml"></mi><mo id="A6.E31.m1.1.1.1" xref="A6.E31.m1.1.1.1.cmml">=</mo><mrow id="A6.E31.m1.1.1.3" xref="A6.E31.m1.1.1.3.cmml"><mi id="A6.E31.m1.1.1.3.2" xref="A6.E31.m1.1.1.3.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.E31.m1.1.1.3.1" xref="A6.E31.m1.1.1.3.1.cmml">​</mo><msup id="A6.E31.m1.1.1.3.3" xref="A6.E31.m1.1.1.3.3.cmml"><mi id="A6.E31.m1.1.1.3.3.2" xref="A6.E31.m1.1.1.3.3.2.cmml">𝚺</mi><mo id="A6.E31.m1.1.1.3.3.3" xref="A6.E31.m1.1.1.3.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A6.E31.m1.1.1.3.1a" xref="A6.E31.m1.1.1.3.1.cmml">​</mo><msup id="A6.E31.m1.1.1.3.4" xref="A6.E31.m1.1.1.3.4.cmml"><mi id="A6.E31.m1.1.1.3.4.2" xref="A6.E31.m1.1.1.3.4.2.cmml">𝑼</mi><mo id="A6.E31.m1.1.1.3.4.3" xref="A6.E31.m1.1.1.3.4.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E31.m1.1b"><apply id="A6.E31.m1.1.1.cmml" xref="A6.E31.m1.1.1"><eq id="A6.E31.m1.1.1.1.cmml" xref="A6.E31.m1.1.1.1"></eq><csymbol cd="latexml" id="A6.E31.m1.1.1.2.cmml" xref="A6.E31.m1.1.1.2">absent</csymbol><apply id="A6.E31.m1.1.1.3.cmml" xref="A6.E31.m1.1.1.3"><times id="A6.E31.m1.1.1.3.1.cmml" xref="A6.E31.m1.1.1.3.1"></times><ci id="A6.E31.m1.1.1.3.2.cmml" xref="A6.E31.m1.1.1.3.2">𝑽</ci><apply id="A6.E31.m1.1.1.3.3.cmml" xref="A6.E31.m1.1.1.3.3"><csymbol cd="ambiguous" id="A6.E31.m1.1.1.3.3.1.cmml" xref="A6.E31.m1.1.1.3.3">superscript</csymbol><ci id="A6.E31.m1.1.1.3.3.2.cmml" xref="A6.E31.m1.1.1.3.3.2">𝚺</ci><ci id="A6.E31.m1.1.1.3.3.3.cmml" xref="A6.E31.m1.1.1.3.3.3">′</ci></apply><apply id="A6.E31.m1.1.1.3.4.cmml" xref="A6.E31.m1.1.1.3.4"><csymbol cd="ambiguous" id="A6.E31.m1.1.1.3.4.1.cmml" xref="A6.E31.m1.1.1.3.4">superscript</csymbol><ci id="A6.E31.m1.1.1.3.4.2.cmml" xref="A6.E31.m1.1.1.3.4.2">𝑼</ci><csymbol cd="latexml" id="A6.E31.m1.1.1.3.4.3.cmml" xref="A6.E31.m1.1.1.3.4.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E31.m1.1c">\displaystyle=\bm{V}\bm{\Sigma}^{\prime}\bm{U}^{\top}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(31)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.5.p1.11" class="ltx_p">where <math id="A6.SS2.5.p1.3.m1.1" class="ltx_Math" alttext="\bm{\Sigma}_{\lambda}\triangleq\bm{\Sigma}^{2}+\lambda\bm{I}" display="inline"><semantics id="A6.SS2.5.p1.3.m1.1a"><mrow id="A6.SS2.5.p1.3.m1.1.1" xref="A6.SS2.5.p1.3.m1.1.1.cmml"><msub id="A6.SS2.5.p1.3.m1.1.1.2" xref="A6.SS2.5.p1.3.m1.1.1.2.cmml"><mi id="A6.SS2.5.p1.3.m1.1.1.2.2" xref="A6.SS2.5.p1.3.m1.1.1.2.2.cmml">𝚺</mi><mi id="A6.SS2.5.p1.3.m1.1.1.2.3" xref="A6.SS2.5.p1.3.m1.1.1.2.3.cmml">λ</mi></msub><mo id="A6.SS2.5.p1.3.m1.1.1.1" xref="A6.SS2.5.p1.3.m1.1.1.1.cmml">≜</mo><mrow id="A6.SS2.5.p1.3.m1.1.1.3" xref="A6.SS2.5.p1.3.m1.1.1.3.cmml"><msup id="A6.SS2.5.p1.3.m1.1.1.3.2" xref="A6.SS2.5.p1.3.m1.1.1.3.2.cmml"><mi id="A6.SS2.5.p1.3.m1.1.1.3.2.2" xref="A6.SS2.5.p1.3.m1.1.1.3.2.2.cmml">𝚺</mi><mn id="A6.SS2.5.p1.3.m1.1.1.3.2.3" xref="A6.SS2.5.p1.3.m1.1.1.3.2.3.cmml">2</mn></msup><mo id="A6.SS2.5.p1.3.m1.1.1.3.1" xref="A6.SS2.5.p1.3.m1.1.1.3.1.cmml">+</mo><mrow id="A6.SS2.5.p1.3.m1.1.1.3.3" xref="A6.SS2.5.p1.3.m1.1.1.3.3.cmml"><mi id="A6.SS2.5.p1.3.m1.1.1.3.3.2" xref="A6.SS2.5.p1.3.m1.1.1.3.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.3.m1.1.1.3.3.1" xref="A6.SS2.5.p1.3.m1.1.1.3.3.1.cmml">​</mo><mi id="A6.SS2.5.p1.3.m1.1.1.3.3.3" xref="A6.SS2.5.p1.3.m1.1.1.3.3.3.cmml">𝑰</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.3.m1.1b"><apply id="A6.SS2.5.p1.3.m1.1.1.cmml" xref="A6.SS2.5.p1.3.m1.1.1"><ci id="A6.SS2.5.p1.3.m1.1.1.1.cmml" xref="A6.SS2.5.p1.3.m1.1.1.1">≜</ci><apply id="A6.SS2.5.p1.3.m1.1.1.2.cmml" xref="A6.SS2.5.p1.3.m1.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.5.p1.3.m1.1.1.2.1.cmml" xref="A6.SS2.5.p1.3.m1.1.1.2">subscript</csymbol><ci id="A6.SS2.5.p1.3.m1.1.1.2.2.cmml" xref="A6.SS2.5.p1.3.m1.1.1.2.2">𝚺</ci><ci id="A6.SS2.5.p1.3.m1.1.1.2.3.cmml" xref="A6.SS2.5.p1.3.m1.1.1.2.3">𝜆</ci></apply><apply id="A6.SS2.5.p1.3.m1.1.1.3.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3"><plus id="A6.SS2.5.p1.3.m1.1.1.3.1.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.1"></plus><apply id="A6.SS2.5.p1.3.m1.1.1.3.2.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.2"><csymbol cd="ambiguous" id="A6.SS2.5.p1.3.m1.1.1.3.2.1.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.2">superscript</csymbol><ci id="A6.SS2.5.p1.3.m1.1.1.3.2.2.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.2.2">𝚺</ci><cn type="integer" id="A6.SS2.5.p1.3.m1.1.1.3.2.3.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.2.3">2</cn></apply><apply id="A6.SS2.5.p1.3.m1.1.1.3.3.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.3"><times id="A6.SS2.5.p1.3.m1.1.1.3.3.1.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.3.1"></times><ci id="A6.SS2.5.p1.3.m1.1.1.3.3.2.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.3.2">𝜆</ci><ci id="A6.SS2.5.p1.3.m1.1.1.3.3.3.cmml" xref="A6.SS2.5.p1.3.m1.1.1.3.3.3">𝑰</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.3.m1.1c">\bm{\Sigma}_{\lambda}\triangleq\bm{\Sigma}^{2}+\lambda\bm{I}</annotation></semantics></math> and <math id="A6.SS2.5.p1.4.m2.1" class="ltx_Math" alttext="\bm{\Sigma}^{\prime}\triangleq(\bm{\Sigma}^{2}+\lambda\bm{I})^{-1}\Sigma" display="inline"><semantics id="A6.SS2.5.p1.4.m2.1a"><mrow id="A6.SS2.5.p1.4.m2.1.1" xref="A6.SS2.5.p1.4.m2.1.1.cmml"><msup id="A6.SS2.5.p1.4.m2.1.1.3" xref="A6.SS2.5.p1.4.m2.1.1.3.cmml"><mi id="A6.SS2.5.p1.4.m2.1.1.3.2" xref="A6.SS2.5.p1.4.m2.1.1.3.2.cmml">𝚺</mi><mo id="A6.SS2.5.p1.4.m2.1.1.3.3" xref="A6.SS2.5.p1.4.m2.1.1.3.3.cmml">′</mo></msup><mo id="A6.SS2.5.p1.4.m2.1.1.2" xref="A6.SS2.5.p1.4.m2.1.1.2.cmml">≜</mo><mrow id="A6.SS2.5.p1.4.m2.1.1.1" xref="A6.SS2.5.p1.4.m2.1.1.1.cmml"><msup id="A6.SS2.5.p1.4.m2.1.1.1.1" xref="A6.SS2.5.p1.4.m2.1.1.1.1.cmml"><mrow id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.2" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.cmml"><msup id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.cmml"><mi id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.2" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.2.cmml">𝚺</mi><mn id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.3" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.3.cmml">2</mn></msup><mo id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.1" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.2" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.1" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.3" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.3.cmml">𝑰</mi></mrow></mrow><mo stretchy="false" id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.3" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.SS2.5.p1.4.m2.1.1.1.1.3" xref="A6.SS2.5.p1.4.m2.1.1.1.1.3.cmml"><mo id="A6.SS2.5.p1.4.m2.1.1.1.1.3a" xref="A6.SS2.5.p1.4.m2.1.1.1.1.3.cmml">−</mo><mn id="A6.SS2.5.p1.4.m2.1.1.1.1.3.2" xref="A6.SS2.5.p1.4.m2.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.4.m2.1.1.1.2" xref="A6.SS2.5.p1.4.m2.1.1.1.2.cmml">​</mo><mi mathvariant="normal" id="A6.SS2.5.p1.4.m2.1.1.1.3" xref="A6.SS2.5.p1.4.m2.1.1.1.3.cmml">Σ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.4.m2.1b"><apply id="A6.SS2.5.p1.4.m2.1.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1"><ci id="A6.SS2.5.p1.4.m2.1.1.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.2">≜</ci><apply id="A6.SS2.5.p1.4.m2.1.1.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.3"><csymbol cd="ambiguous" id="A6.SS2.5.p1.4.m2.1.1.3.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.3">superscript</csymbol><ci id="A6.SS2.5.p1.4.m2.1.1.3.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.3.2">𝚺</ci><ci id="A6.SS2.5.p1.4.m2.1.1.3.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.3.3">′</ci></apply><apply id="A6.SS2.5.p1.4.m2.1.1.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1"><times id="A6.SS2.5.p1.4.m2.1.1.1.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.2"></times><apply id="A6.SS2.5.p1.4.m2.1.1.1.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1"><csymbol cd="ambiguous" id="A6.SS2.5.p1.4.m2.1.1.1.1.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1">superscript</csymbol><apply id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1"><plus id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.1"></plus><apply id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.2">𝚺</ci><cn type="integer" id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.2.3">2</cn></apply><apply id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3"><times id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.1"></times><ci id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.1.1.1.3.3">𝑰</ci></apply></apply><apply id="A6.SS2.5.p1.4.m2.1.1.1.1.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.3"><minus id="A6.SS2.5.p1.4.m2.1.1.1.1.3.1.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.3"></minus><cn type="integer" id="A6.SS2.5.p1.4.m2.1.1.1.1.3.2.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.1.3.2">1</cn></apply></apply><ci id="A6.SS2.5.p1.4.m2.1.1.1.3.cmml" xref="A6.SS2.5.p1.4.m2.1.1.1.3">Σ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.4.m2.1c">\bm{\Sigma}^{\prime}\triangleq(\bm{\Sigma}^{2}+\lambda\bm{I})^{-1}\Sigma</annotation></semantics></math>. Step <a href="#A6.E27" title="In Proof. ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">27</span></a> follows from the equality <math id="A6.SS2.5.p1.5.m3.1" class="ltx_Math" alttext="\bm{V}\lambda\bm{I}\bm{V}^{\top}=\lambda\bm{V}\bm{V}^{\top}=\lambda\bm{I}" display="inline"><semantics id="A6.SS2.5.p1.5.m3.1a"><mrow id="A6.SS2.5.p1.5.m3.1.1" xref="A6.SS2.5.p1.5.m3.1.1.cmml"><mrow id="A6.SS2.5.p1.5.m3.1.1.2" xref="A6.SS2.5.p1.5.m3.1.1.2.cmml"><mi id="A6.SS2.5.p1.5.m3.1.1.2.2" xref="A6.SS2.5.p1.5.m3.1.1.2.2.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.5.m3.1.1.2.1" xref="A6.SS2.5.p1.5.m3.1.1.2.1.cmml">​</mo><mi id="A6.SS2.5.p1.5.m3.1.1.2.3" xref="A6.SS2.5.p1.5.m3.1.1.2.3.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.5.m3.1.1.2.1a" xref="A6.SS2.5.p1.5.m3.1.1.2.1.cmml">​</mo><mi id="A6.SS2.5.p1.5.m3.1.1.2.4" xref="A6.SS2.5.p1.5.m3.1.1.2.4.cmml">𝑰</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.5.m3.1.1.2.1b" xref="A6.SS2.5.p1.5.m3.1.1.2.1.cmml">​</mo><msup id="A6.SS2.5.p1.5.m3.1.1.2.5" xref="A6.SS2.5.p1.5.m3.1.1.2.5.cmml"><mi id="A6.SS2.5.p1.5.m3.1.1.2.5.2" xref="A6.SS2.5.p1.5.m3.1.1.2.5.2.cmml">𝑽</mi><mo id="A6.SS2.5.p1.5.m3.1.1.2.5.3" xref="A6.SS2.5.p1.5.m3.1.1.2.5.3.cmml">⊤</mo></msup></mrow><mo id="A6.SS2.5.p1.5.m3.1.1.3" xref="A6.SS2.5.p1.5.m3.1.1.3.cmml">=</mo><mrow id="A6.SS2.5.p1.5.m3.1.1.4" xref="A6.SS2.5.p1.5.m3.1.1.4.cmml"><mi id="A6.SS2.5.p1.5.m3.1.1.4.2" xref="A6.SS2.5.p1.5.m3.1.1.4.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.5.m3.1.1.4.1" xref="A6.SS2.5.p1.5.m3.1.1.4.1.cmml">​</mo><mi id="A6.SS2.5.p1.5.m3.1.1.4.3" xref="A6.SS2.5.p1.5.m3.1.1.4.3.cmml">𝑽</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.5.m3.1.1.4.1a" xref="A6.SS2.5.p1.5.m3.1.1.4.1.cmml">​</mo><msup id="A6.SS2.5.p1.5.m3.1.1.4.4" xref="A6.SS2.5.p1.5.m3.1.1.4.4.cmml"><mi id="A6.SS2.5.p1.5.m3.1.1.4.4.2" xref="A6.SS2.5.p1.5.m3.1.1.4.4.2.cmml">𝑽</mi><mo id="A6.SS2.5.p1.5.m3.1.1.4.4.3" xref="A6.SS2.5.p1.5.m3.1.1.4.4.3.cmml">⊤</mo></msup></mrow><mo id="A6.SS2.5.p1.5.m3.1.1.5" xref="A6.SS2.5.p1.5.m3.1.1.5.cmml">=</mo><mrow id="A6.SS2.5.p1.5.m3.1.1.6" xref="A6.SS2.5.p1.5.m3.1.1.6.cmml"><mi id="A6.SS2.5.p1.5.m3.1.1.6.2" xref="A6.SS2.5.p1.5.m3.1.1.6.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.5.m3.1.1.6.1" xref="A6.SS2.5.p1.5.m3.1.1.6.1.cmml">​</mo><mi id="A6.SS2.5.p1.5.m3.1.1.6.3" xref="A6.SS2.5.p1.5.m3.1.1.6.3.cmml">𝑰</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.5.m3.1b"><apply id="A6.SS2.5.p1.5.m3.1.1.cmml" xref="A6.SS2.5.p1.5.m3.1.1"><and id="A6.SS2.5.p1.5.m3.1.1a.cmml" xref="A6.SS2.5.p1.5.m3.1.1"></and><apply id="A6.SS2.5.p1.5.m3.1.1b.cmml" xref="A6.SS2.5.p1.5.m3.1.1"><eq id="A6.SS2.5.p1.5.m3.1.1.3.cmml" xref="A6.SS2.5.p1.5.m3.1.1.3"></eq><apply id="A6.SS2.5.p1.5.m3.1.1.2.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2"><times id="A6.SS2.5.p1.5.m3.1.1.2.1.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.1"></times><ci id="A6.SS2.5.p1.5.m3.1.1.2.2.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.2">𝑽</ci><ci id="A6.SS2.5.p1.5.m3.1.1.2.3.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.3">𝜆</ci><ci id="A6.SS2.5.p1.5.m3.1.1.2.4.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.4">𝑰</ci><apply id="A6.SS2.5.p1.5.m3.1.1.2.5.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.5"><csymbol cd="ambiguous" id="A6.SS2.5.p1.5.m3.1.1.2.5.1.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.5">superscript</csymbol><ci id="A6.SS2.5.p1.5.m3.1.1.2.5.2.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.5.2">𝑽</ci><csymbol cd="latexml" id="A6.SS2.5.p1.5.m3.1.1.2.5.3.cmml" xref="A6.SS2.5.p1.5.m3.1.1.2.5.3">top</csymbol></apply></apply><apply id="A6.SS2.5.p1.5.m3.1.1.4.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4"><times id="A6.SS2.5.p1.5.m3.1.1.4.1.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.1"></times><ci id="A6.SS2.5.p1.5.m3.1.1.4.2.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.2">𝜆</ci><ci id="A6.SS2.5.p1.5.m3.1.1.4.3.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.3">𝑽</ci><apply id="A6.SS2.5.p1.5.m3.1.1.4.4.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.4"><csymbol cd="ambiguous" id="A6.SS2.5.p1.5.m3.1.1.4.4.1.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.4">superscript</csymbol><ci id="A6.SS2.5.p1.5.m3.1.1.4.4.2.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.4.2">𝑽</ci><csymbol cd="latexml" id="A6.SS2.5.p1.5.m3.1.1.4.4.3.cmml" xref="A6.SS2.5.p1.5.m3.1.1.4.4.3">top</csymbol></apply></apply></apply><apply id="A6.SS2.5.p1.5.m3.1.1c.cmml" xref="A6.SS2.5.p1.5.m3.1.1"><eq id="A6.SS2.5.p1.5.m3.1.1.5.cmml" xref="A6.SS2.5.p1.5.m3.1.1.5"></eq><share href="#A6.SS2.5.p1.5.m3.1.1.4.cmml" id="A6.SS2.5.p1.5.m3.1.1d.cmml" xref="A6.SS2.5.p1.5.m3.1.1"></share><apply id="A6.SS2.5.p1.5.m3.1.1.6.cmml" xref="A6.SS2.5.p1.5.m3.1.1.6"><times id="A6.SS2.5.p1.5.m3.1.1.6.1.cmml" xref="A6.SS2.5.p1.5.m3.1.1.6.1"></times><ci id="A6.SS2.5.p1.5.m3.1.1.6.2.cmml" xref="A6.SS2.5.p1.5.m3.1.1.6.2">𝜆</ci><ci id="A6.SS2.5.p1.5.m3.1.1.6.3.cmml" xref="A6.SS2.5.p1.5.m3.1.1.6.3">𝑰</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.5.m3.1c">\bm{V}\lambda\bm{I}\bm{V}^{\top}=\lambda\bm{V}\bm{V}^{\top}=\lambda\bm{I}</annotation></semantics></math>. Because the matrices <math id="A6.SS2.5.p1.6.m4.1" class="ltx_Math" alttext="\bm{V}" display="inline"><semantics id="A6.SS2.5.p1.6.m4.1a"><mi id="A6.SS2.5.p1.6.m4.1.1" xref="A6.SS2.5.p1.6.m4.1.1.cmml">𝑽</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.6.m4.1b"><ci id="A6.SS2.5.p1.6.m4.1.1.cmml" xref="A6.SS2.5.p1.6.m4.1.1">𝑽</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.6.m4.1c">\bm{V}</annotation></semantics></math> and <math id="A6.SS2.5.p1.7.m5.1" class="ltx_Math" alttext="\bm{U}^{\top}" display="inline"><semantics id="A6.SS2.5.p1.7.m5.1a"><msup id="A6.SS2.5.p1.7.m5.1.1" xref="A6.SS2.5.p1.7.m5.1.1.cmml"><mi id="A6.SS2.5.p1.7.m5.1.1.2" xref="A6.SS2.5.p1.7.m5.1.1.2.cmml">𝑼</mi><mo id="A6.SS2.5.p1.7.m5.1.1.3" xref="A6.SS2.5.p1.7.m5.1.1.3.cmml">⊤</mo></msup><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.7.m5.1b"><apply id="A6.SS2.5.p1.7.m5.1.1.cmml" xref="A6.SS2.5.p1.7.m5.1.1"><csymbol cd="ambiguous" id="A6.SS2.5.p1.7.m5.1.1.1.cmml" xref="A6.SS2.5.p1.7.m5.1.1">superscript</csymbol><ci id="A6.SS2.5.p1.7.m5.1.1.2.cmml" xref="A6.SS2.5.p1.7.m5.1.1.2">𝑼</ci><csymbol cd="latexml" id="A6.SS2.5.p1.7.m5.1.1.3.cmml" xref="A6.SS2.5.p1.7.m5.1.1.3">top</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.7.m5.1c">\bm{U}^{\top}</annotation></semantics></math> are orthonormal and <math id="A6.SS2.5.p1.8.m6.1" class="ltx_Math" alttext="\bm{\Sigma}^{\prime}" display="inline"><semantics id="A6.SS2.5.p1.8.m6.1a"><msup id="A6.SS2.5.p1.8.m6.1.1" xref="A6.SS2.5.p1.8.m6.1.1.cmml"><mi id="A6.SS2.5.p1.8.m6.1.1.2" xref="A6.SS2.5.p1.8.m6.1.1.2.cmml">𝚺</mi><mo id="A6.SS2.5.p1.8.m6.1.1.3" xref="A6.SS2.5.p1.8.m6.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.8.m6.1b"><apply id="A6.SS2.5.p1.8.m6.1.1.cmml" xref="A6.SS2.5.p1.8.m6.1.1"><csymbol cd="ambiguous" id="A6.SS2.5.p1.8.m6.1.1.1.cmml" xref="A6.SS2.5.p1.8.m6.1.1">superscript</csymbol><ci id="A6.SS2.5.p1.8.m6.1.1.2.cmml" xref="A6.SS2.5.p1.8.m6.1.1.2">𝚺</ci><ci id="A6.SS2.5.p1.8.m6.1.1.3.cmml" xref="A6.SS2.5.p1.8.m6.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.8.m6.1c">\bm{\Sigma}^{\prime}</annotation></semantics></math> is diagonal, the singular values of <math id="A6.SS2.5.p1.9.m7.1" class="ltx_Math" alttext="\bm{Z}" display="inline"><semantics id="A6.SS2.5.p1.9.m7.1a"><mi id="A6.SS2.5.p1.9.m7.1.1" xref="A6.SS2.5.p1.9.m7.1.1.cmml">𝒁</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.9.m7.1b"><ci id="A6.SS2.5.p1.9.m7.1.1.cmml" xref="A6.SS2.5.p1.9.m7.1.1">𝒁</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.9.m7.1c">\bm{Z}</annotation></semantics></math> are equal to the diagonal entries of <math id="A6.SS2.5.p1.10.m8.1" class="ltx_Math" alttext="\bm{\Sigma}^{\prime}" display="inline"><semantics id="A6.SS2.5.p1.10.m8.1a"><msup id="A6.SS2.5.p1.10.m8.1.1" xref="A6.SS2.5.p1.10.m8.1.1.cmml"><mi id="A6.SS2.5.p1.10.m8.1.1.2" xref="A6.SS2.5.p1.10.m8.1.1.2.cmml">𝚺</mi><mo id="A6.SS2.5.p1.10.m8.1.1.3" xref="A6.SS2.5.p1.10.m8.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.10.m8.1b"><apply id="A6.SS2.5.p1.10.m8.1.1.cmml" xref="A6.SS2.5.p1.10.m8.1.1"><csymbol cd="ambiguous" id="A6.SS2.5.p1.10.m8.1.1.1.cmml" xref="A6.SS2.5.p1.10.m8.1.1">superscript</csymbol><ci id="A6.SS2.5.p1.10.m8.1.1.2.cmml" xref="A6.SS2.5.p1.10.m8.1.1.2">𝚺</ci><ci id="A6.SS2.5.p1.10.m8.1.1.3.cmml" xref="A6.SS2.5.p1.10.m8.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.10.m8.1c">\bm{\Sigma}^{\prime}</annotation></semantics></math>. Each entry <math id="A6.SS2.5.p1.11.m9.1" class="ltx_Math" alttext="\sigma_{i}^{\prime}" display="inline"><semantics id="A6.SS2.5.p1.11.m9.1a"><msubsup id="A6.SS2.5.p1.11.m9.1.1" xref="A6.SS2.5.p1.11.m9.1.1.cmml"><mi id="A6.SS2.5.p1.11.m9.1.1.2.2" xref="A6.SS2.5.p1.11.m9.1.1.2.2.cmml">σ</mi><mi id="A6.SS2.5.p1.11.m9.1.1.2.3" xref="A6.SS2.5.p1.11.m9.1.1.2.3.cmml">i</mi><mo id="A6.SS2.5.p1.11.m9.1.1.3" xref="A6.SS2.5.p1.11.m9.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.11.m9.1b"><apply id="A6.SS2.5.p1.11.m9.1.1.cmml" xref="A6.SS2.5.p1.11.m9.1.1"><csymbol cd="ambiguous" id="A6.SS2.5.p1.11.m9.1.1.1.cmml" xref="A6.SS2.5.p1.11.m9.1.1">superscript</csymbol><apply id="A6.SS2.5.p1.11.m9.1.1.2.cmml" xref="A6.SS2.5.p1.11.m9.1.1"><csymbol cd="ambiguous" id="A6.SS2.5.p1.11.m9.1.1.2.1.cmml" xref="A6.SS2.5.p1.11.m9.1.1">subscript</csymbol><ci id="A6.SS2.5.p1.11.m9.1.1.2.2.cmml" xref="A6.SS2.5.p1.11.m9.1.1.2.2">𝜎</ci><ci id="A6.SS2.5.p1.11.m9.1.1.2.3.cmml" xref="A6.SS2.5.p1.11.m9.1.1.2.3">𝑖</ci></apply><ci id="A6.SS2.5.p1.11.m9.1.1.3.cmml" xref="A6.SS2.5.p1.11.m9.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.11.m9.1c">\sigma_{i}^{\prime}</annotation></semantics></math> is equal to</p>
<table id="A6.EGx21" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E32"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E32.m1.1" class="ltx_Math" alttext="\displaystyle\sigma_{i}^{\prime}=\frac{\sigma_{i}}{\sigma_{i}^{2}+\lambda}." display="inline"><semantics id="A6.E32.m1.1a"><mrow id="A6.E32.m1.1.1.1" xref="A6.E32.m1.1.1.1.1.cmml"><mrow id="A6.E32.m1.1.1.1.1" xref="A6.E32.m1.1.1.1.1.cmml"><msubsup id="A6.E32.m1.1.1.1.1.2" xref="A6.E32.m1.1.1.1.1.2.cmml"><mi id="A6.E32.m1.1.1.1.1.2.2.2" xref="A6.E32.m1.1.1.1.1.2.2.2.cmml">σ</mi><mi id="A6.E32.m1.1.1.1.1.2.2.3" xref="A6.E32.m1.1.1.1.1.2.2.3.cmml">i</mi><mo id="A6.E32.m1.1.1.1.1.2.3" xref="A6.E32.m1.1.1.1.1.2.3.cmml">′</mo></msubsup><mo id="A6.E32.m1.1.1.1.1.1" xref="A6.E32.m1.1.1.1.1.1.cmml">=</mo><mstyle displaystyle="true" id="A6.E32.m1.1.1.1.1.3" xref="A6.E32.m1.1.1.1.1.3.cmml"><mfrac id="A6.E32.m1.1.1.1.1.3a" xref="A6.E32.m1.1.1.1.1.3.cmml"><msub id="A6.E32.m1.1.1.1.1.3.2" xref="A6.E32.m1.1.1.1.1.3.2.cmml"><mi id="A6.E32.m1.1.1.1.1.3.2.2" xref="A6.E32.m1.1.1.1.1.3.2.2.cmml">σ</mi><mi id="A6.E32.m1.1.1.1.1.3.2.3" xref="A6.E32.m1.1.1.1.1.3.2.3.cmml">i</mi></msub><mrow id="A6.E32.m1.1.1.1.1.3.3" xref="A6.E32.m1.1.1.1.1.3.3.cmml"><msubsup id="A6.E32.m1.1.1.1.1.3.3.2" xref="A6.E32.m1.1.1.1.1.3.3.2.cmml"><mi id="A6.E32.m1.1.1.1.1.3.3.2.2.2" xref="A6.E32.m1.1.1.1.1.3.3.2.2.2.cmml">σ</mi><mi id="A6.E32.m1.1.1.1.1.3.3.2.2.3" xref="A6.E32.m1.1.1.1.1.3.3.2.2.3.cmml">i</mi><mn id="A6.E32.m1.1.1.1.1.3.3.2.3" xref="A6.E32.m1.1.1.1.1.3.3.2.3.cmml">2</mn></msubsup><mo id="A6.E32.m1.1.1.1.1.3.3.1" xref="A6.E32.m1.1.1.1.1.3.3.1.cmml">+</mo><mi id="A6.E32.m1.1.1.1.1.3.3.3" xref="A6.E32.m1.1.1.1.1.3.3.3.cmml">λ</mi></mrow></mfrac></mstyle></mrow><mo lspace="0em" id="A6.E32.m1.1.1.1.2" xref="A6.E32.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.E32.m1.1b"><apply id="A6.E32.m1.1.1.1.1.cmml" xref="A6.E32.m1.1.1.1"><eq id="A6.E32.m1.1.1.1.1.1.cmml" xref="A6.E32.m1.1.1.1.1.1"></eq><apply id="A6.E32.m1.1.1.1.1.2.cmml" xref="A6.E32.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="A6.E32.m1.1.1.1.1.2.1.cmml" xref="A6.E32.m1.1.1.1.1.2">superscript</csymbol><apply id="A6.E32.m1.1.1.1.1.2.2.cmml" xref="A6.E32.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="A6.E32.m1.1.1.1.1.2.2.1.cmml" xref="A6.E32.m1.1.1.1.1.2">subscript</csymbol><ci id="A6.E32.m1.1.1.1.1.2.2.2.cmml" xref="A6.E32.m1.1.1.1.1.2.2.2">𝜎</ci><ci id="A6.E32.m1.1.1.1.1.2.2.3.cmml" xref="A6.E32.m1.1.1.1.1.2.2.3">𝑖</ci></apply><ci id="A6.E32.m1.1.1.1.1.2.3.cmml" xref="A6.E32.m1.1.1.1.1.2.3">′</ci></apply><apply id="A6.E32.m1.1.1.1.1.3.cmml" xref="A6.E32.m1.1.1.1.1.3"><divide id="A6.E32.m1.1.1.1.1.3.1.cmml" xref="A6.E32.m1.1.1.1.1.3"></divide><apply id="A6.E32.m1.1.1.1.1.3.2.cmml" xref="A6.E32.m1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="A6.E32.m1.1.1.1.1.3.2.1.cmml" xref="A6.E32.m1.1.1.1.1.3.2">subscript</csymbol><ci id="A6.E32.m1.1.1.1.1.3.2.2.cmml" xref="A6.E32.m1.1.1.1.1.3.2.2">𝜎</ci><ci id="A6.E32.m1.1.1.1.1.3.2.3.cmml" xref="A6.E32.m1.1.1.1.1.3.2.3">𝑖</ci></apply><apply id="A6.E32.m1.1.1.1.1.3.3.cmml" xref="A6.E32.m1.1.1.1.1.3.3"><plus id="A6.E32.m1.1.1.1.1.3.3.1.cmml" xref="A6.E32.m1.1.1.1.1.3.3.1"></plus><apply id="A6.E32.m1.1.1.1.1.3.3.2.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="A6.E32.m1.1.1.1.1.3.3.2.1.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2">superscript</csymbol><apply id="A6.E32.m1.1.1.1.1.3.3.2.2.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="A6.E32.m1.1.1.1.1.3.3.2.2.1.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2">subscript</csymbol><ci id="A6.E32.m1.1.1.1.1.3.3.2.2.2.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2.2.2">𝜎</ci><ci id="A6.E32.m1.1.1.1.1.3.3.2.2.3.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2.2.3">𝑖</ci></apply><cn type="integer" id="A6.E32.m1.1.1.1.1.3.3.2.3.cmml" xref="A6.E32.m1.1.1.1.1.3.3.2.3">2</cn></apply><ci id="A6.E32.m1.1.1.1.1.3.3.3.cmml" xref="A6.E32.m1.1.1.1.1.3.3.3">𝜆</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E32.m1.1c">\displaystyle\sigma_{i}^{\prime}=\frac{\sigma_{i}}{\sigma_{i}^{2}+\lambda}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(32)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.5.p1.13" class="ltx_p">This expression attains its maximal value of <math id="A6.SS2.5.p1.12.m1.1" class="ltx_Math" alttext="\frac{1}{2\sqrt{\lambda}}" display="inline"><semantics id="A6.SS2.5.p1.12.m1.1a"><mfrac id="A6.SS2.5.p1.12.m1.1.1" xref="A6.SS2.5.p1.12.m1.1.1.cmml"><mn id="A6.SS2.5.p1.12.m1.1.1.2" xref="A6.SS2.5.p1.12.m1.1.1.2.cmml">1</mn><mrow id="A6.SS2.5.p1.12.m1.1.1.3" xref="A6.SS2.5.p1.12.m1.1.1.3.cmml"><mn id="A6.SS2.5.p1.12.m1.1.1.3.2" xref="A6.SS2.5.p1.12.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.SS2.5.p1.12.m1.1.1.3.1" xref="A6.SS2.5.p1.12.m1.1.1.3.1.cmml">​</mo><msqrt id="A6.SS2.5.p1.12.m1.1.1.3.3" xref="A6.SS2.5.p1.12.m1.1.1.3.3.cmml"><mi id="A6.SS2.5.p1.12.m1.1.1.3.3.2" xref="A6.SS2.5.p1.12.m1.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.12.m1.1b"><apply id="A6.SS2.5.p1.12.m1.1.1.cmml" xref="A6.SS2.5.p1.12.m1.1.1"><divide id="A6.SS2.5.p1.12.m1.1.1.1.cmml" xref="A6.SS2.5.p1.12.m1.1.1"></divide><cn type="integer" id="A6.SS2.5.p1.12.m1.1.1.2.cmml" xref="A6.SS2.5.p1.12.m1.1.1.2">1</cn><apply id="A6.SS2.5.p1.12.m1.1.1.3.cmml" xref="A6.SS2.5.p1.12.m1.1.1.3"><times id="A6.SS2.5.p1.12.m1.1.1.3.1.cmml" xref="A6.SS2.5.p1.12.m1.1.1.3.1"></times><cn type="integer" id="A6.SS2.5.p1.12.m1.1.1.3.2.cmml" xref="A6.SS2.5.p1.12.m1.1.1.3.2">2</cn><apply id="A6.SS2.5.p1.12.m1.1.1.3.3.cmml" xref="A6.SS2.5.p1.12.m1.1.1.3.3"><root id="A6.SS2.5.p1.12.m1.1.1.3.3a.cmml" xref="A6.SS2.5.p1.12.m1.1.1.3.3"></root><ci id="A6.SS2.5.p1.12.m1.1.1.3.3.2.cmml" xref="A6.SS2.5.p1.12.m1.1.1.3.3.2">𝜆</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.12.m1.1c">\frac{1}{2\sqrt{\lambda}}</annotation></semantics></math> when <math id="A6.SS2.5.p1.13.m2.1" class="ltx_Math" alttext="\sigma_{i}^{2}=\lambda" display="inline"><semantics id="A6.SS2.5.p1.13.m2.1a"><mrow id="A6.SS2.5.p1.13.m2.1.1" xref="A6.SS2.5.p1.13.m2.1.1.cmml"><msubsup id="A6.SS2.5.p1.13.m2.1.1.2" xref="A6.SS2.5.p1.13.m2.1.1.2.cmml"><mi id="A6.SS2.5.p1.13.m2.1.1.2.2.2" xref="A6.SS2.5.p1.13.m2.1.1.2.2.2.cmml">σ</mi><mi id="A6.SS2.5.p1.13.m2.1.1.2.2.3" xref="A6.SS2.5.p1.13.m2.1.1.2.2.3.cmml">i</mi><mn id="A6.SS2.5.p1.13.m2.1.1.2.3" xref="A6.SS2.5.p1.13.m2.1.1.2.3.cmml">2</mn></msubsup><mo id="A6.SS2.5.p1.13.m2.1.1.1" xref="A6.SS2.5.p1.13.m2.1.1.1.cmml">=</mo><mi id="A6.SS2.5.p1.13.m2.1.1.3" xref="A6.SS2.5.p1.13.m2.1.1.3.cmml">λ</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.5.p1.13.m2.1b"><apply id="A6.SS2.5.p1.13.m2.1.1.cmml" xref="A6.SS2.5.p1.13.m2.1.1"><eq id="A6.SS2.5.p1.13.m2.1.1.1.cmml" xref="A6.SS2.5.p1.13.m2.1.1.1"></eq><apply id="A6.SS2.5.p1.13.m2.1.1.2.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.5.p1.13.m2.1.1.2.1.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2">superscript</csymbol><apply id="A6.SS2.5.p1.13.m2.1.1.2.2.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.5.p1.13.m2.1.1.2.2.1.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2">subscript</csymbol><ci id="A6.SS2.5.p1.13.m2.1.1.2.2.2.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2.2.2">𝜎</ci><ci id="A6.SS2.5.p1.13.m2.1.1.2.2.3.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2.2.3">𝑖</ci></apply><cn type="integer" id="A6.SS2.5.p1.13.m2.1.1.2.3.cmml" xref="A6.SS2.5.p1.13.m2.1.1.2.3">2</cn></apply><ci id="A6.SS2.5.p1.13.m2.1.1.3.cmml" xref="A6.SS2.5.p1.13.m2.1.1.3">𝜆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.5.p1.13.m2.1c">\sigma_{i}^{2}=\lambda</annotation></semantics></math>.
∎</p>
</div>
</div>
<div id="A6.Thmlemma3" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmlemma3.1.1.1" class="ltx_text ltx_font_bold">Lemma F.3</span></span><span id="A6.Thmlemma3.2.2" class="ltx_text ltx_font_bold"> </span>(Ridge Regression Singular Value Bound)<span id="A6.Thmlemma3.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmlemma3.p1" class="ltx_para">
<p id="A6.Thmlemma3.p1.6" class="ltx_p"><span id="A6.Thmlemma3.p1.6.6" class="ltx_text ltx_font_italic">Let <math id="A6.Thmlemma3.p1.1.1.m1.1" class="ltx_Math" alttext="\bm{X}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="A6.Thmlemma3.p1.1.1.m1.1a"><mrow id="A6.Thmlemma3.p1.1.1.m1.1.1" xref="A6.Thmlemma3.p1.1.1.m1.1.1.cmml"><mi id="A6.Thmlemma3.p1.1.1.m1.1.1.2" xref="A6.Thmlemma3.p1.1.1.m1.1.1.2.cmml">𝐗</mi><mo id="A6.Thmlemma3.p1.1.1.m1.1.1.1" xref="A6.Thmlemma3.p1.1.1.m1.1.1.1.cmml">∈</mo><msup id="A6.Thmlemma3.p1.1.1.m1.1.1.3" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.cmml"><mi id="A6.Thmlemma3.p1.1.1.m1.1.1.3.2" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.cmml"><mi id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.2" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.1" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.3" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma3.p1.1.1.m1.1b"><apply id="A6.Thmlemma3.p1.1.1.m1.1.1.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1"><in id="A6.Thmlemma3.p1.1.1.m1.1.1.1.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.1"></in><ci id="A6.Thmlemma3.p1.1.1.m1.1.1.2.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.2">𝐗</ci><apply id="A6.Thmlemma3.p1.1.1.m1.1.1.3.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.1.1.m1.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3">superscript</csymbol><ci id="A6.Thmlemma3.p1.1.1.m1.1.1.3.2.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.2">ℝ</ci><apply id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3"><times id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.1.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.1"></times><ci id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.2.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.2">𝑁</ci><ci id="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.3.cmml" xref="A6.Thmlemma3.p1.1.1.m1.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma3.p1.1.1.m1.1c">\bm{X}\in\mathbb{R}^{N\times D}</annotation></semantics></math> and <math id="A6.Thmlemma3.p1.2.2.m2.1" class="ltx_Math" alttext="\bm{Y}\in\mathbb{R}^{D\times M}" display="inline"><semantics id="A6.Thmlemma3.p1.2.2.m2.1a"><mrow id="A6.Thmlemma3.p1.2.2.m2.1.1" xref="A6.Thmlemma3.p1.2.2.m2.1.1.cmml"><mi id="A6.Thmlemma3.p1.2.2.m2.1.1.2" xref="A6.Thmlemma3.p1.2.2.m2.1.1.2.cmml">𝐘</mi><mo id="A6.Thmlemma3.p1.2.2.m2.1.1.1" xref="A6.Thmlemma3.p1.2.2.m2.1.1.1.cmml">∈</mo><msup id="A6.Thmlemma3.p1.2.2.m2.1.1.3" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.cmml"><mi id="A6.Thmlemma3.p1.2.2.m2.1.1.3.2" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.cmml"><mi id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.2" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.1" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.1.cmml">×</mo><mi id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.3" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.3.cmml">M</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma3.p1.2.2.m2.1b"><apply id="A6.Thmlemma3.p1.2.2.m2.1.1.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1"><in id="A6.Thmlemma3.p1.2.2.m2.1.1.1.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.1"></in><ci id="A6.Thmlemma3.p1.2.2.m2.1.1.2.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.2">𝐘</ci><apply id="A6.Thmlemma3.p1.2.2.m2.1.1.3.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.2.2.m2.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3">superscript</csymbol><ci id="A6.Thmlemma3.p1.2.2.m2.1.1.3.2.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.2">ℝ</ci><apply id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3"><times id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.1.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.1"></times><ci id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.2.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.2">𝐷</ci><ci id="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.3.cmml" xref="A6.Thmlemma3.p1.2.2.m2.1.1.3.3.3">𝑀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma3.p1.2.2.m2.1c">\bm{Y}\in\mathbb{R}^{D\times M}</annotation></semantics></math> be arbitrary matrices and let <math id="A6.Thmlemma3.p1.3.3.m3.1" class="ltx_Math" alttext="\bm{W}\triangleq(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}\bm{Y}" display="inline"><semantics id="A6.Thmlemma3.p1.3.3.m3.1a"><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.cmml"><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.3.cmml">𝐖</mi><mo id="A6.Thmlemma3.p1.3.3.m3.1.1.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.2.cmml">≜</mo><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.cmml"><msup id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.cmml"><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.cmml"><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.cmml"><msup id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.cmml"><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2.cmml">𝐗</mi><mo id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.3.cmml">𝐗</mi></mrow><mo id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.cmml"><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.1" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.3.cmml">𝐈</mi></mrow></mrow><mo stretchy="false" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.cmml"><mo id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3a" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.cmml">−</mo><mn id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.2.cmml">​</mo><msup id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.cmml"><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.2" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.2.cmml">𝐗</mi><mo id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.3" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.2a" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.2.cmml">​</mo><mi id="A6.Thmlemma3.p1.3.3.m3.1.1.1.4" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.4.cmml">𝐘</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma3.p1.3.3.m3.1b"><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1"><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.2">≜</ci><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.3">𝐖</ci><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1"><times id="A6.Thmlemma3.p1.3.3.m3.1.1.1.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.2"></times><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1">superscript</csymbol><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1"><plus id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.1"></plus><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2"><times id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.1"></times><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.2">𝐗</ci><csymbol cd="latexml" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.2.3">top</csymbol></apply><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.2.3">𝐗</ci></apply><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3"><times id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.1"></times><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.1.1.1.3.3">𝐈</ci></apply></apply><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3"><minus id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3"></minus><cn type="integer" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.1.3.2">1</cn></apply></apply><apply id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3">superscript</csymbol><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.2.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.2">𝐗</ci><csymbol cd="latexml" id="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.3.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.3.3">top</csymbol></apply><ci id="A6.Thmlemma3.p1.3.3.m3.1.1.1.4.cmml" xref="A6.Thmlemma3.p1.3.3.m3.1.1.1.4">𝐘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma3.p1.3.3.m3.1c">\bm{W}\triangleq(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}\bm{Y}</annotation></semantics></math>, <math id="A6.Thmlemma3.p1.4.4.m4.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="A6.Thmlemma3.p1.4.4.m4.1a"><mrow id="A6.Thmlemma3.p1.4.4.m4.1.1" xref="A6.Thmlemma3.p1.4.4.m4.1.1.cmml"><mi id="A6.Thmlemma3.p1.4.4.m4.1.1.2" xref="A6.Thmlemma3.p1.4.4.m4.1.1.2.cmml">λ</mi><mo id="A6.Thmlemma3.p1.4.4.m4.1.1.1" xref="A6.Thmlemma3.p1.4.4.m4.1.1.1.cmml">&gt;</mo><mn id="A6.Thmlemma3.p1.4.4.m4.1.1.3" xref="A6.Thmlemma3.p1.4.4.m4.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma3.p1.4.4.m4.1b"><apply id="A6.Thmlemma3.p1.4.4.m4.1.1.cmml" xref="A6.Thmlemma3.p1.4.4.m4.1.1"><gt id="A6.Thmlemma3.p1.4.4.m4.1.1.1.cmml" xref="A6.Thmlemma3.p1.4.4.m4.1.1.1"></gt><ci id="A6.Thmlemma3.p1.4.4.m4.1.1.2.cmml" xref="A6.Thmlemma3.p1.4.4.m4.1.1.2">𝜆</ci><cn type="integer" id="A6.Thmlemma3.p1.4.4.m4.1.1.3.cmml" xref="A6.Thmlemma3.p1.4.4.m4.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma3.p1.4.4.m4.1c">\lambda&gt;0</annotation></semantics></math> be the ridge regression weight matrix. Then <math id="A6.Thmlemma3.p1.5.5.m5.2" class="ltx_Math" alttext="\lVert\bm{W}\rVert_{\infty}\leq\frac{\lVert\bm{Y}\rVert_{\infty}}{2\sqrt{\lambda}}" display="inline"><semantics id="A6.Thmlemma3.p1.5.5.m5.2a"><mrow id="A6.Thmlemma3.p1.5.5.m5.2.3" xref="A6.Thmlemma3.p1.5.5.m5.2.3.cmml"><msub id="A6.Thmlemma3.p1.5.5.m5.2.3.2" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.cmml"><mrow id="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.2" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.1.cmml"><mo fence="true" rspace="0em" id="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.2.1" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.1.1.cmml">∥</mo><mi id="A6.Thmlemma3.p1.5.5.m5.2.2" xref="A6.Thmlemma3.p1.5.5.m5.2.2.cmml">𝐖</mi><mo fence="true" lspace="0em" id="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.2.2" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.Thmlemma3.p1.5.5.m5.2.3.2.3" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.3.cmml">∞</mi></msub><mo id="A6.Thmlemma3.p1.5.5.m5.2.3.1" xref="A6.Thmlemma3.p1.5.5.m5.2.3.1.cmml">≤</mo><mfrac id="A6.Thmlemma3.p1.5.5.m5.1.1" xref="A6.Thmlemma3.p1.5.5.m5.1.1.cmml"><msub id="A6.Thmlemma3.p1.5.5.m5.1.1.1" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.cmml"><mrow id="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.2" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.1.cmml"><mo fence="true" rspace="0em" id="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.2.1" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.1.1.cmml">∥</mo><mi id="A6.Thmlemma3.p1.5.5.m5.1.1.1.1" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.1.cmml">𝐘</mi><mo fence="true" lspace="0em" id="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.2.2" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.Thmlemma3.p1.5.5.m5.1.1.1.4" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.4.cmml">∞</mi></msub><mrow id="A6.Thmlemma3.p1.5.5.m5.1.1.3" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.cmml"><mn id="A6.Thmlemma3.p1.5.5.m5.1.1.3.2" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.Thmlemma3.p1.5.5.m5.1.1.3.1" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.1.cmml">​</mo><msqrt id="A6.Thmlemma3.p1.5.5.m5.1.1.3.3" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.3.cmml"><mi id="A6.Thmlemma3.p1.5.5.m5.1.1.3.3.2" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma3.p1.5.5.m5.2b"><apply id="A6.Thmlemma3.p1.5.5.m5.2.3.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3"><leq id="A6.Thmlemma3.p1.5.5.m5.2.3.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3.1"></leq><apply id="A6.Thmlemma3.p1.5.5.m5.2.3.2.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.5.5.m5.2.3.2.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2">subscript</csymbol><apply id="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.2"><csymbol cd="latexml" id="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.1.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.Thmlemma3.p1.5.5.m5.2.2.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.2">𝐖</ci></apply><infinity id="A6.Thmlemma3.p1.5.5.m5.2.3.2.3.cmml" xref="A6.Thmlemma3.p1.5.5.m5.2.3.2.3"></infinity></apply><apply id="A6.Thmlemma3.p1.5.5.m5.1.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1"><divide id="A6.Thmlemma3.p1.5.5.m5.1.1.2.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1"></divide><apply id="A6.Thmlemma3.p1.5.5.m5.1.1.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.5.5.m5.1.1.1.2.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1">subscript</csymbol><apply id="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.2"><csymbol cd="latexml" id="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.1.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.3.2.1">delimited-∥∥</csymbol><ci id="A6.Thmlemma3.p1.5.5.m5.1.1.1.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.1">𝐘</ci></apply><infinity id="A6.Thmlemma3.p1.5.5.m5.1.1.1.4.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.1.4"></infinity></apply><apply id="A6.Thmlemma3.p1.5.5.m5.1.1.3.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3"><times id="A6.Thmlemma3.p1.5.5.m5.1.1.3.1.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.1"></times><cn type="integer" id="A6.Thmlemma3.p1.5.5.m5.1.1.3.2.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.2">2</cn><apply id="A6.Thmlemma3.p1.5.5.m5.1.1.3.3.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.3"><root id="A6.Thmlemma3.p1.5.5.m5.1.1.3.3a.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.3"></root><ci id="A6.Thmlemma3.p1.5.5.m5.1.1.3.3.2.cmml" xref="A6.Thmlemma3.p1.5.5.m5.1.1.3.3.2">𝜆</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma3.p1.5.5.m5.2c">\lVert\bm{W}\rVert_{\infty}\leq\frac{\lVert\bm{Y}\rVert_{\infty}}{2\sqrt{\lambda}}</annotation></semantics></math>, where <math id="A6.Thmlemma3.p1.6.6.m6.1" class="ltx_Math" alttext="\lVert\cdot\rVert_{\infty}" display="inline"><semantics id="A6.Thmlemma3.p1.6.6.m6.1a"><msub id="A6.Thmlemma3.p1.6.6.m6.1.2" xref="A6.Thmlemma3.p1.6.6.m6.1.2.cmml"><mrow id="A6.Thmlemma3.p1.6.6.m6.1.2.2.2" xref="A6.Thmlemma3.p1.6.6.m6.1.2.2.1.cmml"><mo fence="true" rspace="0em" id="A6.Thmlemma3.p1.6.6.m6.1.2.2.2.1" xref="A6.Thmlemma3.p1.6.6.m6.1.2.2.1.1.cmml">∥</mo><mo lspace="0em" rspace="0em" id="A6.Thmlemma3.p1.6.6.m6.1.1" xref="A6.Thmlemma3.p1.6.6.m6.1.1.cmml">⋅</mo><mo fence="true" lspace="0em" id="A6.Thmlemma3.p1.6.6.m6.1.2.2.2.2" xref="A6.Thmlemma3.p1.6.6.m6.1.2.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.Thmlemma3.p1.6.6.m6.1.2.3" xref="A6.Thmlemma3.p1.6.6.m6.1.2.3.cmml">∞</mi></msub><annotation-xml encoding="MathML-Content" id="A6.Thmlemma3.p1.6.6.m6.1b"><apply id="A6.Thmlemma3.p1.6.6.m6.1.2.cmml" xref="A6.Thmlemma3.p1.6.6.m6.1.2"><csymbol cd="ambiguous" id="A6.Thmlemma3.p1.6.6.m6.1.2.1.cmml" xref="A6.Thmlemma3.p1.6.6.m6.1.2">subscript</csymbol><apply id="A6.Thmlemma3.p1.6.6.m6.1.2.2.1.cmml" xref="A6.Thmlemma3.p1.6.6.m6.1.2.2.2"><csymbol cd="latexml" id="A6.Thmlemma3.p1.6.6.m6.1.2.2.1.1.cmml" xref="A6.Thmlemma3.p1.6.6.m6.1.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.Thmlemma3.p1.6.6.m6.1.1.cmml" xref="A6.Thmlemma3.p1.6.6.m6.1.1">⋅</ci></apply><infinity id="A6.Thmlemma3.p1.6.6.m6.1.2.3.cmml" xref="A6.Thmlemma3.p1.6.6.m6.1.2.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma3.p1.6.6.m6.1c">\lVert\cdot\rVert_{\infty}</annotation></semantics></math> denotes the largest singular value.</span></p>
</div>
</div>
<div id="A6.SS2.6" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A6.SS2.6.p1" class="ltx_para">
<p id="A6.SS2.6.p1.2" class="ltx_p">Observe that <math id="A6.SS2.6.p1.1.m1.1" class="ltx_Math" alttext="\bm{W}=\bm{Z}\bm{Y}" display="inline"><semantics id="A6.SS2.6.p1.1.m1.1a"><mrow id="A6.SS2.6.p1.1.m1.1.1" xref="A6.SS2.6.p1.1.m1.1.1.cmml"><mi id="A6.SS2.6.p1.1.m1.1.1.2" xref="A6.SS2.6.p1.1.m1.1.1.2.cmml">𝑾</mi><mo id="A6.SS2.6.p1.1.m1.1.1.1" xref="A6.SS2.6.p1.1.m1.1.1.1.cmml">=</mo><mrow id="A6.SS2.6.p1.1.m1.1.1.3" xref="A6.SS2.6.p1.1.m1.1.1.3.cmml"><mi id="A6.SS2.6.p1.1.m1.1.1.3.2" xref="A6.SS2.6.p1.1.m1.1.1.3.2.cmml">𝒁</mi><mo lspace="0em" rspace="0em" id="A6.SS2.6.p1.1.m1.1.1.3.1" xref="A6.SS2.6.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="A6.SS2.6.p1.1.m1.1.1.3.3" xref="A6.SS2.6.p1.1.m1.1.1.3.3.cmml">𝒀</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.6.p1.1.m1.1b"><apply id="A6.SS2.6.p1.1.m1.1.1.cmml" xref="A6.SS2.6.p1.1.m1.1.1"><eq id="A6.SS2.6.p1.1.m1.1.1.1.cmml" xref="A6.SS2.6.p1.1.m1.1.1.1"></eq><ci id="A6.SS2.6.p1.1.m1.1.1.2.cmml" xref="A6.SS2.6.p1.1.m1.1.1.2">𝑾</ci><apply id="A6.SS2.6.p1.1.m1.1.1.3.cmml" xref="A6.SS2.6.p1.1.m1.1.1.3"><times id="A6.SS2.6.p1.1.m1.1.1.3.1.cmml" xref="A6.SS2.6.p1.1.m1.1.1.3.1"></times><ci id="A6.SS2.6.p1.1.m1.1.1.3.2.cmml" xref="A6.SS2.6.p1.1.m1.1.1.3.2">𝒁</ci><ci id="A6.SS2.6.p1.1.m1.1.1.3.3.cmml" xref="A6.SS2.6.p1.1.m1.1.1.3.3">𝒀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.6.p1.1.m1.1c">\bm{W}=\bm{Z}\bm{Y}</annotation></semantics></math>, where <math id="A6.SS2.6.p1.2.m2.1" class="ltx_Math" alttext="\bm{Z}\triangleq(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}" display="inline"><semantics id="A6.SS2.6.p1.2.m2.1a"><mrow id="A6.SS2.6.p1.2.m2.1.1" xref="A6.SS2.6.p1.2.m2.1.1.cmml"><mi id="A6.SS2.6.p1.2.m2.1.1.3" xref="A6.SS2.6.p1.2.m2.1.1.3.cmml">𝒁</mi><mo id="A6.SS2.6.p1.2.m2.1.1.2" xref="A6.SS2.6.p1.2.m2.1.1.2.cmml">≜</mo><mrow id="A6.SS2.6.p1.2.m2.1.1.1" xref="A6.SS2.6.p1.2.m2.1.1.1.cmml"><msup id="A6.SS2.6.p1.2.m2.1.1.1.1" xref="A6.SS2.6.p1.2.m2.1.1.1.1.cmml"><mrow id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.2" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.cmml"><mrow id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.cmml"><msup id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.cmml"><mi id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.2" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.2.cmml">𝑿</mi><mo id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.3" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.1" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.3" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.3.cmml">𝑿</mi></mrow><mo id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.1" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.2" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.1" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.3" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.3.cmml">𝑰</mi></mrow></mrow><mo stretchy="false" id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.3" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="A6.SS2.6.p1.2.m2.1.1.1.1.3" xref="A6.SS2.6.p1.2.m2.1.1.1.1.3.cmml"><mo id="A6.SS2.6.p1.2.m2.1.1.1.1.3a" xref="A6.SS2.6.p1.2.m2.1.1.1.1.3.cmml">−</mo><mn id="A6.SS2.6.p1.2.m2.1.1.1.1.3.2" xref="A6.SS2.6.p1.2.m2.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A6.SS2.6.p1.2.m2.1.1.1.2" xref="A6.SS2.6.p1.2.m2.1.1.1.2.cmml">​</mo><msup id="A6.SS2.6.p1.2.m2.1.1.1.3" xref="A6.SS2.6.p1.2.m2.1.1.1.3.cmml"><mi id="A6.SS2.6.p1.2.m2.1.1.1.3.2" xref="A6.SS2.6.p1.2.m2.1.1.1.3.2.cmml">𝑿</mi><mo id="A6.SS2.6.p1.2.m2.1.1.1.3.3" xref="A6.SS2.6.p1.2.m2.1.1.1.3.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.6.p1.2.m2.1b"><apply id="A6.SS2.6.p1.2.m2.1.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1"><ci id="A6.SS2.6.p1.2.m2.1.1.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.2">≜</ci><ci id="A6.SS2.6.p1.2.m2.1.1.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.3">𝒁</ci><apply id="A6.SS2.6.p1.2.m2.1.1.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1"><times id="A6.SS2.6.p1.2.m2.1.1.1.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.2"></times><apply id="A6.SS2.6.p1.2.m2.1.1.1.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="A6.SS2.6.p1.2.m2.1.1.1.1.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1">superscript</csymbol><apply id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1"><plus id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.1"></plus><apply id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2"><times id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.1"></times><apply id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.2">𝑿</ci><csymbol cd="latexml" id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.2.3">top</csymbol></apply><ci id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.2.3">𝑿</ci></apply><apply id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3"><times id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.1"></times><ci id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.2">𝜆</ci><ci id="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.1.1.1.3.3">𝑰</ci></apply></apply><apply id="A6.SS2.6.p1.2.m2.1.1.1.1.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.3"><minus id="A6.SS2.6.p1.2.m2.1.1.1.1.3.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.3"></minus><cn type="integer" id="A6.SS2.6.p1.2.m2.1.1.1.1.3.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.1.3.2">1</cn></apply></apply><apply id="A6.SS2.6.p1.2.m2.1.1.1.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="A6.SS2.6.p1.2.m2.1.1.1.3.1.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.3">superscript</csymbol><ci id="A6.SS2.6.p1.2.m2.1.1.1.3.2.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.3.2">𝑿</ci><csymbol cd="latexml" id="A6.SS2.6.p1.2.m2.1.1.1.3.3.cmml" xref="A6.SS2.6.p1.2.m2.1.1.1.3.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.6.p1.2.m2.1c">\bm{Z}\triangleq(\bm{X}^{\top}\bm{X}+\lambda\bm{I})^{-1}\bm{X}^{\top}</annotation></semantics></math>. Then by applying Lemma <a href="#A6.Thmlemma2" title="Lemma F.2 (Regularized Pseudoinverse Operator Norm Bound). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.2</span></a> and recalling that Schatten norms are submultiplicative, we have</p>
<table id="A6.EGx22" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E33"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E33.m1.5" class="ltx_Math" alttext="\displaystyle\lVert\bm{W}\rVert_{\infty}\leq\lVert\bm{Z}\rVert_{\infty}\lVert\bm{Y}\rVert_{\infty}\leq\frac{\lVert\bm{Y}\rVert_{\infty}}{2\sqrt{\lambda}}." display="inline"><semantics id="A6.E33.m1.5a"><mrow id="A6.E33.m1.5.5.1" xref="A6.E33.m1.5.5.1.1.cmml"><mrow id="A6.E33.m1.5.5.1.1" xref="A6.E33.m1.5.5.1.1.cmml"><msub id="A6.E33.m1.5.5.1.1.2" xref="A6.E33.m1.5.5.1.1.2.cmml"><mrow id="A6.E33.m1.5.5.1.1.2.2.2" xref="A6.E33.m1.5.5.1.1.2.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E33.m1.5.5.1.1.2.2.2.1" xref="A6.E33.m1.5.5.1.1.2.2.1.1.cmml">∥</mo><mi id="A6.E33.m1.2.2" xref="A6.E33.m1.2.2.cmml">𝑾</mi><mo fence="true" lspace="0em" id="A6.E33.m1.5.5.1.1.2.2.2.2" xref="A6.E33.m1.5.5.1.1.2.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E33.m1.5.5.1.1.2.3" xref="A6.E33.m1.5.5.1.1.2.3.cmml">∞</mi></msub><mo rspace="0.1389em" id="A6.E33.m1.5.5.1.1.3" xref="A6.E33.m1.5.5.1.1.3.cmml">≤</mo><mrow id="A6.E33.m1.5.5.1.1.4" xref="A6.E33.m1.5.5.1.1.4.cmml"><msub id="A6.E33.m1.5.5.1.1.4.2" xref="A6.E33.m1.5.5.1.1.4.2.cmml"><mrow id="A6.E33.m1.5.5.1.1.4.2.2.2" xref="A6.E33.m1.5.5.1.1.4.2.2.1.cmml"><mo fence="true" lspace="0.1389em" rspace="0em" id="A6.E33.m1.5.5.1.1.4.2.2.2.1" xref="A6.E33.m1.5.5.1.1.4.2.2.1.1.cmml">∥</mo><mi id="A6.E33.m1.3.3" xref="A6.E33.m1.3.3.cmml">𝒁</mi><mo fence="true" lspace="0em" id="A6.E33.m1.5.5.1.1.4.2.2.2.2" xref="A6.E33.m1.5.5.1.1.4.2.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E33.m1.5.5.1.1.4.2.3" xref="A6.E33.m1.5.5.1.1.4.2.3.cmml">∞</mi></msub><mo lspace="0em" rspace="0em" id="A6.E33.m1.5.5.1.1.4.1" xref="A6.E33.m1.5.5.1.1.4.1.cmml">​</mo><msub id="A6.E33.m1.5.5.1.1.4.3" xref="A6.E33.m1.5.5.1.1.4.3.cmml"><mrow id="A6.E33.m1.5.5.1.1.4.3.2.2" xref="A6.E33.m1.5.5.1.1.4.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E33.m1.5.5.1.1.4.3.2.2.1" xref="A6.E33.m1.5.5.1.1.4.3.2.1.1.cmml">∥</mo><mi id="A6.E33.m1.4.4" xref="A6.E33.m1.4.4.cmml">𝒀</mi><mo fence="true" lspace="0em" rspace="0.1389em" id="A6.E33.m1.5.5.1.1.4.3.2.2.2" xref="A6.E33.m1.5.5.1.1.4.3.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E33.m1.5.5.1.1.4.3.3" xref="A6.E33.m1.5.5.1.1.4.3.3.cmml">∞</mi></msub></mrow><mo lspace="0.1389em" id="A6.E33.m1.5.5.1.1.5" xref="A6.E33.m1.5.5.1.1.5.cmml">≤</mo><mstyle displaystyle="true" id="A6.E33.m1.1.1" xref="A6.E33.m1.1.1.cmml"><mfrac id="A6.E33.m1.1.1a" xref="A6.E33.m1.1.1.cmml"><msub id="A6.E33.m1.1.1.1" xref="A6.E33.m1.1.1.1.cmml"><mrow id="A6.E33.m1.1.1.1.3.2" xref="A6.E33.m1.1.1.1.3.1.cmml"><mo fence="true" rspace="0em" id="A6.E33.m1.1.1.1.3.2.1" xref="A6.E33.m1.1.1.1.3.1.1.cmml">∥</mo><mi id="A6.E33.m1.1.1.1.1" xref="A6.E33.m1.1.1.1.1.cmml">𝒀</mi><mo fence="true" lspace="0em" id="A6.E33.m1.1.1.1.3.2.2" xref="A6.E33.m1.1.1.1.3.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E33.m1.1.1.1.4" xref="A6.E33.m1.1.1.1.4.cmml">∞</mi></msub><mrow id="A6.E33.m1.1.1.3" xref="A6.E33.m1.1.1.3.cmml"><mn id="A6.E33.m1.1.1.3.2" xref="A6.E33.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E33.m1.1.1.3.1" xref="A6.E33.m1.1.1.3.1.cmml">​</mo><msqrt id="A6.E33.m1.1.1.3.3" xref="A6.E33.m1.1.1.3.3.cmml"><mi id="A6.E33.m1.1.1.3.3.2" xref="A6.E33.m1.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle></mrow><mo lspace="0em" id="A6.E33.m1.5.5.1.2" xref="A6.E33.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.E33.m1.5b"><apply id="A6.E33.m1.5.5.1.1.cmml" xref="A6.E33.m1.5.5.1"><and id="A6.E33.m1.5.5.1.1a.cmml" xref="A6.E33.m1.5.5.1"></and><apply id="A6.E33.m1.5.5.1.1b.cmml" xref="A6.E33.m1.5.5.1"><leq id="A6.E33.m1.5.5.1.1.3.cmml" xref="A6.E33.m1.5.5.1.1.3"></leq><apply id="A6.E33.m1.5.5.1.1.2.cmml" xref="A6.E33.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="A6.E33.m1.5.5.1.1.2.1.cmml" xref="A6.E33.m1.5.5.1.1.2">subscript</csymbol><apply id="A6.E33.m1.5.5.1.1.2.2.1.cmml" xref="A6.E33.m1.5.5.1.1.2.2.2"><csymbol cd="latexml" id="A6.E33.m1.5.5.1.1.2.2.1.1.cmml" xref="A6.E33.m1.5.5.1.1.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.E33.m1.2.2.cmml" xref="A6.E33.m1.2.2">𝑾</ci></apply><infinity id="A6.E33.m1.5.5.1.1.2.3.cmml" xref="A6.E33.m1.5.5.1.1.2.3"></infinity></apply><apply id="A6.E33.m1.5.5.1.1.4.cmml" xref="A6.E33.m1.5.5.1.1.4"><times id="A6.E33.m1.5.5.1.1.4.1.cmml" xref="A6.E33.m1.5.5.1.1.4.1"></times><apply id="A6.E33.m1.5.5.1.1.4.2.cmml" xref="A6.E33.m1.5.5.1.1.4.2"><csymbol cd="ambiguous" id="A6.E33.m1.5.5.1.1.4.2.1.cmml" xref="A6.E33.m1.5.5.1.1.4.2">subscript</csymbol><apply id="A6.E33.m1.5.5.1.1.4.2.2.1.cmml" xref="A6.E33.m1.5.5.1.1.4.2.2.2"><csymbol cd="latexml" id="A6.E33.m1.5.5.1.1.4.2.2.1.1.cmml" xref="A6.E33.m1.5.5.1.1.4.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.E33.m1.3.3.cmml" xref="A6.E33.m1.3.3">𝒁</ci></apply><infinity id="A6.E33.m1.5.5.1.1.4.2.3.cmml" xref="A6.E33.m1.5.5.1.1.4.2.3"></infinity></apply><apply id="A6.E33.m1.5.5.1.1.4.3.cmml" xref="A6.E33.m1.5.5.1.1.4.3"><csymbol cd="ambiguous" id="A6.E33.m1.5.5.1.1.4.3.1.cmml" xref="A6.E33.m1.5.5.1.1.4.3">subscript</csymbol><apply id="A6.E33.m1.5.5.1.1.4.3.2.1.cmml" xref="A6.E33.m1.5.5.1.1.4.3.2.2"><csymbol cd="latexml" id="A6.E33.m1.5.5.1.1.4.3.2.1.1.cmml" xref="A6.E33.m1.5.5.1.1.4.3.2.2.1">delimited-∥∥</csymbol><ci id="A6.E33.m1.4.4.cmml" xref="A6.E33.m1.4.4">𝒀</ci></apply><infinity id="A6.E33.m1.5.5.1.1.4.3.3.cmml" xref="A6.E33.m1.5.5.1.1.4.3.3"></infinity></apply></apply></apply><apply id="A6.E33.m1.5.5.1.1c.cmml" xref="A6.E33.m1.5.5.1"><leq id="A6.E33.m1.5.5.1.1.5.cmml" xref="A6.E33.m1.5.5.1.1.5"></leq><share href="#A6.E33.m1.5.5.1.1.4.cmml" id="A6.E33.m1.5.5.1.1d.cmml" xref="A6.E33.m1.5.5.1"></share><apply id="A6.E33.m1.1.1.cmml" xref="A6.E33.m1.1.1"><divide id="A6.E33.m1.1.1.2.cmml" xref="A6.E33.m1.1.1"></divide><apply id="A6.E33.m1.1.1.1.cmml" xref="A6.E33.m1.1.1.1"><csymbol cd="ambiguous" id="A6.E33.m1.1.1.1.2.cmml" xref="A6.E33.m1.1.1.1">subscript</csymbol><apply id="A6.E33.m1.1.1.1.3.1.cmml" xref="A6.E33.m1.1.1.1.3.2"><csymbol cd="latexml" id="A6.E33.m1.1.1.1.3.1.1.cmml" xref="A6.E33.m1.1.1.1.3.2.1">delimited-∥∥</csymbol><ci id="A6.E33.m1.1.1.1.1.cmml" xref="A6.E33.m1.1.1.1.1">𝒀</ci></apply><infinity id="A6.E33.m1.1.1.1.4.cmml" xref="A6.E33.m1.1.1.1.4"></infinity></apply><apply id="A6.E33.m1.1.1.3.cmml" xref="A6.E33.m1.1.1.3"><times id="A6.E33.m1.1.1.3.1.cmml" xref="A6.E33.m1.1.1.3.1"></times><cn type="integer" id="A6.E33.m1.1.1.3.2.cmml" xref="A6.E33.m1.1.1.3.2">2</cn><apply id="A6.E33.m1.1.1.3.3.cmml" xref="A6.E33.m1.1.1.3.3"><root id="A6.E33.m1.1.1.3.3a.cmml" xref="A6.E33.m1.1.1.3.3"></root><ci id="A6.E33.m1.1.1.3.3.2.cmml" xref="A6.E33.m1.1.1.3.3.2">𝜆</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E33.m1.5c">\displaystyle\lVert\bm{W}\rVert_{\infty}\leq\lVert\bm{Z}\rVert_{\infty}\lVert\bm{Y}\rVert_{\infty}\leq\frac{\lVert\bm{Y}\rVert_{\infty}}{2\sqrt{\lambda}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(33)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.6.p1.3" class="ltx_p">∎</p>
</div>
</div>
<div id="A6.Thmlemma4" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmlemma4.1.1.1" class="ltx_text ltx_font_bold">Lemma F.4</span></span><span id="A6.Thmlemma4.2.2" class="ltx_text ltx_font_bold"> </span>(Bound on <span id="A6.Thmlemma4.3.3" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.Thmlemma4.3.3.1" class="ltx_text ltx_font_upright"> </span></span>Embedding Norm)<span id="A6.Thmlemma4.4.4" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmlemma4.p1" class="ltx_para">
<p id="A6.Thmlemma4.p1.6" class="ltx_p"><span id="A6.Thmlemma4.p1.6.6" class="ltx_text ltx_font_italic">Let <math id="A6.Thmlemma4.p1.1.1.m1.1" class="ltx_Math" alttext="\bm{g}=g(\bm{a})" display="inline"><semantics id="A6.Thmlemma4.p1.1.1.m1.1a"><mrow id="A6.Thmlemma4.p1.1.1.m1.1.2" xref="A6.Thmlemma4.p1.1.1.m1.1.2.cmml"><mi id="A6.Thmlemma4.p1.1.1.m1.1.2.2" xref="A6.Thmlemma4.p1.1.1.m1.1.2.2.cmml">𝐠</mi><mo id="A6.Thmlemma4.p1.1.1.m1.1.2.1" xref="A6.Thmlemma4.p1.1.1.m1.1.2.1.cmml">=</mo><mrow id="A6.Thmlemma4.p1.1.1.m1.1.2.3" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.cmml"><mi id="A6.Thmlemma4.p1.1.1.m1.1.2.3.2" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma4.p1.1.1.m1.1.2.3.1" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.1.cmml">​</mo><mrow id="A6.Thmlemma4.p1.1.1.m1.1.2.3.3.2" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.cmml"><mo stretchy="false" id="A6.Thmlemma4.p1.1.1.m1.1.2.3.3.2.1" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.cmml">(</mo><mi id="A6.Thmlemma4.p1.1.1.m1.1.1" xref="A6.Thmlemma4.p1.1.1.m1.1.1.cmml">𝐚</mi><mo stretchy="false" id="A6.Thmlemma4.p1.1.1.m1.1.2.3.3.2.2" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.1.1.m1.1b"><apply id="A6.Thmlemma4.p1.1.1.m1.1.2.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.2"><eq id="A6.Thmlemma4.p1.1.1.m1.1.2.1.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.2.1"></eq><ci id="A6.Thmlemma4.p1.1.1.m1.1.2.2.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.2.2">𝐠</ci><apply id="A6.Thmlemma4.p1.1.1.m1.1.2.3.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3"><times id="A6.Thmlemma4.p1.1.1.m1.1.2.3.1.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.1"></times><ci id="A6.Thmlemma4.p1.1.1.m1.1.2.3.2.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.2.3.2">𝑔</ci><ci id="A6.Thmlemma4.p1.1.1.m1.1.1.cmml" xref="A6.Thmlemma4.p1.1.1.m1.1.1">𝐚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.1.1.m1.1c">\bm{g}=g(\bm{a})</annotation></semantics></math> be the encoding of an arbitrary vector <math id="A6.Thmlemma4.p1.2.2.m2.1" class="ltx_Math" alttext="\bm{a}" display="inline"><semantics id="A6.Thmlemma4.p1.2.2.m2.1a"><mi id="A6.Thmlemma4.p1.2.2.m2.1.1" xref="A6.Thmlemma4.p1.2.2.m2.1.1.cmml">𝐚</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.2.2.m2.1b"><ci id="A6.Thmlemma4.p1.2.2.m2.1.1.cmml" xref="A6.Thmlemma4.p1.2.2.m2.1.1">𝐚</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.2.2.m2.1c">\bm{a}</annotation></semantics></math> using <math id="A6.Thmlemma4.p1.3.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.Thmlemma4.p1.3.3.m3.1a"><mi id="A6.Thmlemma4.p1.3.3.m3.1.1" xref="A6.Thmlemma4.p1.3.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.3.3.m3.1b"><ci id="A6.Thmlemma4.p1.3.3.m3.1.1.cmml" xref="A6.Thmlemma4.p1.3.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.3.3.m3.1c">C</annotation></semantics></math> codebooks and let <math id="A6.Thmlemma4.p1.4.4.m4.1" class="ltx_Math" alttext="\bm{P}" display="inline"><semantics id="A6.Thmlemma4.p1.4.4.m4.1a"><mi id="A6.Thmlemma4.p1.4.4.m4.1.1" xref="A6.Thmlemma4.p1.4.4.m4.1.1.cmml">𝐏</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.4.4.m4.1b"><ci id="A6.Thmlemma4.p1.4.4.m4.1.1.cmml" xref="A6.Thmlemma4.p1.4.4.m4.1.1">𝐏</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.4.4.m4.1c">\bm{P}</annotation></semantics></math> be the prototype matrix learned by <span id="A6.Thmlemma4.p1.6.6.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.Thmlemma4.p1.6.6.1.1" class="ltx_text ltx_font_italic"> </span></span>using training matrix <math id="A6.Thmlemma4.p1.5.5.m5.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="A6.Thmlemma4.p1.5.5.m5.1a"><mover accent="true" id="A6.Thmlemma4.p1.5.5.m5.1.1" xref="A6.Thmlemma4.p1.5.5.m5.1.1.cmml"><mi id="A6.Thmlemma4.p1.5.5.m5.1.1.2" xref="A6.Thmlemma4.p1.5.5.m5.1.1.2.cmml">𝐀</mi><mo id="A6.Thmlemma4.p1.5.5.m5.1.1.1" xref="A6.Thmlemma4.p1.5.5.m5.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.5.5.m5.1b"><apply id="A6.Thmlemma4.p1.5.5.m5.1.1.cmml" xref="A6.Thmlemma4.p1.5.5.m5.1.1"><ci id="A6.Thmlemma4.p1.5.5.m5.1.1.1.cmml" xref="A6.Thmlemma4.p1.5.5.m5.1.1.1">~</ci><ci id="A6.Thmlemma4.p1.5.5.m5.1.1.2.cmml" xref="A6.Thmlemma4.p1.5.5.m5.1.1.2">𝐀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.5.5.m5.1c">\tilde{\bm{A}}</annotation></semantics></math> with ridge regression parameter <math id="A6.Thmlemma4.p1.6.6.m6.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="A6.Thmlemma4.p1.6.6.m6.1a"><mrow id="A6.Thmlemma4.p1.6.6.m6.1.1" xref="A6.Thmlemma4.p1.6.6.m6.1.1.cmml"><mi id="A6.Thmlemma4.p1.6.6.m6.1.1.2" xref="A6.Thmlemma4.p1.6.6.m6.1.1.2.cmml">λ</mi><mo id="A6.Thmlemma4.p1.6.6.m6.1.1.1" xref="A6.Thmlemma4.p1.6.6.m6.1.1.1.cmml">&gt;</mo><mn id="A6.Thmlemma4.p1.6.6.m6.1.1.3" xref="A6.Thmlemma4.p1.6.6.m6.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.6.6.m6.1b"><apply id="A6.Thmlemma4.p1.6.6.m6.1.1.cmml" xref="A6.Thmlemma4.p1.6.6.m6.1.1"><gt id="A6.Thmlemma4.p1.6.6.m6.1.1.1.cmml" xref="A6.Thmlemma4.p1.6.6.m6.1.1.1"></gt><ci id="A6.Thmlemma4.p1.6.6.m6.1.1.2.cmml" xref="A6.Thmlemma4.p1.6.6.m6.1.1.2">𝜆</ci><cn type="integer" id="A6.Thmlemma4.p1.6.6.m6.1.1.3.cmml" xref="A6.Thmlemma4.p1.6.6.m6.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.6.6.m6.1c">\lambda&gt;0</annotation></semantics></math>. Then</span></p>
<table id="A6.EGx23" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E34"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E34.m1.2" class="ltx_Math" alttext="\displaystyle\lVert\bm{g}^{\top}\bm{P}\rVert_{2}\leq\frac{C}{2\sqrt{\lambda}}\lVert\tilde{\bm{A}}\rVert_{\infty}" display="inline"><semantics id="A6.E34.m1.2a"><mrow id="A6.E34.m1.2.2" xref="A6.E34.m1.2.2.cmml"><msub id="A6.E34.m1.2.2.1" xref="A6.E34.m1.2.2.1.cmml"><mrow id="A6.E34.m1.2.2.1.1.1" xref="A6.E34.m1.2.2.1.1.2.cmml"><mo fence="true" rspace="0em" id="A6.E34.m1.2.2.1.1.1.2" xref="A6.E34.m1.2.2.1.1.2.1.cmml">∥</mo><mrow id="A6.E34.m1.2.2.1.1.1.1" xref="A6.E34.m1.2.2.1.1.1.1.cmml"><msup id="A6.E34.m1.2.2.1.1.1.1.2" xref="A6.E34.m1.2.2.1.1.1.1.2.cmml"><mi id="A6.E34.m1.2.2.1.1.1.1.2.2" xref="A6.E34.m1.2.2.1.1.1.1.2.2.cmml">𝒈</mi><mo id="A6.E34.m1.2.2.1.1.1.1.2.3" xref="A6.E34.m1.2.2.1.1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E34.m1.2.2.1.1.1.1.1" xref="A6.E34.m1.2.2.1.1.1.1.1.cmml">​</mo><mi id="A6.E34.m1.2.2.1.1.1.1.3" xref="A6.E34.m1.2.2.1.1.1.1.3.cmml">𝑷</mi></mrow><mo fence="true" lspace="0em" id="A6.E34.m1.2.2.1.1.1.3" xref="A6.E34.m1.2.2.1.1.2.1.cmml">∥</mo></mrow><mn id="A6.E34.m1.2.2.1.3" xref="A6.E34.m1.2.2.1.3.cmml">2</mn></msub><mo id="A6.E34.m1.2.2.2" xref="A6.E34.m1.2.2.2.cmml">≤</mo><mrow id="A6.E34.m1.2.2.3" xref="A6.E34.m1.2.2.3.cmml"><mstyle displaystyle="true" id="A6.E34.m1.2.2.3.2" xref="A6.E34.m1.2.2.3.2.cmml"><mfrac id="A6.E34.m1.2.2.3.2a" xref="A6.E34.m1.2.2.3.2.cmml"><mi id="A6.E34.m1.2.2.3.2.2" xref="A6.E34.m1.2.2.3.2.2.cmml">C</mi><mrow id="A6.E34.m1.2.2.3.2.3" xref="A6.E34.m1.2.2.3.2.3.cmml"><mn id="A6.E34.m1.2.2.3.2.3.2" xref="A6.E34.m1.2.2.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E34.m1.2.2.3.2.3.1" xref="A6.E34.m1.2.2.3.2.3.1.cmml">​</mo><msqrt id="A6.E34.m1.2.2.3.2.3.3" xref="A6.E34.m1.2.2.3.2.3.3.cmml"><mi id="A6.E34.m1.2.2.3.2.3.3.2" xref="A6.E34.m1.2.2.3.2.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A6.E34.m1.2.2.3.1" xref="A6.E34.m1.2.2.3.1.cmml">​</mo><msub id="A6.E34.m1.2.2.3.3" xref="A6.E34.m1.2.2.3.3.cmml"><mrow id="A6.E34.m1.2.2.3.3.2.2" xref="A6.E34.m1.2.2.3.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E34.m1.2.2.3.3.2.2.1" xref="A6.E34.m1.2.2.3.3.2.1.1.cmml">∥</mo><mover accent="true" id="A6.E34.m1.1.1" xref="A6.E34.m1.1.1.cmml"><mi id="A6.E34.m1.1.1.2" xref="A6.E34.m1.1.1.2.cmml">𝑨</mi><mo id="A6.E34.m1.1.1.1" xref="A6.E34.m1.1.1.1.cmml">~</mo></mover><mo fence="true" lspace="0em" id="A6.E34.m1.2.2.3.3.2.2.2" xref="A6.E34.m1.2.2.3.3.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E34.m1.2.2.3.3.3" xref="A6.E34.m1.2.2.3.3.3.cmml">∞</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E34.m1.2b"><apply id="A6.E34.m1.2.2.cmml" xref="A6.E34.m1.2.2"><leq id="A6.E34.m1.2.2.2.cmml" xref="A6.E34.m1.2.2.2"></leq><apply id="A6.E34.m1.2.2.1.cmml" xref="A6.E34.m1.2.2.1"><csymbol cd="ambiguous" id="A6.E34.m1.2.2.1.2.cmml" xref="A6.E34.m1.2.2.1">subscript</csymbol><apply id="A6.E34.m1.2.2.1.1.2.cmml" xref="A6.E34.m1.2.2.1.1.1"><csymbol cd="latexml" id="A6.E34.m1.2.2.1.1.2.1.cmml" xref="A6.E34.m1.2.2.1.1.1.2">delimited-∥∥</csymbol><apply id="A6.E34.m1.2.2.1.1.1.1.cmml" xref="A6.E34.m1.2.2.1.1.1.1"><times id="A6.E34.m1.2.2.1.1.1.1.1.cmml" xref="A6.E34.m1.2.2.1.1.1.1.1"></times><apply id="A6.E34.m1.2.2.1.1.1.1.2.cmml" xref="A6.E34.m1.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="A6.E34.m1.2.2.1.1.1.1.2.1.cmml" xref="A6.E34.m1.2.2.1.1.1.1.2">superscript</csymbol><ci id="A6.E34.m1.2.2.1.1.1.1.2.2.cmml" xref="A6.E34.m1.2.2.1.1.1.1.2.2">𝒈</ci><csymbol cd="latexml" id="A6.E34.m1.2.2.1.1.1.1.2.3.cmml" xref="A6.E34.m1.2.2.1.1.1.1.2.3">top</csymbol></apply><ci id="A6.E34.m1.2.2.1.1.1.1.3.cmml" xref="A6.E34.m1.2.2.1.1.1.1.3">𝑷</ci></apply></apply><cn type="integer" id="A6.E34.m1.2.2.1.3.cmml" xref="A6.E34.m1.2.2.1.3">2</cn></apply><apply id="A6.E34.m1.2.2.3.cmml" xref="A6.E34.m1.2.2.3"><times id="A6.E34.m1.2.2.3.1.cmml" xref="A6.E34.m1.2.2.3.1"></times><apply id="A6.E34.m1.2.2.3.2.cmml" xref="A6.E34.m1.2.2.3.2"><divide id="A6.E34.m1.2.2.3.2.1.cmml" xref="A6.E34.m1.2.2.3.2"></divide><ci id="A6.E34.m1.2.2.3.2.2.cmml" xref="A6.E34.m1.2.2.3.2.2">𝐶</ci><apply id="A6.E34.m1.2.2.3.2.3.cmml" xref="A6.E34.m1.2.2.3.2.3"><times id="A6.E34.m1.2.2.3.2.3.1.cmml" xref="A6.E34.m1.2.2.3.2.3.1"></times><cn type="integer" id="A6.E34.m1.2.2.3.2.3.2.cmml" xref="A6.E34.m1.2.2.3.2.3.2">2</cn><apply id="A6.E34.m1.2.2.3.2.3.3.cmml" xref="A6.E34.m1.2.2.3.2.3.3"><root id="A6.E34.m1.2.2.3.2.3.3a.cmml" xref="A6.E34.m1.2.2.3.2.3.3"></root><ci id="A6.E34.m1.2.2.3.2.3.3.2.cmml" xref="A6.E34.m1.2.2.3.2.3.3.2">𝜆</ci></apply></apply></apply><apply id="A6.E34.m1.2.2.3.3.cmml" xref="A6.E34.m1.2.2.3.3"><csymbol cd="ambiguous" id="A6.E34.m1.2.2.3.3.1.cmml" xref="A6.E34.m1.2.2.3.3">subscript</csymbol><apply id="A6.E34.m1.2.2.3.3.2.1.cmml" xref="A6.E34.m1.2.2.3.3.2.2"><csymbol cd="latexml" id="A6.E34.m1.2.2.3.3.2.1.1.cmml" xref="A6.E34.m1.2.2.3.3.2.2.1">delimited-∥∥</csymbol><apply id="A6.E34.m1.1.1.cmml" xref="A6.E34.m1.1.1"><ci id="A6.E34.m1.1.1.1.cmml" xref="A6.E34.m1.1.1.1">~</ci><ci id="A6.E34.m1.1.1.2.cmml" xref="A6.E34.m1.1.1.2">𝑨</ci></apply></apply><infinity id="A6.E34.m1.2.2.3.3.3.cmml" xref="A6.E34.m1.2.2.3.3.3"></infinity></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E34.m1.2c">\displaystyle\lVert\bm{g}^{\top}\bm{P}\rVert_{2}\leq\frac{C}{2\sqrt{\lambda}}\lVert\tilde{\bm{A}}\rVert_{\infty}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(34)</span></td>
</tr></tbody>
</table>
<p id="A6.Thmlemma4.p1.8" class="ltx_p"><span id="A6.Thmlemma4.p1.8.2" class="ltx_text ltx_font_italic">where <math id="A6.Thmlemma4.p1.7.1.m1.1" class="ltx_Math" alttext="\lVert\tilde{\bm{A}}\rVert_{\infty}" display="inline"><semantics id="A6.Thmlemma4.p1.7.1.m1.1a"><msub id="A6.Thmlemma4.p1.7.1.m1.1.2" xref="A6.Thmlemma4.p1.7.1.m1.1.2.cmml"><mrow id="A6.Thmlemma4.p1.7.1.m1.1.2.2.2" xref="A6.Thmlemma4.p1.7.1.m1.1.2.2.1.cmml"><mo fence="true" rspace="0em" id="A6.Thmlemma4.p1.7.1.m1.1.2.2.2.1" xref="A6.Thmlemma4.p1.7.1.m1.1.2.2.1.1.cmml">∥</mo><mover accent="true" id="A6.Thmlemma4.p1.7.1.m1.1.1" xref="A6.Thmlemma4.p1.7.1.m1.1.1.cmml"><mi id="A6.Thmlemma4.p1.7.1.m1.1.1.2" xref="A6.Thmlemma4.p1.7.1.m1.1.1.2.cmml">𝐀</mi><mo id="A6.Thmlemma4.p1.7.1.m1.1.1.1" xref="A6.Thmlemma4.p1.7.1.m1.1.1.1.cmml">~</mo></mover><mo fence="true" lspace="0em" id="A6.Thmlemma4.p1.7.1.m1.1.2.2.2.2" xref="A6.Thmlemma4.p1.7.1.m1.1.2.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.Thmlemma4.p1.7.1.m1.1.2.3" xref="A6.Thmlemma4.p1.7.1.m1.1.2.3.cmml">∞</mi></msub><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.7.1.m1.1b"><apply id="A6.Thmlemma4.p1.7.1.m1.1.2.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.2"><csymbol cd="ambiguous" id="A6.Thmlemma4.p1.7.1.m1.1.2.1.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.2">subscript</csymbol><apply id="A6.Thmlemma4.p1.7.1.m1.1.2.2.1.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.2.2.2"><csymbol cd="latexml" id="A6.Thmlemma4.p1.7.1.m1.1.2.2.1.1.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.2.2.2.1">delimited-∥∥</csymbol><apply id="A6.Thmlemma4.p1.7.1.m1.1.1.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.1"><ci id="A6.Thmlemma4.p1.7.1.m1.1.1.1.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.1.1">~</ci><ci id="A6.Thmlemma4.p1.7.1.m1.1.1.2.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.1.2">𝐀</ci></apply></apply><infinity id="A6.Thmlemma4.p1.7.1.m1.1.2.3.cmml" xref="A6.Thmlemma4.p1.7.1.m1.1.2.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.7.1.m1.1c">\lVert\tilde{\bm{A}}\rVert_{\infty}</annotation></semantics></math> denotes the largest singular value of <math id="A6.Thmlemma4.p1.8.2.m2.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="A6.Thmlemma4.p1.8.2.m2.1a"><mover accent="true" id="A6.Thmlemma4.p1.8.2.m2.1.1" xref="A6.Thmlemma4.p1.8.2.m2.1.1.cmml"><mi id="A6.Thmlemma4.p1.8.2.m2.1.1.2" xref="A6.Thmlemma4.p1.8.2.m2.1.1.2.cmml">𝐀</mi><mo id="A6.Thmlemma4.p1.8.2.m2.1.1.1" xref="A6.Thmlemma4.p1.8.2.m2.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="A6.Thmlemma4.p1.8.2.m2.1b"><apply id="A6.Thmlemma4.p1.8.2.m2.1.1.cmml" xref="A6.Thmlemma4.p1.8.2.m2.1.1"><ci id="A6.Thmlemma4.p1.8.2.m2.1.1.1.cmml" xref="A6.Thmlemma4.p1.8.2.m2.1.1.1">~</ci><ci id="A6.Thmlemma4.p1.8.2.m2.1.1.2.cmml" xref="A6.Thmlemma4.p1.8.2.m2.1.1.2">𝐀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma4.p1.8.2.m2.1c">\tilde{\bm{A}}</annotation></semantics></math>.</span></p>
</div>
</div>
<div id="A6.SS2.7" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A6.SS2.7.p1" class="ltx_para">
<p id="A6.SS2.7.p1.3" class="ltx_p">We have</p>
<table id="A6.EGx24" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E35"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E35.m1.1" class="ltx_Math" alttext="\displaystyle\lVert\bm{g}^{\top}\bm{P}\rVert_{2}" display="inline"><semantics id="A6.E35.m1.1a"><msub id="A6.E35.m1.1.1" xref="A6.E35.m1.1.1.cmml"><mrow id="A6.E35.m1.1.1.1.1" xref="A6.E35.m1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="A6.E35.m1.1.1.1.1.2" xref="A6.E35.m1.1.1.1.2.1.cmml">∥</mo><mrow id="A6.E35.m1.1.1.1.1.1" xref="A6.E35.m1.1.1.1.1.1.cmml"><msup id="A6.E35.m1.1.1.1.1.1.2" xref="A6.E35.m1.1.1.1.1.1.2.cmml"><mi id="A6.E35.m1.1.1.1.1.1.2.2" xref="A6.E35.m1.1.1.1.1.1.2.2.cmml">𝒈</mi><mo id="A6.E35.m1.1.1.1.1.1.2.3" xref="A6.E35.m1.1.1.1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E35.m1.1.1.1.1.1.1" xref="A6.E35.m1.1.1.1.1.1.1.cmml">​</mo><mi id="A6.E35.m1.1.1.1.1.1.3" xref="A6.E35.m1.1.1.1.1.1.3.cmml">𝑷</mi></mrow><mo fence="true" lspace="0em" id="A6.E35.m1.1.1.1.1.3" xref="A6.E35.m1.1.1.1.2.1.cmml">∥</mo></mrow><mn id="A6.E35.m1.1.1.3" xref="A6.E35.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.E35.m1.1b"><apply id="A6.E35.m1.1.1.cmml" xref="A6.E35.m1.1.1"><csymbol cd="ambiguous" id="A6.E35.m1.1.1.2.cmml" xref="A6.E35.m1.1.1">subscript</csymbol><apply id="A6.E35.m1.1.1.1.2.cmml" xref="A6.E35.m1.1.1.1.1"><csymbol cd="latexml" id="A6.E35.m1.1.1.1.2.1.cmml" xref="A6.E35.m1.1.1.1.1.2">delimited-∥∥</csymbol><apply id="A6.E35.m1.1.1.1.1.1.cmml" xref="A6.E35.m1.1.1.1.1.1"><times id="A6.E35.m1.1.1.1.1.1.1.cmml" xref="A6.E35.m1.1.1.1.1.1.1"></times><apply id="A6.E35.m1.1.1.1.1.1.2.cmml" xref="A6.E35.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A6.E35.m1.1.1.1.1.1.2.1.cmml" xref="A6.E35.m1.1.1.1.1.1.2">superscript</csymbol><ci id="A6.E35.m1.1.1.1.1.1.2.2.cmml" xref="A6.E35.m1.1.1.1.1.1.2.2">𝒈</ci><csymbol cd="latexml" id="A6.E35.m1.1.1.1.1.1.2.3.cmml" xref="A6.E35.m1.1.1.1.1.1.2.3">top</csymbol></apply><ci id="A6.E35.m1.1.1.1.1.1.3.cmml" xref="A6.E35.m1.1.1.1.1.1.3">𝑷</ci></apply></apply><cn type="integer" id="A6.E35.m1.1.1.3.cmml" xref="A6.E35.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E35.m1.1c">\displaystyle\lVert\bm{g}^{\top}\bm{P}\rVert_{2}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E35.m2.2" class="ltx_Math" alttext="\displaystyle\leq\lVert\bm{g}\rVert_{2}\lVert\bm{P}\rVert_{\infty}" display="inline"><semantics id="A6.E35.m2.2a"><mrow id="A6.E35.m2.2.3" xref="A6.E35.m2.2.3.cmml"><mi id="A6.E35.m2.2.3.2" xref="A6.E35.m2.2.3.2.cmml"></mi><mo rspace="0.1389em" id="A6.E35.m2.2.3.1" xref="A6.E35.m2.2.3.1.cmml">≤</mo><mrow id="A6.E35.m2.2.3.3" xref="A6.E35.m2.2.3.3.cmml"><msub id="A6.E35.m2.2.3.3.2" xref="A6.E35.m2.2.3.3.2.cmml"><mrow id="A6.E35.m2.2.3.3.2.2.2" xref="A6.E35.m2.2.3.3.2.2.1.cmml"><mo fence="true" lspace="0.1389em" rspace="0em" id="A6.E35.m2.2.3.3.2.2.2.1" xref="A6.E35.m2.2.3.3.2.2.1.1.cmml">∥</mo><mi id="A6.E35.m2.1.1" xref="A6.E35.m2.1.1.cmml">𝒈</mi><mo fence="true" lspace="0em" id="A6.E35.m2.2.3.3.2.2.2.2" xref="A6.E35.m2.2.3.3.2.2.1.1.cmml">∥</mo></mrow><mn id="A6.E35.m2.2.3.3.2.3" xref="A6.E35.m2.2.3.3.2.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="A6.E35.m2.2.3.3.1" xref="A6.E35.m2.2.3.3.1.cmml">​</mo><msub id="A6.E35.m2.2.3.3.3" xref="A6.E35.m2.2.3.3.3.cmml"><mrow id="A6.E35.m2.2.3.3.3.2.2" xref="A6.E35.m2.2.3.3.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E35.m2.2.3.3.3.2.2.1" xref="A6.E35.m2.2.3.3.3.2.1.1.cmml">∥</mo><mi id="A6.E35.m2.2.2" xref="A6.E35.m2.2.2.cmml">𝑷</mi><mo fence="true" lspace="0em" id="A6.E35.m2.2.3.3.3.2.2.2" xref="A6.E35.m2.2.3.3.3.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E35.m2.2.3.3.3.3" xref="A6.E35.m2.2.3.3.3.3.cmml">∞</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E35.m2.2b"><apply id="A6.E35.m2.2.3.cmml" xref="A6.E35.m2.2.3"><leq id="A6.E35.m2.2.3.1.cmml" xref="A6.E35.m2.2.3.1"></leq><csymbol cd="latexml" id="A6.E35.m2.2.3.2.cmml" xref="A6.E35.m2.2.3.2">absent</csymbol><apply id="A6.E35.m2.2.3.3.cmml" xref="A6.E35.m2.2.3.3"><times id="A6.E35.m2.2.3.3.1.cmml" xref="A6.E35.m2.2.3.3.1"></times><apply id="A6.E35.m2.2.3.3.2.cmml" xref="A6.E35.m2.2.3.3.2"><csymbol cd="ambiguous" id="A6.E35.m2.2.3.3.2.1.cmml" xref="A6.E35.m2.2.3.3.2">subscript</csymbol><apply id="A6.E35.m2.2.3.3.2.2.1.cmml" xref="A6.E35.m2.2.3.3.2.2.2"><csymbol cd="latexml" id="A6.E35.m2.2.3.3.2.2.1.1.cmml" xref="A6.E35.m2.2.3.3.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.E35.m2.1.1.cmml" xref="A6.E35.m2.1.1">𝒈</ci></apply><cn type="integer" id="A6.E35.m2.2.3.3.2.3.cmml" xref="A6.E35.m2.2.3.3.2.3">2</cn></apply><apply id="A6.E35.m2.2.3.3.3.cmml" xref="A6.E35.m2.2.3.3.3"><csymbol cd="ambiguous" id="A6.E35.m2.2.3.3.3.1.cmml" xref="A6.E35.m2.2.3.3.3">subscript</csymbol><apply id="A6.E35.m2.2.3.3.3.2.1.cmml" xref="A6.E35.m2.2.3.3.3.2.2"><csymbol cd="latexml" id="A6.E35.m2.2.3.3.3.2.1.1.cmml" xref="A6.E35.m2.2.3.3.3.2.2.1">delimited-∥∥</csymbol><ci id="A6.E35.m2.2.2.cmml" xref="A6.E35.m2.2.2">𝑷</ci></apply><infinity id="A6.E35.m2.2.3.3.3.3.cmml" xref="A6.E35.m2.2.3.3.3.3"></infinity></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E35.m2.2c">\displaystyle\leq\lVert\bm{g}\rVert_{2}\lVert\bm{P}\rVert_{\infty}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(35)</span></td>
</tr></tbody>
<tbody id="A6.E36"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E36.m1.1" class="ltx_Math" alttext="\displaystyle=C\lVert\bm{P}\rVert_{\infty}" display="inline"><semantics id="A6.E36.m1.1a"><mrow id="A6.E36.m1.1.2" xref="A6.E36.m1.1.2.cmml"><mi id="A6.E36.m1.1.2.2" xref="A6.E36.m1.1.2.2.cmml"></mi><mo id="A6.E36.m1.1.2.1" xref="A6.E36.m1.1.2.1.cmml">=</mo><mrow id="A6.E36.m1.1.2.3" xref="A6.E36.m1.1.2.3.cmml"><mi id="A6.E36.m1.1.2.3.2" xref="A6.E36.m1.1.2.3.2.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.E36.m1.1.2.3.1" xref="A6.E36.m1.1.2.3.1.cmml">​</mo><msub id="A6.E36.m1.1.2.3.3" xref="A6.E36.m1.1.2.3.3.cmml"><mrow id="A6.E36.m1.1.2.3.3.2.2" xref="A6.E36.m1.1.2.3.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E36.m1.1.2.3.3.2.2.1" xref="A6.E36.m1.1.2.3.3.2.1.1.cmml">∥</mo><mi id="A6.E36.m1.1.1" xref="A6.E36.m1.1.1.cmml">𝑷</mi><mo fence="true" lspace="0em" id="A6.E36.m1.1.2.3.3.2.2.2" xref="A6.E36.m1.1.2.3.3.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E36.m1.1.2.3.3.3" xref="A6.E36.m1.1.2.3.3.3.cmml">∞</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E36.m1.1b"><apply id="A6.E36.m1.1.2.cmml" xref="A6.E36.m1.1.2"><eq id="A6.E36.m1.1.2.1.cmml" xref="A6.E36.m1.1.2.1"></eq><csymbol cd="latexml" id="A6.E36.m1.1.2.2.cmml" xref="A6.E36.m1.1.2.2">absent</csymbol><apply id="A6.E36.m1.1.2.3.cmml" xref="A6.E36.m1.1.2.3"><times id="A6.E36.m1.1.2.3.1.cmml" xref="A6.E36.m1.1.2.3.1"></times><ci id="A6.E36.m1.1.2.3.2.cmml" xref="A6.E36.m1.1.2.3.2">𝐶</ci><apply id="A6.E36.m1.1.2.3.3.cmml" xref="A6.E36.m1.1.2.3.3"><csymbol cd="ambiguous" id="A6.E36.m1.1.2.3.3.1.cmml" xref="A6.E36.m1.1.2.3.3">subscript</csymbol><apply id="A6.E36.m1.1.2.3.3.2.1.cmml" xref="A6.E36.m1.1.2.3.3.2.2"><csymbol cd="latexml" id="A6.E36.m1.1.2.3.3.2.1.1.cmml" xref="A6.E36.m1.1.2.3.3.2.2.1">delimited-∥∥</csymbol><ci id="A6.E36.m1.1.1.cmml" xref="A6.E36.m1.1.1">𝑷</ci></apply><infinity id="A6.E36.m1.1.2.3.3.3.cmml" xref="A6.E36.m1.1.2.3.3.3"></infinity></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E36.m1.1c">\displaystyle=C\lVert\bm{P}\rVert_{\infty}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(36)</span></td>
</tr></tbody>
<tbody id="A6.E37"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E37.m1.2" class="ltx_Math" alttext="\displaystyle\leq\frac{C}{2\sqrt{\lambda}}\lVert\tilde{\bm{A}}\rVert_{\infty}." display="inline"><semantics id="A6.E37.m1.2a"><mrow id="A6.E37.m1.2.2.1" xref="A6.E37.m1.2.2.1.1.cmml"><mrow id="A6.E37.m1.2.2.1.1" xref="A6.E37.m1.2.2.1.1.cmml"><mi id="A6.E37.m1.2.2.1.1.2" xref="A6.E37.m1.2.2.1.1.2.cmml"></mi><mo id="A6.E37.m1.2.2.1.1.1" xref="A6.E37.m1.2.2.1.1.1.cmml">≤</mo><mrow id="A6.E37.m1.2.2.1.1.3" xref="A6.E37.m1.2.2.1.1.3.cmml"><mstyle displaystyle="true" id="A6.E37.m1.2.2.1.1.3.2" xref="A6.E37.m1.2.2.1.1.3.2.cmml"><mfrac id="A6.E37.m1.2.2.1.1.3.2a" xref="A6.E37.m1.2.2.1.1.3.2.cmml"><mi id="A6.E37.m1.2.2.1.1.3.2.2" xref="A6.E37.m1.2.2.1.1.3.2.2.cmml">C</mi><mrow id="A6.E37.m1.2.2.1.1.3.2.3" xref="A6.E37.m1.2.2.1.1.3.2.3.cmml"><mn id="A6.E37.m1.2.2.1.1.3.2.3.2" xref="A6.E37.m1.2.2.1.1.3.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E37.m1.2.2.1.1.3.2.3.1" xref="A6.E37.m1.2.2.1.1.3.2.3.1.cmml">​</mo><msqrt id="A6.E37.m1.2.2.1.1.3.2.3.3" xref="A6.E37.m1.2.2.1.1.3.2.3.3.cmml"><mi id="A6.E37.m1.2.2.1.1.3.2.3.3.2" xref="A6.E37.m1.2.2.1.1.3.2.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A6.E37.m1.2.2.1.1.3.1" xref="A6.E37.m1.2.2.1.1.3.1.cmml">​</mo><msub id="A6.E37.m1.2.2.1.1.3.3" xref="A6.E37.m1.2.2.1.1.3.3.cmml"><mrow id="A6.E37.m1.2.2.1.1.3.3.2.2" xref="A6.E37.m1.2.2.1.1.3.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E37.m1.2.2.1.1.3.3.2.2.1" xref="A6.E37.m1.2.2.1.1.3.3.2.1.1.cmml">∥</mo><mover accent="true" id="A6.E37.m1.1.1" xref="A6.E37.m1.1.1.cmml"><mi id="A6.E37.m1.1.1.2" xref="A6.E37.m1.1.1.2.cmml">𝑨</mi><mo id="A6.E37.m1.1.1.1" xref="A6.E37.m1.1.1.1.cmml">~</mo></mover><mo fence="true" lspace="0em" rspace="0em" id="A6.E37.m1.2.2.1.1.3.3.2.2.2" xref="A6.E37.m1.2.2.1.1.3.3.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E37.m1.2.2.1.1.3.3.3" xref="A6.E37.m1.2.2.1.1.3.3.3.cmml">∞</mi></msub></mrow></mrow><mo lspace="0em" id="A6.E37.m1.2.2.1.2" xref="A6.E37.m1.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.E37.m1.2b"><apply id="A6.E37.m1.2.2.1.1.cmml" xref="A6.E37.m1.2.2.1"><leq id="A6.E37.m1.2.2.1.1.1.cmml" xref="A6.E37.m1.2.2.1.1.1"></leq><csymbol cd="latexml" id="A6.E37.m1.2.2.1.1.2.cmml" xref="A6.E37.m1.2.2.1.1.2">absent</csymbol><apply id="A6.E37.m1.2.2.1.1.3.cmml" xref="A6.E37.m1.2.2.1.1.3"><times id="A6.E37.m1.2.2.1.1.3.1.cmml" xref="A6.E37.m1.2.2.1.1.3.1"></times><apply id="A6.E37.m1.2.2.1.1.3.2.cmml" xref="A6.E37.m1.2.2.1.1.3.2"><divide id="A6.E37.m1.2.2.1.1.3.2.1.cmml" xref="A6.E37.m1.2.2.1.1.3.2"></divide><ci id="A6.E37.m1.2.2.1.1.3.2.2.cmml" xref="A6.E37.m1.2.2.1.1.3.2.2">𝐶</ci><apply id="A6.E37.m1.2.2.1.1.3.2.3.cmml" xref="A6.E37.m1.2.2.1.1.3.2.3"><times id="A6.E37.m1.2.2.1.1.3.2.3.1.cmml" xref="A6.E37.m1.2.2.1.1.3.2.3.1"></times><cn type="integer" id="A6.E37.m1.2.2.1.1.3.2.3.2.cmml" xref="A6.E37.m1.2.2.1.1.3.2.3.2">2</cn><apply id="A6.E37.m1.2.2.1.1.3.2.3.3.cmml" xref="A6.E37.m1.2.2.1.1.3.2.3.3"><root id="A6.E37.m1.2.2.1.1.3.2.3.3a.cmml" xref="A6.E37.m1.2.2.1.1.3.2.3.3"></root><ci id="A6.E37.m1.2.2.1.1.3.2.3.3.2.cmml" xref="A6.E37.m1.2.2.1.1.3.2.3.3.2">𝜆</ci></apply></apply></apply><apply id="A6.E37.m1.2.2.1.1.3.3.cmml" xref="A6.E37.m1.2.2.1.1.3.3"><csymbol cd="ambiguous" id="A6.E37.m1.2.2.1.1.3.3.1.cmml" xref="A6.E37.m1.2.2.1.1.3.3">subscript</csymbol><apply id="A6.E37.m1.2.2.1.1.3.3.2.1.cmml" xref="A6.E37.m1.2.2.1.1.3.3.2.2"><csymbol cd="latexml" id="A6.E37.m1.2.2.1.1.3.3.2.1.1.cmml" xref="A6.E37.m1.2.2.1.1.3.3.2.2.1">delimited-∥∥</csymbol><apply id="A6.E37.m1.1.1.cmml" xref="A6.E37.m1.1.1"><ci id="A6.E37.m1.1.1.1.cmml" xref="A6.E37.m1.1.1.1">~</ci><ci id="A6.E37.m1.1.1.2.cmml" xref="A6.E37.m1.1.1.2">𝑨</ci></apply></apply><infinity id="A6.E37.m1.2.2.1.1.3.3.3.cmml" xref="A6.E37.m1.2.2.1.1.3.3.3"></infinity></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E37.m1.2c">\displaystyle\leq\frac{C}{2\sqrt{\lambda}}\lVert\tilde{\bm{A}}\rVert_{\infty}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(37)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.7.p1.2" class="ltx_p">The first step follows from Cauchy-Schwarz. The second follows from <math id="A6.SS2.7.p1.1.m1.1" class="ltx_Math" alttext="\bm{g}" display="inline"><semantics id="A6.SS2.7.p1.1.m1.1a"><mi id="A6.SS2.7.p1.1.m1.1.1" xref="A6.SS2.7.p1.1.m1.1.1.cmml">𝒈</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.7.p1.1.m1.1b"><ci id="A6.SS2.7.p1.1.m1.1.1.cmml" xref="A6.SS2.7.p1.1.m1.1.1">𝒈</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.7.p1.1.m1.1c">\bm{g}</annotation></semantics></math> being zero except for exactly <math id="A6.SS2.7.p1.2.m2.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.SS2.7.p1.2.m2.1a"><mi id="A6.SS2.7.p1.2.m2.1.1" xref="A6.SS2.7.p1.2.m2.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.7.p1.2.m2.1b"><ci id="A6.SS2.7.p1.2.m2.1.1.cmml" xref="A6.SS2.7.p1.2.m2.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.7.p1.2.m2.1c">C</annotation></semantics></math> ones. The last is an application of Lemma <a href="#A6.Thmlemma3" title="Lemma F.3 (Ridge Regression Singular Value Bound). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.3</span></a>.
∎</p>
</div>
</div>
<div id="A6.Thmlemma5" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmlemma5.1.1.1" class="ltx_text ltx_font_bold">Lemma F.5</span></span><span id="A6.Thmlemma5.2.2" class="ltx_text ltx_font_bold"> </span>(Maximum Table Quantization Loss)<span id="A6.Thmlemma5.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmlemma5.p1" class="ltx_para">
<p id="A6.Thmlemma5.p1.14" class="ltx_p"><span id="A6.Thmlemma5.p1.14.14" class="ltx_text ltx_font_italic">Let <math id="A6.Thmlemma5.p1.1.1.m1.1" class="ltx_Math" alttext="\bm{\hat{a}}=g(\bm{a})^{\top}\bm{P}" display="inline"><semantics id="A6.Thmlemma5.p1.1.1.m1.1a"><mrow id="A6.Thmlemma5.p1.1.1.m1.1.2" xref="A6.Thmlemma5.p1.1.1.m1.1.2.cmml"><mover accent="true" id="A6.Thmlemma5.p1.1.1.m1.1.2.2" xref="A6.Thmlemma5.p1.1.1.m1.1.2.2.cmml"><mi id="A6.Thmlemma5.p1.1.1.m1.1.2.2.2" xref="A6.Thmlemma5.p1.1.1.m1.1.2.2.2.cmml">𝐚</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="A6.Thmlemma5.p1.1.1.m1.1.2.2.1" xref="A6.Thmlemma5.p1.1.1.m1.1.2.2.1.cmml">^</mo></mover><mo id="A6.Thmlemma5.p1.1.1.m1.1.2.1" xref="A6.Thmlemma5.p1.1.1.m1.1.2.1.cmml">=</mo><mrow id="A6.Thmlemma5.p1.1.1.m1.1.2.3" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.cmml"><mi id="A6.Thmlemma5.p1.1.1.m1.1.2.3.2" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.1.1.m1.1.2.3.1" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.1.cmml">​</mo><msup id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.cmml"><mrow id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.2.2" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.cmml"><mo stretchy="false" id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.2.2.1" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.cmml">(</mo><mi id="A6.Thmlemma5.p1.1.1.m1.1.1" xref="A6.Thmlemma5.p1.1.1.m1.1.1.cmml">𝐚</mi><mo stretchy="false" id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.2.2.2" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.cmml">)</mo></mrow><mo id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.3" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.1.1.m1.1.2.3.1a" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.1.cmml">​</mo><mi id="A6.Thmlemma5.p1.1.1.m1.1.2.3.4" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.4.cmml">𝐏</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.1.1.m1.1b"><apply id="A6.Thmlemma5.p1.1.1.m1.1.2.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2"><eq id="A6.Thmlemma5.p1.1.1.m1.1.2.1.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.1"></eq><apply id="A6.Thmlemma5.p1.1.1.m1.1.2.2.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.2"><ci id="A6.Thmlemma5.p1.1.1.m1.1.2.2.1.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.2.1">bold-^</ci><ci id="A6.Thmlemma5.p1.1.1.m1.1.2.2.2.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.2.2">𝐚</ci></apply><apply id="A6.Thmlemma5.p1.1.1.m1.1.2.3.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3"><times id="A6.Thmlemma5.p1.1.1.m1.1.2.3.1.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.1"></times><ci id="A6.Thmlemma5.p1.1.1.m1.1.2.3.2.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.2">𝑔</ci><apply id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.1.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3">superscript</csymbol><ci id="A6.Thmlemma5.p1.1.1.m1.1.1.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.1">𝐚</ci><csymbol cd="latexml" id="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.3.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.3.3">top</csymbol></apply><ci id="A6.Thmlemma5.p1.1.1.m1.1.2.3.4.cmml" xref="A6.Thmlemma5.p1.1.1.m1.1.2.3.4">𝐏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.1.1.m1.1c">\bm{\hat{a}}=g(\bm{a})^{\top}\bm{P}</annotation></semantics></math>, where <math id="A6.Thmlemma5.p1.2.2.m2.1" class="ltx_Math" alttext="g(\cdot)" display="inline"><semantics id="A6.Thmlemma5.p1.2.2.m2.1a"><mrow id="A6.Thmlemma5.p1.2.2.m2.1.2" xref="A6.Thmlemma5.p1.2.2.m2.1.2.cmml"><mi id="A6.Thmlemma5.p1.2.2.m2.1.2.2" xref="A6.Thmlemma5.p1.2.2.m2.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.2.2.m2.1.2.1" xref="A6.Thmlemma5.p1.2.2.m2.1.2.1.cmml">​</mo><mrow id="A6.Thmlemma5.p1.2.2.m2.1.2.3.2" xref="A6.Thmlemma5.p1.2.2.m2.1.2.cmml"><mo stretchy="false" id="A6.Thmlemma5.p1.2.2.m2.1.2.3.2.1" xref="A6.Thmlemma5.p1.2.2.m2.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.2.2.m2.1.1" xref="A6.Thmlemma5.p1.2.2.m2.1.1.cmml">⋅</mo><mo stretchy="false" id="A6.Thmlemma5.p1.2.2.m2.1.2.3.2.2" xref="A6.Thmlemma5.p1.2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.2.2.m2.1b"><apply id="A6.Thmlemma5.p1.2.2.m2.1.2.cmml" xref="A6.Thmlemma5.p1.2.2.m2.1.2"><times id="A6.Thmlemma5.p1.2.2.m2.1.2.1.cmml" xref="A6.Thmlemma5.p1.2.2.m2.1.2.1"></times><ci id="A6.Thmlemma5.p1.2.2.m2.1.2.2.cmml" xref="A6.Thmlemma5.p1.2.2.m2.1.2.2">𝑔</ci><ci id="A6.Thmlemma5.p1.2.2.m2.1.1.cmml" xref="A6.Thmlemma5.p1.2.2.m2.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.2.2.m2.1c">g(\cdot)</annotation></semantics></math> and <math id="A6.Thmlemma5.p1.3.3.m3.1" class="ltx_Math" alttext="P" display="inline"><semantics id="A6.Thmlemma5.p1.3.3.m3.1a"><mi id="A6.Thmlemma5.p1.3.3.m3.1.1" xref="A6.Thmlemma5.p1.3.3.m3.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.3.3.m3.1b"><ci id="A6.Thmlemma5.p1.3.3.m3.1.1.cmml" xref="A6.Thmlemma5.p1.3.3.m3.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.3.3.m3.1c">P</annotation></semantics></math> are trained using <math id="A6.Thmlemma5.p1.4.4.m4.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.Thmlemma5.p1.4.4.m4.1a"><mi id="A6.Thmlemma5.p1.4.4.m4.1.1" xref="A6.Thmlemma5.p1.4.4.m4.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.4.4.m4.1b"><ci id="A6.Thmlemma5.p1.4.4.m4.1.1.cmml" xref="A6.Thmlemma5.p1.4.4.m4.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.4.4.m4.1c">C</annotation></semantics></math> codebooks and ridge regression penalty <math id="A6.Thmlemma5.p1.5.5.m5.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="A6.Thmlemma5.p1.5.5.m5.1a"><mrow id="A6.Thmlemma5.p1.5.5.m5.1.1" xref="A6.Thmlemma5.p1.5.5.m5.1.1.cmml"><mi id="A6.Thmlemma5.p1.5.5.m5.1.1.2" xref="A6.Thmlemma5.p1.5.5.m5.1.1.2.cmml">λ</mi><mo id="A6.Thmlemma5.p1.5.5.m5.1.1.1" xref="A6.Thmlemma5.p1.5.5.m5.1.1.1.cmml">&gt;</mo><mn id="A6.Thmlemma5.p1.5.5.m5.1.1.3" xref="A6.Thmlemma5.p1.5.5.m5.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.5.5.m5.1b"><apply id="A6.Thmlemma5.p1.5.5.m5.1.1.cmml" xref="A6.Thmlemma5.p1.5.5.m5.1.1"><gt id="A6.Thmlemma5.p1.5.5.m5.1.1.1.cmml" xref="A6.Thmlemma5.p1.5.5.m5.1.1.1"></gt><ci id="A6.Thmlemma5.p1.5.5.m5.1.1.2.cmml" xref="A6.Thmlemma5.p1.5.5.m5.1.1.2">𝜆</ci><cn type="integer" id="A6.Thmlemma5.p1.5.5.m5.1.1.3.cmml" xref="A6.Thmlemma5.p1.5.5.m5.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.5.5.m5.1c">\lambda&gt;0</annotation></semantics></math> on a matrix <math id="A6.Thmlemma5.p1.6.6.m6.1" class="ltx_Math" alttext="\tilde{\bm{A}}" display="inline"><semantics id="A6.Thmlemma5.p1.6.6.m6.1a"><mover accent="true" id="A6.Thmlemma5.p1.6.6.m6.1.1" xref="A6.Thmlemma5.p1.6.6.m6.1.1.cmml"><mi id="A6.Thmlemma5.p1.6.6.m6.1.1.2" xref="A6.Thmlemma5.p1.6.6.m6.1.1.2.cmml">𝐀</mi><mo id="A6.Thmlemma5.p1.6.6.m6.1.1.1" xref="A6.Thmlemma5.p1.6.6.m6.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.6.6.m6.1b"><apply id="A6.Thmlemma5.p1.6.6.m6.1.1.cmml" xref="A6.Thmlemma5.p1.6.6.m6.1.1"><ci id="A6.Thmlemma5.p1.6.6.m6.1.1.1.cmml" xref="A6.Thmlemma5.p1.6.6.m6.1.1.1">~</ci><ci id="A6.Thmlemma5.p1.6.6.m6.1.1.2.cmml" xref="A6.Thmlemma5.p1.6.6.m6.1.1.2">𝐀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.6.6.m6.1c">\tilde{\bm{A}}</annotation></semantics></math> with maximum singular value at most <math id="A6.Thmlemma5.p1.7.7.m7.1" class="ltx_Math" alttext="\sigma_{A}" display="inline"><semantics id="A6.Thmlemma5.p1.7.7.m7.1a"><msub id="A6.Thmlemma5.p1.7.7.m7.1.1" xref="A6.Thmlemma5.p1.7.7.m7.1.1.cmml"><mi id="A6.Thmlemma5.p1.7.7.m7.1.1.2" xref="A6.Thmlemma5.p1.7.7.m7.1.1.2.cmml">σ</mi><mi id="A6.Thmlemma5.p1.7.7.m7.1.1.3" xref="A6.Thmlemma5.p1.7.7.m7.1.1.3.cmml">A</mi></msub><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.7.7.m7.1b"><apply id="A6.Thmlemma5.p1.7.7.m7.1.1.cmml" xref="A6.Thmlemma5.p1.7.7.m7.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.7.7.m7.1.1.1.cmml" xref="A6.Thmlemma5.p1.7.7.m7.1.1">subscript</csymbol><ci id="A6.Thmlemma5.p1.7.7.m7.1.1.2.cmml" xref="A6.Thmlemma5.p1.7.7.m7.1.1.2">𝜎</ci><ci id="A6.Thmlemma5.p1.7.7.m7.1.1.3.cmml" xref="A6.Thmlemma5.p1.7.7.m7.1.1.3">𝐴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.7.7.m7.1c">\sigma_{A}</annotation></semantics></math>, and <math id="A6.Thmlemma5.p1.8.8.m8.1" class="ltx_Math" alttext="\bm{a}\in\mathbb{R}^{D}" display="inline"><semantics id="A6.Thmlemma5.p1.8.8.m8.1a"><mrow id="A6.Thmlemma5.p1.8.8.m8.1.1" xref="A6.Thmlemma5.p1.8.8.m8.1.1.cmml"><mi id="A6.Thmlemma5.p1.8.8.m8.1.1.2" xref="A6.Thmlemma5.p1.8.8.m8.1.1.2.cmml">𝐚</mi><mo id="A6.Thmlemma5.p1.8.8.m8.1.1.1" xref="A6.Thmlemma5.p1.8.8.m8.1.1.1.cmml">∈</mo><msup id="A6.Thmlemma5.p1.8.8.m8.1.1.3" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3.cmml"><mi id="A6.Thmlemma5.p1.8.8.m8.1.1.3.2" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3.2.cmml">ℝ</mi><mi id="A6.Thmlemma5.p1.8.8.m8.1.1.3.3" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.8.8.m8.1b"><apply id="A6.Thmlemma5.p1.8.8.m8.1.1.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1"><in id="A6.Thmlemma5.p1.8.8.m8.1.1.1.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1.1"></in><ci id="A6.Thmlemma5.p1.8.8.m8.1.1.2.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1.2">𝐚</ci><apply id="A6.Thmlemma5.p1.8.8.m8.1.1.3.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.8.8.m8.1.1.3.1.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3">superscript</csymbol><ci id="A6.Thmlemma5.p1.8.8.m8.1.1.3.2.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3.2">ℝ</ci><ci id="A6.Thmlemma5.p1.8.8.m8.1.1.3.3.cmml" xref="A6.Thmlemma5.p1.8.8.m8.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.8.8.m8.1c">\bm{a}\in\mathbb{R}^{D}</annotation></semantics></math> is an arbitrary vector. Then for any vector <math id="A6.Thmlemma5.p1.9.9.m9.1" class="ltx_Math" alttext="\bm{b}\in\mathbb{R}^{D}" display="inline"><semantics id="A6.Thmlemma5.p1.9.9.m9.1a"><mrow id="A6.Thmlemma5.p1.9.9.m9.1.1" xref="A6.Thmlemma5.p1.9.9.m9.1.1.cmml"><mi id="A6.Thmlemma5.p1.9.9.m9.1.1.2" xref="A6.Thmlemma5.p1.9.9.m9.1.1.2.cmml">𝐛</mi><mo id="A6.Thmlemma5.p1.9.9.m9.1.1.1" xref="A6.Thmlemma5.p1.9.9.m9.1.1.1.cmml">∈</mo><msup id="A6.Thmlemma5.p1.9.9.m9.1.1.3" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3.cmml"><mi id="A6.Thmlemma5.p1.9.9.m9.1.1.3.2" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3.2.cmml">ℝ</mi><mi id="A6.Thmlemma5.p1.9.9.m9.1.1.3.3" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.9.9.m9.1b"><apply id="A6.Thmlemma5.p1.9.9.m9.1.1.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1"><in id="A6.Thmlemma5.p1.9.9.m9.1.1.1.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1.1"></in><ci id="A6.Thmlemma5.p1.9.9.m9.1.1.2.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1.2">𝐛</ci><apply id="A6.Thmlemma5.p1.9.9.m9.1.1.3.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.9.9.m9.1.1.3.1.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3">superscript</csymbol><ci id="A6.Thmlemma5.p1.9.9.m9.1.1.3.2.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3.2">ℝ</ci><ci id="A6.Thmlemma5.p1.9.9.m9.1.1.3.3.cmml" xref="A6.Thmlemma5.p1.9.9.m9.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.9.9.m9.1c">\bm{b}\in\mathbb{R}^{D}</annotation></semantics></math>, <math id="A6.Thmlemma5.p1.10.10.m10.2" class="ltx_Math" alttext="|\bm{\hat{a}}^{\top}\bm{b}-\hat{\bm{y}}|&lt;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}" display="inline"><semantics id="A6.Thmlemma5.p1.10.10.m10.2a"><mrow id="A6.Thmlemma5.p1.10.10.m10.2.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.cmml"><mrow id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.2.cmml"><mo stretchy="false" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.2.1.cmml">|</mo><mrow id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.cmml"><mrow id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.cmml"><msup id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.cmml"><mover accent="true" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.cmml"><mi id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.2.cmml">𝐚</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.1" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.1.cmml">^</mo></mover><mo id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.3" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.1" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.1.cmml">​</mo><mi id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.3" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.3.cmml">𝐛</mi></mrow><mo id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.1" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.1.cmml">−</mo><mover accent="true" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.cmml"><mi id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.2.cmml">𝐲</mi><mo id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.1" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.1.cmml">^</mo></mover></mrow><mo stretchy="false" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.3" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.2.1.cmml">|</mo></mrow><mo id="A6.Thmlemma5.p1.10.10.m10.2.2.2" xref="A6.Thmlemma5.p1.10.10.m10.2.2.2.cmml">&lt;</mo><mfrac id="A6.Thmlemma5.p1.10.10.m10.1.1" xref="A6.Thmlemma5.p1.10.10.m10.1.1.cmml"><mrow id="A6.Thmlemma5.p1.10.10.m10.1.1.1" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.cmml"><mi id="A6.Thmlemma5.p1.10.10.m10.1.1.1.3" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.2" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.2.cmml">​</mo><msub id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.cmml"><mi id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.2" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.2.cmml">σ</mi><mi id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.3" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.2a" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.2.cmml">​</mo><msub id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.cmml"><mrow id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.2" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.2.1" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.1.1.cmml">∥</mo><mi id="A6.Thmlemma5.p1.10.10.m10.1.1.1.1" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.1.cmml">𝐛</mi><mo fence="true" lspace="0em" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.2.2" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.3" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.Thmlemma5.p1.10.10.m10.1.1.3" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.cmml"><mn id="A6.Thmlemma5.p1.10.10.m10.1.1.3.2" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.2.cmml">512</mn><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.10.10.m10.1.1.3.1" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.1.cmml">​</mo><msqrt id="A6.Thmlemma5.p1.10.10.m10.1.1.3.3" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.3.cmml"><mi id="A6.Thmlemma5.p1.10.10.m10.1.1.3.3.2" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.10.10.m10.2b"><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2"><lt id="A6.Thmlemma5.p1.10.10.m10.2.2.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.2"></lt><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.1.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1"><abs id="A6.Thmlemma5.p1.10.10.m10.2.2.1.2.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.2"></abs><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1"><minus id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.1"></minus><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2"><times id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.1"></times><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2">superscript</csymbol><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2"><ci id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.1">bold-^</ci><ci id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.2.2">𝐚</ci></apply><csymbol cd="latexml" id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.2.3">top</csymbol></apply><ci id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.2.3">𝐛</ci></apply><apply id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3"><ci id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.1">^</ci><ci id="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.2.2.1.1.1.3.2">𝐲</ci></apply></apply></apply><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1"><divide id="A6.Thmlemma5.p1.10.10.m10.1.1.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1"></divide><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1"><times id="A6.Thmlemma5.p1.10.10.m10.1.1.1.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.2"></times><ci id="A6.Thmlemma5.p1.10.10.m10.1.1.1.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.3">𝐶</ci><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4">subscript</csymbol><ci id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.2">𝜎</ci><ci id="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.4.3">𝐴</ci></apply><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5">subscript</csymbol><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.2"><csymbol cd="latexml" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.1.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.Thmlemma5.p1.10.10.m10.1.1.1.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.1">𝐛</ci></apply><cn type="integer" id="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.1.5.3">2</cn></apply></apply><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3"><times id="A6.Thmlemma5.p1.10.10.m10.1.1.3.1.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.1"></times><cn type="integer" id="A6.Thmlemma5.p1.10.10.m10.1.1.3.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.2">512</cn><apply id="A6.Thmlemma5.p1.10.10.m10.1.1.3.3.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.3"><root id="A6.Thmlemma5.p1.10.10.m10.1.1.3.3a.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.3"></root><ci id="A6.Thmlemma5.p1.10.10.m10.1.1.3.3.2.cmml" xref="A6.Thmlemma5.p1.10.10.m10.1.1.3.3.2">𝜆</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.10.10.m10.2c">|\bm{\hat{a}}^{\top}\bm{b}-\hat{\bm{y}}|&lt;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}</annotation></semantics></math>, where <math id="A6.Thmlemma5.p1.11.11.m11.2" class="ltx_Math" alttext="\hat{\bm{y}}\triangleq\alpha g(\bm{a})^{\top}g(\bm{b})+\bm{\beta}" display="inline"><semantics id="A6.Thmlemma5.p1.11.11.m11.2a"><mrow id="A6.Thmlemma5.p1.11.11.m11.2.3" xref="A6.Thmlemma5.p1.11.11.m11.2.3.cmml"><mover accent="true" id="A6.Thmlemma5.p1.11.11.m11.2.3.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.2.cmml"><mi id="A6.Thmlemma5.p1.11.11.m11.2.3.2.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.2.2.cmml">𝐲</mi><mo id="A6.Thmlemma5.p1.11.11.m11.2.3.2.1" xref="A6.Thmlemma5.p1.11.11.m11.2.3.2.1.cmml">^</mo></mover><mo id="A6.Thmlemma5.p1.11.11.m11.2.3.1" xref="A6.Thmlemma5.p1.11.11.m11.2.3.1.cmml">≜</mo><mrow id="A6.Thmlemma5.p1.11.11.m11.2.3.3" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.cmml"><mrow id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.cmml"><mi id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1.cmml">​</mo><mi id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.3" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.3.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1a" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1.cmml">​</mo><msup id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.cmml"><mrow id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.2.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.cmml"><mo stretchy="false" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.2.2.1" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.cmml">(</mo><mi id="A6.Thmlemma5.p1.11.11.m11.1.1" xref="A6.Thmlemma5.p1.11.11.m11.1.1.cmml">𝐚</mi><mo stretchy="false" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.2.2.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.cmml">)</mo></mrow><mo id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.3" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1b" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1.cmml">​</mo><mi id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.5" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.5.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1c" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1.cmml">​</mo><mrow id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.6.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.cmml"><mo stretchy="false" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.6.2.1" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.cmml">(</mo><mi id="A6.Thmlemma5.p1.11.11.m11.2.2" xref="A6.Thmlemma5.p1.11.11.m11.2.2.cmml">𝐛</mi><mo stretchy="false" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.6.2.2" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.cmml">)</mo></mrow></mrow><mo id="A6.Thmlemma5.p1.11.11.m11.2.3.3.1" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.1.cmml">+</mo><mi id="A6.Thmlemma5.p1.11.11.m11.2.3.3.3" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.3.cmml">𝛃</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.11.11.m11.2b"><apply id="A6.Thmlemma5.p1.11.11.m11.2.3.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3"><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.1.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.1">≜</ci><apply id="A6.Thmlemma5.p1.11.11.m11.2.3.2.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.2"><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.2.1.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.2.1">^</ci><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.2.2.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.2.2">𝐲</ci></apply><apply id="A6.Thmlemma5.p1.11.11.m11.2.3.3.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3"><plus id="A6.Thmlemma5.p1.11.11.m11.2.3.3.1.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.1"></plus><apply id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2"><times id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.1"></times><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.2.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.2">𝛼</ci><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.3.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.3">𝑔</ci><apply id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.1.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4">superscript</csymbol><ci id="A6.Thmlemma5.p1.11.11.m11.1.1.cmml" xref="A6.Thmlemma5.p1.11.11.m11.1.1">𝐚</ci><csymbol cd="latexml" id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.3.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.4.3">top</csymbol></apply><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.5.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.2.5">𝑔</ci><ci id="A6.Thmlemma5.p1.11.11.m11.2.2.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.2">𝐛</ci></apply><ci id="A6.Thmlemma5.p1.11.11.m11.2.3.3.3.cmml" xref="A6.Thmlemma5.p1.11.11.m11.2.3.3.3">𝛃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.11.11.m11.2c">\hat{\bm{y}}\triangleq\alpha g(\bm{a})^{\top}g(\bm{b})+\bm{\beta}</annotation></semantics></math> is <span id="A6.Thmlemma5.p1.14.14.1" class="ltx_text ltx_font_smallcaps">Maddness</span>’s approximation to <math id="A6.Thmlemma5.p1.12.12.m12.1" class="ltx_Math" alttext="\bm{a}^{\top}\bm{b}" display="inline"><semantics id="A6.Thmlemma5.p1.12.12.m12.1a"><mrow id="A6.Thmlemma5.p1.12.12.m12.1.1" xref="A6.Thmlemma5.p1.12.12.m12.1.1.cmml"><msup id="A6.Thmlemma5.p1.12.12.m12.1.1.2" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2.cmml"><mi id="A6.Thmlemma5.p1.12.12.m12.1.1.2.2" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2.2.cmml">𝐚</mi><mo id="A6.Thmlemma5.p1.12.12.m12.1.1.2.3" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma5.p1.12.12.m12.1.1.1" xref="A6.Thmlemma5.p1.12.12.m12.1.1.1.cmml">​</mo><mi id="A6.Thmlemma5.p1.12.12.m12.1.1.3" xref="A6.Thmlemma5.p1.12.12.m12.1.1.3.cmml">𝐛</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.12.12.m12.1b"><apply id="A6.Thmlemma5.p1.12.12.m12.1.1.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1"><times id="A6.Thmlemma5.p1.12.12.m12.1.1.1.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1.1"></times><apply id="A6.Thmlemma5.p1.12.12.m12.1.1.2.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2"><csymbol cd="ambiguous" id="A6.Thmlemma5.p1.12.12.m12.1.1.2.1.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2">superscript</csymbol><ci id="A6.Thmlemma5.p1.12.12.m12.1.1.2.2.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2.2">𝐚</ci><csymbol cd="latexml" id="A6.Thmlemma5.p1.12.12.m12.1.1.2.3.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1.2.3">top</csymbol></apply><ci id="A6.Thmlemma5.p1.12.12.m12.1.1.3.cmml" xref="A6.Thmlemma5.p1.12.12.m12.1.1.3">𝐛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.12.12.m12.1c">\bm{a}^{\top}\bm{b}</annotation></semantics></math>. <math id="A6.Thmlemma5.p1.13.13.m13.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="A6.Thmlemma5.p1.13.13.m13.1a"><mi id="A6.Thmlemma5.p1.13.13.m13.1.1" xref="A6.Thmlemma5.p1.13.13.m13.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.13.13.m13.1b"><ci id="A6.Thmlemma5.p1.13.13.m13.1.1.cmml" xref="A6.Thmlemma5.p1.13.13.m13.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.13.13.m13.1c">\alpha</annotation></semantics></math> and <math id="A6.Thmlemma5.p1.14.14.m14.1" class="ltx_Math" alttext="\bm{\beta}" display="inline"><semantics id="A6.Thmlemma5.p1.14.14.m14.1a"><mi id="A6.Thmlemma5.p1.14.14.m14.1.1" xref="A6.Thmlemma5.p1.14.14.m14.1.1.cmml">𝛃</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma5.p1.14.14.m14.1b"><ci id="A6.Thmlemma5.p1.14.14.m14.1.1.cmml" xref="A6.Thmlemma5.p1.14.14.m14.1.1">𝛃</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma5.p1.14.14.m14.1c">\bm{\beta}</annotation></semantics></math> are the scale and offsets used to quantize the lookup tables.
</span></p>
</div>
</div>
<div id="A6.SS2.8" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A6.SS2.8.p1" class="ltx_para">
<p id="A6.SS2.8.p1.3" class="ltx_p">If <span id="A6.SS2.8.p1.3.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.SS2.8.p1.3.1.1" class="ltx_text ltx_font_upright"> </span></span>had infinite-precision lookup tables, <math id="A6.SS2.8.p1.1.m1.1" class="ltx_Math" alttext="\hat{\bm{y}}" display="inline"><semantics id="A6.SS2.8.p1.1.m1.1a"><mover accent="true" id="A6.SS2.8.p1.1.m1.1.1" xref="A6.SS2.8.p1.1.m1.1.1.cmml"><mi id="A6.SS2.8.p1.1.m1.1.1.2" xref="A6.SS2.8.p1.1.m1.1.1.2.cmml">𝒚</mi><mo id="A6.SS2.8.p1.1.m1.1.1.1" xref="A6.SS2.8.p1.1.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.1.m1.1b"><apply id="A6.SS2.8.p1.1.m1.1.1.cmml" xref="A6.SS2.8.p1.1.m1.1.1"><ci id="A6.SS2.8.p1.1.m1.1.1.1.cmml" xref="A6.SS2.8.p1.1.m1.1.1.1">^</ci><ci id="A6.SS2.8.p1.1.m1.1.1.2.cmml" xref="A6.SS2.8.p1.1.m1.1.1.2">𝒚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.1.m1.1c">\hat{\bm{y}}</annotation></semantics></math> would exactly equal <math id="A6.SS2.8.p1.2.m2.1" class="ltx_Math" alttext="\bm{\hat{a}}^{\top}\bm{b}" display="inline"><semantics id="A6.SS2.8.p1.2.m2.1a"><mrow id="A6.SS2.8.p1.2.m2.1.1" xref="A6.SS2.8.p1.2.m2.1.1.cmml"><msup id="A6.SS2.8.p1.2.m2.1.1.2" xref="A6.SS2.8.p1.2.m2.1.1.2.cmml"><mover accent="true" id="A6.SS2.8.p1.2.m2.1.1.2.2" xref="A6.SS2.8.p1.2.m2.1.1.2.2.cmml"><mi id="A6.SS2.8.p1.2.m2.1.1.2.2.2" xref="A6.SS2.8.p1.2.m2.1.1.2.2.2.cmml">𝒂</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="A6.SS2.8.p1.2.m2.1.1.2.2.1" xref="A6.SS2.8.p1.2.m2.1.1.2.2.1.cmml">^</mo></mover><mo id="A6.SS2.8.p1.2.m2.1.1.2.3" xref="A6.SS2.8.p1.2.m2.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.2.m2.1.1.1" xref="A6.SS2.8.p1.2.m2.1.1.1.cmml">​</mo><mi id="A6.SS2.8.p1.2.m2.1.1.3" xref="A6.SS2.8.p1.2.m2.1.1.3.cmml">𝒃</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.2.m2.1b"><apply id="A6.SS2.8.p1.2.m2.1.1.cmml" xref="A6.SS2.8.p1.2.m2.1.1"><times id="A6.SS2.8.p1.2.m2.1.1.1.cmml" xref="A6.SS2.8.p1.2.m2.1.1.1"></times><apply id="A6.SS2.8.p1.2.m2.1.1.2.cmml" xref="A6.SS2.8.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A6.SS2.8.p1.2.m2.1.1.2.1.cmml" xref="A6.SS2.8.p1.2.m2.1.1.2">superscript</csymbol><apply id="A6.SS2.8.p1.2.m2.1.1.2.2.cmml" xref="A6.SS2.8.p1.2.m2.1.1.2.2"><ci id="A6.SS2.8.p1.2.m2.1.1.2.2.1.cmml" xref="A6.SS2.8.p1.2.m2.1.1.2.2.1">bold-^</ci><ci id="A6.SS2.8.p1.2.m2.1.1.2.2.2.cmml" xref="A6.SS2.8.p1.2.m2.1.1.2.2.2">𝒂</ci></apply><csymbol cd="latexml" id="A6.SS2.8.p1.2.m2.1.1.2.3.cmml" xref="A6.SS2.8.p1.2.m2.1.1.2.3">top</csymbol></apply><ci id="A6.SS2.8.p1.2.m2.1.1.3.cmml" xref="A6.SS2.8.p1.2.m2.1.1.3">𝒃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.2.m2.1c">\bm{\hat{a}}^{\top}\bm{b}</annotation></semantics></math>. We therefore need only bound the error introduced by the quantization. By Lemma <a href="#A6.Thmlemma4" title="Lemma F.4 (Bound on Maddness Embedding Norm). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.4</span></a>, <math id="A6.SS2.8.p1.3.m3.1" class="ltx_Math" alttext="\lVert\bm{\hat{a}}\rVert_{2}\leq\frac{C\sigma_{A}}{2\sqrt{\lambda}}" display="inline"><semantics id="A6.SS2.8.p1.3.m3.1a"><mrow id="A6.SS2.8.p1.3.m3.1.2" xref="A6.SS2.8.p1.3.m3.1.2.cmml"><msub id="A6.SS2.8.p1.3.m3.1.2.2" xref="A6.SS2.8.p1.3.m3.1.2.2.cmml"><mrow id="A6.SS2.8.p1.3.m3.1.2.2.2.2" xref="A6.SS2.8.p1.3.m3.1.2.2.2.1.cmml"><mo fence="true" rspace="0em" id="A6.SS2.8.p1.3.m3.1.2.2.2.2.1" xref="A6.SS2.8.p1.3.m3.1.2.2.2.1.1.cmml">∥</mo><mover accent="true" id="A6.SS2.8.p1.3.m3.1.1" xref="A6.SS2.8.p1.3.m3.1.1.cmml"><mi id="A6.SS2.8.p1.3.m3.1.1.2" xref="A6.SS2.8.p1.3.m3.1.1.2.cmml">𝒂</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="A6.SS2.8.p1.3.m3.1.1.1" xref="A6.SS2.8.p1.3.m3.1.1.1.cmml">^</mo></mover><mo fence="true" lspace="0em" id="A6.SS2.8.p1.3.m3.1.2.2.2.2.2" xref="A6.SS2.8.p1.3.m3.1.2.2.2.1.1.cmml">∥</mo></mrow><mn id="A6.SS2.8.p1.3.m3.1.2.2.3" xref="A6.SS2.8.p1.3.m3.1.2.2.3.cmml">2</mn></msub><mo id="A6.SS2.8.p1.3.m3.1.2.1" xref="A6.SS2.8.p1.3.m3.1.2.1.cmml">≤</mo><mfrac id="A6.SS2.8.p1.3.m3.1.2.3" xref="A6.SS2.8.p1.3.m3.1.2.3.cmml"><mrow id="A6.SS2.8.p1.3.m3.1.2.3.2" xref="A6.SS2.8.p1.3.m3.1.2.3.2.cmml"><mi id="A6.SS2.8.p1.3.m3.1.2.3.2.2" xref="A6.SS2.8.p1.3.m3.1.2.3.2.2.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.3.m3.1.2.3.2.1" xref="A6.SS2.8.p1.3.m3.1.2.3.2.1.cmml">​</mo><msub id="A6.SS2.8.p1.3.m3.1.2.3.2.3" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3.cmml"><mi id="A6.SS2.8.p1.3.m3.1.2.3.2.3.2" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3.2.cmml">σ</mi><mi id="A6.SS2.8.p1.3.m3.1.2.3.2.3.3" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3.3.cmml">A</mi></msub></mrow><mrow id="A6.SS2.8.p1.3.m3.1.2.3.3" xref="A6.SS2.8.p1.3.m3.1.2.3.3.cmml"><mn id="A6.SS2.8.p1.3.m3.1.2.3.3.2" xref="A6.SS2.8.p1.3.m3.1.2.3.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.3.m3.1.2.3.3.1" xref="A6.SS2.8.p1.3.m3.1.2.3.3.1.cmml">​</mo><msqrt id="A6.SS2.8.p1.3.m3.1.2.3.3.3" xref="A6.SS2.8.p1.3.m3.1.2.3.3.3.cmml"><mi id="A6.SS2.8.p1.3.m3.1.2.3.3.3.2" xref="A6.SS2.8.p1.3.m3.1.2.3.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.3.m3.1b"><apply id="A6.SS2.8.p1.3.m3.1.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2"><leq id="A6.SS2.8.p1.3.m3.1.2.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.1"></leq><apply id="A6.SS2.8.p1.3.m3.1.2.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2.2"><csymbol cd="ambiguous" id="A6.SS2.8.p1.3.m3.1.2.2.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.2">subscript</csymbol><apply id="A6.SS2.8.p1.3.m3.1.2.2.2.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.2.2.2"><csymbol cd="latexml" id="A6.SS2.8.p1.3.m3.1.2.2.2.1.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.2.2.2.1">delimited-∥∥</csymbol><apply id="A6.SS2.8.p1.3.m3.1.1.cmml" xref="A6.SS2.8.p1.3.m3.1.1"><ci id="A6.SS2.8.p1.3.m3.1.1.1.cmml" xref="A6.SS2.8.p1.3.m3.1.1.1">bold-^</ci><ci id="A6.SS2.8.p1.3.m3.1.1.2.cmml" xref="A6.SS2.8.p1.3.m3.1.1.2">𝒂</ci></apply></apply><cn type="integer" id="A6.SS2.8.p1.3.m3.1.2.2.3.cmml" xref="A6.SS2.8.p1.3.m3.1.2.2.3">2</cn></apply><apply id="A6.SS2.8.p1.3.m3.1.2.3.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3"><divide id="A6.SS2.8.p1.3.m3.1.2.3.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3"></divide><apply id="A6.SS2.8.p1.3.m3.1.2.3.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2"><times id="A6.SS2.8.p1.3.m3.1.2.3.2.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2.1"></times><ci id="A6.SS2.8.p1.3.m3.1.2.3.2.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2.2">𝐶</ci><apply id="A6.SS2.8.p1.3.m3.1.2.3.2.3.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3"><csymbol cd="ambiguous" id="A6.SS2.8.p1.3.m3.1.2.3.2.3.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3">subscript</csymbol><ci id="A6.SS2.8.p1.3.m3.1.2.3.2.3.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3.2">𝜎</ci><ci id="A6.SS2.8.p1.3.m3.1.2.3.2.3.3.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.2.3.3">𝐴</ci></apply></apply><apply id="A6.SS2.8.p1.3.m3.1.2.3.3.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.3"><times id="A6.SS2.8.p1.3.m3.1.2.3.3.1.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.3.1"></times><cn type="integer" id="A6.SS2.8.p1.3.m3.1.2.3.3.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.3.2">2</cn><apply id="A6.SS2.8.p1.3.m3.1.2.3.3.3.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.3.3"><root id="A6.SS2.8.p1.3.m3.1.2.3.3.3a.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.3.3"></root><ci id="A6.SS2.8.p1.3.m3.1.2.3.3.3.2.cmml" xref="A6.SS2.8.p1.3.m3.1.2.3.3.3.2">𝜆</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.3.m3.1c">\lVert\bm{\hat{a}}\rVert_{2}\leq\frac{C\sigma_{A}}{2\sqrt{\lambda}}</annotation></semantics></math>. This implies that</p>
<table id="A6.EGx25" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E38"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E38.m1.2" class="ltx_Math" alttext="\displaystyle\lVert\bm{\hat{a}}^{\top}\bm{b}\rVert\leq\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}" display="inline"><semantics id="A6.E38.m1.2a"><mrow id="A6.E38.m1.2.2" xref="A6.E38.m1.2.2.cmml"><mrow id="A6.E38.m1.2.2.1.1" xref="A6.E38.m1.2.2.1.2.cmml"><mo fence="true" rspace="0em" id="A6.E38.m1.2.2.1.1.2" xref="A6.E38.m1.2.2.1.2.1.cmml">∥</mo><mrow id="A6.E38.m1.2.2.1.1.1" xref="A6.E38.m1.2.2.1.1.1.cmml"><msup id="A6.E38.m1.2.2.1.1.1.2" xref="A6.E38.m1.2.2.1.1.1.2.cmml"><mover accent="true" id="A6.E38.m1.2.2.1.1.1.2.2" xref="A6.E38.m1.2.2.1.1.1.2.2.cmml"><mi id="A6.E38.m1.2.2.1.1.1.2.2.2" xref="A6.E38.m1.2.2.1.1.1.2.2.2.cmml">𝒂</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="A6.E38.m1.2.2.1.1.1.2.2.1" xref="A6.E38.m1.2.2.1.1.1.2.2.1.cmml">^</mo></mover><mo id="A6.E38.m1.2.2.1.1.1.2.3" xref="A6.E38.m1.2.2.1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E38.m1.2.2.1.1.1.1" xref="A6.E38.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A6.E38.m1.2.2.1.1.1.3" xref="A6.E38.m1.2.2.1.1.1.3.cmml">𝒃</mi></mrow><mo fence="true" lspace="0em" rspace="0.1389em" id="A6.E38.m1.2.2.1.1.3" xref="A6.E38.m1.2.2.1.2.1.cmml">∥</mo></mrow><mo lspace="0.1389em" id="A6.E38.m1.2.2.2" xref="A6.E38.m1.2.2.2.cmml">≤</mo><mstyle displaystyle="true" id="A6.E38.m1.1.1" xref="A6.E38.m1.1.1.cmml"><mfrac id="A6.E38.m1.1.1a" xref="A6.E38.m1.1.1.cmml"><mrow id="A6.E38.m1.1.1.1" xref="A6.E38.m1.1.1.1.cmml"><mi id="A6.E38.m1.1.1.1.3" xref="A6.E38.m1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.E38.m1.1.1.1.2" xref="A6.E38.m1.1.1.1.2.cmml">​</mo><msub id="A6.E38.m1.1.1.1.4" xref="A6.E38.m1.1.1.1.4.cmml"><mi id="A6.E38.m1.1.1.1.4.2" xref="A6.E38.m1.1.1.1.4.2.cmml">σ</mi><mi id="A6.E38.m1.1.1.1.4.3" xref="A6.E38.m1.1.1.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.E38.m1.1.1.1.2a" xref="A6.E38.m1.1.1.1.2.cmml">​</mo><msub id="A6.E38.m1.1.1.1.5" xref="A6.E38.m1.1.1.1.5.cmml"><mrow id="A6.E38.m1.1.1.1.5.2.2" xref="A6.E38.m1.1.1.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E38.m1.1.1.1.5.2.2.1" xref="A6.E38.m1.1.1.1.5.2.1.1.cmml">∥</mo><mi id="A6.E38.m1.1.1.1.1" xref="A6.E38.m1.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.E38.m1.1.1.1.5.2.2.2" xref="A6.E38.m1.1.1.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.E38.m1.1.1.1.5.3" xref="A6.E38.m1.1.1.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.E38.m1.1.1.3" xref="A6.E38.m1.1.1.3.cmml"><mn id="A6.E38.m1.1.1.3.2" xref="A6.E38.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E38.m1.1.1.3.1" xref="A6.E38.m1.1.1.3.1.cmml">​</mo><msqrt id="A6.E38.m1.1.1.3.3" xref="A6.E38.m1.1.1.3.3.cmml"><mi id="A6.E38.m1.1.1.3.3.2" xref="A6.E38.m1.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle></mrow><annotation-xml encoding="MathML-Content" id="A6.E38.m1.2b"><apply id="A6.E38.m1.2.2.cmml" xref="A6.E38.m1.2.2"><leq id="A6.E38.m1.2.2.2.cmml" xref="A6.E38.m1.2.2.2"></leq><apply id="A6.E38.m1.2.2.1.2.cmml" xref="A6.E38.m1.2.2.1.1"><csymbol cd="latexml" id="A6.E38.m1.2.2.1.2.1.cmml" xref="A6.E38.m1.2.2.1.1.2">delimited-∥∥</csymbol><apply id="A6.E38.m1.2.2.1.1.1.cmml" xref="A6.E38.m1.2.2.1.1.1"><times id="A6.E38.m1.2.2.1.1.1.1.cmml" xref="A6.E38.m1.2.2.1.1.1.1"></times><apply id="A6.E38.m1.2.2.1.1.1.2.cmml" xref="A6.E38.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="A6.E38.m1.2.2.1.1.1.2.1.cmml" xref="A6.E38.m1.2.2.1.1.1.2">superscript</csymbol><apply id="A6.E38.m1.2.2.1.1.1.2.2.cmml" xref="A6.E38.m1.2.2.1.1.1.2.2"><ci id="A6.E38.m1.2.2.1.1.1.2.2.1.cmml" xref="A6.E38.m1.2.2.1.1.1.2.2.1">bold-^</ci><ci id="A6.E38.m1.2.2.1.1.1.2.2.2.cmml" xref="A6.E38.m1.2.2.1.1.1.2.2.2">𝒂</ci></apply><csymbol cd="latexml" id="A6.E38.m1.2.2.1.1.1.2.3.cmml" xref="A6.E38.m1.2.2.1.1.1.2.3">top</csymbol></apply><ci id="A6.E38.m1.2.2.1.1.1.3.cmml" xref="A6.E38.m1.2.2.1.1.1.3">𝒃</ci></apply></apply><apply id="A6.E38.m1.1.1.cmml" xref="A6.E38.m1.1.1"><divide id="A6.E38.m1.1.1.2.cmml" xref="A6.E38.m1.1.1"></divide><apply id="A6.E38.m1.1.1.1.cmml" xref="A6.E38.m1.1.1.1"><times id="A6.E38.m1.1.1.1.2.cmml" xref="A6.E38.m1.1.1.1.2"></times><ci id="A6.E38.m1.1.1.1.3.cmml" xref="A6.E38.m1.1.1.1.3">𝐶</ci><apply id="A6.E38.m1.1.1.1.4.cmml" xref="A6.E38.m1.1.1.1.4"><csymbol cd="ambiguous" id="A6.E38.m1.1.1.1.4.1.cmml" xref="A6.E38.m1.1.1.1.4">subscript</csymbol><ci id="A6.E38.m1.1.1.1.4.2.cmml" xref="A6.E38.m1.1.1.1.4.2">𝜎</ci><ci id="A6.E38.m1.1.1.1.4.3.cmml" xref="A6.E38.m1.1.1.1.4.3">𝐴</ci></apply><apply id="A6.E38.m1.1.1.1.5.cmml" xref="A6.E38.m1.1.1.1.5"><csymbol cd="ambiguous" id="A6.E38.m1.1.1.1.5.1.cmml" xref="A6.E38.m1.1.1.1.5">subscript</csymbol><apply id="A6.E38.m1.1.1.1.5.2.1.cmml" xref="A6.E38.m1.1.1.1.5.2.2"><csymbol cd="latexml" id="A6.E38.m1.1.1.1.5.2.1.1.cmml" xref="A6.E38.m1.1.1.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.E38.m1.1.1.1.1.cmml" xref="A6.E38.m1.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.E38.m1.1.1.1.5.3.cmml" xref="A6.E38.m1.1.1.1.5.3">2</cn></apply></apply><apply id="A6.E38.m1.1.1.3.cmml" xref="A6.E38.m1.1.1.3"><times id="A6.E38.m1.1.1.3.1.cmml" xref="A6.E38.m1.1.1.3.1"></times><cn type="integer" id="A6.E38.m1.1.1.3.2.cmml" xref="A6.E38.m1.1.1.3.2">2</cn><apply id="A6.E38.m1.1.1.3.3.cmml" xref="A6.E38.m1.1.1.3.3"><root id="A6.E38.m1.1.1.3.3a.cmml" xref="A6.E38.m1.1.1.3.3"></root><ci id="A6.E38.m1.1.1.3.3.2.cmml" xref="A6.E38.m1.1.1.3.3.2">𝜆</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E38.m1.2c">\displaystyle\lVert\bm{\hat{a}}^{\top}\bm{b}\rVert\leq\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(38)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.8.p1.12" class="ltx_p">and therefore</p>
<table id="A6.EGx26" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E39"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E39.m1.3" class="ltx_Math" alttext="\displaystyle\frac{-C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}\leq\bm{\hat{a}}^{\top}\bm{b}\leq\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}." display="inline"><semantics id="A6.E39.m1.3a"><mrow id="A6.E39.m1.3.3.1" xref="A6.E39.m1.3.3.1.1.cmml"><mrow id="A6.E39.m1.3.3.1.1" xref="A6.E39.m1.3.3.1.1.cmml"><mstyle displaystyle="true" id="A6.E39.m1.1.1" xref="A6.E39.m1.1.1.cmml"><mfrac id="A6.E39.m1.1.1a" xref="A6.E39.m1.1.1.cmml"><mrow id="A6.E39.m1.1.1.1" xref="A6.E39.m1.1.1.1.cmml"><mo id="A6.E39.m1.1.1.1a" xref="A6.E39.m1.1.1.1.cmml">−</mo><mrow id="A6.E39.m1.1.1.1.3" xref="A6.E39.m1.1.1.1.3.cmml"><mi id="A6.E39.m1.1.1.1.3.2" xref="A6.E39.m1.1.1.1.3.2.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.E39.m1.1.1.1.3.1" xref="A6.E39.m1.1.1.1.3.1.cmml">​</mo><msub id="A6.E39.m1.1.1.1.3.3" xref="A6.E39.m1.1.1.1.3.3.cmml"><mi id="A6.E39.m1.1.1.1.3.3.2" xref="A6.E39.m1.1.1.1.3.3.2.cmml">σ</mi><mi id="A6.E39.m1.1.1.1.3.3.3" xref="A6.E39.m1.1.1.1.3.3.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.E39.m1.1.1.1.3.1a" xref="A6.E39.m1.1.1.1.3.1.cmml">​</mo><msub id="A6.E39.m1.1.1.1.3.4" xref="A6.E39.m1.1.1.1.3.4.cmml"><mrow id="A6.E39.m1.1.1.1.3.4.2.2" xref="A6.E39.m1.1.1.1.3.4.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E39.m1.1.1.1.3.4.2.2.1" xref="A6.E39.m1.1.1.1.3.4.2.1.1.cmml">∥</mo><mi id="A6.E39.m1.1.1.1.1" xref="A6.E39.m1.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.E39.m1.1.1.1.3.4.2.2.2" xref="A6.E39.m1.1.1.1.3.4.2.1.1.cmml">∥</mo></mrow><mn id="A6.E39.m1.1.1.1.3.4.3" xref="A6.E39.m1.1.1.1.3.4.3.cmml">2</mn></msub></mrow></mrow><mrow id="A6.E39.m1.1.1.3" xref="A6.E39.m1.1.1.3.cmml"><mn id="A6.E39.m1.1.1.3.2" xref="A6.E39.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E39.m1.1.1.3.1" xref="A6.E39.m1.1.1.3.1.cmml">​</mo><msqrt id="A6.E39.m1.1.1.3.3" xref="A6.E39.m1.1.1.3.3.cmml"><mi id="A6.E39.m1.1.1.3.3.2" xref="A6.E39.m1.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle><mo id="A6.E39.m1.3.3.1.1.2" xref="A6.E39.m1.3.3.1.1.2.cmml">≤</mo><mrow id="A6.E39.m1.3.3.1.1.3" xref="A6.E39.m1.3.3.1.1.3.cmml"><msup id="A6.E39.m1.3.3.1.1.3.2" xref="A6.E39.m1.3.3.1.1.3.2.cmml"><mover accent="true" id="A6.E39.m1.3.3.1.1.3.2.2" xref="A6.E39.m1.3.3.1.1.3.2.2.cmml"><mi id="A6.E39.m1.3.3.1.1.3.2.2.2" xref="A6.E39.m1.3.3.1.1.3.2.2.2.cmml">𝒂</mi><mo class="ltx_mathvariant_bold" mathvariant="bold" id="A6.E39.m1.3.3.1.1.3.2.2.1" xref="A6.E39.m1.3.3.1.1.3.2.2.1.cmml">^</mo></mover><mo id="A6.E39.m1.3.3.1.1.3.2.3" xref="A6.E39.m1.3.3.1.1.3.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E39.m1.3.3.1.1.3.1" xref="A6.E39.m1.3.3.1.1.3.1.cmml">​</mo><mi id="A6.E39.m1.3.3.1.1.3.3" xref="A6.E39.m1.3.3.1.1.3.3.cmml">𝒃</mi></mrow><mo id="A6.E39.m1.3.3.1.1.4" xref="A6.E39.m1.3.3.1.1.4.cmml">≤</mo><mstyle displaystyle="true" id="A6.E39.m1.2.2" xref="A6.E39.m1.2.2.cmml"><mfrac id="A6.E39.m1.2.2a" xref="A6.E39.m1.2.2.cmml"><mrow id="A6.E39.m1.2.2.1" xref="A6.E39.m1.2.2.1.cmml"><mi id="A6.E39.m1.2.2.1.3" xref="A6.E39.m1.2.2.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.E39.m1.2.2.1.2" xref="A6.E39.m1.2.2.1.2.cmml">​</mo><msub id="A6.E39.m1.2.2.1.4" xref="A6.E39.m1.2.2.1.4.cmml"><mi id="A6.E39.m1.2.2.1.4.2" xref="A6.E39.m1.2.2.1.4.2.cmml">σ</mi><mi id="A6.E39.m1.2.2.1.4.3" xref="A6.E39.m1.2.2.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.E39.m1.2.2.1.2a" xref="A6.E39.m1.2.2.1.2.cmml">​</mo><msub id="A6.E39.m1.2.2.1.5" xref="A6.E39.m1.2.2.1.5.cmml"><mrow id="A6.E39.m1.2.2.1.5.2.2" xref="A6.E39.m1.2.2.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E39.m1.2.2.1.5.2.2.1" xref="A6.E39.m1.2.2.1.5.2.1.1.cmml">∥</mo><mi id="A6.E39.m1.2.2.1.1" xref="A6.E39.m1.2.2.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.E39.m1.2.2.1.5.2.2.2" xref="A6.E39.m1.2.2.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.E39.m1.2.2.1.5.3" xref="A6.E39.m1.2.2.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.E39.m1.2.2.3" xref="A6.E39.m1.2.2.3.cmml"><mn id="A6.E39.m1.2.2.3.2" xref="A6.E39.m1.2.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E39.m1.2.2.3.1" xref="A6.E39.m1.2.2.3.1.cmml">​</mo><msqrt id="A6.E39.m1.2.2.3.3" xref="A6.E39.m1.2.2.3.3.cmml"><mi id="A6.E39.m1.2.2.3.3.2" xref="A6.E39.m1.2.2.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle></mrow><mo lspace="0em" id="A6.E39.m1.3.3.1.2" xref="A6.E39.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.E39.m1.3b"><apply id="A6.E39.m1.3.3.1.1.cmml" xref="A6.E39.m1.3.3.1"><and id="A6.E39.m1.3.3.1.1a.cmml" xref="A6.E39.m1.3.3.1"></and><apply id="A6.E39.m1.3.3.1.1b.cmml" xref="A6.E39.m1.3.3.1"><leq id="A6.E39.m1.3.3.1.1.2.cmml" xref="A6.E39.m1.3.3.1.1.2"></leq><apply id="A6.E39.m1.1.1.cmml" xref="A6.E39.m1.1.1"><divide id="A6.E39.m1.1.1.2.cmml" xref="A6.E39.m1.1.1"></divide><apply id="A6.E39.m1.1.1.1.cmml" xref="A6.E39.m1.1.1.1"><minus id="A6.E39.m1.1.1.1.2.cmml" xref="A6.E39.m1.1.1.1"></minus><apply id="A6.E39.m1.1.1.1.3.cmml" xref="A6.E39.m1.1.1.1.3"><times id="A6.E39.m1.1.1.1.3.1.cmml" xref="A6.E39.m1.1.1.1.3.1"></times><ci id="A6.E39.m1.1.1.1.3.2.cmml" xref="A6.E39.m1.1.1.1.3.2">𝐶</ci><apply id="A6.E39.m1.1.1.1.3.3.cmml" xref="A6.E39.m1.1.1.1.3.3"><csymbol cd="ambiguous" id="A6.E39.m1.1.1.1.3.3.1.cmml" xref="A6.E39.m1.1.1.1.3.3">subscript</csymbol><ci id="A6.E39.m1.1.1.1.3.3.2.cmml" xref="A6.E39.m1.1.1.1.3.3.2">𝜎</ci><ci id="A6.E39.m1.1.1.1.3.3.3.cmml" xref="A6.E39.m1.1.1.1.3.3.3">𝐴</ci></apply><apply id="A6.E39.m1.1.1.1.3.4.cmml" xref="A6.E39.m1.1.1.1.3.4"><csymbol cd="ambiguous" id="A6.E39.m1.1.1.1.3.4.1.cmml" xref="A6.E39.m1.1.1.1.3.4">subscript</csymbol><apply id="A6.E39.m1.1.1.1.3.4.2.1.cmml" xref="A6.E39.m1.1.1.1.3.4.2.2"><csymbol cd="latexml" id="A6.E39.m1.1.1.1.3.4.2.1.1.cmml" xref="A6.E39.m1.1.1.1.3.4.2.2.1">delimited-∥∥</csymbol><ci id="A6.E39.m1.1.1.1.1.cmml" xref="A6.E39.m1.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.E39.m1.1.1.1.3.4.3.cmml" xref="A6.E39.m1.1.1.1.3.4.3">2</cn></apply></apply></apply><apply id="A6.E39.m1.1.1.3.cmml" xref="A6.E39.m1.1.1.3"><times id="A6.E39.m1.1.1.3.1.cmml" xref="A6.E39.m1.1.1.3.1"></times><cn type="integer" id="A6.E39.m1.1.1.3.2.cmml" xref="A6.E39.m1.1.1.3.2">2</cn><apply id="A6.E39.m1.1.1.3.3.cmml" xref="A6.E39.m1.1.1.3.3"><root id="A6.E39.m1.1.1.3.3a.cmml" xref="A6.E39.m1.1.1.3.3"></root><ci id="A6.E39.m1.1.1.3.3.2.cmml" xref="A6.E39.m1.1.1.3.3.2">𝜆</ci></apply></apply></apply><apply id="A6.E39.m1.3.3.1.1.3.cmml" xref="A6.E39.m1.3.3.1.1.3"><times id="A6.E39.m1.3.3.1.1.3.1.cmml" xref="A6.E39.m1.3.3.1.1.3.1"></times><apply id="A6.E39.m1.3.3.1.1.3.2.cmml" xref="A6.E39.m1.3.3.1.1.3.2"><csymbol cd="ambiguous" id="A6.E39.m1.3.3.1.1.3.2.1.cmml" xref="A6.E39.m1.3.3.1.1.3.2">superscript</csymbol><apply id="A6.E39.m1.3.3.1.1.3.2.2.cmml" xref="A6.E39.m1.3.3.1.1.3.2.2"><ci id="A6.E39.m1.3.3.1.1.3.2.2.1.cmml" xref="A6.E39.m1.3.3.1.1.3.2.2.1">bold-^</ci><ci id="A6.E39.m1.3.3.1.1.3.2.2.2.cmml" xref="A6.E39.m1.3.3.1.1.3.2.2.2">𝒂</ci></apply><csymbol cd="latexml" id="A6.E39.m1.3.3.1.1.3.2.3.cmml" xref="A6.E39.m1.3.3.1.1.3.2.3">top</csymbol></apply><ci id="A6.E39.m1.3.3.1.1.3.3.cmml" xref="A6.E39.m1.3.3.1.1.3.3">𝒃</ci></apply></apply><apply id="A6.E39.m1.3.3.1.1c.cmml" xref="A6.E39.m1.3.3.1"><leq id="A6.E39.m1.3.3.1.1.4.cmml" xref="A6.E39.m1.3.3.1.1.4"></leq><share href="#A6.E39.m1.3.3.1.1.3.cmml" id="A6.E39.m1.3.3.1.1d.cmml" xref="A6.E39.m1.3.3.1"></share><apply id="A6.E39.m1.2.2.cmml" xref="A6.E39.m1.2.2"><divide id="A6.E39.m1.2.2.2.cmml" xref="A6.E39.m1.2.2"></divide><apply id="A6.E39.m1.2.2.1.cmml" xref="A6.E39.m1.2.2.1"><times id="A6.E39.m1.2.2.1.2.cmml" xref="A6.E39.m1.2.2.1.2"></times><ci id="A6.E39.m1.2.2.1.3.cmml" xref="A6.E39.m1.2.2.1.3">𝐶</ci><apply id="A6.E39.m1.2.2.1.4.cmml" xref="A6.E39.m1.2.2.1.4"><csymbol cd="ambiguous" id="A6.E39.m1.2.2.1.4.1.cmml" xref="A6.E39.m1.2.2.1.4">subscript</csymbol><ci id="A6.E39.m1.2.2.1.4.2.cmml" xref="A6.E39.m1.2.2.1.4.2">𝜎</ci><ci id="A6.E39.m1.2.2.1.4.3.cmml" xref="A6.E39.m1.2.2.1.4.3">𝐴</ci></apply><apply id="A6.E39.m1.2.2.1.5.cmml" xref="A6.E39.m1.2.2.1.5"><csymbol cd="ambiguous" id="A6.E39.m1.2.2.1.5.1.cmml" xref="A6.E39.m1.2.2.1.5">subscript</csymbol><apply id="A6.E39.m1.2.2.1.5.2.1.cmml" xref="A6.E39.m1.2.2.1.5.2.2"><csymbol cd="latexml" id="A6.E39.m1.2.2.1.5.2.1.1.cmml" xref="A6.E39.m1.2.2.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.E39.m1.2.2.1.1.cmml" xref="A6.E39.m1.2.2.1.1">𝒃</ci></apply><cn type="integer" id="A6.E39.m1.2.2.1.5.3.cmml" xref="A6.E39.m1.2.2.1.5.3">2</cn></apply></apply><apply id="A6.E39.m1.2.2.3.cmml" xref="A6.E39.m1.2.2.3"><times id="A6.E39.m1.2.2.3.1.cmml" xref="A6.E39.m1.2.2.3.1"></times><cn type="integer" id="A6.E39.m1.2.2.3.2.cmml" xref="A6.E39.m1.2.2.3.2">2</cn><apply id="A6.E39.m1.2.2.3.3.cmml" xref="A6.E39.m1.2.2.3.3"><root id="A6.E39.m1.2.2.3.3a.cmml" xref="A6.E39.m1.2.2.3.3"></root><ci id="A6.E39.m1.2.2.3.3.2.cmml" xref="A6.E39.m1.2.2.3.3.2">𝜆</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E39.m1.3c">\displaystyle\frac{-C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}\leq\bm{\hat{a}}^{\top}\bm{b}\leq\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(39)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.8.p1.11" class="ltx_p">For each of the <math id="A6.SS2.8.p1.4.m1.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.SS2.8.p1.4.m1.1a"><mi id="A6.SS2.8.p1.4.m1.1.1" xref="A6.SS2.8.p1.4.m1.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.4.m1.1b"><ci id="A6.SS2.8.p1.4.m1.1.1.cmml" xref="A6.SS2.8.p1.4.m1.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.4.m1.1c">C</annotation></semantics></math> codebooks, this means that the value to be quantized lies in the interval <math id="A6.SS2.8.p1.5.m2.2" class="ltx_Math" alttext="[\frac{-\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}},\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}]" display="inline"><semantics id="A6.SS2.8.p1.5.m2.2a"><mrow id="A6.SS2.8.p1.5.m2.2.3.2" xref="A6.SS2.8.p1.5.m2.2.3.1.cmml"><mo stretchy="false" id="A6.SS2.8.p1.5.m2.2.3.2.1" xref="A6.SS2.8.p1.5.m2.2.3.1.cmml">[</mo><mfrac id="A6.SS2.8.p1.5.m2.1.1" xref="A6.SS2.8.p1.5.m2.1.1.cmml"><mrow id="A6.SS2.8.p1.5.m2.1.1.1" xref="A6.SS2.8.p1.5.m2.1.1.1.cmml"><mo id="A6.SS2.8.p1.5.m2.1.1.1a" xref="A6.SS2.8.p1.5.m2.1.1.1.cmml">−</mo><mrow id="A6.SS2.8.p1.5.m2.1.1.1.3" xref="A6.SS2.8.p1.5.m2.1.1.1.3.cmml"><msub id="A6.SS2.8.p1.5.m2.1.1.1.3.2" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2.cmml"><mi id="A6.SS2.8.p1.5.m2.1.1.1.3.2.2" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2.2.cmml">σ</mi><mi id="A6.SS2.8.p1.5.m2.1.1.1.3.2.3" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.5.m2.1.1.1.3.1" xref="A6.SS2.8.p1.5.m2.1.1.1.3.1.cmml">​</mo><msub id="A6.SS2.8.p1.5.m2.1.1.1.3.3" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.cmml"><mrow id="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.2" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.2.1" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.1.1.cmml">∥</mo><mi id="A6.SS2.8.p1.5.m2.1.1.1.1" xref="A6.SS2.8.p1.5.m2.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.2.2" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.1.1.cmml">∥</mo></mrow><mn id="A6.SS2.8.p1.5.m2.1.1.1.3.3.3" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.3.cmml">2</mn></msub></mrow></mrow><mrow id="A6.SS2.8.p1.5.m2.1.1.3" xref="A6.SS2.8.p1.5.m2.1.1.3.cmml"><mn id="A6.SS2.8.p1.5.m2.1.1.3.2" xref="A6.SS2.8.p1.5.m2.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.5.m2.1.1.3.1" xref="A6.SS2.8.p1.5.m2.1.1.3.1.cmml">​</mo><msqrt id="A6.SS2.8.p1.5.m2.1.1.3.3" xref="A6.SS2.8.p1.5.m2.1.1.3.3.cmml"><mi id="A6.SS2.8.p1.5.m2.1.1.3.3.2" xref="A6.SS2.8.p1.5.m2.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac><mo id="A6.SS2.8.p1.5.m2.2.3.2.2" xref="A6.SS2.8.p1.5.m2.2.3.1.cmml">,</mo><mfrac id="A6.SS2.8.p1.5.m2.2.2" xref="A6.SS2.8.p1.5.m2.2.2.cmml"><mrow id="A6.SS2.8.p1.5.m2.2.2.1" xref="A6.SS2.8.p1.5.m2.2.2.1.cmml"><msub id="A6.SS2.8.p1.5.m2.2.2.1.3" xref="A6.SS2.8.p1.5.m2.2.2.1.3.cmml"><mi id="A6.SS2.8.p1.5.m2.2.2.1.3.2" xref="A6.SS2.8.p1.5.m2.2.2.1.3.2.cmml">σ</mi><mi id="A6.SS2.8.p1.5.m2.2.2.1.3.3" xref="A6.SS2.8.p1.5.m2.2.2.1.3.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.5.m2.2.2.1.2" xref="A6.SS2.8.p1.5.m2.2.2.1.2.cmml">​</mo><msub id="A6.SS2.8.p1.5.m2.2.2.1.4" xref="A6.SS2.8.p1.5.m2.2.2.1.4.cmml"><mrow id="A6.SS2.8.p1.5.m2.2.2.1.4.2.2" xref="A6.SS2.8.p1.5.m2.2.2.1.4.2.1.cmml"><mo fence="true" rspace="0em" id="A6.SS2.8.p1.5.m2.2.2.1.4.2.2.1" xref="A6.SS2.8.p1.5.m2.2.2.1.4.2.1.1.cmml">∥</mo><mi id="A6.SS2.8.p1.5.m2.2.2.1.1" xref="A6.SS2.8.p1.5.m2.2.2.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.SS2.8.p1.5.m2.2.2.1.4.2.2.2" xref="A6.SS2.8.p1.5.m2.2.2.1.4.2.1.1.cmml">∥</mo></mrow><mn id="A6.SS2.8.p1.5.m2.2.2.1.4.3" xref="A6.SS2.8.p1.5.m2.2.2.1.4.3.cmml">2</mn></msub></mrow><mrow id="A6.SS2.8.p1.5.m2.2.2.3" xref="A6.SS2.8.p1.5.m2.2.2.3.cmml"><mn id="A6.SS2.8.p1.5.m2.2.2.3.2" xref="A6.SS2.8.p1.5.m2.2.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.5.m2.2.2.3.1" xref="A6.SS2.8.p1.5.m2.2.2.3.1.cmml">​</mo><msqrt id="A6.SS2.8.p1.5.m2.2.2.3.3" xref="A6.SS2.8.p1.5.m2.2.2.3.3.cmml"><mi id="A6.SS2.8.p1.5.m2.2.2.3.3.2" xref="A6.SS2.8.p1.5.m2.2.2.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac><mo stretchy="false" id="A6.SS2.8.p1.5.m2.2.3.2.3" xref="A6.SS2.8.p1.5.m2.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.5.m2.2b"><interval closure="closed" id="A6.SS2.8.p1.5.m2.2.3.1.cmml" xref="A6.SS2.8.p1.5.m2.2.3.2"><apply id="A6.SS2.8.p1.5.m2.1.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1"><divide id="A6.SS2.8.p1.5.m2.1.1.2.cmml" xref="A6.SS2.8.p1.5.m2.1.1"></divide><apply id="A6.SS2.8.p1.5.m2.1.1.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1"><minus id="A6.SS2.8.p1.5.m2.1.1.1.2.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1"></minus><apply id="A6.SS2.8.p1.5.m2.1.1.1.3.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3"><times id="A6.SS2.8.p1.5.m2.1.1.1.3.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.1"></times><apply id="A6.SS2.8.p1.5.m2.1.1.1.3.2.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2"><csymbol cd="ambiguous" id="A6.SS2.8.p1.5.m2.1.1.1.3.2.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2">subscript</csymbol><ci id="A6.SS2.8.p1.5.m2.1.1.1.3.2.2.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2.2">𝜎</ci><ci id="A6.SS2.8.p1.5.m2.1.1.1.3.2.3.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.2.3">𝐴</ci></apply><apply id="A6.SS2.8.p1.5.m2.1.1.1.3.3.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3"><csymbol cd="ambiguous" id="A6.SS2.8.p1.5.m2.1.1.1.3.3.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3">subscript</csymbol><apply id="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.2"><csymbol cd="latexml" id="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.1.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.2.2.1">delimited-∥∥</csymbol><ci id="A6.SS2.8.p1.5.m2.1.1.1.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.SS2.8.p1.5.m2.1.1.1.3.3.3.cmml" xref="A6.SS2.8.p1.5.m2.1.1.1.3.3.3">2</cn></apply></apply></apply><apply id="A6.SS2.8.p1.5.m2.1.1.3.cmml" xref="A6.SS2.8.p1.5.m2.1.1.3"><times id="A6.SS2.8.p1.5.m2.1.1.3.1.cmml" xref="A6.SS2.8.p1.5.m2.1.1.3.1"></times><cn type="integer" id="A6.SS2.8.p1.5.m2.1.1.3.2.cmml" xref="A6.SS2.8.p1.5.m2.1.1.3.2">2</cn><apply id="A6.SS2.8.p1.5.m2.1.1.3.3.cmml" xref="A6.SS2.8.p1.5.m2.1.1.3.3"><root id="A6.SS2.8.p1.5.m2.1.1.3.3a.cmml" xref="A6.SS2.8.p1.5.m2.1.1.3.3"></root><ci id="A6.SS2.8.p1.5.m2.1.1.3.3.2.cmml" xref="A6.SS2.8.p1.5.m2.1.1.3.3.2">𝜆</ci></apply></apply></apply><apply id="A6.SS2.8.p1.5.m2.2.2.cmml" xref="A6.SS2.8.p1.5.m2.2.2"><divide id="A6.SS2.8.p1.5.m2.2.2.2.cmml" xref="A6.SS2.8.p1.5.m2.2.2"></divide><apply id="A6.SS2.8.p1.5.m2.2.2.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1"><times id="A6.SS2.8.p1.5.m2.2.2.1.2.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.2"></times><apply id="A6.SS2.8.p1.5.m2.2.2.1.3.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.3"><csymbol cd="ambiguous" id="A6.SS2.8.p1.5.m2.2.2.1.3.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.3">subscript</csymbol><ci id="A6.SS2.8.p1.5.m2.2.2.1.3.2.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.3.2">𝜎</ci><ci id="A6.SS2.8.p1.5.m2.2.2.1.3.3.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.3.3">𝐴</ci></apply><apply id="A6.SS2.8.p1.5.m2.2.2.1.4.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.4"><csymbol cd="ambiguous" id="A6.SS2.8.p1.5.m2.2.2.1.4.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.4">subscript</csymbol><apply id="A6.SS2.8.p1.5.m2.2.2.1.4.2.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.4.2.2"><csymbol cd="latexml" id="A6.SS2.8.p1.5.m2.2.2.1.4.2.1.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.4.2.2.1">delimited-∥∥</csymbol><ci id="A6.SS2.8.p1.5.m2.2.2.1.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.1">𝒃</ci></apply><cn type="integer" id="A6.SS2.8.p1.5.m2.2.2.1.4.3.cmml" xref="A6.SS2.8.p1.5.m2.2.2.1.4.3">2</cn></apply></apply><apply id="A6.SS2.8.p1.5.m2.2.2.3.cmml" xref="A6.SS2.8.p1.5.m2.2.2.3"><times id="A6.SS2.8.p1.5.m2.2.2.3.1.cmml" xref="A6.SS2.8.p1.5.m2.2.2.3.1"></times><cn type="integer" id="A6.SS2.8.p1.5.m2.2.2.3.2.cmml" xref="A6.SS2.8.p1.5.m2.2.2.3.2">2</cn><apply id="A6.SS2.8.p1.5.m2.2.2.3.3.cmml" xref="A6.SS2.8.p1.5.m2.2.2.3.3"><root id="A6.SS2.8.p1.5.m2.2.2.3.3a.cmml" xref="A6.SS2.8.p1.5.m2.2.2.3.3"></root><ci id="A6.SS2.8.p1.5.m2.2.2.3.3.2.cmml" xref="A6.SS2.8.p1.5.m2.2.2.3.3.2">𝜆</ci></apply></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.5.m2.2c">[\frac{-\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}},\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}]</annotation></semantics></math> of width <math id="A6.SS2.8.p1.6.m3.1" class="ltx_Math" alttext="\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{\sqrt{\lambda}}" display="inline"><semantics id="A6.SS2.8.p1.6.m3.1a"><mfrac id="A6.SS2.8.p1.6.m3.1.1" xref="A6.SS2.8.p1.6.m3.1.1.cmml"><mrow id="A6.SS2.8.p1.6.m3.1.1.1" xref="A6.SS2.8.p1.6.m3.1.1.1.cmml"><msub id="A6.SS2.8.p1.6.m3.1.1.1.3" xref="A6.SS2.8.p1.6.m3.1.1.1.3.cmml"><mi id="A6.SS2.8.p1.6.m3.1.1.1.3.2" xref="A6.SS2.8.p1.6.m3.1.1.1.3.2.cmml">σ</mi><mi id="A6.SS2.8.p1.6.m3.1.1.1.3.3" xref="A6.SS2.8.p1.6.m3.1.1.1.3.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.6.m3.1.1.1.2" xref="A6.SS2.8.p1.6.m3.1.1.1.2.cmml">​</mo><msub id="A6.SS2.8.p1.6.m3.1.1.1.4" xref="A6.SS2.8.p1.6.m3.1.1.1.4.cmml"><mrow id="A6.SS2.8.p1.6.m3.1.1.1.4.2.2" xref="A6.SS2.8.p1.6.m3.1.1.1.4.2.1.cmml"><mo fence="true" rspace="0em" id="A6.SS2.8.p1.6.m3.1.1.1.4.2.2.1" xref="A6.SS2.8.p1.6.m3.1.1.1.4.2.1.1.cmml">∥</mo><mi id="A6.SS2.8.p1.6.m3.1.1.1.1" xref="A6.SS2.8.p1.6.m3.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.SS2.8.p1.6.m3.1.1.1.4.2.2.2" xref="A6.SS2.8.p1.6.m3.1.1.1.4.2.1.1.cmml">∥</mo></mrow><mn id="A6.SS2.8.p1.6.m3.1.1.1.4.3" xref="A6.SS2.8.p1.6.m3.1.1.1.4.3.cmml">2</mn></msub></mrow><msqrt id="A6.SS2.8.p1.6.m3.1.1.3" xref="A6.SS2.8.p1.6.m3.1.1.3.cmml"><mi id="A6.SS2.8.p1.6.m3.1.1.3.2" xref="A6.SS2.8.p1.6.m3.1.1.3.2.cmml">λ</mi></msqrt></mfrac><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.6.m3.1b"><apply id="A6.SS2.8.p1.6.m3.1.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1"><divide id="A6.SS2.8.p1.6.m3.1.1.2.cmml" xref="A6.SS2.8.p1.6.m3.1.1"></divide><apply id="A6.SS2.8.p1.6.m3.1.1.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1"><times id="A6.SS2.8.p1.6.m3.1.1.1.2.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.2"></times><apply id="A6.SS2.8.p1.6.m3.1.1.1.3.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.3"><csymbol cd="ambiguous" id="A6.SS2.8.p1.6.m3.1.1.1.3.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.3">subscript</csymbol><ci id="A6.SS2.8.p1.6.m3.1.1.1.3.2.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.3.2">𝜎</ci><ci id="A6.SS2.8.p1.6.m3.1.1.1.3.3.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.3.3">𝐴</ci></apply><apply id="A6.SS2.8.p1.6.m3.1.1.1.4.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.4"><csymbol cd="ambiguous" id="A6.SS2.8.p1.6.m3.1.1.1.4.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.4">subscript</csymbol><apply id="A6.SS2.8.p1.6.m3.1.1.1.4.2.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.4.2.2"><csymbol cd="latexml" id="A6.SS2.8.p1.6.m3.1.1.1.4.2.1.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.4.2.2.1">delimited-∥∥</csymbol><ci id="A6.SS2.8.p1.6.m3.1.1.1.1.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.SS2.8.p1.6.m3.1.1.1.4.3.cmml" xref="A6.SS2.8.p1.6.m3.1.1.1.4.3">2</cn></apply></apply><apply id="A6.SS2.8.p1.6.m3.1.1.3.cmml" xref="A6.SS2.8.p1.6.m3.1.1.3"><root id="A6.SS2.8.p1.6.m3.1.1.3a.cmml" xref="A6.SS2.8.p1.6.m3.1.1.3"></root><ci id="A6.SS2.8.p1.6.m3.1.1.3.2.cmml" xref="A6.SS2.8.p1.6.m3.1.1.3.2">𝜆</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.6.m3.1c">\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{\sqrt{\lambda}}</annotation></semantics></math>.
Because <span id="A6.SS2.8.p1.11.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.SS2.8.p1.11.1.1" class="ltx_text ltx_font_upright"> </span></span>quantizes the lookup tables such that largest and smallest entries for any row of <math id="A6.SS2.8.p1.7.m4.1" class="ltx_Math" alttext="\bm{P}" display="inline"><semantics id="A6.SS2.8.p1.7.m4.1a"><mi id="A6.SS2.8.p1.7.m4.1.1" xref="A6.SS2.8.p1.7.m4.1.1.cmml">𝑷</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.7.m4.1b"><ci id="A6.SS2.8.p1.7.m4.1.1.cmml" xref="A6.SS2.8.p1.7.m4.1.1">𝑷</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.7.m4.1c">\bm{P}</annotation></semantics></math> are linearly mapped to <math id="A6.SS2.8.p1.8.m5.1" class="ltx_Math" alttext="255.5" display="inline"><semantics id="A6.SS2.8.p1.8.m5.1a"><mn id="A6.SS2.8.p1.8.m5.1.1" xref="A6.SS2.8.p1.8.m5.1.1.cmml">255.5</mn><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.8.m5.1b"><cn type="float" id="A6.SS2.8.p1.8.m5.1.1.cmml" xref="A6.SS2.8.p1.8.m5.1.1">255.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.8.m5.1c">255.5</annotation></semantics></math> and <math id="A6.SS2.8.p1.9.m6.1" class="ltx_Math" alttext="-0.5" display="inline"><semantics id="A6.SS2.8.p1.9.m6.1a"><mrow id="A6.SS2.8.p1.9.m6.1.1" xref="A6.SS2.8.p1.9.m6.1.1.cmml"><mo id="A6.SS2.8.p1.9.m6.1.1a" xref="A6.SS2.8.p1.9.m6.1.1.cmml">−</mo><mn id="A6.SS2.8.p1.9.m6.1.1.2" xref="A6.SS2.8.p1.9.m6.1.1.2.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.9.m6.1b"><apply id="A6.SS2.8.p1.9.m6.1.1.cmml" xref="A6.SS2.8.p1.9.m6.1.1"><minus id="A6.SS2.8.p1.9.m6.1.1.1.cmml" xref="A6.SS2.8.p1.9.m6.1.1"></minus><cn type="float" id="A6.SS2.8.p1.9.m6.1.1.2.cmml" xref="A6.SS2.8.p1.9.m6.1.1.2">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.9.m6.1c">-0.5</annotation></semantics></math>,<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>We use <math id="footnote4.m1.1" class="ltx_Math" alttext="255.5" display="inline"><semantics id="footnote4.m1.1b"><mn id="footnote4.m1.1.1" xref="footnote4.m1.1.1.cmml">255.5</mn><annotation-xml encoding="MathML-Content" id="footnote4.m1.1c"><cn type="float" id="footnote4.m1.1.1.cmml" xref="footnote4.m1.1.1">255.5</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote4.m1.1d">255.5</annotation></semantics></math> and <math id="footnote4.m2.1" class="ltx_Math" alttext="-0.5" display="inline"><semantics id="footnote4.m2.1b"><mrow id="footnote4.m2.1.1" xref="footnote4.m2.1.1.cmml"><mo id="footnote4.m2.1.1b" xref="footnote4.m2.1.1.cmml">−</mo><mn id="footnote4.m2.1.1.2" xref="footnote4.m2.1.1.2.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote4.m2.1c"><apply id="footnote4.m2.1.1.cmml" xref="footnote4.m2.1.1"><minus id="footnote4.m2.1.1.1.cmml" xref="footnote4.m2.1.1"></minus><cn type="float" id="footnote4.m2.1.1.2.cmml" xref="footnote4.m2.1.1.2">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote4.m2.1d">-0.5</annotation></semantics></math> rather than <math id="footnote4.m3.1" class="ltx_Math" alttext="255" display="inline"><semantics id="footnote4.m3.1b"><mn id="footnote4.m3.1.1" xref="footnote4.m3.1.1.cmml">255</mn><annotation-xml encoding="MathML-Content" id="footnote4.m3.1c"><cn type="integer" id="footnote4.m3.1.1.cmml" xref="footnote4.m3.1.1">255</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote4.m3.1d">255</annotation></semantics></math> and <math id="footnote4.m4.1" class="ltx_Math" alttext="0" display="inline"><semantics id="footnote4.m4.1b"><mn id="footnote4.m4.1.1" xref="footnote4.m4.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="footnote4.m4.1c"><cn type="integer" id="footnote4.m4.1.1.cmml" xref="footnote4.m4.1.1">0</cn></annotation-xml></semantics></math> because the latter only guarantees that a point is within <math id="footnote4.m5.1" class="ltx_Math" alttext="1/510" display="inline"><semantics id="footnote4.m5.1b"><mrow id="footnote4.m5.1.1" xref="footnote4.m5.1.1.cmml"><mn id="footnote4.m5.1.1.2" xref="footnote4.m5.1.1.2.cmml">1</mn><mo id="footnote4.m5.1.1.1" xref="footnote4.m5.1.1.1.cmml">/</mo><mn id="footnote4.m5.1.1.3" xref="footnote4.m5.1.1.3.cmml">510</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote4.m5.1c"><apply id="footnote4.m5.1.1.cmml" xref="footnote4.m5.1.1"><divide id="footnote4.m5.1.1.1.cmml" xref="footnote4.m5.1.1.1"></divide><cn type="integer" id="footnote4.m5.1.1.2.cmml" xref="footnote4.m5.1.1.2">1</cn><cn type="integer" id="footnote4.m5.1.1.3.cmml" xref="footnote4.m5.1.1.3">510</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote4.m5.1d">1/510</annotation></semantics></math> of the interval width, not <math id="footnote4.m6.1" class="ltx_Math" alttext="1/512" display="inline"><semantics id="footnote4.m6.1b"><mrow id="footnote4.m6.1.1" xref="footnote4.m6.1.1.cmml"><mn id="footnote4.m6.1.1.2" xref="footnote4.m6.1.1.2.cmml">1</mn><mo id="footnote4.m6.1.1.1" xref="footnote4.m6.1.1.1.cmml">/</mo><mn id="footnote4.m6.1.1.3" xref="footnote4.m6.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote4.m6.1c"><apply id="footnote4.m6.1.1.cmml" xref="footnote4.m6.1.1"><divide id="footnote4.m6.1.1.1.cmml" xref="footnote4.m6.1.1.1"></divide><cn type="integer" id="footnote4.m6.1.1.2.cmml" xref="footnote4.m6.1.1.2">1</cn><cn type="integer" id="footnote4.m6.1.1.3.cmml" xref="footnote4.m6.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote4.m6.1d">1/512</annotation></semantics></math>. This is not an important choice and either option would be fine.</span></span></span> respectively, the worst-case quantization error is when the quantized value lies exactly between two quantization levels. We therefore need to compute the largest possible gap between a value and its quantization. Using 256 quantization levels, the largest possible gap is <math id="A6.SS2.8.p1.10.m7.1" class="ltx_Math" alttext="1/(256/.5)=1/512" display="inline"><semantics id="A6.SS2.8.p1.10.m7.1a"><mrow id="A6.SS2.8.p1.10.m7.1.1" xref="A6.SS2.8.p1.10.m7.1.1.cmml"><mrow id="A6.SS2.8.p1.10.m7.1.1.1" xref="A6.SS2.8.p1.10.m7.1.1.1.cmml"><mn id="A6.SS2.8.p1.10.m7.1.1.1.3" xref="A6.SS2.8.p1.10.m7.1.1.1.3.cmml">1</mn><mo id="A6.SS2.8.p1.10.m7.1.1.1.2" xref="A6.SS2.8.p1.10.m7.1.1.1.2.cmml">/</mo><mrow id="A6.SS2.8.p1.10.m7.1.1.1.1.1" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.SS2.8.p1.10.m7.1.1.1.1.1.2" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.cmml">(</mo><mrow id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.cmml"><mn id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.2" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.2.cmml">256</mn><mo id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.1" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.1.cmml">/</mo><mn id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.3" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.3.cmml">.5</mn></mrow><mo stretchy="false" id="A6.SS2.8.p1.10.m7.1.1.1.1.1.3" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.SS2.8.p1.10.m7.1.1.2" xref="A6.SS2.8.p1.10.m7.1.1.2.cmml">=</mo><mrow id="A6.SS2.8.p1.10.m7.1.1.3" xref="A6.SS2.8.p1.10.m7.1.1.3.cmml"><mn id="A6.SS2.8.p1.10.m7.1.1.3.2" xref="A6.SS2.8.p1.10.m7.1.1.3.2.cmml">1</mn><mo id="A6.SS2.8.p1.10.m7.1.1.3.1" xref="A6.SS2.8.p1.10.m7.1.1.3.1.cmml">/</mo><mn id="A6.SS2.8.p1.10.m7.1.1.3.3" xref="A6.SS2.8.p1.10.m7.1.1.3.3.cmml">512</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.10.m7.1b"><apply id="A6.SS2.8.p1.10.m7.1.1.cmml" xref="A6.SS2.8.p1.10.m7.1.1"><eq id="A6.SS2.8.p1.10.m7.1.1.2.cmml" xref="A6.SS2.8.p1.10.m7.1.1.2"></eq><apply id="A6.SS2.8.p1.10.m7.1.1.1.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1"><divide id="A6.SS2.8.p1.10.m7.1.1.1.2.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1.2"></divide><cn type="integer" id="A6.SS2.8.p1.10.m7.1.1.1.3.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1.3">1</cn><apply id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1"><divide id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.1.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.1"></divide><cn type="integer" id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.2.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.2">256</cn><cn type="float" id="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.3.cmml" xref="A6.SS2.8.p1.10.m7.1.1.1.1.1.1.3">.5</cn></apply></apply><apply id="A6.SS2.8.p1.10.m7.1.1.3.cmml" xref="A6.SS2.8.p1.10.m7.1.1.3"><divide id="A6.SS2.8.p1.10.m7.1.1.3.1.cmml" xref="A6.SS2.8.p1.10.m7.1.1.3.1"></divide><cn type="integer" id="A6.SS2.8.p1.10.m7.1.1.3.2.cmml" xref="A6.SS2.8.p1.10.m7.1.1.3.2">1</cn><cn type="integer" id="A6.SS2.8.p1.10.m7.1.1.3.3.cmml" xref="A6.SS2.8.p1.10.m7.1.1.3.3">512</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.10.m7.1c">1/(256/.5)=1/512</annotation></semantics></math> of the interval width. Multiplying by the above interval width yields a maximum quantization error for a given codebook of <math id="A6.SS2.8.p1.11.m8.1" class="ltx_Math" alttext="\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}" display="inline"><semantics id="A6.SS2.8.p1.11.m8.1a"><mfrac id="A6.SS2.8.p1.11.m8.1.1" xref="A6.SS2.8.p1.11.m8.1.1.cmml"><mrow id="A6.SS2.8.p1.11.m8.1.1.1" xref="A6.SS2.8.p1.11.m8.1.1.1.cmml"><msub id="A6.SS2.8.p1.11.m8.1.1.1.3" xref="A6.SS2.8.p1.11.m8.1.1.1.3.cmml"><mi id="A6.SS2.8.p1.11.m8.1.1.1.3.2" xref="A6.SS2.8.p1.11.m8.1.1.1.3.2.cmml">σ</mi><mi id="A6.SS2.8.p1.11.m8.1.1.1.3.3" xref="A6.SS2.8.p1.11.m8.1.1.1.3.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.11.m8.1.1.1.2" xref="A6.SS2.8.p1.11.m8.1.1.1.2.cmml">​</mo><msub id="A6.SS2.8.p1.11.m8.1.1.1.4" xref="A6.SS2.8.p1.11.m8.1.1.1.4.cmml"><mrow id="A6.SS2.8.p1.11.m8.1.1.1.4.2.2" xref="A6.SS2.8.p1.11.m8.1.1.1.4.2.1.cmml"><mo fence="true" rspace="0em" id="A6.SS2.8.p1.11.m8.1.1.1.4.2.2.1" xref="A6.SS2.8.p1.11.m8.1.1.1.4.2.1.1.cmml">∥</mo><mi id="A6.SS2.8.p1.11.m8.1.1.1.1" xref="A6.SS2.8.p1.11.m8.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.SS2.8.p1.11.m8.1.1.1.4.2.2.2" xref="A6.SS2.8.p1.11.m8.1.1.1.4.2.1.1.cmml">∥</mo></mrow><mn id="A6.SS2.8.p1.11.m8.1.1.1.4.3" xref="A6.SS2.8.p1.11.m8.1.1.1.4.3.cmml">2</mn></msub></mrow><mrow id="A6.SS2.8.p1.11.m8.1.1.3" xref="A6.SS2.8.p1.11.m8.1.1.3.cmml"><mn id="A6.SS2.8.p1.11.m8.1.1.3.2" xref="A6.SS2.8.p1.11.m8.1.1.3.2.cmml">512</mn><mo lspace="0em" rspace="0em" id="A6.SS2.8.p1.11.m8.1.1.3.1" xref="A6.SS2.8.p1.11.m8.1.1.3.1.cmml">​</mo><msqrt id="A6.SS2.8.p1.11.m8.1.1.3.3" xref="A6.SS2.8.p1.11.m8.1.1.3.3.cmml"><mi id="A6.SS2.8.p1.11.m8.1.1.3.3.2" xref="A6.SS2.8.p1.11.m8.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A6.SS2.8.p1.11.m8.1b"><apply id="A6.SS2.8.p1.11.m8.1.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1"><divide id="A6.SS2.8.p1.11.m8.1.1.2.cmml" xref="A6.SS2.8.p1.11.m8.1.1"></divide><apply id="A6.SS2.8.p1.11.m8.1.1.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1"><times id="A6.SS2.8.p1.11.m8.1.1.1.2.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.2"></times><apply id="A6.SS2.8.p1.11.m8.1.1.1.3.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.3"><csymbol cd="ambiguous" id="A6.SS2.8.p1.11.m8.1.1.1.3.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.3">subscript</csymbol><ci id="A6.SS2.8.p1.11.m8.1.1.1.3.2.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.3.2">𝜎</ci><ci id="A6.SS2.8.p1.11.m8.1.1.1.3.3.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.3.3">𝐴</ci></apply><apply id="A6.SS2.8.p1.11.m8.1.1.1.4.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.4"><csymbol cd="ambiguous" id="A6.SS2.8.p1.11.m8.1.1.1.4.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.4">subscript</csymbol><apply id="A6.SS2.8.p1.11.m8.1.1.1.4.2.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.4.2.2"><csymbol cd="latexml" id="A6.SS2.8.p1.11.m8.1.1.1.4.2.1.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.4.2.2.1">delimited-∥∥</csymbol><ci id="A6.SS2.8.p1.11.m8.1.1.1.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.SS2.8.p1.11.m8.1.1.1.4.3.cmml" xref="A6.SS2.8.p1.11.m8.1.1.1.4.3">2</cn></apply></apply><apply id="A6.SS2.8.p1.11.m8.1.1.3.cmml" xref="A6.SS2.8.p1.11.m8.1.1.3"><times id="A6.SS2.8.p1.11.m8.1.1.3.1.cmml" xref="A6.SS2.8.p1.11.m8.1.1.3.1"></times><cn type="integer" id="A6.SS2.8.p1.11.m8.1.1.3.2.cmml" xref="A6.SS2.8.p1.11.m8.1.1.3.2">512</cn><apply id="A6.SS2.8.p1.11.m8.1.1.3.3.cmml" xref="A6.SS2.8.p1.11.m8.1.1.3.3"><root id="A6.SS2.8.p1.11.m8.1.1.3.3a.cmml" xref="A6.SS2.8.p1.11.m8.1.1.3.3"></root><ci id="A6.SS2.8.p1.11.m8.1.1.3.3.2.cmml" xref="A6.SS2.8.p1.11.m8.1.1.3.3.2">𝜆</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.8.p1.11.m8.1c">\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}</annotation></semantics></math>.
Because the errors in each subspace may not agree in sign, their sum is an upper bound on the overall quantization error.
∎</p>
</div>
</div>
<div id="A6.SS2.p4" class="ltx_para">
<p id="A6.SS2.p4.1" class="ltx_p">At this point, we have all of the pieces necessary to prove a generalization guarantee for <span id="A6.SS2.p4.1.1" class="ltx_text ltx_font_typewriter">Maddness-Regress</span> save one: a theorem linking the norms of the various vectors and matrices involved to a probabilistic guarantee. <cite class="ltx_cite ltx_citemacro_citet">Kakade et al. (<a href="#bib.bib44" title="" class="ltx_ref">2009</a>)</cite> provide such a gaurantee, based on Rademacher complexity <cite class="ltx_cite ltx_citemacro_citep">(Bartlett &amp; Mendelson, <a href="#bib.bib10" title="" class="ltx_ref">2002</a>)</cite>.</p>
</div>
<div id="A6.Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem F.1</span></span><span id="A6.Thmtheorem1.2.2" class="ltx_text ltx_font_bold"> </span>(<cite class="ltx_cite ltx_citemacro_citep">(Kakade et al., <a href="#bib.bib44" title="" class="ltx_ref">2009</a>)</cite>, Corollary 5)<span id="A6.Thmtheorem1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmtheorem1.p1" class="ltx_para">
<p id="A6.Thmtheorem1.p1.12" class="ltx_p"><span id="A6.Thmtheorem1.p1.12.12" class="ltx_text ltx_font_italic">Let <math id="A6.Thmtheorem1.p1.1.1.m1.3" class="ltx_Math" alttext="\mathcal{F}=\{\bm{w}^{\top}\bm{x}:\lVert\bm{w}\rVert_{2}\leq W\}" display="inline"><semantics id="A6.Thmtheorem1.p1.1.1.m1.3a"><mrow id="A6.Thmtheorem1.p1.1.1.m1.3.3" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.Thmtheorem1.p1.1.1.m1.3.3.4" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.4.cmml">ℱ</mi><mo id="A6.Thmtheorem1.p1.1.1.m1.3.3.3" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.3.cmml">=</mo><mrow id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.3.cmml"><mo stretchy="false" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.3" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.3.1.cmml">{</mo><mrow id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.cmml"><msup id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.cmml"><mi id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.2" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.2.cmml">𝐰</mi><mo id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.3" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.1" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.3" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.3.cmml">𝐱</mi></mrow><mo lspace="0.278em" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.4" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.3.1.cmml">:</mo><mrow id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.cmml"><msub id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.cmml"><mrow id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.2" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.1.cmml"><mo fence="true" lspace="0.111em" rspace="0em" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.2.1" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.1.1.cmml">∥</mo><mi id="A6.Thmtheorem1.p1.1.1.m1.1.1" xref="A6.Thmtheorem1.p1.1.1.m1.1.1.cmml">𝐰</mi><mo fence="true" lspace="0em" rspace="0.1389em" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.2.2" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.1.1.cmml">∥</mo></mrow><mn id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.3" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.3.cmml">2</mn></msub><mo lspace="0.1389em" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.1" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.1.cmml">≤</mo><mi id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.3" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.3.cmml">W</mi></mrow><mo stretchy="false" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.5" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.1.1.m1.3b"><apply id="A6.Thmtheorem1.p1.1.1.m1.3.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3"><eq id="A6.Thmtheorem1.p1.1.1.m1.3.3.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.3"></eq><ci id="A6.Thmtheorem1.p1.1.1.m1.3.3.4.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.4">ℱ</ci><apply id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2"><csymbol cd="latexml" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.3.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.3">conditional-set</csymbol><apply id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1"><times id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.1"></times><apply id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2">superscript</csymbol><ci id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.2.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.2">𝐰</ci><csymbol cd="latexml" id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.2.3">top</csymbol></apply><ci id="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.2.2.1.1.1.3">𝐱</ci></apply><apply id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2"><leq id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.1"></leq><apply id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2">subscript</csymbol><apply id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.2"><csymbol cd="latexml" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.1.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.Thmtheorem1.p1.1.1.m1.1.1.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.1.1">𝐰</ci></apply><cn type="integer" id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.2.3">2</cn></apply><ci id="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.3.cmml" xref="A6.Thmtheorem1.p1.1.1.m1.3.3.2.2.2.3">𝑊</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.1.1.m1.3c">\mathcal{F}=\{\bm{w}^{\top}\bm{x}:\lVert\bm{w}\rVert_{2}\leq W\}</annotation></semantics></math> be the class of linear functions with bounded <math id="A6.Thmtheorem1.p1.2.2.m2.1" class="ltx_Math" alttext="L_{2}" display="inline"><semantics id="A6.Thmtheorem1.p1.2.2.m2.1a"><msub id="A6.Thmtheorem1.p1.2.2.m2.1.1" xref="A6.Thmtheorem1.p1.2.2.m2.1.1.cmml"><mi id="A6.Thmtheorem1.p1.2.2.m2.1.1.2" xref="A6.Thmtheorem1.p1.2.2.m2.1.1.2.cmml">L</mi><mn id="A6.Thmtheorem1.p1.2.2.m2.1.1.3" xref="A6.Thmtheorem1.p1.2.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.2.2.m2.1b"><apply id="A6.Thmtheorem1.p1.2.2.m2.1.1.cmml" xref="A6.Thmtheorem1.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="A6.Thmtheorem1.p1.2.2.m2.1.1.1.cmml" xref="A6.Thmtheorem1.p1.2.2.m2.1.1">subscript</csymbol><ci id="A6.Thmtheorem1.p1.2.2.m2.1.1.2.cmml" xref="A6.Thmtheorem1.p1.2.2.m2.1.1.2">𝐿</ci><cn type="integer" id="A6.Thmtheorem1.p1.2.2.m2.1.1.3.cmml" xref="A6.Thmtheorem1.p1.2.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.2.2.m2.1c">L_{2}</annotation></semantics></math> norms, let <math id="A6.Thmtheorem1.p1.3.3.m3.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="A6.Thmtheorem1.p1.3.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="A6.Thmtheorem1.p1.3.3.m3.1.1" xref="A6.Thmtheorem1.p1.3.3.m3.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.3.3.m3.1b"><ci id="A6.Thmtheorem1.p1.3.3.m3.1.1.cmml" xref="A6.Thmtheorem1.p1.3.3.m3.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.3.3.m3.1c">\mathcal{S}</annotation></semantics></math> be a set of <math id="A6.Thmtheorem1.p1.4.4.m4.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A6.Thmtheorem1.p1.4.4.m4.1a"><mi id="A6.Thmtheorem1.p1.4.4.m4.1.1" xref="A6.Thmtheorem1.p1.4.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.4.4.m4.1b"><ci id="A6.Thmtheorem1.p1.4.4.m4.1.1.cmml" xref="A6.Thmtheorem1.p1.4.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.4.4.m4.1c">n</annotation></semantics></math> samples drawn i.i.d. from some distribution <math id="A6.Thmtheorem1.p1.5.5.m5.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="A6.Thmtheorem1.p1.5.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="A6.Thmtheorem1.p1.5.5.m5.1.1" xref="A6.Thmtheorem1.p1.5.5.m5.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.5.5.m5.1b"><ci id="A6.Thmtheorem1.p1.5.5.m5.1.1.cmml" xref="A6.Thmtheorem1.p1.5.5.m5.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.5.5.m5.1c">\mathcal{D}</annotation></semantics></math> over the <math id="A6.Thmtheorem1.p1.6.6.m6.1" class="ltx_Math" alttext="L_{2}" display="inline"><semantics id="A6.Thmtheorem1.p1.6.6.m6.1a"><msub id="A6.Thmtheorem1.p1.6.6.m6.1.1" xref="A6.Thmtheorem1.p1.6.6.m6.1.1.cmml"><mi id="A6.Thmtheorem1.p1.6.6.m6.1.1.2" xref="A6.Thmtheorem1.p1.6.6.m6.1.1.2.cmml">L</mi><mn id="A6.Thmtheorem1.p1.6.6.m6.1.1.3" xref="A6.Thmtheorem1.p1.6.6.m6.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.6.6.m6.1b"><apply id="A6.Thmtheorem1.p1.6.6.m6.1.1.cmml" xref="A6.Thmtheorem1.p1.6.6.m6.1.1"><csymbol cd="ambiguous" id="A6.Thmtheorem1.p1.6.6.m6.1.1.1.cmml" xref="A6.Thmtheorem1.p1.6.6.m6.1.1">subscript</csymbol><ci id="A6.Thmtheorem1.p1.6.6.m6.1.1.2.cmml" xref="A6.Thmtheorem1.p1.6.6.m6.1.1.2">𝐿</ci><cn type="integer" id="A6.Thmtheorem1.p1.6.6.m6.1.1.3.cmml" xref="A6.Thmtheorem1.p1.6.6.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.6.6.m6.1c">L_{2}</annotation></semantics></math> ball of radius <math id="A6.Thmtheorem1.p1.7.7.m7.1" class="ltx_Math" alttext="X" display="inline"><semantics id="A6.Thmtheorem1.p1.7.7.m7.1a"><mi id="A6.Thmtheorem1.p1.7.7.m7.1.1" xref="A6.Thmtheorem1.p1.7.7.m7.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.7.7.m7.1b"><ci id="A6.Thmtheorem1.p1.7.7.m7.1.1.cmml" xref="A6.Thmtheorem1.p1.7.7.m7.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.7.7.m7.1c">X</annotation></semantics></math>,
and let <math id="A6.Thmtheorem1.p1.8.8.m8.3" class="ltx_Math" alttext="\mathcal{L}(f),f\in\mathcal{F}" display="inline"><semantics id="A6.Thmtheorem1.p1.8.8.m8.3a"><mrow id="A6.Thmtheorem1.p1.8.8.m8.3.3" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.cmml"><mrow id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.2.cmml"><mrow id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.2" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.1" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.1.cmml">​</mo><mrow id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.3.2" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.cmml"><mo stretchy="false" id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.3.2.1" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.cmml">(</mo><mi id="A6.Thmtheorem1.p1.8.8.m8.1.1" xref="A6.Thmtheorem1.p1.8.8.m8.1.1.cmml">f</mi><mo stretchy="false" id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.3.2.2" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.2" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.2.cmml">,</mo><mi id="A6.Thmtheorem1.p1.8.8.m8.2.2" xref="A6.Thmtheorem1.p1.8.8.m8.2.2.cmml">f</mi></mrow><mo id="A6.Thmtheorem1.p1.8.8.m8.3.3.2" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.2.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="A6.Thmtheorem1.p1.8.8.m8.3.3.3" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.3.cmml">ℱ</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.8.8.m8.3b"><apply id="A6.Thmtheorem1.p1.8.8.m8.3.3.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3"><in id="A6.Thmtheorem1.p1.8.8.m8.3.3.2.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.2"></in><list id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.2.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1"><apply id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1"><times id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.1.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.1"></times><ci id="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.2.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.1.1.1.2">ℒ</ci><ci id="A6.Thmtheorem1.p1.8.8.m8.1.1.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.1.1">𝑓</ci></apply><ci id="A6.Thmtheorem1.p1.8.8.m8.2.2.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.2.2">𝑓</ci></list><ci id="A6.Thmtheorem1.p1.8.8.m8.3.3.3.cmml" xref="A6.Thmtheorem1.p1.8.8.m8.3.3.3">ℱ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.8.8.m8.3c">\mathcal{L}(f),f\in\mathcal{F}</annotation></semantics></math> be a loss function with Lipschitz constant <math id="A6.Thmtheorem1.p1.9.9.m9.1" class="ltx_Math" alttext="L" display="inline"><semantics id="A6.Thmtheorem1.p1.9.9.m9.1a"><mi id="A6.Thmtheorem1.p1.9.9.m9.1.1" xref="A6.Thmtheorem1.p1.9.9.m9.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.9.9.m9.1b"><ci id="A6.Thmtheorem1.p1.9.9.m9.1.1.cmml" xref="A6.Thmtheorem1.p1.9.9.m9.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.9.9.m9.1c">L</annotation></semantics></math>. Then for any <math id="A6.Thmtheorem1.p1.10.10.m10.1" class="ltx_Math" alttext="0&lt;\delta&lt;1" display="inline"><semantics id="A6.Thmtheorem1.p1.10.10.m10.1a"><mrow id="A6.Thmtheorem1.p1.10.10.m10.1.1" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.cmml"><mn id="A6.Thmtheorem1.p1.10.10.m10.1.1.2" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.2.cmml">0</mn><mo id="A6.Thmtheorem1.p1.10.10.m10.1.1.3" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.3.cmml">&lt;</mo><mi id="A6.Thmtheorem1.p1.10.10.m10.1.1.4" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.4.cmml">δ</mi><mo id="A6.Thmtheorem1.p1.10.10.m10.1.1.5" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.5.cmml">&lt;</mo><mn id="A6.Thmtheorem1.p1.10.10.m10.1.1.6" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.6.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.10.10.m10.1b"><apply id="A6.Thmtheorem1.p1.10.10.m10.1.1.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1"><and id="A6.Thmtheorem1.p1.10.10.m10.1.1a.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1"></and><apply id="A6.Thmtheorem1.p1.10.10.m10.1.1b.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1"><lt id="A6.Thmtheorem1.p1.10.10.m10.1.1.3.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.3"></lt><cn type="integer" id="A6.Thmtheorem1.p1.10.10.m10.1.1.2.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.2">0</cn><ci id="A6.Thmtheorem1.p1.10.10.m10.1.1.4.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.4">𝛿</ci></apply><apply id="A6.Thmtheorem1.p1.10.10.m10.1.1c.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1"><lt id="A6.Thmtheorem1.p1.10.10.m10.1.1.5.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.5"></lt><share href="#A6.Thmtheorem1.p1.10.10.m10.1.1.4.cmml" id="A6.Thmtheorem1.p1.10.10.m10.1.1d.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1"></share><cn type="integer" id="A6.Thmtheorem1.p1.10.10.m10.1.1.6.cmml" xref="A6.Thmtheorem1.p1.10.10.m10.1.1.6">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.10.10.m10.1c">0&lt;\delta&lt;1</annotation></semantics></math>, it holds with probability at least <math id="A6.Thmtheorem1.p1.11.11.m11.1" class="ltx_Math" alttext="1-\delta" display="inline"><semantics id="A6.Thmtheorem1.p1.11.11.m11.1a"><mrow id="A6.Thmtheorem1.p1.11.11.m11.1.1" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.cmml"><mn id="A6.Thmtheorem1.p1.11.11.m11.1.1.2" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.2.cmml">1</mn><mo id="A6.Thmtheorem1.p1.11.11.m11.1.1.1" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.1.cmml">−</mo><mi id="A6.Thmtheorem1.p1.11.11.m11.1.1.3" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.3.cmml">δ</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.11.11.m11.1b"><apply id="A6.Thmtheorem1.p1.11.11.m11.1.1.cmml" xref="A6.Thmtheorem1.p1.11.11.m11.1.1"><minus id="A6.Thmtheorem1.p1.11.11.m11.1.1.1.cmml" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.1"></minus><cn type="integer" id="A6.Thmtheorem1.p1.11.11.m11.1.1.2.cmml" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.2">1</cn><ci id="A6.Thmtheorem1.p1.11.11.m11.1.1.3.cmml" xref="A6.Thmtheorem1.p1.11.11.m11.1.1.3">𝛿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.11.11.m11.1c">1-\delta</annotation></semantics></math> over the sample <math id="A6.Thmtheorem1.p1.12.12.m12.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="A6.Thmtheorem1.p1.12.12.m12.1a"><mi class="ltx_font_mathcaligraphic" id="A6.Thmtheorem1.p1.12.12.m12.1.1" xref="A6.Thmtheorem1.p1.12.12.m12.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="A6.Thmtheorem1.p1.12.12.m12.1b"><ci id="A6.Thmtheorem1.p1.12.12.m12.1.1.cmml" xref="A6.Thmtheorem1.p1.12.12.m12.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmtheorem1.p1.12.12.m12.1c">\mathcal{S}</annotation></semantics></math> that</span></p>
<table id="A6.EGx27" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E40"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E40.m1.4" class="ltx_Math" alttext="\displaystyle\mathbb{E}_{\mathcal{D}}[\mathcal{L}(f)]\leq\mathbb{E}_{S}[\mathcal{L}(f)]+\frac{LXW}{\sqrt{2n}}\left(8+\sqrt{-log(\delta)}\right)." display="inline"><semantics id="A6.E40.m1.4a"><mrow id="A6.E40.m1.4.4.1" xref="A6.E40.m1.4.4.1.1.cmml"><mrow id="A6.E40.m1.4.4.1.1" xref="A6.E40.m1.4.4.1.1.cmml"><mrow id="A6.E40.m1.4.4.1.1.1" xref="A6.E40.m1.4.4.1.1.1.cmml"><msub id="A6.E40.m1.4.4.1.1.1.3" xref="A6.E40.m1.4.4.1.1.1.3.cmml"><mi id="A6.E40.m1.4.4.1.1.1.3.2" xref="A6.E40.m1.4.4.1.1.1.3.2.cmml">𝔼</mi><mi class="ltx_font_mathcaligraphic" id="A6.E40.m1.4.4.1.1.1.3.3" xref="A6.E40.m1.4.4.1.1.1.3.3.cmml">𝒟</mi></msub><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.1.2" xref="A6.E40.m1.4.4.1.1.1.2.cmml">​</mo><mrow id="A6.E40.m1.4.4.1.1.1.1.1" xref="A6.E40.m1.4.4.1.1.1.1.2.cmml"><mo stretchy="false" id="A6.E40.m1.4.4.1.1.1.1.1.2" xref="A6.E40.m1.4.4.1.1.1.1.2.1.cmml">[</mo><mrow id="A6.E40.m1.4.4.1.1.1.1.1.1" xref="A6.E40.m1.4.4.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.E40.m1.4.4.1.1.1.1.1.1.2" xref="A6.E40.m1.4.4.1.1.1.1.1.1.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.1.1.1.1.1" xref="A6.E40.m1.4.4.1.1.1.1.1.1.1.cmml">​</mo><mrow id="A6.E40.m1.4.4.1.1.1.1.1.1.3.2" xref="A6.E40.m1.4.4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.E40.m1.4.4.1.1.1.1.1.1.3.2.1" xref="A6.E40.m1.4.4.1.1.1.1.1.1.cmml">(</mo><mi id="A6.E40.m1.2.2" xref="A6.E40.m1.2.2.cmml">f</mi><mo stretchy="false" id="A6.E40.m1.4.4.1.1.1.1.1.1.3.2.2" xref="A6.E40.m1.4.4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A6.E40.m1.4.4.1.1.1.1.1.3" xref="A6.E40.m1.4.4.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A6.E40.m1.4.4.1.1.4" xref="A6.E40.m1.4.4.1.1.4.cmml">≤</mo><mrow id="A6.E40.m1.4.4.1.1.3" xref="A6.E40.m1.4.4.1.1.3.cmml"><mrow id="A6.E40.m1.4.4.1.1.2.1" xref="A6.E40.m1.4.4.1.1.2.1.cmml"><msub id="A6.E40.m1.4.4.1.1.2.1.3" xref="A6.E40.m1.4.4.1.1.2.1.3.cmml"><mi id="A6.E40.m1.4.4.1.1.2.1.3.2" xref="A6.E40.m1.4.4.1.1.2.1.3.2.cmml">𝔼</mi><mi id="A6.E40.m1.4.4.1.1.2.1.3.3" xref="A6.E40.m1.4.4.1.1.2.1.3.3.cmml">S</mi></msub><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.2.1.2" xref="A6.E40.m1.4.4.1.1.2.1.2.cmml">​</mo><mrow id="A6.E40.m1.4.4.1.1.2.1.1.1" xref="A6.E40.m1.4.4.1.1.2.1.1.2.cmml"><mo stretchy="false" id="A6.E40.m1.4.4.1.1.2.1.1.1.2" xref="A6.E40.m1.4.4.1.1.2.1.1.2.1.cmml">[</mo><mrow id="A6.E40.m1.4.4.1.1.2.1.1.1.1" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.E40.m1.4.4.1.1.2.1.1.1.1.2" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.2.1.1.1.1.1" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.1.cmml">​</mo><mrow id="A6.E40.m1.4.4.1.1.2.1.1.1.1.3.2" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.cmml"><mo stretchy="false" id="A6.E40.m1.4.4.1.1.2.1.1.1.1.3.2.1" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.cmml">(</mo><mi id="A6.E40.m1.3.3" xref="A6.E40.m1.3.3.cmml">f</mi><mo stretchy="false" id="A6.E40.m1.4.4.1.1.2.1.1.1.1.3.2.2" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A6.E40.m1.4.4.1.1.2.1.1.1.3" xref="A6.E40.m1.4.4.1.1.2.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A6.E40.m1.4.4.1.1.3.3" xref="A6.E40.m1.4.4.1.1.3.3.cmml">+</mo><mrow id="A6.E40.m1.4.4.1.1.3.2" xref="A6.E40.m1.4.4.1.1.3.2.cmml"><mstyle displaystyle="true" id="A6.E40.m1.4.4.1.1.3.2.3" xref="A6.E40.m1.4.4.1.1.3.2.3.cmml"><mfrac id="A6.E40.m1.4.4.1.1.3.2.3a" xref="A6.E40.m1.4.4.1.1.3.2.3.cmml"><mrow id="A6.E40.m1.4.4.1.1.3.2.3.2" xref="A6.E40.m1.4.4.1.1.3.2.3.2.cmml"><mi id="A6.E40.m1.4.4.1.1.3.2.3.2.2" xref="A6.E40.m1.4.4.1.1.3.2.3.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.3.2.3.2.1" xref="A6.E40.m1.4.4.1.1.3.2.3.2.1.cmml">​</mo><mi id="A6.E40.m1.4.4.1.1.3.2.3.2.3" xref="A6.E40.m1.4.4.1.1.3.2.3.2.3.cmml">X</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.3.2.3.2.1a" xref="A6.E40.m1.4.4.1.1.3.2.3.2.1.cmml">​</mo><mi id="A6.E40.m1.4.4.1.1.3.2.3.2.4" xref="A6.E40.m1.4.4.1.1.3.2.3.2.4.cmml">W</mi></mrow><msqrt id="A6.E40.m1.4.4.1.1.3.2.3.3" xref="A6.E40.m1.4.4.1.1.3.2.3.3.cmml"><mrow id="A6.E40.m1.4.4.1.1.3.2.3.3.2" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.cmml"><mn id="A6.E40.m1.4.4.1.1.3.2.3.3.2.2" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.3.2.3.3.2.1" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.1.cmml">​</mo><mi id="A6.E40.m1.4.4.1.1.3.2.3.3.2.3" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.3.cmml">n</mi></mrow></msqrt></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A6.E40.m1.4.4.1.1.3.2.2" xref="A6.E40.m1.4.4.1.1.3.2.2.cmml">​</mo><mrow id="A6.E40.m1.4.4.1.1.3.2.1.1" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.cmml"><mo id="A6.E40.m1.4.4.1.1.3.2.1.1.2" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.cmml">(</mo><mrow id="A6.E40.m1.4.4.1.1.3.2.1.1.1" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.cmml"><mn id="A6.E40.m1.4.4.1.1.3.2.1.1.1.2" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.2.cmml">8</mn><mo id="A6.E40.m1.4.4.1.1.3.2.1.1.1.1" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.1.cmml">+</mo><msqrt id="A6.E40.m1.1.1" xref="A6.E40.m1.1.1.cmml"><mrow id="A6.E40.m1.1.1.1" xref="A6.E40.m1.1.1.1.cmml"><mo id="A6.E40.m1.1.1.1a" xref="A6.E40.m1.1.1.1.cmml">−</mo><mrow id="A6.E40.m1.1.1.1.3" xref="A6.E40.m1.1.1.1.3.cmml"><mi id="A6.E40.m1.1.1.1.3.2" xref="A6.E40.m1.1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.1.1.1.3.1" xref="A6.E40.m1.1.1.1.3.1.cmml">​</mo><mi id="A6.E40.m1.1.1.1.3.3" xref="A6.E40.m1.1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.1.1.1.3.1a" xref="A6.E40.m1.1.1.1.3.1.cmml">​</mo><mi id="A6.E40.m1.1.1.1.3.4" xref="A6.E40.m1.1.1.1.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.E40.m1.1.1.1.3.1b" xref="A6.E40.m1.1.1.1.3.1.cmml">​</mo><mrow id="A6.E40.m1.1.1.1.3.5.2" xref="A6.E40.m1.1.1.1.3.cmml"><mo stretchy="false" id="A6.E40.m1.1.1.1.3.5.2.1" xref="A6.E40.m1.1.1.1.3.cmml">(</mo><mi id="A6.E40.m1.1.1.1.1" xref="A6.E40.m1.1.1.1.1.cmml">δ</mi><mo stretchy="false" id="A6.E40.m1.1.1.1.3.5.2.2" xref="A6.E40.m1.1.1.1.3.cmml">)</mo></mrow></mrow></mrow></msqrt></mrow><mo id="A6.E40.m1.4.4.1.1.3.2.1.1.3" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="A6.E40.m1.4.4.1.2" xref="A6.E40.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.E40.m1.4b"><apply id="A6.E40.m1.4.4.1.1.cmml" xref="A6.E40.m1.4.4.1"><leq id="A6.E40.m1.4.4.1.1.4.cmml" xref="A6.E40.m1.4.4.1.1.4"></leq><apply id="A6.E40.m1.4.4.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.1"><times id="A6.E40.m1.4.4.1.1.1.2.cmml" xref="A6.E40.m1.4.4.1.1.1.2"></times><apply id="A6.E40.m1.4.4.1.1.1.3.cmml" xref="A6.E40.m1.4.4.1.1.1.3"><csymbol cd="ambiguous" id="A6.E40.m1.4.4.1.1.1.3.1.cmml" xref="A6.E40.m1.4.4.1.1.1.3">subscript</csymbol><ci id="A6.E40.m1.4.4.1.1.1.3.2.cmml" xref="A6.E40.m1.4.4.1.1.1.3.2">𝔼</ci><ci id="A6.E40.m1.4.4.1.1.1.3.3.cmml" xref="A6.E40.m1.4.4.1.1.1.3.3">𝒟</ci></apply><apply id="A6.E40.m1.4.4.1.1.1.1.2.cmml" xref="A6.E40.m1.4.4.1.1.1.1.1"><csymbol cd="latexml" id="A6.E40.m1.4.4.1.1.1.1.2.1.cmml" xref="A6.E40.m1.4.4.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A6.E40.m1.4.4.1.1.1.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.1.1.1.1"><times id="A6.E40.m1.4.4.1.1.1.1.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.1.1.1.1.1"></times><ci id="A6.E40.m1.4.4.1.1.1.1.1.1.2.cmml" xref="A6.E40.m1.4.4.1.1.1.1.1.1.2">ℒ</ci><ci id="A6.E40.m1.2.2.cmml" xref="A6.E40.m1.2.2">𝑓</ci></apply></apply></apply><apply id="A6.E40.m1.4.4.1.1.3.cmml" xref="A6.E40.m1.4.4.1.1.3"><plus id="A6.E40.m1.4.4.1.1.3.3.cmml" xref="A6.E40.m1.4.4.1.1.3.3"></plus><apply id="A6.E40.m1.4.4.1.1.2.1.cmml" xref="A6.E40.m1.4.4.1.1.2.1"><times id="A6.E40.m1.4.4.1.1.2.1.2.cmml" xref="A6.E40.m1.4.4.1.1.2.1.2"></times><apply id="A6.E40.m1.4.4.1.1.2.1.3.cmml" xref="A6.E40.m1.4.4.1.1.2.1.3"><csymbol cd="ambiguous" id="A6.E40.m1.4.4.1.1.2.1.3.1.cmml" xref="A6.E40.m1.4.4.1.1.2.1.3">subscript</csymbol><ci id="A6.E40.m1.4.4.1.1.2.1.3.2.cmml" xref="A6.E40.m1.4.4.1.1.2.1.3.2">𝔼</ci><ci id="A6.E40.m1.4.4.1.1.2.1.3.3.cmml" xref="A6.E40.m1.4.4.1.1.2.1.3.3">𝑆</ci></apply><apply id="A6.E40.m1.4.4.1.1.2.1.1.2.cmml" xref="A6.E40.m1.4.4.1.1.2.1.1.1"><csymbol cd="latexml" id="A6.E40.m1.4.4.1.1.2.1.1.2.1.cmml" xref="A6.E40.m1.4.4.1.1.2.1.1.1.2">delimited-[]</csymbol><apply id="A6.E40.m1.4.4.1.1.2.1.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1"><times id="A6.E40.m1.4.4.1.1.2.1.1.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.1"></times><ci id="A6.E40.m1.4.4.1.1.2.1.1.1.1.2.cmml" xref="A6.E40.m1.4.4.1.1.2.1.1.1.1.2">ℒ</ci><ci id="A6.E40.m1.3.3.cmml" xref="A6.E40.m1.3.3">𝑓</ci></apply></apply></apply><apply id="A6.E40.m1.4.4.1.1.3.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2"><times id="A6.E40.m1.4.4.1.1.3.2.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2.2"></times><apply id="A6.E40.m1.4.4.1.1.3.2.3.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3"><divide id="A6.E40.m1.4.4.1.1.3.2.3.1.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3"></divide><apply id="A6.E40.m1.4.4.1.1.3.2.3.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.2"><times id="A6.E40.m1.4.4.1.1.3.2.3.2.1.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.2.1"></times><ci id="A6.E40.m1.4.4.1.1.3.2.3.2.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.2.2">𝐿</ci><ci id="A6.E40.m1.4.4.1.1.3.2.3.2.3.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.2.3">𝑋</ci><ci id="A6.E40.m1.4.4.1.1.3.2.3.2.4.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.2.4">𝑊</ci></apply><apply id="A6.E40.m1.4.4.1.1.3.2.3.3.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.3"><root id="A6.E40.m1.4.4.1.1.3.2.3.3a.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.3"></root><apply id="A6.E40.m1.4.4.1.1.3.2.3.3.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2"><times id="A6.E40.m1.4.4.1.1.3.2.3.3.2.1.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.1"></times><cn type="integer" id="A6.E40.m1.4.4.1.1.3.2.3.3.2.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.2">2</cn><ci id="A6.E40.m1.4.4.1.1.3.2.3.3.2.3.cmml" xref="A6.E40.m1.4.4.1.1.3.2.3.3.2.3">𝑛</ci></apply></apply></apply><apply id="A6.E40.m1.4.4.1.1.3.2.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.3.2.1.1"><plus id="A6.E40.m1.4.4.1.1.3.2.1.1.1.1.cmml" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.1"></plus><cn type="integer" id="A6.E40.m1.4.4.1.1.3.2.1.1.1.2.cmml" xref="A6.E40.m1.4.4.1.1.3.2.1.1.1.2">8</cn><apply id="A6.E40.m1.1.1.cmml" xref="A6.E40.m1.1.1"><root id="A6.E40.m1.1.1a.cmml" xref="A6.E40.m1.1.1"></root><apply id="A6.E40.m1.1.1.1.cmml" xref="A6.E40.m1.1.1.1"><minus id="A6.E40.m1.1.1.1.2.cmml" xref="A6.E40.m1.1.1.1"></minus><apply id="A6.E40.m1.1.1.1.3.cmml" xref="A6.E40.m1.1.1.1.3"><times id="A6.E40.m1.1.1.1.3.1.cmml" xref="A6.E40.m1.1.1.1.3.1"></times><ci id="A6.E40.m1.1.1.1.3.2.cmml" xref="A6.E40.m1.1.1.1.3.2">𝑙</ci><ci id="A6.E40.m1.1.1.1.3.3.cmml" xref="A6.E40.m1.1.1.1.3.3">𝑜</ci><ci id="A6.E40.m1.1.1.1.3.4.cmml" xref="A6.E40.m1.1.1.1.3.4">𝑔</ci><ci id="A6.E40.m1.1.1.1.1.cmml" xref="A6.E40.m1.1.1.1.1">𝛿</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E40.m1.4c">\displaystyle\mathbb{E}_{\mathcal{D}}[\mathcal{L}(f)]\leq\mathbb{E}_{S}[\mathcal{L}(f)]+\frac{LXW}{\sqrt{2n}}\left(8+\sqrt{-log(\delta)}\right).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(40)</span></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A6.SS2.p5" class="ltx_para">
<p id="A6.SS2.p5.1" class="ltx_p">We can now obtain our desired guarantee for the regression step.</p>
</div>
<div id="A6.Thmlemma6" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A6.Thmlemma6.1.1.1" class="ltx_text ltx_font_bold">Lemma F.6</span></span><span id="A6.Thmlemma6.2.2" class="ltx_text ltx_font_bold"> </span>(Generalization Error of <span id="A6.Thmlemma6.3.3" class="ltx_text ltx_font_typewriter">Maddness-Regress</span>)<span id="A6.Thmlemma6.4.4" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A6.Thmlemma6.p1" class="ltx_para">
<p id="A6.Thmlemma6.p1.13" class="ltx_p"><span id="A6.Thmlemma6.p1.13.13" class="ltx_text ltx_font_italic">Let <math id="A6.Thmlemma6.p1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="A6.Thmlemma6.p1.1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A6.Thmlemma6.p1.1.1.m1.1.1" xref="A6.Thmlemma6.p1.1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.1.1.m1.1b"><ci id="A6.Thmlemma6.p1.1.1.m1.1.1.cmml" xref="A6.Thmlemma6.p1.1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.1.1.m1.1c">\mathcal{D}</annotation></semantics></math> be a probability distribution over <math id="A6.Thmlemma6.p1.2.2.m2.1" class="ltx_Math" alttext="\mathbb{R}^{D}" display="inline"><semantics id="A6.Thmlemma6.p1.2.2.m2.1a"><msup id="A6.Thmlemma6.p1.2.2.m2.1.1" xref="A6.Thmlemma6.p1.2.2.m2.1.1.cmml"><mi id="A6.Thmlemma6.p1.2.2.m2.1.1.2" xref="A6.Thmlemma6.p1.2.2.m2.1.1.2.cmml">ℝ</mi><mi id="A6.Thmlemma6.p1.2.2.m2.1.1.3" xref="A6.Thmlemma6.p1.2.2.m2.1.1.3.cmml">D</mi></msup><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.2.2.m2.1b"><apply id="A6.Thmlemma6.p1.2.2.m2.1.1.cmml" xref="A6.Thmlemma6.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma6.p1.2.2.m2.1.1.1.cmml" xref="A6.Thmlemma6.p1.2.2.m2.1.1">superscript</csymbol><ci id="A6.Thmlemma6.p1.2.2.m2.1.1.2.cmml" xref="A6.Thmlemma6.p1.2.2.m2.1.1.2">ℝ</ci><ci id="A6.Thmlemma6.p1.2.2.m2.1.1.3.cmml" xref="A6.Thmlemma6.p1.2.2.m2.1.1.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.2.2.m2.1c">\mathbb{R}^{D}</annotation></semantics></math> and suppose that <span id="A6.Thmlemma6.p1.13.13.1" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.Thmlemma6.p1.13.13.1.1" class="ltx_text ltx_font_italic"> </span></span>is trained on a matrix <math id="A6.Thmlemma6.p1.3.3.m3.1" class="ltx_Math" alttext="\tilde{\bm{A}}\in R^{N\times D}" display="inline"><semantics id="A6.Thmlemma6.p1.3.3.m3.1a"><mrow id="A6.Thmlemma6.p1.3.3.m3.1.1" xref="A6.Thmlemma6.p1.3.3.m3.1.1.cmml"><mover accent="true" id="A6.Thmlemma6.p1.3.3.m3.1.1.2" xref="A6.Thmlemma6.p1.3.3.m3.1.1.2.cmml"><mi id="A6.Thmlemma6.p1.3.3.m3.1.1.2.2" xref="A6.Thmlemma6.p1.3.3.m3.1.1.2.2.cmml">𝐀</mi><mo id="A6.Thmlemma6.p1.3.3.m3.1.1.2.1" xref="A6.Thmlemma6.p1.3.3.m3.1.1.2.1.cmml">~</mo></mover><mo id="A6.Thmlemma6.p1.3.3.m3.1.1.1" xref="A6.Thmlemma6.p1.3.3.m3.1.1.1.cmml">∈</mo><msup id="A6.Thmlemma6.p1.3.3.m3.1.1.3" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.cmml"><mi id="A6.Thmlemma6.p1.3.3.m3.1.1.3.2" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.2.cmml">R</mi><mrow id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.cmml"><mi id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.2" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.1" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.1.cmml">×</mo><mi id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.3" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.3.3.m3.1b"><apply id="A6.Thmlemma6.p1.3.3.m3.1.1.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1"><in id="A6.Thmlemma6.p1.3.3.m3.1.1.1.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.1"></in><apply id="A6.Thmlemma6.p1.3.3.m3.1.1.2.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.2"><ci id="A6.Thmlemma6.p1.3.3.m3.1.1.2.1.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.2.1">~</ci><ci id="A6.Thmlemma6.p1.3.3.m3.1.1.2.2.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.2.2">𝐀</ci></apply><apply id="A6.Thmlemma6.p1.3.3.m3.1.1.3.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3"><csymbol cd="ambiguous" id="A6.Thmlemma6.p1.3.3.m3.1.1.3.1.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3">superscript</csymbol><ci id="A6.Thmlemma6.p1.3.3.m3.1.1.3.2.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.2">𝑅</ci><apply id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3"><times id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.1.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.1"></times><ci id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.2.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.2">𝑁</ci><ci id="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.3.cmml" xref="A6.Thmlemma6.p1.3.3.m3.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.3.3.m3.1c">\tilde{\bm{A}}\in R^{N\times D}</annotation></semantics></math> whose rows are drawn independently from <math id="A6.Thmlemma6.p1.4.4.m4.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="A6.Thmlemma6.p1.4.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="A6.Thmlemma6.p1.4.4.m4.1.1" xref="A6.Thmlemma6.p1.4.4.m4.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.4.4.m4.1b"><ci id="A6.Thmlemma6.p1.4.4.m4.1.1.cmml" xref="A6.Thmlemma6.p1.4.4.m4.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.4.4.m4.1c">\mathcal{D}</annotation></semantics></math> and with maximum singular value bounded by <math id="A6.Thmlemma6.p1.5.5.m5.1" class="ltx_Math" alttext="\sigma_{A}" display="inline"><semantics id="A6.Thmlemma6.p1.5.5.m5.1a"><msub id="A6.Thmlemma6.p1.5.5.m5.1.1" xref="A6.Thmlemma6.p1.5.5.m5.1.1.cmml"><mi id="A6.Thmlemma6.p1.5.5.m5.1.1.2" xref="A6.Thmlemma6.p1.5.5.m5.1.1.2.cmml">σ</mi><mi id="A6.Thmlemma6.p1.5.5.m5.1.1.3" xref="A6.Thmlemma6.p1.5.5.m5.1.1.3.cmml">A</mi></msub><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.5.5.m5.1b"><apply id="A6.Thmlemma6.p1.5.5.m5.1.1.cmml" xref="A6.Thmlemma6.p1.5.5.m5.1.1"><csymbol cd="ambiguous" id="A6.Thmlemma6.p1.5.5.m5.1.1.1.cmml" xref="A6.Thmlemma6.p1.5.5.m5.1.1">subscript</csymbol><ci id="A6.Thmlemma6.p1.5.5.m5.1.1.2.cmml" xref="A6.Thmlemma6.p1.5.5.m5.1.1.2">𝜎</ci><ci id="A6.Thmlemma6.p1.5.5.m5.1.1.3.cmml" xref="A6.Thmlemma6.p1.5.5.m5.1.1.3">𝐴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.5.5.m5.1c">\sigma_{A}</annotation></semantics></math>. Let <math id="A6.Thmlemma6.p1.6.6.m6.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.Thmlemma6.p1.6.6.m6.1a"><mi id="A6.Thmlemma6.p1.6.6.m6.1.1" xref="A6.Thmlemma6.p1.6.6.m6.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.6.6.m6.1b"><ci id="A6.Thmlemma6.p1.6.6.m6.1.1.cmml" xref="A6.Thmlemma6.p1.6.6.m6.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.6.6.m6.1c">C</annotation></semantics></math> be the number of codebooks used by <span id="A6.Thmlemma6.p1.13.13.2" class="ltx_text ltx_font_smallcaps">Maddness<span id="A6.Thmlemma6.p1.13.13.2.1" class="ltx_text ltx_font_italic"> </span></span>and <math id="A6.Thmlemma6.p1.7.7.m7.1" class="ltx_Math" alttext="\lambda&gt;0" display="inline"><semantics id="A6.Thmlemma6.p1.7.7.m7.1a"><mrow id="A6.Thmlemma6.p1.7.7.m7.1.1" xref="A6.Thmlemma6.p1.7.7.m7.1.1.cmml"><mi id="A6.Thmlemma6.p1.7.7.m7.1.1.2" xref="A6.Thmlemma6.p1.7.7.m7.1.1.2.cmml">λ</mi><mo id="A6.Thmlemma6.p1.7.7.m7.1.1.1" xref="A6.Thmlemma6.p1.7.7.m7.1.1.1.cmml">&gt;</mo><mn id="A6.Thmlemma6.p1.7.7.m7.1.1.3" xref="A6.Thmlemma6.p1.7.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.7.7.m7.1b"><apply id="A6.Thmlemma6.p1.7.7.m7.1.1.cmml" xref="A6.Thmlemma6.p1.7.7.m7.1.1"><gt id="A6.Thmlemma6.p1.7.7.m7.1.1.1.cmml" xref="A6.Thmlemma6.p1.7.7.m7.1.1.1"></gt><ci id="A6.Thmlemma6.p1.7.7.m7.1.1.2.cmml" xref="A6.Thmlemma6.p1.7.7.m7.1.1.2">𝜆</ci><cn type="integer" id="A6.Thmlemma6.p1.7.7.m7.1.1.3.cmml" xref="A6.Thmlemma6.p1.7.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.7.7.m7.1c">\lambda&gt;0</annotation></semantics></math> the regularization parameter used in the ridge regression step. Further let <math id="A6.Thmlemma6.p1.8.8.m8.1" class="ltx_Math" alttext="g(\bm{a})" display="inline"><semantics id="A6.Thmlemma6.p1.8.8.m8.1a"><mrow id="A6.Thmlemma6.p1.8.8.m8.1.2" xref="A6.Thmlemma6.p1.8.8.m8.1.2.cmml"><mi id="A6.Thmlemma6.p1.8.8.m8.1.2.2" xref="A6.Thmlemma6.p1.8.8.m8.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma6.p1.8.8.m8.1.2.1" xref="A6.Thmlemma6.p1.8.8.m8.1.2.1.cmml">​</mo><mrow id="A6.Thmlemma6.p1.8.8.m8.1.2.3.2" xref="A6.Thmlemma6.p1.8.8.m8.1.2.cmml"><mo stretchy="false" id="A6.Thmlemma6.p1.8.8.m8.1.2.3.2.1" xref="A6.Thmlemma6.p1.8.8.m8.1.2.cmml">(</mo><mi id="A6.Thmlemma6.p1.8.8.m8.1.1" xref="A6.Thmlemma6.p1.8.8.m8.1.1.cmml">𝐚</mi><mo stretchy="false" id="A6.Thmlemma6.p1.8.8.m8.1.2.3.2.2" xref="A6.Thmlemma6.p1.8.8.m8.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.8.8.m8.1b"><apply id="A6.Thmlemma6.p1.8.8.m8.1.2.cmml" xref="A6.Thmlemma6.p1.8.8.m8.1.2"><times id="A6.Thmlemma6.p1.8.8.m8.1.2.1.cmml" xref="A6.Thmlemma6.p1.8.8.m8.1.2.1"></times><ci id="A6.Thmlemma6.p1.8.8.m8.1.2.2.cmml" xref="A6.Thmlemma6.p1.8.8.m8.1.2.2">𝑔</ci><ci id="A6.Thmlemma6.p1.8.8.m8.1.1.cmml" xref="A6.Thmlemma6.p1.8.8.m8.1.1">𝐚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.8.8.m8.1c">g(\bm{a})</annotation></semantics></math> be a fixed (data-independent) function and <math id="A6.Thmlemma6.p1.9.9.m9.5" class="ltx_Math" alttext="\mathcal{L}(\bm{a},\bm{b})\triangleq|\bm{a}^{\top}\bm{b}-f(g(\bm{a}),h(\bm{b}))|" display="inline"><semantics id="A6.Thmlemma6.p1.9.9.m9.5a"><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5" xref="A6.Thmlemma6.p1.9.9.m9.5.5.cmml"><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.Thmlemma6.p1.9.9.m9.5.5.3.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma6.p1.9.9.m9.5.5.3.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.1.cmml">​</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.1.cmml"><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.2.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.1.cmml">(</mo><mi id="A6.Thmlemma6.p1.9.9.m9.1.1" xref="A6.Thmlemma6.p1.9.9.m9.1.1.cmml">𝐚</mi><mo id="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.1.cmml">,</mo><mi id="A6.Thmlemma6.p1.9.9.m9.2.2" xref="A6.Thmlemma6.p1.9.9.m9.2.2.cmml">𝐛</mi><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.2.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.1.cmml">)</mo></mrow></mrow><mo id="A6.Thmlemma6.p1.9.9.m9.5.5.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.2.cmml">≜</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.2.cmml"><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.2.1.cmml">|</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.cmml"><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.cmml"><msup id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.cmml"><mi id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.2.cmml">𝐚</mi><mo id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.1.cmml">​</mo><mi id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.3.cmml">𝐛</mi></mrow><mo id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.3.cmml">−</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.cmml"><mi id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.4" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.3.cmml">​</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.3.cmml"><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.3.cmml">(</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.cmml"><mi id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.3.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.3.2.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.cmml">(</mo><mi id="A6.Thmlemma6.p1.9.9.m9.3.3" xref="A6.Thmlemma6.p1.9.9.m9.3.3.cmml">𝐚</mi><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.3.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.4" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.3.cmml">,</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.cmml"><mi id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.1.cmml">​</mo><mrow id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.3.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.cmml"><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.3.2.1" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.cmml">(</mo><mi id="A6.Thmlemma6.p1.9.9.m9.4.4" xref="A6.Thmlemma6.p1.9.9.m9.4.4.cmml">𝐛</mi><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.3.2.2" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.5" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.3" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.9.9.m9.5b"><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5"><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.2">≜</ci><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3"><times id="A6.Thmlemma6.p1.9.9.m9.5.5.3.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.1"></times><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.3.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.2">ℒ</ci><interval closure="open" id="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.3.3.2"><ci id="A6.Thmlemma6.p1.9.9.m9.1.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.1.1">𝐚</ci><ci id="A6.Thmlemma6.p1.9.9.m9.2.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.2.2">𝐛</ci></interval></apply><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1"><abs id="A6.Thmlemma6.p1.9.9.m9.5.5.1.2.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.2"></abs><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1"><minus id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.3"></minus><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4"><times id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.1"></times><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2"><csymbol cd="ambiguous" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2">superscript</csymbol><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.2">𝐚</ci><csymbol cd="latexml" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.2.3">top</csymbol></apply><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.4.3">𝐛</ci></apply><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2"><times id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.3"></times><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.4.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.4">𝑓</ci><interval closure="open" id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2"><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1"><times id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.1"></times><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.1.1.1.1.2">𝑔</ci><ci id="A6.Thmlemma6.p1.9.9.m9.3.3.cmml" xref="A6.Thmlemma6.p1.9.9.m9.3.3">𝐚</ci></apply><apply id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2"><times id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.1.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.1"></times><ci id="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.2.cmml" xref="A6.Thmlemma6.p1.9.9.m9.5.5.1.1.1.2.2.2.2.2">ℎ</ci><ci id="A6.Thmlemma6.p1.9.9.m9.4.4.cmml" xref="A6.Thmlemma6.p1.9.9.m9.4.4">𝐛</ci></apply></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.9.9.m9.5c">\mathcal{L}(\bm{a},\bm{b})\triangleq|\bm{a}^{\top}\bm{b}-f(g(\bm{a}),h(\bm{b}))|</annotation></semantics></math>. Then for all vectors <math id="A6.Thmlemma6.p1.10.10.m10.1" class="ltx_Math" alttext="\bm{b}" display="inline"><semantics id="A6.Thmlemma6.p1.10.10.m10.1a"><mi id="A6.Thmlemma6.p1.10.10.m10.1.1" xref="A6.Thmlemma6.p1.10.10.m10.1.1.cmml">𝐛</mi><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.10.10.m10.1b"><ci id="A6.Thmlemma6.p1.10.10.m10.1.1.cmml" xref="A6.Thmlemma6.p1.10.10.m10.1.1">𝐛</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.10.10.m10.1c">\bm{b}</annotation></semantics></math>, any vector <math id="A6.Thmlemma6.p1.11.11.m11.1" class="ltx_Math" alttext="\bm{a}\sim\mathcal{D}" display="inline"><semantics id="A6.Thmlemma6.p1.11.11.m11.1a"><mrow id="A6.Thmlemma6.p1.11.11.m11.1.1" xref="A6.Thmlemma6.p1.11.11.m11.1.1.cmml"><mi id="A6.Thmlemma6.p1.11.11.m11.1.1.2" xref="A6.Thmlemma6.p1.11.11.m11.1.1.2.cmml">𝐚</mi><mo id="A6.Thmlemma6.p1.11.11.m11.1.1.1" xref="A6.Thmlemma6.p1.11.11.m11.1.1.1.cmml">∼</mo><mi class="ltx_font_mathcaligraphic" id="A6.Thmlemma6.p1.11.11.m11.1.1.3" xref="A6.Thmlemma6.p1.11.11.m11.1.1.3.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.11.11.m11.1b"><apply id="A6.Thmlemma6.p1.11.11.m11.1.1.cmml" xref="A6.Thmlemma6.p1.11.11.m11.1.1"><csymbol cd="latexml" id="A6.Thmlemma6.p1.11.11.m11.1.1.1.cmml" xref="A6.Thmlemma6.p1.11.11.m11.1.1.1">similar-to</csymbol><ci id="A6.Thmlemma6.p1.11.11.m11.1.1.2.cmml" xref="A6.Thmlemma6.p1.11.11.m11.1.1.2">𝐚</ci><ci id="A6.Thmlemma6.p1.11.11.m11.1.1.3.cmml" xref="A6.Thmlemma6.p1.11.11.m11.1.1.3">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.11.11.m11.1c">\bm{a}\sim\mathcal{D}</annotation></semantics></math>, and any <math id="A6.Thmlemma6.p1.12.12.m12.1" class="ltx_Math" alttext="0&lt;\delta&lt;1" display="inline"><semantics id="A6.Thmlemma6.p1.12.12.m12.1a"><mrow id="A6.Thmlemma6.p1.12.12.m12.1.1" xref="A6.Thmlemma6.p1.12.12.m12.1.1.cmml"><mn id="A6.Thmlemma6.p1.12.12.m12.1.1.2" xref="A6.Thmlemma6.p1.12.12.m12.1.1.2.cmml">0</mn><mo id="A6.Thmlemma6.p1.12.12.m12.1.1.3" xref="A6.Thmlemma6.p1.12.12.m12.1.1.3.cmml">&lt;</mo><mi id="A6.Thmlemma6.p1.12.12.m12.1.1.4" xref="A6.Thmlemma6.p1.12.12.m12.1.1.4.cmml">δ</mi><mo id="A6.Thmlemma6.p1.12.12.m12.1.1.5" xref="A6.Thmlemma6.p1.12.12.m12.1.1.5.cmml">&lt;</mo><mn id="A6.Thmlemma6.p1.12.12.m12.1.1.6" xref="A6.Thmlemma6.p1.12.12.m12.1.1.6.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.12.12.m12.1b"><apply id="A6.Thmlemma6.p1.12.12.m12.1.1.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1"><and id="A6.Thmlemma6.p1.12.12.m12.1.1a.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1"></and><apply id="A6.Thmlemma6.p1.12.12.m12.1.1b.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1"><lt id="A6.Thmlemma6.p1.12.12.m12.1.1.3.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1.3"></lt><cn type="integer" id="A6.Thmlemma6.p1.12.12.m12.1.1.2.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1.2">0</cn><ci id="A6.Thmlemma6.p1.12.12.m12.1.1.4.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1.4">𝛿</ci></apply><apply id="A6.Thmlemma6.p1.12.12.m12.1.1c.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1"><lt id="A6.Thmlemma6.p1.12.12.m12.1.1.5.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1.5"></lt><share href="#A6.Thmlemma6.p1.12.12.m12.1.1.4.cmml" id="A6.Thmlemma6.p1.12.12.m12.1.1d.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1"></share><cn type="integer" id="A6.Thmlemma6.p1.12.12.m12.1.1.6.cmml" xref="A6.Thmlemma6.p1.12.12.m12.1.1.6">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.12.12.m12.1c">0&lt;\delta&lt;1</annotation></semantics></math>, we have with probability at least <math id="A6.Thmlemma6.p1.13.13.m13.1" class="ltx_Math" alttext="1-\delta" display="inline"><semantics id="A6.Thmlemma6.p1.13.13.m13.1a"><mrow id="A6.Thmlemma6.p1.13.13.m13.1.1" xref="A6.Thmlemma6.p1.13.13.m13.1.1.cmml"><mn id="A6.Thmlemma6.p1.13.13.m13.1.1.2" xref="A6.Thmlemma6.p1.13.13.m13.1.1.2.cmml">1</mn><mo id="A6.Thmlemma6.p1.13.13.m13.1.1.1" xref="A6.Thmlemma6.p1.13.13.m13.1.1.1.cmml">−</mo><mi id="A6.Thmlemma6.p1.13.13.m13.1.1.3" xref="A6.Thmlemma6.p1.13.13.m13.1.1.3.cmml">δ</mi></mrow><annotation-xml encoding="MathML-Content" id="A6.Thmlemma6.p1.13.13.m13.1b"><apply id="A6.Thmlemma6.p1.13.13.m13.1.1.cmml" xref="A6.Thmlemma6.p1.13.13.m13.1.1"><minus id="A6.Thmlemma6.p1.13.13.m13.1.1.1.cmml" xref="A6.Thmlemma6.p1.13.13.m13.1.1.1"></minus><cn type="integer" id="A6.Thmlemma6.p1.13.13.m13.1.1.2.cmml" xref="A6.Thmlemma6.p1.13.13.m13.1.1.2">1</cn><ci id="A6.Thmlemma6.p1.13.13.m13.1.1.3.cmml" xref="A6.Thmlemma6.p1.13.13.m13.1.1.3">𝛿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.Thmlemma6.p1.13.13.m13.1c">1-\delta</annotation></semantics></math> that</span></p>
<table id="A6.EGx28" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E41"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td colspan="2" class="ltx_td ltx_align_center ltx_eqn_cell"><math id="A6.E41.m1.33" class="ltx_Math" alttext="\displaystyle\begin{split}\mathbb{E}_{\mathcal{D}}[&amp;\mathcal{L}(\bm{a},\bm{b})]\leq\mathbb{E}_{\tilde{\bm{A}}}[\mathcal{L}(\bm{a},\bm{b})]+\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}+\\
&amp;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{2n\lambda}}\left(8+\sqrt{-log(\delta)}\right).\end{split}" display="inline"><semantics id="A6.E41.m1.33a"><mtable columnspacing="0pt" rowspacing="0pt" id="A6.E41.m1.33.33.2"><mtr id="A6.E41.m1.33.33.2a"><mtd class="ltx_align_right" columnalign="right" id="A6.E41.m1.33.33.2b"><mrow id="A6.E41.m1.3.3.3.3.3a"><msub id="A6.E41.m1.3.3.3.3.3a.4"><mi id="A6.E41.m1.1.1.1.1.1.1" xref="A6.E41.m1.1.1.1.1.1.1.cmml">𝔼</mi><mi class="ltx_font_mathcaligraphic" id="A6.E41.m1.2.2.2.2.2.2.1" xref="A6.E41.m1.2.2.2.2.2.2.1.cmml">𝒟</mi></msub><mo stretchy="false" id="A6.E41.m1.3.3.3.3.3.3" xref="A6.E41.m1.32.32.1.1.1.cmml">[</mo></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="A6.E41.m1.33.33.2c"><mrow id="A6.E41.m1.24.24.24.24.21a"><mi class="ltx_font_mathcaligraphic" id="A6.E41.m1.4.4.4.4.1.1" xref="A6.E41.m1.4.4.4.4.1.1.cmml">ℒ</mi><mrow id="A6.E41.m1.24.24.24.24.21a.22"><mo stretchy="false" id="A6.E41.m1.5.5.5.5.2.2" xref="A6.E41.m1.32.32.1.1.1.cmml">(</mo><mi id="A6.E41.m1.6.6.6.6.3.3" xref="A6.E41.m1.6.6.6.6.3.3.cmml">𝒂</mi><mo id="A6.E41.m1.7.7.7.7.4.4" xref="A6.E41.m1.32.32.1.1.1.cmml">,</mo><mi id="A6.E41.m1.8.8.8.8.5.5" xref="A6.E41.m1.8.8.8.8.5.5.cmml">𝒃</mi><mo stretchy="false" id="A6.E41.m1.9.9.9.9.6.6" xref="A6.E41.m1.32.32.1.1.1.cmml">)</mo></mrow><mo stretchy="false" id="A6.E41.m1.10.10.10.10.7.7" xref="A6.E41.m1.32.32.1.1.1.cmml">]</mo><mo id="A6.E41.m1.11.11.11.11.8.8" xref="A6.E41.m1.11.11.11.11.8.8.cmml">≤</mo><mi id="A6.E41.m1.12.12.12.12.9.9" xref="A6.E41.m1.12.12.12.12.9.9.cmml">𝔼</mi><msub id="A6.E41.m1.24.24.24.24.21a.23"><mi id="A6.E41.m1.24.24.24.24.21a.23a" xref="A6.E41.m1.32.32.1.1.1.cmml"></mi><mover accent="true" id="A6.E41.m1.13.13.13.13.10.10.1" xref="A6.E41.m1.13.13.13.13.10.10.1.cmml"><mi id="A6.E41.m1.13.13.13.13.10.10.1.2" xref="A6.E41.m1.13.13.13.13.10.10.1.2.cmml">𝑨</mi><mo id="A6.E41.m1.13.13.13.13.10.10.1.1" xref="A6.E41.m1.13.13.13.13.10.10.1.1.cmml">~</mo></mover></msub><mo stretchy="false" id="A6.E41.m1.14.14.14.14.11.11" xref="A6.E41.m1.32.32.1.1.1.cmml">[</mo><mi class="ltx_font_mathcaligraphic" id="A6.E41.m1.15.15.15.15.12.12" xref="A6.E41.m1.15.15.15.15.12.12.cmml">ℒ</mi><mrow id="A6.E41.m1.24.24.24.24.21a.24"><mo stretchy="false" id="A6.E41.m1.16.16.16.16.13.13" xref="A6.E41.m1.32.32.1.1.1.cmml">(</mo><mi id="A6.E41.m1.17.17.17.17.14.14" xref="A6.E41.m1.17.17.17.17.14.14.cmml">𝒂</mi><mo id="A6.E41.m1.18.18.18.18.15.15" xref="A6.E41.m1.32.32.1.1.1.cmml">,</mo><mi id="A6.E41.m1.19.19.19.19.16.16" xref="A6.E41.m1.19.19.19.19.16.16.cmml">𝒃</mi><mo stretchy="false" id="A6.E41.m1.20.20.20.20.17.17" xref="A6.E41.m1.32.32.1.1.1.cmml">)</mo></mrow><mo stretchy="false" id="A6.E41.m1.21.21.21.21.18.18" xref="A6.E41.m1.32.32.1.1.1.cmml">]</mo><mo id="A6.E41.m1.22.22.22.22.19.19" xref="A6.E41.m1.22.22.22.22.19.19.cmml">+</mo><mstyle displaystyle="true" id="A6.E41.m1.23.23.23.23.20.20" xref="A6.E41.m1.23.23.23.23.20.20.cmml"><mfrac id="A6.E41.m1.23.23.23.23.20.20a" xref="A6.E41.m1.23.23.23.23.20.20.cmml"><mrow id="A6.E41.m1.23.23.23.23.20.20.1" xref="A6.E41.m1.23.23.23.23.20.20.1.cmml"><mi id="A6.E41.m1.23.23.23.23.20.20.1.3" xref="A6.E41.m1.23.23.23.23.20.20.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.E41.m1.23.23.23.23.20.20.1.2" xref="A6.E41.m1.23.23.23.23.20.20.1.2.cmml">​</mo><msub id="A6.E41.m1.23.23.23.23.20.20.1.4" xref="A6.E41.m1.23.23.23.23.20.20.1.4.cmml"><mi id="A6.E41.m1.23.23.23.23.20.20.1.4.2" xref="A6.E41.m1.23.23.23.23.20.20.1.4.2.cmml">σ</mi><mi id="A6.E41.m1.23.23.23.23.20.20.1.4.3" xref="A6.E41.m1.23.23.23.23.20.20.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.E41.m1.23.23.23.23.20.20.1.2a" xref="A6.E41.m1.23.23.23.23.20.20.1.2.cmml">​</mo><msub id="A6.E41.m1.23.23.23.23.20.20.1.5" xref="A6.E41.m1.23.23.23.23.20.20.1.5.cmml"><mrow id="A6.E41.m1.23.23.23.23.20.20.1.5.2.2" xref="A6.E41.m1.23.23.23.23.20.20.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E41.m1.23.23.23.23.20.20.1.5.2.2.1" xref="A6.E41.m1.23.23.23.23.20.20.1.5.2.1.1.cmml">∥</mo><mi id="A6.E41.m1.23.23.23.23.20.20.1.1" xref="A6.E41.m1.23.23.23.23.20.20.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.E41.m1.23.23.23.23.20.20.1.5.2.2.2" xref="A6.E41.m1.23.23.23.23.20.20.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.E41.m1.23.23.23.23.20.20.1.5.3" xref="A6.E41.m1.23.23.23.23.20.20.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.E41.m1.23.23.23.23.20.20.3" xref="A6.E41.m1.23.23.23.23.20.20.3.cmml"><mn id="A6.E41.m1.23.23.23.23.20.20.3.2" xref="A6.E41.m1.23.23.23.23.20.20.3.2.cmml">512</mn><mo lspace="0em" rspace="0em" id="A6.E41.m1.23.23.23.23.20.20.3.1" xref="A6.E41.m1.23.23.23.23.20.20.3.1.cmml">​</mo><msqrt id="A6.E41.m1.23.23.23.23.20.20.3.3" xref="A6.E41.m1.23.23.23.23.20.20.3.3.cmml"><mi id="A6.E41.m1.23.23.23.23.20.20.3.3.2" xref="A6.E41.m1.23.23.23.23.20.20.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle><mo id="A6.E41.m1.24.24.24.24.21.21" xref="A6.E41.m1.32.32.1.1.1.cmml">+</mo></mrow></mtd></mtr><mtr id="A6.E41.m1.33.33.2d"><mtd id="A6.E41.m1.33.33.2e" xref="A6.E41.m1.32.32.1.1.1.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="A6.E41.m1.33.33.2f"><mrow id="A6.E41.m1.33.33.2.32.8.8.8"><mrow id="A6.E41.m1.33.33.2.32.8.8.8.1"><mstyle displaystyle="true" id="A6.E41.m1.25.25.25.1.1.1" xref="A6.E41.m1.25.25.25.1.1.1.cmml"><mfrac id="A6.E41.m1.25.25.25.1.1.1a" xref="A6.E41.m1.25.25.25.1.1.1.cmml"><mrow id="A6.E41.m1.25.25.25.1.1.1.1" xref="A6.E41.m1.25.25.25.1.1.1.1.cmml"><mi id="A6.E41.m1.25.25.25.1.1.1.1.3" xref="A6.E41.m1.25.25.25.1.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.E41.m1.25.25.25.1.1.1.1.2" xref="A6.E41.m1.25.25.25.1.1.1.1.2.cmml">​</mo><msub id="A6.E41.m1.25.25.25.1.1.1.1.4" xref="A6.E41.m1.25.25.25.1.1.1.1.4.cmml"><mi id="A6.E41.m1.25.25.25.1.1.1.1.4.2" xref="A6.E41.m1.25.25.25.1.1.1.1.4.2.cmml">σ</mi><mi id="A6.E41.m1.25.25.25.1.1.1.1.4.3" xref="A6.E41.m1.25.25.25.1.1.1.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.E41.m1.25.25.25.1.1.1.1.2a" xref="A6.E41.m1.25.25.25.1.1.1.1.2.cmml">​</mo><msub id="A6.E41.m1.25.25.25.1.1.1.1.5" xref="A6.E41.m1.25.25.25.1.1.1.1.5.cmml"><mrow id="A6.E41.m1.25.25.25.1.1.1.1.5.2.2" xref="A6.E41.m1.25.25.25.1.1.1.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E41.m1.25.25.25.1.1.1.1.5.2.2.1" xref="A6.E41.m1.25.25.25.1.1.1.1.5.2.1.1.cmml">∥</mo><mi id="A6.E41.m1.25.25.25.1.1.1.1.1" xref="A6.E41.m1.25.25.25.1.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.E41.m1.25.25.25.1.1.1.1.5.2.2.2" xref="A6.E41.m1.25.25.25.1.1.1.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.E41.m1.25.25.25.1.1.1.1.5.3" xref="A6.E41.m1.25.25.25.1.1.1.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.E41.m1.25.25.25.1.1.1.3" xref="A6.E41.m1.25.25.25.1.1.1.3.cmml"><mn id="A6.E41.m1.25.25.25.1.1.1.3.2" xref="A6.E41.m1.25.25.25.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E41.m1.25.25.25.1.1.1.3.1" xref="A6.E41.m1.25.25.25.1.1.1.3.1.cmml">​</mo><msqrt id="A6.E41.m1.25.25.25.1.1.1.3.3" xref="A6.E41.m1.25.25.25.1.1.1.3.3.cmml"><mrow id="A6.E41.m1.25.25.25.1.1.1.3.3.2" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.cmml"><mn id="A6.E41.m1.25.25.25.1.1.1.3.3.2.2" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E41.m1.25.25.25.1.1.1.3.3.2.1" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.1.cmml">​</mo><mi id="A6.E41.m1.25.25.25.1.1.1.3.3.2.3" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="A6.E41.m1.25.25.25.1.1.1.3.3.2.1a" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.1.cmml">​</mo><mi id="A6.E41.m1.25.25.25.1.1.1.3.3.2.4" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.4.cmml">λ</mi></mrow></msqrt></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A6.E41.m1.33.33.2.32.8.8.8.1.2" xref="A6.E41.m1.32.32.1.1.1.cmml">​</mo><mrow id="A6.E41.m1.33.33.2.32.8.8.8.1.1.1"><mo id="A6.E41.m1.26.26.26.2.2.2" xref="A6.E41.m1.32.32.1.1.1.cmml">(</mo><mrow id="A6.E41.m1.33.33.2.32.8.8.8.1.1.1.1"><mn id="A6.E41.m1.27.27.27.3.3.3" xref="A6.E41.m1.27.27.27.3.3.3.cmml">8</mn><mo id="A6.E41.m1.28.28.28.4.4.4" xref="A6.E41.m1.28.28.28.4.4.4.cmml">+</mo><msqrt id="A6.E41.m1.29.29.29.5.5.5" xref="A6.E41.m1.29.29.29.5.5.5.cmml"><mrow id="A6.E41.m1.29.29.29.5.5.5.1" xref="A6.E41.m1.29.29.29.5.5.5.1.cmml"><mo id="A6.E41.m1.29.29.29.5.5.5.1a" xref="A6.E41.m1.29.29.29.5.5.5.1.cmml">−</mo><mrow id="A6.E41.m1.29.29.29.5.5.5.1.3" xref="A6.E41.m1.29.29.29.5.5.5.1.3.cmml"><mi id="A6.E41.m1.29.29.29.5.5.5.1.3.2" xref="A6.E41.m1.29.29.29.5.5.5.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="A6.E41.m1.29.29.29.5.5.5.1.3.1" xref="A6.E41.m1.29.29.29.5.5.5.1.3.1.cmml">​</mo><mi id="A6.E41.m1.29.29.29.5.5.5.1.3.3" xref="A6.E41.m1.29.29.29.5.5.5.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A6.E41.m1.29.29.29.5.5.5.1.3.1a" xref="A6.E41.m1.29.29.29.5.5.5.1.3.1.cmml">​</mo><mi id="A6.E41.m1.29.29.29.5.5.5.1.3.4" xref="A6.E41.m1.29.29.29.5.5.5.1.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.E41.m1.29.29.29.5.5.5.1.3.1b" xref="A6.E41.m1.29.29.29.5.5.5.1.3.1.cmml">​</mo><mrow id="A6.E41.m1.29.29.29.5.5.5.1.3.5.2" xref="A6.E41.m1.29.29.29.5.5.5.1.3.cmml"><mo stretchy="false" id="A6.E41.m1.29.29.29.5.5.5.1.3.5.2.1" xref="A6.E41.m1.29.29.29.5.5.5.1.3.cmml">(</mo><mi id="A6.E41.m1.29.29.29.5.5.5.1.1" xref="A6.E41.m1.29.29.29.5.5.5.1.1.cmml">δ</mi><mo stretchy="false" id="A6.E41.m1.29.29.29.5.5.5.1.3.5.2.2" xref="A6.E41.m1.29.29.29.5.5.5.1.3.cmml">)</mo></mrow></mrow></mrow></msqrt></mrow><mo id="A6.E41.m1.30.30.30.6.6.6" xref="A6.E41.m1.32.32.1.1.1.cmml">)</mo></mrow></mrow><mo lspace="0em" id="A6.E41.m1.31.31.31.7.7.7" xref="A6.E41.m1.32.32.1.1.1.cmml">.</mo></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="A6.E41.m1.33b"><apply id="A6.E41.m1.32.32.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><leq id="A6.E41.m1.11.11.11.11.8.8.cmml" xref="A6.E41.m1.11.11.11.11.8.8"></leq><apply id="A6.E41.m1.32.32.1.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><times id="A6.E41.m1.32.32.1.1.1.1.2.cmml" xref="A6.E41.m1.3.3.3.3.3.3"></times><apply id="A6.E41.m1.32.32.1.1.1.1.3.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><csymbol cd="ambiguous" id="A6.E41.m1.32.32.1.1.1.1.3.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3">subscript</csymbol><ci id="A6.E41.m1.1.1.1.1.1.1.cmml" xref="A6.E41.m1.1.1.1.1.1.1">𝔼</ci><ci id="A6.E41.m1.2.2.2.2.2.2.1.cmml" xref="A6.E41.m1.2.2.2.2.2.2.1">𝒟</ci></apply><apply id="A6.E41.m1.32.32.1.1.1.1.1.2.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><csymbol cd="latexml" id="A6.E41.m1.32.32.1.1.1.1.1.2.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3">delimited-[]</csymbol><apply id="A6.E41.m1.32.32.1.1.1.1.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><times id="A6.E41.m1.32.32.1.1.1.1.1.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"></times><ci id="A6.E41.m1.4.4.4.4.1.1.cmml" xref="A6.E41.m1.4.4.4.4.1.1">ℒ</ci><interval closure="open" id="A6.E41.m1.32.32.1.1.1.1.1.1.1.3.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><ci id="A6.E41.m1.6.6.6.6.3.3.cmml" xref="A6.E41.m1.6.6.6.6.3.3">𝒂</ci><ci id="A6.E41.m1.8.8.8.8.5.5.cmml" xref="A6.E41.m1.8.8.8.8.5.5">𝒃</ci></interval></apply></apply></apply><apply id="A6.E41.m1.32.32.1.1.1.3.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><plus id="A6.E41.m1.22.22.22.22.19.19.cmml" xref="A6.E41.m1.22.22.22.22.19.19"></plus><apply id="A6.E41.m1.32.32.1.1.1.2.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><times id="A6.E41.m1.32.32.1.1.1.2.1.2.cmml" xref="A6.E41.m1.3.3.3.3.3.3"></times><apply id="A6.E41.m1.32.32.1.1.1.2.1.3.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><csymbol cd="ambiguous" id="A6.E41.m1.32.32.1.1.1.2.1.3.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3">subscript</csymbol><ci id="A6.E41.m1.12.12.12.12.9.9.cmml" xref="A6.E41.m1.12.12.12.12.9.9">𝔼</ci><apply id="A6.E41.m1.13.13.13.13.10.10.1.cmml" xref="A6.E41.m1.13.13.13.13.10.10.1"><ci id="A6.E41.m1.13.13.13.13.10.10.1.1.cmml" xref="A6.E41.m1.13.13.13.13.10.10.1.1">~</ci><ci id="A6.E41.m1.13.13.13.13.10.10.1.2.cmml" xref="A6.E41.m1.13.13.13.13.10.10.1.2">𝑨</ci></apply></apply><apply id="A6.E41.m1.32.32.1.1.1.2.1.1.2.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><csymbol cd="latexml" id="A6.E41.m1.32.32.1.1.1.2.1.1.2.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3">delimited-[]</csymbol><apply id="A6.E41.m1.32.32.1.1.1.2.1.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><times id="A6.E41.m1.32.32.1.1.1.2.1.1.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"></times><ci id="A6.E41.m1.15.15.15.15.12.12.cmml" xref="A6.E41.m1.15.15.15.15.12.12">ℒ</ci><interval closure="open" id="A6.E41.m1.32.32.1.1.1.2.1.1.1.1.3.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><ci id="A6.E41.m1.17.17.17.17.14.14.cmml" xref="A6.E41.m1.17.17.17.17.14.14">𝒂</ci><ci id="A6.E41.m1.19.19.19.19.16.16.cmml" xref="A6.E41.m1.19.19.19.19.16.16">𝒃</ci></interval></apply></apply></apply><apply id="A6.E41.m1.23.23.23.23.20.20.cmml" xref="A6.E41.m1.23.23.23.23.20.20"><divide id="A6.E41.m1.23.23.23.23.20.20.2.cmml" xref="A6.E41.m1.23.23.23.23.20.20"></divide><apply id="A6.E41.m1.23.23.23.23.20.20.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1"><times id="A6.E41.m1.23.23.23.23.20.20.1.2.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.2"></times><ci id="A6.E41.m1.23.23.23.23.20.20.1.3.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.3">𝐶</ci><apply id="A6.E41.m1.23.23.23.23.20.20.1.4.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.4"><csymbol cd="ambiguous" id="A6.E41.m1.23.23.23.23.20.20.1.4.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.4">subscript</csymbol><ci id="A6.E41.m1.23.23.23.23.20.20.1.4.2.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.4.2">𝜎</ci><ci id="A6.E41.m1.23.23.23.23.20.20.1.4.3.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.4.3">𝐴</ci></apply><apply id="A6.E41.m1.23.23.23.23.20.20.1.5.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.5"><csymbol cd="ambiguous" id="A6.E41.m1.23.23.23.23.20.20.1.5.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.5">subscript</csymbol><apply id="A6.E41.m1.23.23.23.23.20.20.1.5.2.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.5.2.2"><csymbol cd="latexml" id="A6.E41.m1.23.23.23.23.20.20.1.5.2.1.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.E41.m1.23.23.23.23.20.20.1.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.1">𝒃</ci></apply><cn type="integer" id="A6.E41.m1.23.23.23.23.20.20.1.5.3.cmml" xref="A6.E41.m1.23.23.23.23.20.20.1.5.3">2</cn></apply></apply><apply id="A6.E41.m1.23.23.23.23.20.20.3.cmml" xref="A6.E41.m1.23.23.23.23.20.20.3"><times id="A6.E41.m1.23.23.23.23.20.20.3.1.cmml" xref="A6.E41.m1.23.23.23.23.20.20.3.1"></times><cn type="integer" id="A6.E41.m1.23.23.23.23.20.20.3.2.cmml" xref="A6.E41.m1.23.23.23.23.20.20.3.2">512</cn><apply id="A6.E41.m1.23.23.23.23.20.20.3.3.cmml" xref="A6.E41.m1.23.23.23.23.20.20.3.3"><root id="A6.E41.m1.23.23.23.23.20.20.3.3a.cmml" xref="A6.E41.m1.23.23.23.23.20.20.3.3"></root><ci id="A6.E41.m1.23.23.23.23.20.20.3.3.2.cmml" xref="A6.E41.m1.23.23.23.23.20.20.3.3.2">𝜆</ci></apply></apply></apply><apply id="A6.E41.m1.32.32.1.1.1.3.2.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><times id="A6.E41.m1.32.32.1.1.1.3.2.2.cmml" xref="A6.E41.m1.3.3.3.3.3.3"></times><apply id="A6.E41.m1.25.25.25.1.1.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1"><divide id="A6.E41.m1.25.25.25.1.1.1.2.cmml" xref="A6.E41.m1.25.25.25.1.1.1"></divide><apply id="A6.E41.m1.25.25.25.1.1.1.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1"><times id="A6.E41.m1.25.25.25.1.1.1.1.2.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.2"></times><ci id="A6.E41.m1.25.25.25.1.1.1.1.3.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.3">𝐶</ci><apply id="A6.E41.m1.25.25.25.1.1.1.1.4.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.4"><csymbol cd="ambiguous" id="A6.E41.m1.25.25.25.1.1.1.1.4.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.4">subscript</csymbol><ci id="A6.E41.m1.25.25.25.1.1.1.1.4.2.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.4.2">𝜎</ci><ci id="A6.E41.m1.25.25.25.1.1.1.1.4.3.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.4.3">𝐴</ci></apply><apply id="A6.E41.m1.25.25.25.1.1.1.1.5.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.5"><csymbol cd="ambiguous" id="A6.E41.m1.25.25.25.1.1.1.1.5.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.5">subscript</csymbol><apply id="A6.E41.m1.25.25.25.1.1.1.1.5.2.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.5.2.2"><csymbol cd="latexml" id="A6.E41.m1.25.25.25.1.1.1.1.5.2.1.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.E41.m1.25.25.25.1.1.1.1.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.E41.m1.25.25.25.1.1.1.1.5.3.cmml" xref="A6.E41.m1.25.25.25.1.1.1.1.5.3">2</cn></apply></apply><apply id="A6.E41.m1.25.25.25.1.1.1.3.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3"><times id="A6.E41.m1.25.25.25.1.1.1.3.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.1"></times><cn type="integer" id="A6.E41.m1.25.25.25.1.1.1.3.2.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.2">2</cn><apply id="A6.E41.m1.25.25.25.1.1.1.3.3.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3"><root id="A6.E41.m1.25.25.25.1.1.1.3.3a.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3"></root><apply id="A6.E41.m1.25.25.25.1.1.1.3.3.2.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2"><times id="A6.E41.m1.25.25.25.1.1.1.3.3.2.1.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.1"></times><cn type="integer" id="A6.E41.m1.25.25.25.1.1.1.3.3.2.2.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.2">2</cn><ci id="A6.E41.m1.25.25.25.1.1.1.3.3.2.3.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.3">𝑛</ci><ci id="A6.E41.m1.25.25.25.1.1.1.3.3.2.4.cmml" xref="A6.E41.m1.25.25.25.1.1.1.3.3.2.4">𝜆</ci></apply></apply></apply></apply><apply id="A6.E41.m1.32.32.1.1.1.3.2.1.1.1.cmml" xref="A6.E41.m1.3.3.3.3.3.3"><plus id="A6.E41.m1.28.28.28.4.4.4.cmml" xref="A6.E41.m1.28.28.28.4.4.4"></plus><cn type="integer" id="A6.E41.m1.27.27.27.3.3.3.cmml" xref="A6.E41.m1.27.27.27.3.3.3">8</cn><apply id="A6.E41.m1.29.29.29.5.5.5.cmml" xref="A6.E41.m1.29.29.29.5.5.5"><root id="A6.E41.m1.29.29.29.5.5.5a.cmml" xref="A6.E41.m1.29.29.29.5.5.5"></root><apply id="A6.E41.m1.29.29.29.5.5.5.1.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1"><minus id="A6.E41.m1.29.29.29.5.5.5.1.2.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1"></minus><apply id="A6.E41.m1.29.29.29.5.5.5.1.3.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1.3"><times id="A6.E41.m1.29.29.29.5.5.5.1.3.1.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1.3.1"></times><ci id="A6.E41.m1.29.29.29.5.5.5.1.3.2.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1.3.2">𝑙</ci><ci id="A6.E41.m1.29.29.29.5.5.5.1.3.3.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1.3.3">𝑜</ci><ci id="A6.E41.m1.29.29.29.5.5.5.1.3.4.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1.3.4">𝑔</ci><ci id="A6.E41.m1.29.29.29.5.5.5.1.1.cmml" xref="A6.E41.m1.29.29.29.5.5.5.1.1">𝛿</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E41.m1.33c">\displaystyle\begin{split}\mathbb{E}_{\mathcal{D}}[&amp;\mathcal{L}(\bm{a},\bm{b})]\leq\mathbb{E}_{\tilde{\bm{A}}}[\mathcal{L}(\bm{a},\bm{b})]+\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}+\\
&amp;\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{2n\lambda}}\left(8+\sqrt{-log(\delta)}\right).\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(41)</span></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A6.SS2.9" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="A6.SS2.9.p1" class="ltx_para">
<p id="A6.SS2.9.p1.17" class="ltx_p">The output of <span id="A6.SS2.9.p1.17.1" class="ltx_text ltx_font_typewriter">Maddness-Regress</span> can be decomposed into</p>
<table id="A6.EGx29" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E42"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E42.m1.4" class="ltx_Math" alttext="\displaystyle\hat{\bm{y}}\triangleq f(g(\bm{a}),h(\bm{b}))=\bm{g}^{\top}\bm{P}\bm{b}+\varepsilon+\zeta" display="inline"><semantics id="A6.E42.m1.4a"><mrow id="A6.E42.m1.4.4" xref="A6.E42.m1.4.4.cmml"><mover accent="true" id="A6.E42.m1.4.4.4" xref="A6.E42.m1.4.4.4.cmml"><mi id="A6.E42.m1.4.4.4.2" xref="A6.E42.m1.4.4.4.2.cmml">𝒚</mi><mo id="A6.E42.m1.4.4.4.1" xref="A6.E42.m1.4.4.4.1.cmml">^</mo></mover><mo id="A6.E42.m1.4.4.5" xref="A6.E42.m1.4.4.5.cmml">≜</mo><mrow id="A6.E42.m1.4.4.2" xref="A6.E42.m1.4.4.2.cmml"><mi id="A6.E42.m1.4.4.2.4" xref="A6.E42.m1.4.4.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="A6.E42.m1.4.4.2.3" xref="A6.E42.m1.4.4.2.3.cmml">​</mo><mrow id="A6.E42.m1.4.4.2.2.2" xref="A6.E42.m1.4.4.2.2.3.cmml"><mo stretchy="false" id="A6.E42.m1.4.4.2.2.2.3" xref="A6.E42.m1.4.4.2.2.3.cmml">(</mo><mrow id="A6.E42.m1.3.3.1.1.1.1" xref="A6.E42.m1.3.3.1.1.1.1.cmml"><mi id="A6.E42.m1.3.3.1.1.1.1.2" xref="A6.E42.m1.3.3.1.1.1.1.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.E42.m1.3.3.1.1.1.1.1" xref="A6.E42.m1.3.3.1.1.1.1.1.cmml">​</mo><mrow id="A6.E42.m1.3.3.1.1.1.1.3.2" xref="A6.E42.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="A6.E42.m1.3.3.1.1.1.1.3.2.1" xref="A6.E42.m1.3.3.1.1.1.1.cmml">(</mo><mi id="A6.E42.m1.1.1" xref="A6.E42.m1.1.1.cmml">𝒂</mi><mo stretchy="false" id="A6.E42.m1.3.3.1.1.1.1.3.2.2" xref="A6.E42.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A6.E42.m1.4.4.2.2.2.4" xref="A6.E42.m1.4.4.2.2.3.cmml">,</mo><mrow id="A6.E42.m1.4.4.2.2.2.2" xref="A6.E42.m1.4.4.2.2.2.2.cmml"><mi id="A6.E42.m1.4.4.2.2.2.2.2" xref="A6.E42.m1.4.4.2.2.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A6.E42.m1.4.4.2.2.2.2.1" xref="A6.E42.m1.4.4.2.2.2.2.1.cmml">​</mo><mrow id="A6.E42.m1.4.4.2.2.2.2.3.2" xref="A6.E42.m1.4.4.2.2.2.2.cmml"><mo stretchy="false" id="A6.E42.m1.4.4.2.2.2.2.3.2.1" xref="A6.E42.m1.4.4.2.2.2.2.cmml">(</mo><mi id="A6.E42.m1.2.2" xref="A6.E42.m1.2.2.cmml">𝒃</mi><mo stretchy="false" id="A6.E42.m1.4.4.2.2.2.2.3.2.2" xref="A6.E42.m1.4.4.2.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A6.E42.m1.4.4.2.2.2.5" xref="A6.E42.m1.4.4.2.2.3.cmml">)</mo></mrow></mrow><mo id="A6.E42.m1.4.4.6" xref="A6.E42.m1.4.4.6.cmml">=</mo><mrow id="A6.E42.m1.4.4.7" xref="A6.E42.m1.4.4.7.cmml"><mrow id="A6.E42.m1.4.4.7.2" xref="A6.E42.m1.4.4.7.2.cmml"><msup id="A6.E42.m1.4.4.7.2.2" xref="A6.E42.m1.4.4.7.2.2.cmml"><mi id="A6.E42.m1.4.4.7.2.2.2" xref="A6.E42.m1.4.4.7.2.2.2.cmml">𝒈</mi><mo id="A6.E42.m1.4.4.7.2.2.3" xref="A6.E42.m1.4.4.7.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.E42.m1.4.4.7.2.1" xref="A6.E42.m1.4.4.7.2.1.cmml">​</mo><mi id="A6.E42.m1.4.4.7.2.3" xref="A6.E42.m1.4.4.7.2.3.cmml">𝑷</mi><mo lspace="0em" rspace="0em" id="A6.E42.m1.4.4.7.2.1a" xref="A6.E42.m1.4.4.7.2.1.cmml">​</mo><mi id="A6.E42.m1.4.4.7.2.4" xref="A6.E42.m1.4.4.7.2.4.cmml">𝒃</mi></mrow><mo id="A6.E42.m1.4.4.7.1" xref="A6.E42.m1.4.4.7.1.cmml">+</mo><mi id="A6.E42.m1.4.4.7.3" xref="A6.E42.m1.4.4.7.3.cmml">ε</mi><mo id="A6.E42.m1.4.4.7.1a" xref="A6.E42.m1.4.4.7.1.cmml">+</mo><mi id="A6.E42.m1.4.4.7.4" xref="A6.E42.m1.4.4.7.4.cmml">ζ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E42.m1.4b"><apply id="A6.E42.m1.4.4.cmml" xref="A6.E42.m1.4.4"><and id="A6.E42.m1.4.4a.cmml" xref="A6.E42.m1.4.4"></and><apply id="A6.E42.m1.4.4b.cmml" xref="A6.E42.m1.4.4"><ci id="A6.E42.m1.4.4.5.cmml" xref="A6.E42.m1.4.4.5">≜</ci><apply id="A6.E42.m1.4.4.4.cmml" xref="A6.E42.m1.4.4.4"><ci id="A6.E42.m1.4.4.4.1.cmml" xref="A6.E42.m1.4.4.4.1">^</ci><ci id="A6.E42.m1.4.4.4.2.cmml" xref="A6.E42.m1.4.4.4.2">𝒚</ci></apply><apply id="A6.E42.m1.4.4.2.cmml" xref="A6.E42.m1.4.4.2"><times id="A6.E42.m1.4.4.2.3.cmml" xref="A6.E42.m1.4.4.2.3"></times><ci id="A6.E42.m1.4.4.2.4.cmml" xref="A6.E42.m1.4.4.2.4">𝑓</ci><interval closure="open" id="A6.E42.m1.4.4.2.2.3.cmml" xref="A6.E42.m1.4.4.2.2.2"><apply id="A6.E42.m1.3.3.1.1.1.1.cmml" xref="A6.E42.m1.3.3.1.1.1.1"><times id="A6.E42.m1.3.3.1.1.1.1.1.cmml" xref="A6.E42.m1.3.3.1.1.1.1.1"></times><ci id="A6.E42.m1.3.3.1.1.1.1.2.cmml" xref="A6.E42.m1.3.3.1.1.1.1.2">𝑔</ci><ci id="A6.E42.m1.1.1.cmml" xref="A6.E42.m1.1.1">𝒂</ci></apply><apply id="A6.E42.m1.4.4.2.2.2.2.cmml" xref="A6.E42.m1.4.4.2.2.2.2"><times id="A6.E42.m1.4.4.2.2.2.2.1.cmml" xref="A6.E42.m1.4.4.2.2.2.2.1"></times><ci id="A6.E42.m1.4.4.2.2.2.2.2.cmml" xref="A6.E42.m1.4.4.2.2.2.2.2">ℎ</ci><ci id="A6.E42.m1.2.2.cmml" xref="A6.E42.m1.2.2">𝒃</ci></apply></interval></apply></apply><apply id="A6.E42.m1.4.4c.cmml" xref="A6.E42.m1.4.4"><eq id="A6.E42.m1.4.4.6.cmml" xref="A6.E42.m1.4.4.6"></eq><share href="#A6.E42.m1.4.4.2.cmml" id="A6.E42.m1.4.4d.cmml" xref="A6.E42.m1.4.4"></share><apply id="A6.E42.m1.4.4.7.cmml" xref="A6.E42.m1.4.4.7"><plus id="A6.E42.m1.4.4.7.1.cmml" xref="A6.E42.m1.4.4.7.1"></plus><apply id="A6.E42.m1.4.4.7.2.cmml" xref="A6.E42.m1.4.4.7.2"><times id="A6.E42.m1.4.4.7.2.1.cmml" xref="A6.E42.m1.4.4.7.2.1"></times><apply id="A6.E42.m1.4.4.7.2.2.cmml" xref="A6.E42.m1.4.4.7.2.2"><csymbol cd="ambiguous" id="A6.E42.m1.4.4.7.2.2.1.cmml" xref="A6.E42.m1.4.4.7.2.2">superscript</csymbol><ci id="A6.E42.m1.4.4.7.2.2.2.cmml" xref="A6.E42.m1.4.4.7.2.2.2">𝒈</ci><csymbol cd="latexml" id="A6.E42.m1.4.4.7.2.2.3.cmml" xref="A6.E42.m1.4.4.7.2.2.3">top</csymbol></apply><ci id="A6.E42.m1.4.4.7.2.3.cmml" xref="A6.E42.m1.4.4.7.2.3">𝑷</ci><ci id="A6.E42.m1.4.4.7.2.4.cmml" xref="A6.E42.m1.4.4.7.2.4">𝒃</ci></apply><ci id="A6.E42.m1.4.4.7.3.cmml" xref="A6.E42.m1.4.4.7.3">𝜀</ci><ci id="A6.E42.m1.4.4.7.4.cmml" xref="A6.E42.m1.4.4.7.4">𝜁</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E42.m1.4c">\displaystyle\hat{\bm{y}}\triangleq f(g(\bm{a}),h(\bm{b}))=\bm{g}^{\top}\bm{P}\bm{b}+\varepsilon+\zeta</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(42)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.9.p1.16" class="ltx_p">where <math id="A6.SS2.9.p1.1.m1.1" class="ltx_Math" alttext="\bm{g}=g(\bm{a})" display="inline"><semantics id="A6.SS2.9.p1.1.m1.1a"><mrow id="A6.SS2.9.p1.1.m1.1.2" xref="A6.SS2.9.p1.1.m1.1.2.cmml"><mi id="A6.SS2.9.p1.1.m1.1.2.2" xref="A6.SS2.9.p1.1.m1.1.2.2.cmml">𝒈</mi><mo id="A6.SS2.9.p1.1.m1.1.2.1" xref="A6.SS2.9.p1.1.m1.1.2.1.cmml">=</mo><mrow id="A6.SS2.9.p1.1.m1.1.2.3" xref="A6.SS2.9.p1.1.m1.1.2.3.cmml"><mi id="A6.SS2.9.p1.1.m1.1.2.3.2" xref="A6.SS2.9.p1.1.m1.1.2.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.1.m1.1.2.3.1" xref="A6.SS2.9.p1.1.m1.1.2.3.1.cmml">​</mo><mrow id="A6.SS2.9.p1.1.m1.1.2.3.3.2" xref="A6.SS2.9.p1.1.m1.1.2.3.cmml"><mo stretchy="false" id="A6.SS2.9.p1.1.m1.1.2.3.3.2.1" xref="A6.SS2.9.p1.1.m1.1.2.3.cmml">(</mo><mi id="A6.SS2.9.p1.1.m1.1.1" xref="A6.SS2.9.p1.1.m1.1.1.cmml">𝒂</mi><mo stretchy="false" id="A6.SS2.9.p1.1.m1.1.2.3.3.2.2" xref="A6.SS2.9.p1.1.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.1.m1.1b"><apply id="A6.SS2.9.p1.1.m1.1.2.cmml" xref="A6.SS2.9.p1.1.m1.1.2"><eq id="A6.SS2.9.p1.1.m1.1.2.1.cmml" xref="A6.SS2.9.p1.1.m1.1.2.1"></eq><ci id="A6.SS2.9.p1.1.m1.1.2.2.cmml" xref="A6.SS2.9.p1.1.m1.1.2.2">𝒈</ci><apply id="A6.SS2.9.p1.1.m1.1.2.3.cmml" xref="A6.SS2.9.p1.1.m1.1.2.3"><times id="A6.SS2.9.p1.1.m1.1.2.3.1.cmml" xref="A6.SS2.9.p1.1.m1.1.2.3.1"></times><ci id="A6.SS2.9.p1.1.m1.1.2.3.2.cmml" xref="A6.SS2.9.p1.1.m1.1.2.3.2">𝑔</ci><ci id="A6.SS2.9.p1.1.m1.1.1.cmml" xref="A6.SS2.9.p1.1.m1.1.1">𝒂</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.1.m1.1c">\bm{g}=g(\bm{a})</annotation></semantics></math>, <math id="A6.SS2.9.p1.2.m2.1" class="ltx_Math" alttext="\bm{P}" display="inline"><semantics id="A6.SS2.9.p1.2.m2.1a"><mi id="A6.SS2.9.p1.2.m2.1.1" xref="A6.SS2.9.p1.2.m2.1.1.cmml">𝑷</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.2.m2.1b"><ci id="A6.SS2.9.p1.2.m2.1.1.cmml" xref="A6.SS2.9.p1.2.m2.1.1">𝑷</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.2.m2.1c">\bm{P}</annotation></semantics></math> is the matrix of prototypes, <math id="A6.SS2.9.p1.3.m3.1" class="ltx_Math" alttext="\varepsilon" display="inline"><semantics id="A6.SS2.9.p1.3.m3.1a"><mi id="A6.SS2.9.p1.3.m3.1.1" xref="A6.SS2.9.p1.3.m3.1.1.cmml">ε</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.3.m3.1b"><ci id="A6.SS2.9.p1.3.m3.1.1.cmml" xref="A6.SS2.9.p1.3.m3.1.1">𝜀</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.3.m3.1c">\varepsilon</annotation></semantics></math> is data-independent noise from the averaging process<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>We continue to make the assumption that the least significant bits of the lookup table entries are independent Bernoulli(0.5) random variables, which is nearly true in practice. Even if this assumption does not hold, this noise does not contribute to the generalization gap unless it differs between train and test sets.</span></span></span>, and <math id="A6.SS2.9.p1.4.m4.1" class="ltx_Math" alttext="\zeta" display="inline"><semantics id="A6.SS2.9.p1.4.m4.1a"><mi id="A6.SS2.9.p1.4.m4.1.1" xref="A6.SS2.9.p1.4.m4.1.1.cmml">ζ</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.4.m4.1b"><ci id="A6.SS2.9.p1.4.m4.1.1.cmml" xref="A6.SS2.9.p1.4.m4.1.1">𝜁</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.4.m4.1c">\zeta</annotation></semantics></math> is noise from quantizing the lookup table entries. By Lemma <a href="#A6.Thmlemma5" title="Lemma F.5 (Maximum Table Quantization Loss). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.5</span></a>, <math id="A6.SS2.9.p1.5.m5.1" class="ltx_Math" alttext="\zeta\leq\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}" display="inline"><semantics id="A6.SS2.9.p1.5.m5.1a"><mrow id="A6.SS2.9.p1.5.m5.1.2" xref="A6.SS2.9.p1.5.m5.1.2.cmml"><mi id="A6.SS2.9.p1.5.m5.1.2.2" xref="A6.SS2.9.p1.5.m5.1.2.2.cmml">ζ</mi><mo id="A6.SS2.9.p1.5.m5.1.2.1" xref="A6.SS2.9.p1.5.m5.1.2.1.cmml">≤</mo><mfrac id="A6.SS2.9.p1.5.m5.1.1" xref="A6.SS2.9.p1.5.m5.1.1.cmml"><mrow id="A6.SS2.9.p1.5.m5.1.1.1" xref="A6.SS2.9.p1.5.m5.1.1.1.cmml"><mi id="A6.SS2.9.p1.5.m5.1.1.1.3" xref="A6.SS2.9.p1.5.m5.1.1.1.3.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.5.m5.1.1.1.2" xref="A6.SS2.9.p1.5.m5.1.1.1.2.cmml">​</mo><msub id="A6.SS2.9.p1.5.m5.1.1.1.4" xref="A6.SS2.9.p1.5.m5.1.1.1.4.cmml"><mi id="A6.SS2.9.p1.5.m5.1.1.1.4.2" xref="A6.SS2.9.p1.5.m5.1.1.1.4.2.cmml">σ</mi><mi id="A6.SS2.9.p1.5.m5.1.1.1.4.3" xref="A6.SS2.9.p1.5.m5.1.1.1.4.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.5.m5.1.1.1.2a" xref="A6.SS2.9.p1.5.m5.1.1.1.2.cmml">​</mo><msub id="A6.SS2.9.p1.5.m5.1.1.1.5" xref="A6.SS2.9.p1.5.m5.1.1.1.5.cmml"><mrow id="A6.SS2.9.p1.5.m5.1.1.1.5.2.2" xref="A6.SS2.9.p1.5.m5.1.1.1.5.2.1.cmml"><mo fence="true" rspace="0em" id="A6.SS2.9.p1.5.m5.1.1.1.5.2.2.1" xref="A6.SS2.9.p1.5.m5.1.1.1.5.2.1.1.cmml">∥</mo><mi id="A6.SS2.9.p1.5.m5.1.1.1.1" xref="A6.SS2.9.p1.5.m5.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.SS2.9.p1.5.m5.1.1.1.5.2.2.2" xref="A6.SS2.9.p1.5.m5.1.1.1.5.2.1.1.cmml">∥</mo></mrow><mn id="A6.SS2.9.p1.5.m5.1.1.1.5.3" xref="A6.SS2.9.p1.5.m5.1.1.1.5.3.cmml">2</mn></msub></mrow><mrow id="A6.SS2.9.p1.5.m5.1.1.3" xref="A6.SS2.9.p1.5.m5.1.1.3.cmml"><mn id="A6.SS2.9.p1.5.m5.1.1.3.2" xref="A6.SS2.9.p1.5.m5.1.1.3.2.cmml">512</mn><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.5.m5.1.1.3.1" xref="A6.SS2.9.p1.5.m5.1.1.3.1.cmml">​</mo><msqrt id="A6.SS2.9.p1.5.m5.1.1.3.3" xref="A6.SS2.9.p1.5.m5.1.1.3.3.cmml"><mi id="A6.SS2.9.p1.5.m5.1.1.3.3.2" xref="A6.SS2.9.p1.5.m5.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.5.m5.1b"><apply id="A6.SS2.9.p1.5.m5.1.2.cmml" xref="A6.SS2.9.p1.5.m5.1.2"><leq id="A6.SS2.9.p1.5.m5.1.2.1.cmml" xref="A6.SS2.9.p1.5.m5.1.2.1"></leq><ci id="A6.SS2.9.p1.5.m5.1.2.2.cmml" xref="A6.SS2.9.p1.5.m5.1.2.2">𝜁</ci><apply id="A6.SS2.9.p1.5.m5.1.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1"><divide id="A6.SS2.9.p1.5.m5.1.1.2.cmml" xref="A6.SS2.9.p1.5.m5.1.1"></divide><apply id="A6.SS2.9.p1.5.m5.1.1.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1"><times id="A6.SS2.9.p1.5.m5.1.1.1.2.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.2"></times><ci id="A6.SS2.9.p1.5.m5.1.1.1.3.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.3">𝐶</ci><apply id="A6.SS2.9.p1.5.m5.1.1.1.4.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.4"><csymbol cd="ambiguous" id="A6.SS2.9.p1.5.m5.1.1.1.4.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.4">subscript</csymbol><ci id="A6.SS2.9.p1.5.m5.1.1.1.4.2.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.4.2">𝜎</ci><ci id="A6.SS2.9.p1.5.m5.1.1.1.4.3.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.4.3">𝐴</ci></apply><apply id="A6.SS2.9.p1.5.m5.1.1.1.5.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.5"><csymbol cd="ambiguous" id="A6.SS2.9.p1.5.m5.1.1.1.5.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.5">subscript</csymbol><apply id="A6.SS2.9.p1.5.m5.1.1.1.5.2.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.5.2.2"><csymbol cd="latexml" id="A6.SS2.9.p1.5.m5.1.1.1.5.2.1.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.5.2.2.1">delimited-∥∥</csymbol><ci id="A6.SS2.9.p1.5.m5.1.1.1.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.SS2.9.p1.5.m5.1.1.1.5.3.cmml" xref="A6.SS2.9.p1.5.m5.1.1.1.5.3">2</cn></apply></apply><apply id="A6.SS2.9.p1.5.m5.1.1.3.cmml" xref="A6.SS2.9.p1.5.m5.1.1.3"><times id="A6.SS2.9.p1.5.m5.1.1.3.1.cmml" xref="A6.SS2.9.p1.5.m5.1.1.3.1"></times><cn type="integer" id="A6.SS2.9.p1.5.m5.1.1.3.2.cmml" xref="A6.SS2.9.p1.5.m5.1.1.3.2">512</cn><apply id="A6.SS2.9.p1.5.m5.1.1.3.3.cmml" xref="A6.SS2.9.p1.5.m5.1.1.3.3"><root id="A6.SS2.9.p1.5.m5.1.1.3.3a.cmml" xref="A6.SS2.9.p1.5.m5.1.1.3.3"></root><ci id="A6.SS2.9.p1.5.m5.1.1.3.3.2.cmml" xref="A6.SS2.9.p1.5.m5.1.1.3.3.2">𝜆</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.5.m5.1c">\zeta\leq\frac{C\sigma_{A}\lVert\bm{b}\rVert_{2}}{512\sqrt{\lambda}}</annotation></semantics></math> (accounting for the second term in Equation <a href="#A6.E41" title="In Lemma F.6 (Generalization Error of Maddness-Regress). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">41</span></a>). We therefore need only obtain a guarantee for <math id="A6.SS2.9.p1.6.m6.1" class="ltx_Math" alttext="|\bm{g}^{\top}\bm{P}\bm{b}-\bm{a}^{\top}\bm{b}|" display="inline"><semantics id="A6.SS2.9.p1.6.m6.1a"><mrow id="A6.SS2.9.p1.6.m6.1.1.1" xref="A6.SS2.9.p1.6.m6.1.1.2.cmml"><mo stretchy="false" id="A6.SS2.9.p1.6.m6.1.1.1.2" xref="A6.SS2.9.p1.6.m6.1.1.2.1.cmml">|</mo><mrow id="A6.SS2.9.p1.6.m6.1.1.1.1" xref="A6.SS2.9.p1.6.m6.1.1.1.1.cmml"><mrow id="A6.SS2.9.p1.6.m6.1.1.1.1.2" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.cmml"><msup id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.cmml"><mi id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.2" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.2.cmml">𝒈</mi><mo id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.3" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.6.m6.1.1.1.1.2.1" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.1.cmml">​</mo><mi id="A6.SS2.9.p1.6.m6.1.1.1.1.2.3" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.3.cmml">𝑷</mi><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.6.m6.1.1.1.1.2.1a" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.1.cmml">​</mo><mi id="A6.SS2.9.p1.6.m6.1.1.1.1.2.4" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.4.cmml">𝒃</mi></mrow><mo id="A6.SS2.9.p1.6.m6.1.1.1.1.1" xref="A6.SS2.9.p1.6.m6.1.1.1.1.1.cmml">−</mo><mrow id="A6.SS2.9.p1.6.m6.1.1.1.1.3" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.cmml"><msup id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.cmml"><mi id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.2" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.2.cmml">𝒂</mi><mo id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.3" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.6.m6.1.1.1.1.3.1" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.1.cmml">​</mo><mi id="A6.SS2.9.p1.6.m6.1.1.1.1.3.3" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.3.cmml">𝒃</mi></mrow></mrow><mo stretchy="false" id="A6.SS2.9.p1.6.m6.1.1.1.3" xref="A6.SS2.9.p1.6.m6.1.1.2.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.6.m6.1b"><apply id="A6.SS2.9.p1.6.m6.1.1.2.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1"><abs id="A6.SS2.9.p1.6.m6.1.1.2.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.2"></abs><apply id="A6.SS2.9.p1.6.m6.1.1.1.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1"><minus id="A6.SS2.9.p1.6.m6.1.1.1.1.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.1"></minus><apply id="A6.SS2.9.p1.6.m6.1.1.1.1.2.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2"><times id="A6.SS2.9.p1.6.m6.1.1.1.1.2.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.1"></times><apply id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2">superscript</csymbol><ci id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.2.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.2">𝒈</ci><csymbol cd="latexml" id="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.3.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.2.3">top</csymbol></apply><ci id="A6.SS2.9.p1.6.m6.1.1.1.1.2.3.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.3">𝑷</ci><ci id="A6.SS2.9.p1.6.m6.1.1.1.1.2.4.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.2.4">𝒃</ci></apply><apply id="A6.SS2.9.p1.6.m6.1.1.1.1.3.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3"><times id="A6.SS2.9.p1.6.m6.1.1.1.1.3.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.1"></times><apply id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2"><csymbol cd="ambiguous" id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.1.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2">superscript</csymbol><ci id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.2.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.2">𝒂</ci><csymbol cd="latexml" id="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.3.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.2.3">top</csymbol></apply><ci id="A6.SS2.9.p1.6.m6.1.1.1.1.3.3.cmml" xref="A6.SS2.9.p1.6.m6.1.1.1.1.3.3">𝒃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.6.m6.1c">|\bm{g}^{\top}\bm{P}\bm{b}-\bm{a}^{\top}\bm{b}|</annotation></semantics></math>. Defining <math id="A6.SS2.9.p1.7.m7.1" class="ltx_Math" alttext="\bm{w}\triangleq\bm{P}\bm{b}" display="inline"><semantics id="A6.SS2.9.p1.7.m7.1a"><mrow id="A6.SS2.9.p1.7.m7.1.1" xref="A6.SS2.9.p1.7.m7.1.1.cmml"><mi id="A6.SS2.9.p1.7.m7.1.1.2" xref="A6.SS2.9.p1.7.m7.1.1.2.cmml">𝒘</mi><mo id="A6.SS2.9.p1.7.m7.1.1.1" xref="A6.SS2.9.p1.7.m7.1.1.1.cmml">≜</mo><mrow id="A6.SS2.9.p1.7.m7.1.1.3" xref="A6.SS2.9.p1.7.m7.1.1.3.cmml"><mi id="A6.SS2.9.p1.7.m7.1.1.3.2" xref="A6.SS2.9.p1.7.m7.1.1.3.2.cmml">𝑷</mi><mo lspace="0em" rspace="0em" id="A6.SS2.9.p1.7.m7.1.1.3.1" xref="A6.SS2.9.p1.7.m7.1.1.3.1.cmml">​</mo><mi id="A6.SS2.9.p1.7.m7.1.1.3.3" xref="A6.SS2.9.p1.7.m7.1.1.3.3.cmml">𝒃</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.7.m7.1b"><apply id="A6.SS2.9.p1.7.m7.1.1.cmml" xref="A6.SS2.9.p1.7.m7.1.1"><ci id="A6.SS2.9.p1.7.m7.1.1.1.cmml" xref="A6.SS2.9.p1.7.m7.1.1.1">≜</ci><ci id="A6.SS2.9.p1.7.m7.1.1.2.cmml" xref="A6.SS2.9.p1.7.m7.1.1.2">𝒘</ci><apply id="A6.SS2.9.p1.7.m7.1.1.3.cmml" xref="A6.SS2.9.p1.7.m7.1.1.3"><times id="A6.SS2.9.p1.7.m7.1.1.3.1.cmml" xref="A6.SS2.9.p1.7.m7.1.1.3.1"></times><ci id="A6.SS2.9.p1.7.m7.1.1.3.2.cmml" xref="A6.SS2.9.p1.7.m7.1.1.3.2">𝑷</ci><ci id="A6.SS2.9.p1.7.m7.1.1.3.3.cmml" xref="A6.SS2.9.p1.7.m7.1.1.3.3">𝒃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.7.m7.1c">\bm{w}\triangleq\bm{P}\bm{b}</annotation></semantics></math>, we see that <span id="A6.SS2.9.p1.16.1" class="ltx_text ltx_font_typewriter">Maddness-Regress</span> is a linear model, and therefore subject to Theorem <a href="#A6.Thmtheorem1" title="Theorem F.1 ((Kakade et al., 2009), Corollary 5). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.1</span></a>. Given an upper bound on the Lipschitz constant of the loss, a bound on the <math id="A6.SS2.9.p1.8.m8.1" class="ltx_Math" alttext="L_{2}" display="inline"><semantics id="A6.SS2.9.p1.8.m8.1a"><msub id="A6.SS2.9.p1.8.m8.1.1" xref="A6.SS2.9.p1.8.m8.1.1.cmml"><mi id="A6.SS2.9.p1.8.m8.1.1.2" xref="A6.SS2.9.p1.8.m8.1.1.2.cmml">L</mi><mn id="A6.SS2.9.p1.8.m8.1.1.3" xref="A6.SS2.9.p1.8.m8.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.8.m8.1b"><apply id="A6.SS2.9.p1.8.m8.1.1.cmml" xref="A6.SS2.9.p1.8.m8.1.1"><csymbol cd="ambiguous" id="A6.SS2.9.p1.8.m8.1.1.1.cmml" xref="A6.SS2.9.p1.8.m8.1.1">subscript</csymbol><ci id="A6.SS2.9.p1.8.m8.1.1.2.cmml" xref="A6.SS2.9.p1.8.m8.1.1.2">𝐿</ci><cn type="integer" id="A6.SS2.9.p1.8.m8.1.1.3.cmml" xref="A6.SS2.9.p1.8.m8.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.8.m8.1c">L_{2}</annotation></semantics></math> norm of <math id="A6.SS2.9.p1.9.m9.1" class="ltx_Math" alttext="\bm{g}" display="inline"><semantics id="A6.SS2.9.p1.9.m9.1a"><mi id="A6.SS2.9.p1.9.m9.1.1" xref="A6.SS2.9.p1.9.m9.1.1.cmml">𝒈</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.9.m9.1b"><ci id="A6.SS2.9.p1.9.m9.1.1.cmml" xref="A6.SS2.9.p1.9.m9.1.1">𝒈</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.9.m9.1c">\bm{g}</annotation></semantics></math>, and a bound on the <math id="A6.SS2.9.p1.10.m10.1" class="ltx_Math" alttext="L_{2}" display="inline"><semantics id="A6.SS2.9.p1.10.m10.1a"><msub id="A6.SS2.9.p1.10.m10.1.1" xref="A6.SS2.9.p1.10.m10.1.1.cmml"><mi id="A6.SS2.9.p1.10.m10.1.1.2" xref="A6.SS2.9.p1.10.m10.1.1.2.cmml">L</mi><mn id="A6.SS2.9.p1.10.m10.1.1.3" xref="A6.SS2.9.p1.10.m10.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.10.m10.1b"><apply id="A6.SS2.9.p1.10.m10.1.1.cmml" xref="A6.SS2.9.p1.10.m10.1.1"><csymbol cd="ambiguous" id="A6.SS2.9.p1.10.m10.1.1.1.cmml" xref="A6.SS2.9.p1.10.m10.1.1">subscript</csymbol><ci id="A6.SS2.9.p1.10.m10.1.1.2.cmml" xref="A6.SS2.9.p1.10.m10.1.1.2">𝐿</ci><cn type="integer" id="A6.SS2.9.p1.10.m10.1.1.3.cmml" xref="A6.SS2.9.p1.10.m10.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.10.m10.1c">L_{2}</annotation></semantics></math> norm of <math id="A6.SS2.9.p1.11.m11.1" class="ltx_Math" alttext="\bm{w}" display="inline"><semantics id="A6.SS2.9.p1.11.m11.1a"><mi id="A6.SS2.9.p1.11.m11.1.1" xref="A6.SS2.9.p1.11.m11.1.1.cmml">𝒘</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.11.m11.1b"><ci id="A6.SS2.9.p1.11.m11.1.1.cmml" xref="A6.SS2.9.p1.11.m11.1.1">𝒘</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.11.m11.1c">\bm{w}</annotation></semantics></math>, we can apply this theorem. The Lipschitz constant for the absolute loss is 1. The <math id="A6.SS2.9.p1.12.m12.1" class="ltx_Math" alttext="L_{2}" display="inline"><semantics id="A6.SS2.9.p1.12.m12.1a"><msub id="A6.SS2.9.p1.12.m12.1.1" xref="A6.SS2.9.p1.12.m12.1.1.cmml"><mi id="A6.SS2.9.p1.12.m12.1.1.2" xref="A6.SS2.9.p1.12.m12.1.1.2.cmml">L</mi><mn id="A6.SS2.9.p1.12.m12.1.1.3" xref="A6.SS2.9.p1.12.m12.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.12.m12.1b"><apply id="A6.SS2.9.p1.12.m12.1.1.cmml" xref="A6.SS2.9.p1.12.m12.1.1"><csymbol cd="ambiguous" id="A6.SS2.9.p1.12.m12.1.1.1.cmml" xref="A6.SS2.9.p1.12.m12.1.1">subscript</csymbol><ci id="A6.SS2.9.p1.12.m12.1.1.2.cmml" xref="A6.SS2.9.p1.12.m12.1.1.2">𝐿</ci><cn type="integer" id="A6.SS2.9.p1.12.m12.1.1.3.cmml" xref="A6.SS2.9.p1.12.m12.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.12.m12.1c">L_{2}</annotation></semantics></math> norm of <math id="A6.SS2.9.p1.13.m13.1" class="ltx_Math" alttext="\bm{g}" display="inline"><semantics id="A6.SS2.9.p1.13.m13.1a"><mi id="A6.SS2.9.p1.13.m13.1.1" xref="A6.SS2.9.p1.13.m13.1.1.cmml">𝒈</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.13.m13.1b"><ci id="A6.SS2.9.p1.13.m13.1.1.cmml" xref="A6.SS2.9.p1.13.m13.1.1">𝒈</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.13.m13.1c">\bm{g}</annotation></semantics></math> is exactly <math id="A6.SS2.9.p1.14.m14.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A6.SS2.9.p1.14.m14.1a"><mi id="A6.SS2.9.p1.14.m14.1.1" xref="A6.SS2.9.p1.14.m14.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.14.m14.1b"><ci id="A6.SS2.9.p1.14.m14.1.1.cmml" xref="A6.SS2.9.p1.14.m14.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.14.m14.1c">C</annotation></semantics></math>. The <math id="A6.SS2.9.p1.15.m15.1" class="ltx_Math" alttext="L_{2}" display="inline"><semantics id="A6.SS2.9.p1.15.m15.1a"><msub id="A6.SS2.9.p1.15.m15.1.1" xref="A6.SS2.9.p1.15.m15.1.1.cmml"><mi id="A6.SS2.9.p1.15.m15.1.1.2" xref="A6.SS2.9.p1.15.m15.1.1.2.cmml">L</mi><mn id="A6.SS2.9.p1.15.m15.1.1.3" xref="A6.SS2.9.p1.15.m15.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.15.m15.1b"><apply id="A6.SS2.9.p1.15.m15.1.1.cmml" xref="A6.SS2.9.p1.15.m15.1.1"><csymbol cd="ambiguous" id="A6.SS2.9.p1.15.m15.1.1.1.cmml" xref="A6.SS2.9.p1.15.m15.1.1">subscript</csymbol><ci id="A6.SS2.9.p1.15.m15.1.1.2.cmml" xref="A6.SS2.9.p1.15.m15.1.1.2">𝐿</ci><cn type="integer" id="A6.SS2.9.p1.15.m15.1.1.3.cmml" xref="A6.SS2.9.p1.15.m15.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.15.m15.1c">L_{2}</annotation></semantics></math> norm of <math id="A6.SS2.9.p1.16.m16.1" class="ltx_Math" alttext="\bm{w}" display="inline"><semantics id="A6.SS2.9.p1.16.m16.1a"><mi id="A6.SS2.9.p1.16.m16.1.1" xref="A6.SS2.9.p1.16.m16.1.1.cmml">𝒘</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.9.p1.16.m16.1b"><ci id="A6.SS2.9.p1.16.m16.1.1.cmml" xref="A6.SS2.9.p1.16.m16.1.1">𝒘</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.9.p1.16.m16.1c">\bm{w}</annotation></semantics></math> can be bounded as</p>
<table id="A6.EGx30" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E43"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E43.m1.1" class="ltx_Math" alttext="\displaystyle\lVert\bm{w}\rVert_{2}" display="inline"><semantics id="A6.E43.m1.1a"><msub id="A6.E43.m1.1.2" xref="A6.E43.m1.1.2.cmml"><mrow id="A6.E43.m1.1.2.2.2" xref="A6.E43.m1.1.2.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E43.m1.1.2.2.2.1" xref="A6.E43.m1.1.2.2.1.1.cmml">∥</mo><mi id="A6.E43.m1.1.1" xref="A6.E43.m1.1.1.cmml">𝒘</mi><mo fence="true" lspace="0em" id="A6.E43.m1.1.2.2.2.2" xref="A6.E43.m1.1.2.2.1.1.cmml">∥</mo></mrow><mn id="A6.E43.m1.1.2.3" xref="A6.E43.m1.1.2.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A6.E43.m1.1b"><apply id="A6.E43.m1.1.2.cmml" xref="A6.E43.m1.1.2"><csymbol cd="ambiguous" id="A6.E43.m1.1.2.1.cmml" xref="A6.E43.m1.1.2">subscript</csymbol><apply id="A6.E43.m1.1.2.2.1.cmml" xref="A6.E43.m1.1.2.2.2"><csymbol cd="latexml" id="A6.E43.m1.1.2.2.1.1.cmml" xref="A6.E43.m1.1.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.E43.m1.1.1.cmml" xref="A6.E43.m1.1.1">𝒘</ci></apply><cn type="integer" id="A6.E43.m1.1.2.3.cmml" xref="A6.E43.m1.1.2.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E43.m1.1c">\displaystyle\lVert\bm{w}\rVert_{2}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A6.E43.m2.4" class="ltx_Math" alttext="\displaystyle=\lVert\bm{P}\bm{b}\rVert_{2}\leq\lVert\bm{P}\rVert_{\infty}\lVert\bm{b}\rVert_{2}\leq\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}" display="inline"><semantics id="A6.E43.m2.4a"><mrow id="A6.E43.m2.4.4" xref="A6.E43.m2.4.4.cmml"><mi id="A6.E43.m2.4.4.3" xref="A6.E43.m2.4.4.3.cmml"></mi><mo rspace="0.1389em" id="A6.E43.m2.4.4.4" xref="A6.E43.m2.4.4.4.cmml">=</mo><msub id="A6.E43.m2.4.4.1" xref="A6.E43.m2.4.4.1.cmml"><mrow id="A6.E43.m2.4.4.1.1.1" xref="A6.E43.m2.4.4.1.1.2.cmml"><mo fence="true" lspace="0.1389em" rspace="0em" id="A6.E43.m2.4.4.1.1.1.2" xref="A6.E43.m2.4.4.1.1.2.1.cmml">∥</mo><mrow id="A6.E43.m2.4.4.1.1.1.1" xref="A6.E43.m2.4.4.1.1.1.1.cmml"><mi id="A6.E43.m2.4.4.1.1.1.1.2" xref="A6.E43.m2.4.4.1.1.1.1.2.cmml">𝑷</mi><mo lspace="0em" rspace="0em" id="A6.E43.m2.4.4.1.1.1.1.1" xref="A6.E43.m2.4.4.1.1.1.1.1.cmml">​</mo><mi id="A6.E43.m2.4.4.1.1.1.1.3" xref="A6.E43.m2.4.4.1.1.1.1.3.cmml">𝒃</mi></mrow><mo fence="true" lspace="0em" rspace="0.1389em" id="A6.E43.m2.4.4.1.1.1.3" xref="A6.E43.m2.4.4.1.1.2.1.cmml">∥</mo></mrow><mn id="A6.E43.m2.4.4.1.3" xref="A6.E43.m2.4.4.1.3.cmml">2</mn></msub><mo lspace="0.1389em" rspace="0.1389em" id="A6.E43.m2.4.4.5" xref="A6.E43.m2.4.4.5.cmml">≤</mo><mrow id="A6.E43.m2.4.4.6" xref="A6.E43.m2.4.4.6.cmml"><msub id="A6.E43.m2.4.4.6.2" xref="A6.E43.m2.4.4.6.2.cmml"><mrow id="A6.E43.m2.4.4.6.2.2.2" xref="A6.E43.m2.4.4.6.2.2.1.cmml"><mo fence="true" lspace="0.1389em" rspace="0em" id="A6.E43.m2.4.4.6.2.2.2.1" xref="A6.E43.m2.4.4.6.2.2.1.1.cmml">∥</mo><mi id="A6.E43.m2.2.2" xref="A6.E43.m2.2.2.cmml">𝑷</mi><mo fence="true" lspace="0em" id="A6.E43.m2.4.4.6.2.2.2.2" xref="A6.E43.m2.4.4.6.2.2.1.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A6.E43.m2.4.4.6.2.3" xref="A6.E43.m2.4.4.6.2.3.cmml">∞</mi></msub><mo lspace="0em" rspace="0em" id="A6.E43.m2.4.4.6.1" xref="A6.E43.m2.4.4.6.1.cmml">​</mo><msub id="A6.E43.m2.4.4.6.3" xref="A6.E43.m2.4.4.6.3.cmml"><mrow id="A6.E43.m2.4.4.6.3.2.2" xref="A6.E43.m2.4.4.6.3.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E43.m2.4.4.6.3.2.2.1" xref="A6.E43.m2.4.4.6.3.2.1.1.cmml">∥</mo><mi id="A6.E43.m2.3.3" xref="A6.E43.m2.3.3.cmml">𝒃</mi><mo fence="true" lspace="0em" rspace="0.1389em" id="A6.E43.m2.4.4.6.3.2.2.2" xref="A6.E43.m2.4.4.6.3.2.1.1.cmml">∥</mo></mrow><mn id="A6.E43.m2.4.4.6.3.3" xref="A6.E43.m2.4.4.6.3.3.cmml">2</mn></msub></mrow><mo lspace="0.1389em" id="A6.E43.m2.4.4.7" xref="A6.E43.m2.4.4.7.cmml">≤</mo><mstyle displaystyle="true" id="A6.E43.m2.1.1" xref="A6.E43.m2.1.1.cmml"><mfrac id="A6.E43.m2.1.1a" xref="A6.E43.m2.1.1.cmml"><mrow id="A6.E43.m2.1.1.1" xref="A6.E43.m2.1.1.1.cmml"><msub id="A6.E43.m2.1.1.1.3" xref="A6.E43.m2.1.1.1.3.cmml"><mi id="A6.E43.m2.1.1.1.3.2" xref="A6.E43.m2.1.1.1.3.2.cmml">σ</mi><mi id="A6.E43.m2.1.1.1.3.3" xref="A6.E43.m2.1.1.1.3.3.cmml">A</mi></msub><mo lspace="0em" rspace="0em" id="A6.E43.m2.1.1.1.2" xref="A6.E43.m2.1.1.1.2.cmml">​</mo><msub id="A6.E43.m2.1.1.1.4" xref="A6.E43.m2.1.1.1.4.cmml"><mrow id="A6.E43.m2.1.1.1.4.2.2" xref="A6.E43.m2.1.1.1.4.2.1.cmml"><mo fence="true" rspace="0em" id="A6.E43.m2.1.1.1.4.2.2.1" xref="A6.E43.m2.1.1.1.4.2.1.1.cmml">∥</mo><mi id="A6.E43.m2.1.1.1.1" xref="A6.E43.m2.1.1.1.1.cmml">𝒃</mi><mo fence="true" lspace="0em" id="A6.E43.m2.1.1.1.4.2.2.2" xref="A6.E43.m2.1.1.1.4.2.1.1.cmml">∥</mo></mrow><mn id="A6.E43.m2.1.1.1.4.3" xref="A6.E43.m2.1.1.1.4.3.cmml">2</mn></msub></mrow><mrow id="A6.E43.m2.1.1.3" xref="A6.E43.m2.1.1.3.cmml"><mn id="A6.E43.m2.1.1.3.2" xref="A6.E43.m2.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.E43.m2.1.1.3.1" xref="A6.E43.m2.1.1.3.1.cmml">​</mo><msqrt id="A6.E43.m2.1.1.3.3" xref="A6.E43.m2.1.1.3.3.cmml"><mi id="A6.E43.m2.1.1.3.3.2" xref="A6.E43.m2.1.1.3.3.2.cmml">λ</mi></msqrt></mrow></mfrac></mstyle></mrow><annotation-xml encoding="MathML-Content" id="A6.E43.m2.4b"><apply id="A6.E43.m2.4.4.cmml" xref="A6.E43.m2.4.4"><and id="A6.E43.m2.4.4a.cmml" xref="A6.E43.m2.4.4"></and><apply id="A6.E43.m2.4.4b.cmml" xref="A6.E43.m2.4.4"><eq id="A6.E43.m2.4.4.4.cmml" xref="A6.E43.m2.4.4.4"></eq><csymbol cd="latexml" id="A6.E43.m2.4.4.3.cmml" xref="A6.E43.m2.4.4.3">absent</csymbol><apply id="A6.E43.m2.4.4.1.cmml" xref="A6.E43.m2.4.4.1"><csymbol cd="ambiguous" id="A6.E43.m2.4.4.1.2.cmml" xref="A6.E43.m2.4.4.1">subscript</csymbol><apply id="A6.E43.m2.4.4.1.1.2.cmml" xref="A6.E43.m2.4.4.1.1.1"><csymbol cd="latexml" id="A6.E43.m2.4.4.1.1.2.1.cmml" xref="A6.E43.m2.4.4.1.1.1.2">delimited-∥∥</csymbol><apply id="A6.E43.m2.4.4.1.1.1.1.cmml" xref="A6.E43.m2.4.4.1.1.1.1"><times id="A6.E43.m2.4.4.1.1.1.1.1.cmml" xref="A6.E43.m2.4.4.1.1.1.1.1"></times><ci id="A6.E43.m2.4.4.1.1.1.1.2.cmml" xref="A6.E43.m2.4.4.1.1.1.1.2">𝑷</ci><ci id="A6.E43.m2.4.4.1.1.1.1.3.cmml" xref="A6.E43.m2.4.4.1.1.1.1.3">𝒃</ci></apply></apply><cn type="integer" id="A6.E43.m2.4.4.1.3.cmml" xref="A6.E43.m2.4.4.1.3">2</cn></apply></apply><apply id="A6.E43.m2.4.4c.cmml" xref="A6.E43.m2.4.4"><leq id="A6.E43.m2.4.4.5.cmml" xref="A6.E43.m2.4.4.5"></leq><share href="#A6.E43.m2.4.4.1.cmml" id="A6.E43.m2.4.4d.cmml" xref="A6.E43.m2.4.4"></share><apply id="A6.E43.m2.4.4.6.cmml" xref="A6.E43.m2.4.4.6"><times id="A6.E43.m2.4.4.6.1.cmml" xref="A6.E43.m2.4.4.6.1"></times><apply id="A6.E43.m2.4.4.6.2.cmml" xref="A6.E43.m2.4.4.6.2"><csymbol cd="ambiguous" id="A6.E43.m2.4.4.6.2.1.cmml" xref="A6.E43.m2.4.4.6.2">subscript</csymbol><apply id="A6.E43.m2.4.4.6.2.2.1.cmml" xref="A6.E43.m2.4.4.6.2.2.2"><csymbol cd="latexml" id="A6.E43.m2.4.4.6.2.2.1.1.cmml" xref="A6.E43.m2.4.4.6.2.2.2.1">delimited-∥∥</csymbol><ci id="A6.E43.m2.2.2.cmml" xref="A6.E43.m2.2.2">𝑷</ci></apply><infinity id="A6.E43.m2.4.4.6.2.3.cmml" xref="A6.E43.m2.4.4.6.2.3"></infinity></apply><apply id="A6.E43.m2.4.4.6.3.cmml" xref="A6.E43.m2.4.4.6.3"><csymbol cd="ambiguous" id="A6.E43.m2.4.4.6.3.1.cmml" xref="A6.E43.m2.4.4.6.3">subscript</csymbol><apply id="A6.E43.m2.4.4.6.3.2.1.cmml" xref="A6.E43.m2.4.4.6.3.2.2"><csymbol cd="latexml" id="A6.E43.m2.4.4.6.3.2.1.1.cmml" xref="A6.E43.m2.4.4.6.3.2.2.1">delimited-∥∥</csymbol><ci id="A6.E43.m2.3.3.cmml" xref="A6.E43.m2.3.3">𝒃</ci></apply><cn type="integer" id="A6.E43.m2.4.4.6.3.3.cmml" xref="A6.E43.m2.4.4.6.3.3">2</cn></apply></apply></apply><apply id="A6.E43.m2.4.4e.cmml" xref="A6.E43.m2.4.4"><leq id="A6.E43.m2.4.4.7.cmml" xref="A6.E43.m2.4.4.7"></leq><share href="#A6.E43.m2.4.4.6.cmml" id="A6.E43.m2.4.4f.cmml" xref="A6.E43.m2.4.4"></share><apply id="A6.E43.m2.1.1.cmml" xref="A6.E43.m2.1.1"><divide id="A6.E43.m2.1.1.2.cmml" xref="A6.E43.m2.1.1"></divide><apply id="A6.E43.m2.1.1.1.cmml" xref="A6.E43.m2.1.1.1"><times id="A6.E43.m2.1.1.1.2.cmml" xref="A6.E43.m2.1.1.1.2"></times><apply id="A6.E43.m2.1.1.1.3.cmml" xref="A6.E43.m2.1.1.1.3"><csymbol cd="ambiguous" id="A6.E43.m2.1.1.1.3.1.cmml" xref="A6.E43.m2.1.1.1.3">subscript</csymbol><ci id="A6.E43.m2.1.1.1.3.2.cmml" xref="A6.E43.m2.1.1.1.3.2">𝜎</ci><ci id="A6.E43.m2.1.1.1.3.3.cmml" xref="A6.E43.m2.1.1.1.3.3">𝐴</ci></apply><apply id="A6.E43.m2.1.1.1.4.cmml" xref="A6.E43.m2.1.1.1.4"><csymbol cd="ambiguous" id="A6.E43.m2.1.1.1.4.1.cmml" xref="A6.E43.m2.1.1.1.4">subscript</csymbol><apply id="A6.E43.m2.1.1.1.4.2.1.cmml" xref="A6.E43.m2.1.1.1.4.2.2"><csymbol cd="latexml" id="A6.E43.m2.1.1.1.4.2.1.1.cmml" xref="A6.E43.m2.1.1.1.4.2.2.1">delimited-∥∥</csymbol><ci id="A6.E43.m2.1.1.1.1.cmml" xref="A6.E43.m2.1.1.1.1">𝒃</ci></apply><cn type="integer" id="A6.E43.m2.1.1.1.4.3.cmml" xref="A6.E43.m2.1.1.1.4.3">2</cn></apply></apply><apply id="A6.E43.m2.1.1.3.cmml" xref="A6.E43.m2.1.1.3"><times id="A6.E43.m2.1.1.3.1.cmml" xref="A6.E43.m2.1.1.3.1"></times><cn type="integer" id="A6.E43.m2.1.1.3.2.cmml" xref="A6.E43.m2.1.1.3.2">2</cn><apply id="A6.E43.m2.1.1.3.3.cmml" xref="A6.E43.m2.1.1.3.3"><root id="A6.E43.m2.1.1.3.3a.cmml" xref="A6.E43.m2.1.1.3.3"></root><ci id="A6.E43.m2.1.1.3.3.2.cmml" xref="A6.E43.m2.1.1.3.3.2">𝜆</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E43.m2.4c">\displaystyle=\lVert\bm{P}\bm{b}\rVert_{2}\leq\lVert\bm{P}\rVert_{\infty}\lVert\bm{b}\rVert_{2}\leq\frac{\sigma_{A}\lVert\bm{b}\rVert_{2}}{2\sqrt{\lambda}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(43)</span></td>
</tr></tbody>
</table>
<p id="A6.SS2.9.p1.18" class="ltx_p">using Lemma <a href="#A6.Thmlemma3" title="Lemma F.3 (Ridge Regression Singular Value Bound). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.3</span></a>.
∎</p>
</div>
</div>
<div id="A6.SS2.p6" class="ltx_para">
<p id="A6.SS2.p6.1" class="ltx_p">Using this lemma, the proof of Theorem <a href="#S4.Thmtheorem1" title="Theorem 4.1 (Generalization Error of Maddness). ‣ 4.5 Theoretical Guarantees ‣ 4 Our Method ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a> is immediate; we begin with Lemma <a href="#A6.Thmlemma6" title="Lemma F.6 (Generalization Error of Maddness-Regress). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.6</span></a> and simply union bound over all <math id="A6.SS2.p6.1.m1.2" class="ltx_Math" alttext="2^{C(4\left\lceil\log_{2}(D)\right\rceil+120)}" display="inline"><semantics id="A6.SS2.p6.1.m1.2a"><msup id="A6.SS2.p6.1.m1.2.3" xref="A6.SS2.p6.1.m1.2.3.cmml"><mn id="A6.SS2.p6.1.m1.2.3.2" xref="A6.SS2.p6.1.m1.2.3.2.cmml">2</mn><mrow id="A6.SS2.p6.1.m1.2.2.2" xref="A6.SS2.p6.1.m1.2.2.2.cmml"><mi id="A6.SS2.p6.1.m1.2.2.2.4" xref="A6.SS2.p6.1.m1.2.2.2.4.cmml">C</mi><mo lspace="0em" rspace="0em" id="A6.SS2.p6.1.m1.2.2.2.3" xref="A6.SS2.p6.1.m1.2.2.2.3.cmml">​</mo><mrow id="A6.SS2.p6.1.m1.2.2.2.2.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.cmml"><mo stretchy="false" id="A6.SS2.p6.1.m1.2.2.2.2.1.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.cmml">(</mo><mrow id="A6.SS2.p6.1.m1.2.2.2.2.1.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.cmml"><mrow id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.cmml"><mn id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.3" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.2.cmml">​</mo><mrow id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.2.cmml"><mo id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.2.1.cmml">⌈</mo><mrow id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.2.cmml"><msub id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2.cmml">log</mi><mn id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1a" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.2.1" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.2.cmml">(</mo><mi id="A6.SS2.p6.1.m1.1.1.1.1" xref="A6.SS2.p6.1.m1.1.1.1.1.cmml">D</mi><mo stretchy="false" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.2.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.3" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.2.1.cmml">⌉</mo></mrow></mrow><mo id="A6.SS2.p6.1.m1.2.2.2.2.1.1.2" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.2.cmml">+</mo><mn id="A6.SS2.p6.1.m1.2.2.2.2.1.1.3" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.3.cmml">120</mn></mrow><mo stretchy="false" id="A6.SS2.p6.1.m1.2.2.2.2.1.3" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.cmml">)</mo></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="A6.SS2.p6.1.m1.2b"><apply id="A6.SS2.p6.1.m1.2.3.cmml" xref="A6.SS2.p6.1.m1.2.3"><csymbol cd="ambiguous" id="A6.SS2.p6.1.m1.2.3.1.cmml" xref="A6.SS2.p6.1.m1.2.3">superscript</csymbol><cn type="integer" id="A6.SS2.p6.1.m1.2.3.2.cmml" xref="A6.SS2.p6.1.m1.2.3.2">2</cn><apply id="A6.SS2.p6.1.m1.2.2.2.cmml" xref="A6.SS2.p6.1.m1.2.2.2"><times id="A6.SS2.p6.1.m1.2.2.2.3.cmml" xref="A6.SS2.p6.1.m1.2.2.2.3"></times><ci id="A6.SS2.p6.1.m1.2.2.2.4.cmml" xref="A6.SS2.p6.1.m1.2.2.2.4">𝐶</ci><apply id="A6.SS2.p6.1.m1.2.2.2.2.1.1.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1"><plus id="A6.SS2.p6.1.m1.2.2.2.2.1.1.2.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.2"></plus><apply id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1"><times id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.2.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.2"></times><cn type="integer" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.3.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.3">4</cn><apply id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.2.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1"><ceiling id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.2.1.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.2"></ceiling><apply id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.2.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1"><apply id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1">subscript</csymbol><log id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.2"></log><cn type="integer" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.1.1.1.1.1.1.3">2</cn></apply><ci id="A6.SS2.p6.1.m1.1.1.1.1.cmml" xref="A6.SS2.p6.1.m1.1.1.1.1">𝐷</ci></apply></apply></apply><cn type="integer" id="A6.SS2.p6.1.m1.2.2.2.2.1.1.3.cmml" xref="A6.SS2.p6.1.m1.2.2.2.2.1.1.3">120</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p6.1.m1.2c">2^{C(4\left\lceil\log_{2}(D)\right\rceil+120)}</annotation></semantics></math> hypotheses from Lemma <a href="#A6.Thmlemma1" title="Lemma F.1 (Number of Hypotheses for Maddness-Build-Tree). ‣ F.2 Proof of Generalization Guarantee ‣ Appendix F Theoretical Analysis of Maddness ‣ Multiplying Matrices Without Multiplying" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F.1</span></a>.</p>
</div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2106.10859" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2106.10860" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2106.10860">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2106.10860" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2106.10861" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sat Mar  2 02:54:20 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2106.10860
Multiplying matrices is among the most fundamental and compute-intensive operations in machine learning. Consequently, there has been significant work on efficiently approximating matrix multiplies.
We introduce a learning-based algorithm for this task that greatly outperforms existing methods.
Experiments using hundreds of matrices from diverse domains show that it often runsfaster than exact matrix products andfaster than current approximate methods. In the common case that one matrix is known ahead of time,
our method also has the interesting property that it requires zero multiply-adds.
These results suggest that a mixture of hashing, averaging, and byte shuffling—–the core operations of our method—–could be a more promising building block for machine learning than the sparsified, factorized, and/or scalar quantized matrix products that have recently been the focus of substantial research and hardware investment.
abstract
Multiplying Matrices Without Multiplying
Abstract
2106.10860
1IntroductionMatrix multiplication is among the most fundamental subroutines used in machine learning and scientific computing. As a result, there has been a great deal of work on implementing high-speed matrix multiplication libraries, designing custom hardware to accelerate multiplication of certain classes of matrices,
speeding up distributed matrix multiplication,
and designing efficient Approximate Matrix Multiplication (AMM)  algorithms under various assumptions.We focus on the AMM task under the assumptions that the matrices are tall, relatively dense, and resident in a single machine’s memory. In this setting, the primary challenge is minimizing the amount of compute time required to approximate linear operations with a given level of fidelity.
This setting arises naturally in machine learning and data mining when one has a data matrixwhose rows are samples and a linear operatorone wishes to apply to these samples.could be a linear classifier, linear regressor, or an embedding matrix, among other possibilities.As a concrete example, consider the task of approximating a softmax classifier trained to predict image labels given embeddings derived from a neural network. Here, the rows ofare the embeddings for each image, and the columns ofare the weight vectors for each class. Classification is performed by computing the productand taking the argmax within each row of the result.
In Figure1, we see the results of approximatingusing our method and its best-performing rivalson the CIFAR-10 and CIFAR-100 datasets.Figure 1:Our method achieves a dramatically better speed-accuracy tradeoff than the best existing methods when approximating two linear classifiers.Our method represents a significant methodological departure from most traditional approaches to this problem. Traditional AMM methods construct matricessuch that(1) Often,andare sparse, embody some sort of sampling scheme, or have other structure such that these projection operations are faster than a dense matrix multiply. In short, these methods use linear functions to preprocessandand reduce the problem to exact matrix multiplication in a lower-dimensional space.Our proposed method,Maddness111Multiply-ADDitioN-lESS, instead employs anonlinearpreprocessing function and reduces the problem to table lookups. Moreover, in the case thatis known ahead of time—which happens when applying a trained linear model to new data, among other situations—Maddnessdoes not require any multiply-add operations.
Our method is most closely related to vector quantization methods used for similarity search (e.g.,) . However, instead of using an expensive quantization function that requires many multiply-adds, we introduce a family of quantization functions that require no multiply-adds.Our contributions can be summarized as follows:•An efficient family of learned vector quantization functions that can encode over 100GB of data per second in a single CPU thread.•A high-speed summation algorithm for low-bitwidth integers that avoids upcasting, saturation, and overflow.•An algorithm based on these functions for approximate matrix multiplication. Experiments across hundreds of diverse matrices demonstrate that this algorithm significantly outperforms existing alternatives. It also features theoretical quality guarantees.1.1Problem FormulationLetandbe two matrices, with. Given a computation time budget, our task is to
construct three functions,, and, along with constantsand, such that(2) for as small an errorpossible. The constantsandare separated fromso thatcan produce low-bitwidth outputs (e.g., in the range)  even when the entries ofdo not fall in this range.We assume the existence of a training set, whose rows are drawn from the same distribution as the rows of. This is a natural assumption in the case that rows ofrepresent examples in training data, or structured subsets thereof (such as patches of images) .
introduction
Multiplying Matrices Without Multiplying
1Introduction
2106.10860
2Related WorkBecause our work draws on ideas from randomized algorithms, approximate matrix multiplication, vector quantization, and other fields, the body of work related to our own is vast. Here, we provide only a high-level overview, and refer the interested reader tofor more detailed surveys. We also defer discussion of related vector quantization methods to the following sections.2.1Linear ApproximationMost AMM methods work by projectingandinto lower-dimensional spaces and then performing an exact matrix multiply.
One simple option for choosing the projection matrices is to use matrix sketching algorithms. The most prominent deterministic matrix sketching methods are the Frequent Directions algorithmand its many variations. There are also many randomized sketching methodsand sampling methods.A weakness of matrix sketching methods in the context of matrix multiplication is that they consider each matrix in isolation. To exploit information about both matrices simultaneously,sample columns ofand rows ofaccording to a sampling distribution dependent upon both matrices. Later work byreduces approximation of the matrices to an optimization problem, which is solved by steepest descent.,, andintroduce variations of the Frequent Directions algorithm that take into account both matrices.All of the above methods differ from our own not merely in specifics, but also in problem formulation. These methods all assume that there is no training setand nearly all focus on large matrices, where provably reduced asymptotic complexity for a given level of error is the goal.2.2Hashing to Avoid Linear OperationsIn the neural network acceleration literature, there have been several efforts to accelerate dense linear layers using some form of hashing. These methods differ from our own in the hash functions chosen, in not exploiting a training set, and in the overall goal of the algorithm. While we seek to approximate the entire output matrix, these methods seek to either sample outputs, approximate only the largest outputs, or implement a fixed, sparse linear operator.
body
Multiplying Matrices Without Multiplying
2Related Work
2106.10860
3Background - Product QuantizationTo lay the groundwork for our own method, we begin by reviewing Product Quantization (PQ) . PQ is a classic vector quantization algorithm for approximating inner products and Euclidean distances and serves as the basis for nearly all vector quantization methods similar to our own.The basic intuition behind PQ is that, whereis small buthas special structure allowing the product to be computed quickly. This structure consists ofbeing formed by concatenating learned prototypes in disjoint subspaces; one obtains a speedup by precomputing the dot products betweenand the prototypes once, and then reusing these values across manyvectors. Thevectors here are the (transposed)  rows ofand thevectors are the columns of.In somewhat more detail, PQ consists of the following:1.Prototype Learning- In an initial, offline training phase, cluster the rows of(or a training set)  using K-means to create prototypes. A separate K-means is run in each ofdisjoint subspaces to producesets ofprototypes.2.Encoding Function,- Determine the most similar prototype toin each subspace. Store these assignments as integer indices usingbits.3.Table Construction,- Precompute the dot products betweenand each prototype in each subspace. Store these partial dot products inlookup tables of size.4.Aggregation,- Use the indices and tables tolookupthe estimated partialin each subspace, then sum the results across allsubspaces.PQ is depicted for a single pair of vectorsandin Figure2. We elaborate upon each of these steps below.Prototype Learning:Letbe a training set,be a number of prototypes per subspace,be a number of subspaces, andbe the mutually exclusive and collectively exhaustive sets of indices associated with each subspace. The training-time task of PQ is to learnsets of prototypesand assignmentssuch that:(3) is minimized. It does this by running K-means separately in each subspaceand using the resulting centroids and assignments to populateand.Figure 2:Product Quantization. Thefunction returns the index of the most similar prototype to the data vector a in each subspace. Thefunction computes a lookup table of dot products between the query vector b and each protoype in each subspace. The aggregation functionsums the table entries corresponding to each index.Encoding Function,:Given the learned prototypes, PQ replaces each rowofwith the concatenation of itsK-means centroid assignments in each of thesubspaces. Formally:(4) We will refer to the resulting sequence of indices as theencodingofand the set ofcentroids as acodebook. For convenience, we will also refer to the vectoras thesubvectorofin subspace.Table Construction,:Using these same prototypes, PQ constructs a lookup tablein each of thesubspaces for each columnof, where(5) Existing work has shown that settingand quantizing the lookup tables to 8 bits can offer enormous speedups compared to largerand/or floating-point tables. This is because 16 1-byte entries can be stored in a SIMD register, allowing 16 or more table lookups to be performed in parallel using a byte shuffle instruction. Since the table entries naturally occupy more than 8 bits even for 8-bit data, some means of quantizing these entries is necessary. This can easily be done by subtracting off the minimum entry in each table and linearly rescaling such that the maximum entry in any table is at most 255. Ignoring rounding error, this affine transform is invertible, and is reflected by the constantsandin equation2. See AppendixAfor further details.Aggregation,:Given the encoding ofand the lookup tables for, the product can be approximated as(6) 
body
Multiplying Matrices Without Multiplying
3Background - Product Quantization
2106.10860
4Our MethodProduct Quantization and its variants yield a large speedup with. However, we require an algorithm that only needs, a more relaxed scenario common when using linear models and transforms. In this setting, the preprocessing timecan be significant, sincemay be similar to (or even larger than) .To address this case, we introduce a newfunction that yields large speedups even on much smaller matrices. The main idea behind our function is to determine the “most similar” prototype through locality-sensitive hashing; i.e., rather than compute the Euclidean distance between a subvectorand each prototype, we hashto one ofbuckets where similar subvectors tend to hash to the same bucket. The prototypes are set to the means of the subvectors hashing to each bucket.4.1Hash Function Family,Because we seek to exploit a training set while also doing far less work than even a single linear transform, we found that existing hash functions did not meet our requirements. Consequently, we designed our own family of trainable hash functions.
The family of hash functions we choose is balanced binary regression trees, with each leaf of the tree acting as one hash bucket. The leaf for a vectoris chosen by traversing the tree from the root and moving to the left child if the valueat some indexis below a node-specific threshold, and to the right child otherwise. To enable the use of SIMD instructions, the tree is limited to 16 leaves and all nodes at a given level of the tree are required to split on the same index. The number 16 holds across many processor architectures, and we refer the reader to AppendixBfor further vectorization details.Formally, consider a set of four indicesand four arrays of split thresholds, withhaving length. A vectoris mapped to an index using Algorithm1.Algorithm 1MaddnessHash1:Input:vector, split indices, split thresholds2:// node index within level of tree3:fordo4:// lookup split threshold for nodeat level5:// above split threshold?6:// assign to left or right child7:endfor8:returnThis function is simple, only depends on a constant number of indices in the input vector, and can easily be vectorized provided that the matrix whose rows are being encoded is stored in column-major order.4.2Learning the Hash Function ParametersThe split indicesand split thresholdsare optimized on the training matrixusing a greedy tree construction algorithm.
To describe this algorithm, we introduce the notion of abucket, which is the set of vectors mapped to nodein levelof the tree. The root of the tree is level 0 andcontains all the vectors. It will also be helpful to define the sum of squared errors (SSE)  loss associated with a bucket, or a specific (index, bucket)  pair:(7) Using this notation, it suffices to characterize the learning algorithm by describing the construction of levelof the tree given the bucketsfrom the previous level. This procedure is given in Algorithm2.In line 2, we select a fixed number of indices to evaluate. Several heuristics are possible, including evaluating all indices. We found that simply selecting the topindices that contributed the most loss summed across all buckets was difficult to beat. In preliminary experiments, we found that usingindices offered no clear benefit (and even choosingwas nearly as good) , so we fix.In lines 4-15, we find the minimal loss obtainable by splitting all buckets along that index, but with bucket-specific cutoffs. This loss is minimal not in the sense that it leads to the globally optimal tree, but in that it minimizes the sum of the losses in the buckets produced in this iteration. To do this, we invoke the subroutineoptimal_split_threshold, which takes in a bucketand an indexand tests all possible thresholds to find one minimizing. This can be done in time. The pseudocode for this subroutine is given in Algorithms3and4in AppendixC.Once a split indexand an array of split thresholdsare chosen, all that remains is to split the buckets to form the next level of the tree (lines 16-21) . This entails forming two child buckets from each current bucket by grouping vectors whoseth entries are above or below the bucket’s split threshold.Algorithm 2Adding The Next Level to the Hashing Tree1:Input:buckets, training matrix// greedily choose next split index and thresholds2:3:4:fordo5:// initialize loss for this index to 06:// empty list of split thresholds7:fordo8:9:// append threshold for bucket10:// accumulate loss from bucket11:endfor12:ifthen13:// new best split14:endif15:endfor// create new buckets using chosen split16:[ ]17:fordo18:19:20:21:endfor22:return4.3Optimizing the PrototypesAt this point, we have a complete algorithm. We could simply drop our hash-based encoding function into PQ and approximate matrix products. However, we contribute two additional enhancements: a means of optimizing the prototypes with no runtime overhead, and a means of quickly summing low-bitwidth integers.Several works propose prototype or table optimizations based on knowledge of, and others optimize them at the expense of slowing down the function. In contrast, we introduce a method that does not do either of these. The idea is to choose prototypes such thatcan be reconstructed from its prototypes with as little squared error as possible—this improves results since less error means that less information aboutis being lost.Letbe a matrix whose diagonal blocks of sizeconsist of thelearned prototypes in each subspace. The training matrixcan be approximately reconstructed as,
whereserves to select the appropriate prototype in each subspace. Rows ofare formed by concatenating the one-hot encoded representations of each assignment for the corresponding row of. For example, if a row were assigned prototypes3 1 2with,, its row inwould be0010 1000 0100. Our idea is to optimizeconditioned onand. This is an ordinary least squares problem, and we solve it with ridge regression:(8) One could obtain better performance by cross-validating to find, but for simplicity, we fix.This procedure allows the prototypes to be nonzero outside of their original subspaces. Because of our hashing procedure, we avoid the dramatically increased overhead faced by other methods with non-orthogonal prototypes (c.f.) .4.4Fast 8-bit Aggregation,Letbe the tensor of lookup tables for allcolumns of.
Given the encodings, the functionis defined as(9) Because the entries ofare stored as 8-bit values, exact summation requires immediately upcasting each looked-up entry to 16 bits before performing any addition instructions. This not only imposes overhead directly, but also means that one must perform 16-bit additions, which have half the throughput of 8-bit additions.We propose an alternative that sacrifices a small amount of accuracy for a significant increase in speed. Instead of usingadditioninstructions, we useaveraginginstructions, such asvpavgbon x86 orvrhaddon ARM. While non-saturating additions compute, these instructions compute. This means that they lose information about the low bit instead of the high bit of the sum. We estimate the overall mean by averaging pairs of values, then pairs of pairs, and so on. We refer the reader to AppendixDfor details.The challenging part of this approach is computing the bias in the estimated sum in order to correct for it. We prove in AppendixDthat this bias has a closed-form solution under the realistic assumption that the low bits are equally likely to be 0 or 1.4.5Theoretical GuaranteesOur central theoretical result is a generalization guarantee for the overall approximation error ofMaddness, stated below. See AppendixFfor a proof and additional analysis, including a discussion of algorithmic complexity. Besides this main guarantee, we also inherit all of the guarantees for Bolt, modulo a small amount of additional error from averaging integers rather than summing exactly. This follows from Bolt’s guarantees depending only on the quantization errors, rather than the method used to obtain them.Theorem 4.1(Generalization Error ofMaddness) .Letbe a probability distribution overand suppose thatMaddnessis trained on a matrixwhose rows are drawn independently fromand with maximum singular value bounded by. Letbe the number of codebooks used byMaddnessandbe the regularization parameter used in the ridge regression step. Then for any, any, and any, we have with probability at leastthat(10) where,is the scale used for quantizing the lookup tables,is the constants used in quantizing the lookup tables plus the debiasing constant of Section4.4, and(11) 
body
Multiplying Matrices Without Multiplying
4Our Method
2106.10860
5ExperimentsTo assessMaddness’s effectiveness, we implemented both it and existing algorithms in C++ and Python. All of our code and raw numerical results are publicly available athttps://smarturl.it/Maddness. All experiments use a single thread on a Macbook Pro with a 2.6GHz Intel Core i7-4960HQ processor. Unless stated otherwise, all timing results use five trials, with each trial reporting the fastest among 20 executions. We use the best, rather than the average, since this is standard practice in performance benchmarking and is robust to the purely additive noise introduced by competing CPU tasks. Standard deviations are shown for all curves as shaded areas.
Since training can be performed offline and all methods except SparsePCAtrain in at most a few minutes, we omit profiling of training times. We also do not profile the time to preprocess, since 1)  this time is inconsequential in most cases, and 2) is fixed and could be processed offline in all the problems we consider.
In order to avoid implementation bias, we build upon the source code provided by222https://github.com/dblalock/bolt, which includes highly tuned implementations of many algorithms to which we compare.We do not need to tune any hyperparameters forMaddness, but we do take steps to ensure that other methods are not hindered by insufficient hyperparameter tuning. Concretely, we sweep a wide range of hyperparameter values and allow them to cherry-pick their best hyperparameters on each test matrix.
Further details regarding our experimental setup and choices (e.g., use of a single thread)  can be found in AppendixE.5.1Methods TestedRecall that most baselines take the form of selecting a matrixsuch that. Hereis a free parameter that adjusts the quality versus speed tradeoff. We therefore characterize most of these methods by how they set.•PCA. Setequal to the top principal components of.•SparsePCA. Set, where. This is not the only dictionary learning formulation referred to as SparsePCA, but it is a good representative and is the only one with support in a major Python library.•FastJL.is set to Rademacher random variables composed with a Fast Hadamard Transform (FHT) . For simplicity, we exclude the FHT in the timing.•HashJL.is zero except for ain each row, with both sign and position chosen uniformly at random.•ScalarQuantize. The matrices are not projected, but instead linearly quantized to eight bits such that the smallest and largest entries map to
either 0 and 255 or -128 and 127, as appropriate.
We use FBGEMMto perform the quantized matrix multiplies. We neglect the time required to convert from other formats to eight bits, reflecting the optimistic scenario in which the matrices are already of the appropriate types.•Bolt. Bolt
is the most similar method to our own, differing only in the encoding function, the use of averaging instead of upcasting, and the optimization of centroids.•Exact Multiplication. We simply compute the matrix productusing a modern BLAS implementation.•Maddness-PQ. A handicapped version ofMaddnesswithout the prototype optimization step. The gap betweenMaddnessandMaddness-PQ is the gain from optimizing the prototypes.We also compared to many additional methods (see AppendixE) , but omit their results since they were not competitive with those listed here.5.2How Fast isMaddness?We begin by profiling the raw speed of our method. In Figure3, we time thefunctions for various vector quantization methods. Thematrices haverows and varying numbers of columns. Following, we also vary the number of codebooks, profiling 8-, 16-, and 32-byte encodings. We measure in bytes rather than codebooks since PQ and OPQ use eight bits per codebook while Bolt andMaddnessuse four.Maddnessis up to two orders of magnitude faster than existing methods, and its throughput increases with row length. This latter property is because its encoding cost per row israther than.Figure 3:Maddnessencodes thematrix orders of magnitude more quickly than existing vector quantization methods.We also profile the speed of our aggregation functionusing the same baselines as. As Figure4shows, our average-based, matrix-aware aggregation is significantly faster than the upcasting-based method of Bolt, its nearest rival.Figure 4:Given the preprocessed matrices,Maddnesscomputes the approximate output twice as fast as the fastest existing method.5.3Softmax ClassifierAs described in Section1, we approximated linear classifiers on the widely used CIFAR-10 and CIFAR-100 datasets. The classifiers use as input features the 512-dimensional activations of open-source, VGG-like neural networks trained on each dataset. The matricesare the-dimensional floating point activations for the full test sets, and the matricesare each network’s final dense layer. The-dimensional activations from the training set serve as the training matrices.
As shown in Figure5,Maddnesssignificantly outperforms all existing methods, achieving virtually the same accuracy as exact multiplication more than an order of magnitude faster.Figure 5:Maddnessachieves a far better speed-accuracy tradeoff than any existing method when approximating two softmax classifiers.Moreover, our method achieves this performance despite having worse support from the hardware. More precisely, it obtains speedups much smaller than the level of compression it provides. For example, the third points from the right in both plots correspond to speedups of roughly. However, they are compressing eachrow of the input down to a mere 4B, a savings of(sizes not shown in figure) .If the hardware could lookup-accumulate as many bytes per cycle as it can multiply-accumulate, our method could be overfaster. Combined with the fact that multiplexers require many fewer transistors than multipliers, this suggests that a hardware implementation of our method might offer large efficiency gains compared to existing accelerators.5.4Kernel-Based ClassificationTo assess the efficacy of our method on a larger and more diverse set of datasets than CIFAR-10 and CIFAR-100, we trained kernel classifiers on the datasets from the UCR Time Series Archive. To enable meaningful speed comparison across datasets, we resampled the time series in all datasets to the median length and obtained the matrixfor each dataset by running Stochastic Neighbor Compressionon the training set with an RBF kernel of bandwidth one.
We approximate the Euclidean distances used by the kernel via the identity, which consists only of dot products.
This is not the state-of-the-art means of classifying time series, but it does yield fixed-sized matrices and is representative of several modern techniques for constructing highly efficient classifiers.
Because Stochastic Neighbor Compression optimizes the classifiers to avoid redundancy, this task is quite difficult.As shown in Figure6,Maddnessis significantly faster than alternatives at a given level of accuracy. A counter-intuitive result, however, is that optimization of the prototypes occasionally reduces accuracy—see the red line dipping below the blue one in the lowest subplot. Since the optimization strictly increases the expressive power, we believe that this is a product of overfitting and could be corrected by not fixingin the ridge regression.Figure 6:Fraction of UCR datasets for which each method preserves a given fraction of the original accuracy.Maddnessenables much greater speedups for a given level of accuracy degradation.5.5Image FilteringTo test the extreme limits ofMaddness, we benchmarked the various techniques’ ability to apply small filters to images (after an offline im2row transform to reduce the task to matrix multiplication) . This task is extreme in thatandare tiny, affording almost no opportunity to amortize preprocessing costs. As representative example filters, we choseSobel kernels andGaussian kernels. These are common high-pass and low-pass filters, respectively. We took the first 10 images from the first 50 classes of the Caltech101 datasetas a single training set, and the first 10 images from the remaining 51 classes as 510 test sets. We constructed thematrices by extracting each patch of each image as one row. Thematrices have two columns, corresponding to one pair of Sobel or Gaussian filters (since using these filters in pairs is common) . We report the normalized mean-squared error (NMSE) , defined as, whereis the method’s estimate of. An NMSE of 0 is perfect and an NMSE of 1 corresponds to always predicting 0.In Figure7, we see that it is onlyMaddnessthat offers any advantage over exact matrix products. This is likely because two columns afford almost no time to preprocess; indeed, rival vector quantization methods cannot logically do less work than brute force in this setting, and dense linear methods can only save work by embedding rows ofin one-dimensional space.Maddnessperforms much worse on the high-pass filters (top)  than the low-pass filters (bottom) . This is likely because the former produce outputs with variance that is orders of magnitude lower than that of the original image, making the NMSE denominator tiny.Figure 7:Despite there being only two columns in the matrix,Maddnessstill achieves a significant speedup with reasonable accuracy. Methods that are Pareto dominated by exact matrix multiplication on both tasks are not shown; this includes all methods butMaddnessand SparsePCA.
body
Multiplying Matrices Without Multiplying
5Experiments
2106.10860
6Discussion and ConclusionBecause our work draws on a number of different fields but does not fit cleanly into any of them, it is useful to discuss what we have and have not demonstrated, as well as possible implications and extensions of our work.Our main empirical finding is that our proposed method,Maddness, achieves order-of-magnitude speedups compared to existing AMM methods and up to two-order-of-magnitude speedups compared to the dense baseline. It also compresses matrices by up to three orders of magnitude. These results are evaluated on a CPU, and are obtainable only when there is a training set for one matrix. We also claim superior performance only when one matrix is larger than the other, and both matrices are tall—the regime wherein our extremely fast (but less accurate)  encoding function is beneficial. Our method also loses utility when the larger matrix is known ahead of time; this assumption is common in similarity search, and eliminates the need for a fast encoding function entirely. Our approximate integer summation and fused table lookups would likely be useful independent of any of these assumptions, but demonstrating this is future work.We also have several theoretical findings, taking the form of guarantees regarding the errors introduced by our method and its constituent subroutines. While we do obtain an overall generalization guarantee, this guarantee is not tight. In particular, it should grow looser with the large matrix’s Frobenius norm and tighter as its singular values become more concentrated; at present, however, it simply grows looser as the largest singular value grows. The missing step is a guarantee that our encoding function will yield lower quantization errors when the singular values are more concentrated, which is its behavior in practice.We have not demonstrated results using GPUs or other accelerators. While such accelerators are a small minority of hardware, they are often used in machine learning. Our method is not inherently tied to CPUs, but the differing performance characteristics of accelerators mean that adapting our method to them would require both algorithmic and implementation work, with the details depending on the device. We also have not evaluated a multi-CPU-threaded extension of our algorithm, though this is because our method is intended to serve as the low-level, compute-bound, block matrix product routine called by individual threads.Finally, we have not demonstrated results using convolutional layers in neural networks, or results accelerating full networks. The weight reuse in convolutional layers presents many opportunities for algorithmic optimizations, and we hope to exploit them using a specialized extension of our method in future work. Accelerating overall networks will require two significant undertakings: first, the engineering work of building and integrating custom operators, data layouts, etc., into existing frameworks and networks; and second, the research necessary to determine when, how, and to what extent to include approximate kernels inspired by our approach. A particular difficulty with the latter is that our hash function is not differentiable.We believe that accelerating full networks with our ideas is a promising direction, particularly for inference. This is especially true at the hardware level—our method requires only multiplexers, not multipliers, and can therefore be implemented easily and with far less power than current matrix product logic. Moreover, our encoded representation and lookup tables have contiguous and uniformly-sized elements, making our approximate GEMM inner loops nearly identical to their dense counterparts–i.e., there is no need for complex access patterns or sparsity handling.In summary, we introducedMaddness, an algorithm that achieves up to abetter speed-quality tradeoff than existing methods for the well-studied problem of approximate matrix multiplication (AMM) , as measured on a large, diverse, and challenging set of real-world matrices. Our approach is a significant departure from existing AMM work in that it relies on hashing and table lookups rather than multiply-add operations. Our results suggest that future methods similar to our own might hold promise for accelerating convolution, deep learning, and other workloads bottlenecked by linear transforms.
conclusion
Multiplying Matrices Without Multiplying
6Discussion and Conclusion