<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2302.03213] LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup</title><meta property="og:description" content="On-device Deep Neural Network (DNN) inference consumes significant computing resources and development efforts. To alleviate that, we propose LUT-NN, the first system to empower inference by table lookup, to reduce inf…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2302.03213">

<!--Generated on Fri Mar  1 03:03:23 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="keywords" lang="en" content="Neural Network,  Product Quantization,  Table Lookup,  Centroid Learning">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line ltx_leqno">
<h1 class="ltx_title ltx_title_document">
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Xiaohu Tang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id6.1.id1" class="ltx_text ltx_affiliation_institution">State Key Lab of CAD&amp;CG, Zhejiang University
<br class="ltx_break">Microsoft Research</span><span id="id7.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:tigertang@zju.edu.cn">tigertang@zju.edu.cn</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yang Wang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id8.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft Research</span><span id="id9.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:Yang.Wang92@microsoft.com">Yang.Wang92@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ting Cao
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id10.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft Research</span><span id="id11.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:ting.cao@microsoft.com">ting.cao@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Li Lyna Zhang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id12.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft Research</span><span id="id13.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:lzhani@microsoft.com">lzhani@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Qi Chen
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id14.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft Research</span><span id="id15.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:cheqi@microsoft.com">cheqi@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Deng Cai
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id16.1.id1" class="ltx_text ltx_affiliation_institution">State Key Lab of CAD&amp;CG, Zhejiang University</span><span id="id17.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:dengcai@cad.zju.edu.cn">dengcai@cad.zju.edu.cn</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yunxin Liu
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id18.1.id1" class="ltx_text ltx_affiliation_institution">Institute for AI Industry Research (AIR), Tsinghua University</span><span id="id19.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:liuyunxin@air.tsinghua.edu.cn">liuyunxin@air.tsinghua.edu.cn</a>
</span></span></span>
<span class="ltx_author_before"> and </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Mao Yang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id20.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft Research</span><span id="id21.2.id2" class="ltx_text ltx_affiliation_country"></span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:maoyang@microsoft.com">maoyang@microsoft.com</a>
</span></span></span>
</div>
<div class="ltx_dates">(2018; 2023)</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract.</h6>
<p id="id22.id1" class="ltx_p">On-device Deep Neural Network (DNN) inference consumes significant computing resources and development efforts. To alleviate that, we propose LUT-NN, the first system to empower inference by table lookup, to reduce inference cost. LUT-NN learns the typical features for each operator, named centroid, and precompute the results for these centroids to save in lookup tables. During inference, the results of the closest centroids with the inputs can be read directly from the table, as the approximated outputs without computations.</p>
<p id="id5.5" class="ltx_p">LUT-NN integrates two major novel techniques: (1) differentiable centroid learning through backpropagation, which adapts three levels of approximation to minimize the accuracy impact by centroids; (2) table lookup inference execution, which comprehensively considers different levels of parallelism, memory access reduction, and dedicated hardware units for optimal performance. LUT-NN is evaluated on multiple real tasks, covering image and speech recognition, and nature language processing. Compared to related work, LUT-NN improves accuracy by 66% to 92%, achieving similar level with the original models. LUT-NN reduces the cost at all dimensions, including FLOPs (<math id="id1.1.m1.1" class="ltx_math_unparsed" alttext="\leq 16\times" display="inline"><semantics id="id1.1.m1.1a"><mrow id="id1.1.m1.1b"><mo id="id1.1.m1.1.1">≤</mo><mn id="id1.1.m1.1.2">16</mn><mo lspace="0.222em" id="id1.1.m1.1.3">×</mo></mrow><annotation encoding="application/x-tex" id="id1.1.m1.1c">\leq 16\times</annotation></semantics></math>), model size (<math id="id2.2.m2.1" class="ltx_math_unparsed" alttext="\leq 7\times" display="inline"><semantics id="id2.2.m2.1a"><mrow id="id2.2.m2.1b"><mo id="id2.2.m2.1.1">≤</mo><mn id="id2.2.m2.1.2">7</mn><mo lspace="0.222em" id="id2.2.m2.1.3">×</mo></mrow><annotation encoding="application/x-tex" id="id2.2.m2.1c">\leq 7\times</annotation></semantics></math>), latency (<math id="id3.3.m3.1" class="ltx_math_unparsed" alttext="\leq 6.8\times" display="inline"><semantics id="id3.3.m3.1a"><mrow id="id3.3.m3.1b"><mo id="id3.3.m3.1.1">≤</mo><mn id="id3.3.m3.1.2">6.8</mn><mo lspace="0.222em" id="id3.3.m3.1.3">×</mo></mrow><annotation encoding="application/x-tex" id="id3.3.m3.1c">\leq 6.8\times</annotation></semantics></math>), memory (<math id="id4.4.m4.1" class="ltx_math_unparsed" alttext="\leq 6.5\times" display="inline"><semantics id="id4.4.m4.1a"><mrow id="id4.4.m4.1b"><mo id="id4.4.m4.1.1">≤</mo><mn id="id4.4.m4.1.2">6.5</mn><mo lspace="0.222em" id="id4.4.m4.1.3">×</mo></mrow><annotation encoding="application/x-tex" id="id4.4.m4.1c">\leq 6.5\times</annotation></semantics></math>), and power (<math id="id5.5.m5.1" class="ltx_Math" alttext="\leq 41.7\%" display="inline"><semantics id="id5.5.m5.1a"><mrow id="id5.5.m5.1.1" xref="id5.5.m5.1.1.cmml"><mi id="id5.5.m5.1.1.2" xref="id5.5.m5.1.1.2.cmml"></mi><mo id="id5.5.m5.1.1.1" xref="id5.5.m5.1.1.1.cmml">≤</mo><mrow id="id5.5.m5.1.1.3" xref="id5.5.m5.1.1.3.cmml"><mn id="id5.5.m5.1.1.3.2" xref="id5.5.m5.1.1.3.2.cmml">41.7</mn><mo id="id5.5.m5.1.1.3.1" xref="id5.5.m5.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="id5.5.m5.1b"><apply id="id5.5.m5.1.1.cmml" xref="id5.5.m5.1.1"><leq id="id5.5.m5.1.1.1.cmml" xref="id5.5.m5.1.1.1"></leq><csymbol cd="latexml" id="id5.5.m5.1.1.2.cmml" xref="id5.5.m5.1.1.2">absent</csymbol><apply id="id5.5.m5.1.1.3.cmml" xref="id5.5.m5.1.1.3"><csymbol cd="latexml" id="id5.5.m5.1.1.3.1.cmml" xref="id5.5.m5.1.1.3.1">percent</csymbol><cn type="float" id="id5.5.m5.1.1.3.2.cmml" xref="id5.5.m5.1.1.3.2">41.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="id5.5.m5.1c">\leq 41.7\%</annotation></semantics></math>).</p>
</div>
<div class="ltx_keywords">Neural Network, Product Quantization, Table Lookup, Centroid Learning
</div>
<span id="id1" class="ltx_note ltx_note_frontmatter ltx_role_copyright"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">copyright: </span>acmcopyright</span></span></span><span id="id2" class="ltx_note ltx_note_frontmatter ltx_role_journalyear"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">journalyear: </span>2018</span></span></span><span id="id3" class="ltx_note ltx_note_frontmatter ltx_role_doi"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">doi: </span>XXXXXXX.XXXXXXX</span></span></span><span id="id4" class="ltx_note ltx_note_frontmatter ltx_role_conference"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">conference: </span>Make sure to enter the correct
conference title from your rights confirmation emai; June 03–05,
2018; Woodstock, NY</span></span></span><span id="id5" class="ltx_note ltx_note_frontmatter ltx_role_price"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">price: </span>15.00</span></span></span><span id="id6" class="ltx_note ltx_note_frontmatter ltx_role_isbn"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">isbn: </span>978-1-4503-XXXX-X/18/06</span></span></span><span id="id7" class="ltx_note ltx_note_frontmatter ltx_role_ccs"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">ccs: </span>Computing methodologies Artificial intelligence</span></span></span><span id="id8" class="ltx_note ltx_note_frontmatter ltx_role_journalyear"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">journalyear: </span>2023</span></span></span><span id="id9" class="ltx_note ltx_note_frontmatter ltx_role_copyright"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">copyright: </span>acmlicensed</span></span></span><span id="id10" class="ltx_note ltx_note_frontmatter ltx_role_conference"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">conference: </span>The 29th Annual International Conference on Mobile Computing and Networking; October 2–6, 2023; Madrid, Spain</span></span></span><span id="id11" class="ltx_note ltx_note_frontmatter ltx_role_booktitle"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">booktitle: </span>The 29th Annual International Conference on Mobile Computing and Networking (ACM MobiCom ’23), October 2–6, 2023, Madrid, Spain</span></span></span><span id="id12" class="ltx_note ltx_note_frontmatter ltx_role_price"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">price: </span>15.00</span></span></span><span id="id13" class="ltx_note ltx_note_frontmatter ltx_role_doi"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">doi: </span>10.1145/3570361.3613285</span></span></span><span id="id14" class="ltx_note ltx_note_frontmatter ltx_role_isbn"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">isbn: </span>978-1-4503-9990-6/23/10</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1. </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">DNNs are widely used in mobile applications, offering users unparalleled intelligent services. However, DNNs are computation hungry workloads, mainly composed of linear computation operators, heavily stressing the limited hardware resources on mobile devices.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">As a result, huge efforts have been taken for efficient and affordable DNN inference on mobile devices, such as model compression <cite class="ltx_cite ltx_citemacro_citep">(Blalock et al<span class="ltx_text">.</span>, <a href="#bib.bib5" title="" class="ltx_ref">2020</a>; Gou et al<span class="ltx_text">.</span>, <a href="#bib.bib20" title="" class="ltx_ref">2021</a>)</cite>, sophisticated computing operator optimization <cite class="ltx_cite ltx_citemacro_citep">(Microsoft, <a href="#bib.bib47" title="" class="ltx_ref">2019</a>; Google, <a href="#bib.bib19" title="" class="ltx_ref">2019</a>; MACE, <a href="#bib.bib42" title="" class="ltx_ref">2020</a>; Wang et al<span class="ltx_text">.</span>, <a href="#bib.bib56" title="" class="ltx_ref">2021</a>)</cite>, tensor compilers for operator generation <cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib10" title="" class="ltx_ref">2018</a>; Liang et al<span class="ltx_text">.</span>, <a href="#bib.bib40" title="" class="ltx_ref">2022</a>)</cite>, and customized DNN accelerators <cite class="ltx_cite ltx_citemacro_citep">(Jiao et al<span class="ltx_text">.</span>, <a href="#bib.bib35" title="" class="ltx_ref">2020</a>; Jouppi et al<span class="ltx_text">.</span>, <a href="#bib.bib36" title="" class="ltx_ref">2021</a>; Wechsler et al<span class="ltx_text">.</span>, <a href="#bib.bib58" title="" class="ltx_ref">2019</a>)</cite>. These methods require redesigning or re-implementing model structures, computation operators, or accelerators repeatedly for diverse deployment scenarios.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Different from these directions, this paper explores a new possibility to replace computation operators of DNNs, for reduced inference cost, as well as the tedious efforts of operator development. To achieve this, we first rethink the essence of inference. Each layer of a DNN model is to output another level of features given the input features. For example, the front layers of an image recognition model output low-level features (e.g., edges and lines), and subsequent layers output high-level features (e.g., faces and objects). The fact is that the features of different images for each layer have semantic similarity. A vivid example is that though cats and horses are different input images for a model, their ear features result in similar output for the layer that extracts it. Same for language tasks, similar words of different sentences could have similar embeddings and output results.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Based on this similarity, the question is that whether the typical features can be learned for each computation operator of DNNs, so that the output of these typical features can represent the output of diverse features. If so, by precomputing and saving the output of typical features, the output of future inputs can be read directly without computation.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Towards answering this question, we propose LUT-NN, a novel system to empower DNN inference by table lookup. The system learns the typical features, named <em id="S1.p5.1.1" class="ltx_emph ltx_font_italic">centroid</em>, for each linear computation operator, and precompute the results for these centroids to save in lookup tables. During inference, the results of the closest centroids with the input features can be read directly from the table as the approximated output of this operator. The required computation cost for inference is significantly reduced. Fig. <a href="#S1.F1" title="Figure 1 ‣ 1. Introduction ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows an overview of LUT-NN using real data samples of a model. LUT-NN integrates two major techniques: (1) differentiable centroid learning through backpropagation, and (2) table-lookup inference execution.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p"><span id="S1.p6.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span> <cite class="ltx_cite ltx_citemacro_citep">(Blalock and Guttag, <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite> initiates the centroid learning for matrix multiplication (MM), leveraging the technique of Product Quantization (PQ). PQ is an effective vector quantization algorithm widely used for dataset compression <cite class="ltx_cite ltx_citemacro_citep">(Jegou et al<span class="ltx_text">.</span>, <a href="#bib.bib34" title="" class="ltx_ref">2010</a>; Guo et al<span class="ltx_text">.</span>, <a href="#bib.bib25" title="" class="ltx_ref">2015</a>)</cite>. It compresses the dataset by clustering the vectors in the dataset first, and then learning the centroids to represent vectors in each cluster. By using PQ, <span id="S1.p6.1.2" class="ltx_text ltx_font_smallcaps">Maddness</span> learns the centroids from the training dataset for a single-operator image classifier, and precomputes the MM results for the centroids, as the approximation for future input matrices.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">However, as we will show in the paper, directly applying <span id="S1.p7.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span> to the computation operators of a DNN model results in poor accuracy (80% accuracy drops for CIFAR-10). A similar conclusion is also from McCarter <span id="S1.p7.1.2" class="ltx_text"><em id="S1.p7.1.2.1" class="ltx_emph ltx_font_italic">et al.</em></span> <cite class="ltx_cite ltx_citemacro_citep">(McCarter and Dronen, <a href="#bib.bib45" title="" class="ltx_ref">2022</a>)</cite>.
We first analyze the issue and expose the main reason for the poor result is <em id="S1.p7.1.3" class="ltx_emph ltx_font_italic">the different optimization goal of centroid learning for PQ and DNNs.</em> The goal of PQ is to learn the centroids that can minimize the total error/distance between centroids and vectors, while the goal of DNNs is to minimize the final loss function. The two goals have no direct relationship. Without considering the loss function, errors introduced by centroids in <span id="S1.p7.1.4" class="ltx_text ltx_font_smallcaps">Maddness</span> are accumulated layer by layer, resulting in poor accuracy, making the resulting DNN model unreliable and undeployable in real-world applications.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2302.03213/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="368" height="325" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1. </span>LUT-NN transforms model linear-computation layers to table lookup for inference. </figcaption>
</figure>
<div id="S1.p8" class="ltx_para">
<p id="S1.p8.1" class="ltx_p">Therefore, we identify the key factor for DNN centroid learning is to <em id="S1.p8.1.1" class="ltx_emph ltx_font_italic">pass the model loss to each operator through backpropagation</em>, and iteratively adjust the centroids by the gradients to minimize the model loss. However, the challenge is that currently it is not possible to learn centroids through backpropogation, since encoding a vector to the closest centroid e.g., <code id="S1.p8.1.2" class="ltx_verbatim ltx_font_typewriter">argmin</code> or hashing in <span id="S1.p8.1.3" class="ltx_text ltx_font_smallcaps">Maddness</span>, is not differentiable, and thus the gradients cannot be calculated.</p>
</div>
<div id="S1.p9" class="ltx_para">
<p id="S1.p9.1" class="ltx_p">To solve this challenge, we propose the novel <span id="S1.p9.1.1" class="ltx_text ltx_font_bold">differentiable centroid learning</span> technique for DNN. Through backpropagation, it adapts three different levels of approximation introduced by centroids to model inference by three methods, and minimize the accuracy loss. (1) To enable gradient calculation and iteratively adjust centroids, LUT-NN uses <em id="S1.p9.1.2" class="ltx_emph ltx_font_italic">soft-PQ</em> method. It uses the continuous approximation of <code id="S1.p9.1.3" class="ltx_verbatim ltx_font_typewriter">argmax</code>, i.e., <code id="S1.p9.1.4" class="ltx_verbatim ltx_font_typewriter">softmax</code>, for backward pass. The forward pass still uses <code id="S1.p9.1.5" class="ltx_verbatim ltx_font_typewriter">argmin</code> for model loss calculation, since <code id="S1.p9.1.6" class="ltx_verbatim ltx_font_typewriter">argmin</code> will be used for inference. Soft-PQ can adapt the approximation introduced by centroids to reduce accuracy loss. (2) Since the use of <code id="S1.p9.1.7" class="ltx_verbatim ltx_font_typewriter">softmax</code> introduces approximation to <code id="S1.p9.1.8" class="ltx_verbatim ltx_font_typewriter">argmin</code>, LUT-NN uses <em id="S1.p9.1.9" class="ltx_emph ltx_font_italic">learned temperature</em> method, to learn the hyperparameter <code id="S1.p9.1.10" class="ltx_verbatim ltx_font_typewriter">temperature</code> of <code id="S1.p9.1.11" class="ltx_verbatim ltx_font_typewriter">softmax</code> for each operator, to tradeoff accuracy loss and learning convergence. (3) Since the lookup tables for precomputed results are the main cost for LUT-NN, scalar quantization is used to reduce the table size. To adapt the approximation introduced by this scalar quantization, LUT-NN uses <em id="S1.p9.1.12" class="ltx_emph ltx_font_italic">quantization-aware training</em>, which uses quantized tables in the forward pass and real-value tables in the backward pass, to reduce accuracy loss. By the techniques above, the centroids can be learned, which empowers table lookup to replace the diverse DNN linear computations while maintain similar-level of model accuracy.</p>
</div>
<div id="S1.p10" class="ltx_para">
<p id="S1.p10.1" class="ltx_p">LUT-NN achieves a high FLOPs reduction (up to 16<math id="S1.p10.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p10.1.m1.1a"><mo id="S1.p10.1.m1.1.1" xref="S1.p10.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p10.1.m1.1b"><times id="S1.p10.1.m1.1.1.cmml" xref="S1.p10.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p10.1.m1.1c">\times</annotation></semantics></math>). To run LUT-NN on resource-limited devices, we design and implement the <span id="S1.p10.1.1" class="ltx_text ltx_font_bold">table lookup inference execution</span> on commodity ARM and x86 CPUs. The challenge is that table lookup is a new inference paradigm, and no direct support from current systems and accelerators. Our design comprehensively considers different levels of hardware parallelism, memory access reduction, and dedicated hardware units, to fully utilize hardware resources. Specifically, we align the size of codebook and lookup table to the width of SIMD instructions for data-level parallelism, interleave the instructions of <code id="S1.p10.1.2" class="ltx_verbatim ltx_font_typewriter">argmin</code> reduction for instruction-level parallelism, reside the codebook in an inner loop for cache locality, and leverage the shuffle instruction for fast table access.</p>
</div>
<div id="S1.p11" class="ltx_para">
<p id="S1.p11.1" class="ltx_p">We evaluate LUT-NN on a range of tasks, including image recognition tasks: CIFAR <cite class="ltx_cite ltx_citemacro_citep">(Krizhevsky, <a href="#bib.bib38" title="" class="ltx_ref">2009</a>)</cite>, GTSRB <cite class="ltx_cite ltx_citemacro_citep">(Stallkamp et al<span class="ltx_text">.</span>, <a href="#bib.bib51" title="" class="ltx_ref">2011</a>)</cite>, SVHN <cite class="ltx_cite ltx_citemacro_citep">(Netzer et al<span class="ltx_text">.</span>, <a href="#bib.bib48" title="" class="ltx_ref">2011</a>)</cite>, and ImageNet <cite class="ltx_cite ltx_citemacro_citep">(Deng et al<span class="ltx_text">.</span>, <a href="#bib.bib13" title="" class="ltx_ref">2009</a>)</cite>; speech recognition task: Google Speech Command <cite class="ltx_cite ltx_citemacro_citep">(Warden, <a href="#bib.bib57" title="" class="ltx_ref">2018</a>)</cite>; nature language processing (NLP) tasks: GLUE <cite class="ltx_cite ltx_citemacro_citep">(Wang et al<span class="ltx_text">.</span>, <a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite> and a regression task: UTKFace <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al<span class="ltx_text">.</span>, <a href="#bib.bib61" title="" class="ltx_ref">2017</a>)</cite> age prediction, and cover both CNN and BERT models. For all these tasks, compared to directly applying <span id="S1.p11.1.1" class="ltx_text ltx_font_smallcaps">Maddness</span>, LUT-NN can improve the accuracy by 66% to 92%. The accuracy difference between LUT-NN and the original DNN models ranges from +1.98% (Speech Command) to -2.42% (ImageNet).</p>
</div>
<div id="S1.p12" class="ltx_para">
<p id="S1.p12.5" class="ltx_p">For inference efficiency, LUT-NN can outperform baselines in all dimensions.
The selected baselines include both the state-of-the-art hand-written inference system ONNX Runtime <cite class="ltx_cite ltx_citemacro_citep">(Microsoft, <a href="#bib.bib47" title="" class="ltx_ref">2019</a>)</cite>, and also the well-tuned TVM <cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib10" title="" class="ltx_ref">2018</a>)</cite> generated kernels as the performance upper-bound of linear computations. The detailed results include: FLOPs reduces <math id="S1.p12.1.m1.1" class="ltx_math_unparsed" alttext="5.7\times\sim 16\times" display="inline"><semantics id="S1.p12.1.m1.1a"><mrow id="S1.p12.1.m1.1b"><mn id="S1.p12.1.m1.1.1">5.7</mn><mo lspace="0.222em" rspace="0em" id="S1.p12.1.m1.1.2">×</mo><mo lspace="0em" id="S1.p12.1.m1.1.3">∼</mo><mn id="S1.p12.1.m1.1.4">16</mn><mo lspace="0.222em" id="S1.p12.1.m1.1.5">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p12.1.m1.1c">5.7\times\sim 16\times</annotation></semantics></math>; model (or disk) size reduces <math id="S1.p12.2.m2.1" class="ltx_math_unparsed" alttext="3.4\times\sim 7\times" display="inline"><semantics id="S1.p12.2.m2.1a"><mrow id="S1.p12.2.m2.1b"><mn id="S1.p12.2.m2.1.1">3.4</mn><mo lspace="0.222em" rspace="0em" id="S1.p12.2.m2.1.2">×</mo><mo lspace="0em" id="S1.p12.2.m2.1.3">∼</mo><mn id="S1.p12.2.m2.1.4">7</mn><mo lspace="0.222em" id="S1.p12.2.m2.1.5">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p12.2.m2.1c">3.4\times\sim 7\times</annotation></semantics></math>; end-to-end inference latency reduces <math id="S1.p12.3.m3.1" class="ltx_math_unparsed" alttext="1.3\times\sim 6.8\times" display="inline"><semantics id="S1.p12.3.m3.1a"><mrow id="S1.p12.3.m3.1b"><mn id="S1.p12.3.m3.1.1">1.3</mn><mo lspace="0.222em" rspace="0em" id="S1.p12.3.m3.1.2">×</mo><mo lspace="0em" id="S1.p12.3.m3.1.3">∼</mo><mn id="S1.p12.3.m3.1.4">6.8</mn><mo lspace="0.222em" id="S1.p12.3.m3.1.5">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p12.3.m3.1c">1.3\times\sim 6.8\times</annotation></semantics></math>; memory usage reduces <math id="S1.p12.4.m4.1" class="ltx_math_unparsed" alttext="1.4\times\sim 6.5\times" display="inline"><semantics id="S1.p12.4.m4.1a"><mrow id="S1.p12.4.m4.1b"><mn id="S1.p12.4.m4.1.1">1.4</mn><mo lspace="0.222em" rspace="0em" id="S1.p12.4.m4.1.2">×</mo><mo lspace="0em" id="S1.p12.4.m4.1.3">∼</mo><mn id="S1.p12.4.m4.1.4">6.5</mn><mo lspace="0.222em" id="S1.p12.4.m4.1.5">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p12.4.m4.1c">1.4\times\sim 6.5\times</annotation></semantics></math>; and peak power reduces <math id="S1.p12.5.m5.1" class="ltx_Math" alttext="15\%\sim 41.7\%" display="inline"><semantics id="S1.p12.5.m5.1a"><mrow id="S1.p12.5.m5.1.1" xref="S1.p12.5.m5.1.1.cmml"><mrow id="S1.p12.5.m5.1.1.2" xref="S1.p12.5.m5.1.1.2.cmml"><mn id="S1.p12.5.m5.1.1.2.2" xref="S1.p12.5.m5.1.1.2.2.cmml">15</mn><mo id="S1.p12.5.m5.1.1.2.1" xref="S1.p12.5.m5.1.1.2.1.cmml">%</mo></mrow><mo id="S1.p12.5.m5.1.1.1" xref="S1.p12.5.m5.1.1.1.cmml">∼</mo><mrow id="S1.p12.5.m5.1.1.3" xref="S1.p12.5.m5.1.1.3.cmml"><mn id="S1.p12.5.m5.1.1.3.2" xref="S1.p12.5.m5.1.1.3.2.cmml">41.7</mn><mo id="S1.p12.5.m5.1.1.3.1" xref="S1.p12.5.m5.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p12.5.m5.1b"><apply id="S1.p12.5.m5.1.1.cmml" xref="S1.p12.5.m5.1.1"><csymbol cd="latexml" id="S1.p12.5.m5.1.1.1.cmml" xref="S1.p12.5.m5.1.1.1">similar-to</csymbol><apply id="S1.p12.5.m5.1.1.2.cmml" xref="S1.p12.5.m5.1.1.2"><csymbol cd="latexml" id="S1.p12.5.m5.1.1.2.1.cmml" xref="S1.p12.5.m5.1.1.2.1">percent</csymbol><cn type="integer" id="S1.p12.5.m5.1.1.2.2.cmml" xref="S1.p12.5.m5.1.1.2.2">15</cn></apply><apply id="S1.p12.5.m5.1.1.3.cmml" xref="S1.p12.5.m5.1.1.3"><csymbol cd="latexml" id="S1.p12.5.m5.1.1.3.1.cmml" xref="S1.p12.5.m5.1.1.3.1">percent</csymbol><cn type="float" id="S1.p12.5.m5.1.1.3.2.cmml" xref="S1.p12.5.m5.1.1.3.2">41.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p12.5.m5.1c">15\%\sim 41.7\%</annotation></semantics></math>.
The real speedup is less than the FLOPs reduction due to the unfriendly support for table lookup in hardware. A customized table-lookup unit or accelerator could unleash the full potential of LUT-NN.</p>
</div>
<div id="S1.p13" class="ltx_para">
<p id="S1.p13.1" class="ltx_p">LUT-NN takes the first trial towards this new table-lookup inference paradigm, which potentially much simplifies the inference system and hardware design and improves efficiency. To sum up, the key contributions of this paper are as follows. The code is open sourced at <a target="_blank" href="https://github.com/lutnn" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/lutnn</a>.</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">LUT-NN is the first to empower DNN inference by table lookup to reduce inference cost.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">It can achieve similar level of model accuracy by the novel differentiable centroid learning technique for DNNs through backpropagation.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">It designs the new table lookup operation to enable this new inference paradigm on commodity CPUs.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p">We implement the learning and inference pipelines. Superior results are shown on all evaluated dimensions.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2. </span>Background and Motivation</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">Each data sample in the training set can be assumed as a vector. This section will first introduce the concept of vector quantization and its efficient solution, PQ, and then show the poor results of directly applying PQ to DNN inference.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1. </span>Product Quantization: Background</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Quantization has been well studied in information theory <cite class="ltx_cite ltx_citemacro_citep">(Gray and Neuhoff, <a href="#bib.bib21" title="" class="ltx_ref">1998</a>)</cite>. It reduces the cardinality of a dataset by using <em id="S2.SS1.p1.1.1" class="ltx_emph ltx_font_italic">centroids</em> to represent the data samples. The set of centroids is called a <em id="S2.SS1.p1.1.2" class="ltx_emph ltx_font_italic">codebook</em>. For vector quantization, the dataset is composed of <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mi id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><ci id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">D</annotation></semantics></math>-dimension vectors. The vector quantizer can encode each vector to a centroid in the codebook.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.6" class="ltx_p">To reduce the cost of vector quantization, PQ is proposed. The essence of it is to decompose the high-dimensional vector space into the Cartesian product of sub-vector spaces and then quantize these sub-vector spaces separately. As shown in Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>(a), it splits the <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mi id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><ci id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">D</annotation></semantics></math>-dimension vector into <math id="S2.SS1.p2.2.m2.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S2.SS1.p2.2.m2.1a"><mi id="S2.SS1.p2.2.m2.1.1" xref="S2.SS1.p2.2.m2.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.2.m2.1b"><ci id="S2.SS1.p2.2.m2.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.2.m2.1c">C</annotation></semantics></math> distinct <math id="S2.SS1.p2.3.m3.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.SS1.p2.3.m3.1a"><mi id="S2.SS1.p2.3.m3.1.1" xref="S2.SS1.p2.3.m3.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.3.m3.1b"><ci id="S2.SS1.p2.3.m3.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.3.m3.1c">V</annotation></semantics></math>-dimension sub-vectors (<math id="S2.SS1.p2.4.m4.1" class="ltx_Math" alttext="D=C\cdot V" display="inline"><semantics id="S2.SS1.p2.4.m4.1a"><mrow id="S2.SS1.p2.4.m4.1.1" xref="S2.SS1.p2.4.m4.1.1.cmml"><mi id="S2.SS1.p2.4.m4.1.1.2" xref="S2.SS1.p2.4.m4.1.1.2.cmml">D</mi><mo id="S2.SS1.p2.4.m4.1.1.1" xref="S2.SS1.p2.4.m4.1.1.1.cmml">=</mo><mrow id="S2.SS1.p2.4.m4.1.1.3" xref="S2.SS1.p2.4.m4.1.1.3.cmml"><mi id="S2.SS1.p2.4.m4.1.1.3.2" xref="S2.SS1.p2.4.m4.1.1.3.2.cmml">C</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p2.4.m4.1.1.3.1" xref="S2.SS1.p2.4.m4.1.1.3.1.cmml">⋅</mo><mi id="S2.SS1.p2.4.m4.1.1.3.3" xref="S2.SS1.p2.4.m4.1.1.3.3.cmml">V</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.4.m4.1b"><apply id="S2.SS1.p2.4.m4.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1"><eq id="S2.SS1.p2.4.m4.1.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1.1"></eq><ci id="S2.SS1.p2.4.m4.1.1.2.cmml" xref="S2.SS1.p2.4.m4.1.1.2">𝐷</ci><apply id="S2.SS1.p2.4.m4.1.1.3.cmml" xref="S2.SS1.p2.4.m4.1.1.3"><ci id="S2.SS1.p2.4.m4.1.1.3.1.cmml" xref="S2.SS1.p2.4.m4.1.1.3.1">⋅</ci><ci id="S2.SS1.p2.4.m4.1.1.3.2.cmml" xref="S2.SS1.p2.4.m4.1.1.3.2">𝐶</ci><ci id="S2.SS1.p2.4.m4.1.1.3.3.cmml" xref="S2.SS1.p2.4.m4.1.1.3.3">𝑉</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.4.m4.1c">D=C\cdot V</annotation></semantics></math>). The sub-vectors are quantized separately using <math id="S2.SS1.p2.5.m5.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S2.SS1.p2.5.m5.1a"><mi id="S2.SS1.p2.5.m5.1.1" xref="S2.SS1.p2.5.m5.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.5.m5.1b"><ci id="S2.SS1.p2.5.m5.1.1.cmml" xref="S2.SS1.p2.5.m5.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.5.m5.1c">C</annotation></semantics></math> codebooks. The quantization result of the vector is the concatenation of the <math id="S2.SS1.p2.6.m6.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S2.SS1.p2.6.m6.1a"><mi id="S2.SS1.p2.6.m6.1.1" xref="S2.SS1.p2.6.m6.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.6.m6.1b"><ci id="S2.SS1.p2.6.m6.1.1.cmml" xref="S2.SS1.p2.6.m6.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.6.m6.1c">C</annotation></semantics></math> centroids.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.14" class="ltx_p"><span id="S2.SS1.p3.14.1" class="ltx_text ltx_font_bold">Centroid learning</span> We now formulize the PQ process. To quantize a <math id="S2.SS1.p3.1.m1.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S2.SS1.p3.1.m1.1a"><mi id="S2.SS1.p3.1.m1.1.1" xref="S2.SS1.p3.1.m1.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.1.m1.1b"><ci id="S2.SS1.p3.1.m1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.1.m1.1c">D</annotation></semantics></math>-dimension vector <math id="S2.SS1.p3.2.m2.1" class="ltx_Math" alttext="a\in\mathbb{R}^{D}" display="inline"><semantics id="S2.SS1.p3.2.m2.1a"><mrow id="S2.SS1.p3.2.m2.1.1" xref="S2.SS1.p3.2.m2.1.1.cmml"><mi id="S2.SS1.p3.2.m2.1.1.2" xref="S2.SS1.p3.2.m2.1.1.2.cmml">a</mi><mo id="S2.SS1.p3.2.m2.1.1.1" xref="S2.SS1.p3.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS1.p3.2.m2.1.1.3" xref="S2.SS1.p3.2.m2.1.1.3.cmml"><mi id="S2.SS1.p3.2.m2.1.1.3.2" xref="S2.SS1.p3.2.m2.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS1.p3.2.m2.1.1.3.3" xref="S2.SS1.p3.2.m2.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.2.m2.1b"><apply id="S2.SS1.p3.2.m2.1.1.cmml" xref="S2.SS1.p3.2.m2.1.1"><in id="S2.SS1.p3.2.m2.1.1.1.cmml" xref="S2.SS1.p3.2.m2.1.1.1"></in><ci id="S2.SS1.p3.2.m2.1.1.2.cmml" xref="S2.SS1.p3.2.m2.1.1.2">𝑎</ci><apply id="S2.SS1.p3.2.m2.1.1.3.cmml" xref="S2.SS1.p3.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p3.2.m2.1.1.3.1.cmml" xref="S2.SS1.p3.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS1.p3.2.m2.1.1.3.2.cmml" xref="S2.SS1.p3.2.m2.1.1.3.2">ℝ</ci><ci id="S2.SS1.p3.2.m2.1.1.3.3.cmml" xref="S2.SS1.p3.2.m2.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.2.m2.1c">a\in\mathbb{R}^{D}</annotation></semantics></math>, PQ needs to learn the centroids from a training dataset <math id="S2.SS1.p3.3.m3.1" class="ltx_Math" alttext="\hat{A}\in\mathbb{R}^{\hat{N}\times D}" display="inline"><semantics id="S2.SS1.p3.3.m3.1a"><mrow id="S2.SS1.p3.3.m3.1.1" xref="S2.SS1.p3.3.m3.1.1.cmml"><mover accent="true" id="S2.SS1.p3.3.m3.1.1.2" xref="S2.SS1.p3.3.m3.1.1.2.cmml"><mi id="S2.SS1.p3.3.m3.1.1.2.2" xref="S2.SS1.p3.3.m3.1.1.2.2.cmml">A</mi><mo id="S2.SS1.p3.3.m3.1.1.2.1" xref="S2.SS1.p3.3.m3.1.1.2.1.cmml">^</mo></mover><mo id="S2.SS1.p3.3.m3.1.1.1" xref="S2.SS1.p3.3.m3.1.1.1.cmml">∈</mo><msup id="S2.SS1.p3.3.m3.1.1.3" xref="S2.SS1.p3.3.m3.1.1.3.cmml"><mi id="S2.SS1.p3.3.m3.1.1.3.2" xref="S2.SS1.p3.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p3.3.m3.1.1.3.3" xref="S2.SS1.p3.3.m3.1.1.3.3.cmml"><mover accent="true" id="S2.SS1.p3.3.m3.1.1.3.3.2" xref="S2.SS1.p3.3.m3.1.1.3.3.2.cmml"><mi id="S2.SS1.p3.3.m3.1.1.3.3.2.2" xref="S2.SS1.p3.3.m3.1.1.3.3.2.2.cmml">N</mi><mo id="S2.SS1.p3.3.m3.1.1.3.3.2.1" xref="S2.SS1.p3.3.m3.1.1.3.3.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p3.3.m3.1.1.3.3.1" xref="S2.SS1.p3.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p3.3.m3.1.1.3.3.3" xref="S2.SS1.p3.3.m3.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.3.m3.1b"><apply id="S2.SS1.p3.3.m3.1.1.cmml" xref="S2.SS1.p3.3.m3.1.1"><in id="S2.SS1.p3.3.m3.1.1.1.cmml" xref="S2.SS1.p3.3.m3.1.1.1"></in><apply id="S2.SS1.p3.3.m3.1.1.2.cmml" xref="S2.SS1.p3.3.m3.1.1.2"><ci id="S2.SS1.p3.3.m3.1.1.2.1.cmml" xref="S2.SS1.p3.3.m3.1.1.2.1">^</ci><ci id="S2.SS1.p3.3.m3.1.1.2.2.cmml" xref="S2.SS1.p3.3.m3.1.1.2.2">𝐴</ci></apply><apply id="S2.SS1.p3.3.m3.1.1.3.cmml" xref="S2.SS1.p3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p3.3.m3.1.1.3.1.cmml" xref="S2.SS1.p3.3.m3.1.1.3">superscript</csymbol><ci id="S2.SS1.p3.3.m3.1.1.3.2.cmml" xref="S2.SS1.p3.3.m3.1.1.3.2">ℝ</ci><apply id="S2.SS1.p3.3.m3.1.1.3.3.cmml" xref="S2.SS1.p3.3.m3.1.1.3.3"><times id="S2.SS1.p3.3.m3.1.1.3.3.1.cmml" xref="S2.SS1.p3.3.m3.1.1.3.3.1"></times><apply id="S2.SS1.p3.3.m3.1.1.3.3.2.cmml" xref="S2.SS1.p3.3.m3.1.1.3.3.2"><ci id="S2.SS1.p3.3.m3.1.1.3.3.2.1.cmml" xref="S2.SS1.p3.3.m3.1.1.3.3.2.1">^</ci><ci id="S2.SS1.p3.3.m3.1.1.3.3.2.2.cmml" xref="S2.SS1.p3.3.m3.1.1.3.3.2.2">𝑁</ci></apply><ci id="S2.SS1.p3.3.m3.1.1.3.3.3.cmml" xref="S2.SS1.p3.3.m3.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.3.m3.1c">\hat{A}\in\mathbb{R}^{\hat{N}\times D}</annotation></semantics></math> composed of vectors with the same distribution as <math id="S2.SS1.p3.4.m4.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.SS1.p3.4.m4.1a"><mi id="S2.SS1.p3.4.m4.1.1" xref="S2.SS1.p3.4.m4.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.4.m4.1b"><ci id="S2.SS1.p3.4.m4.1.1.cmml" xref="S2.SS1.p3.4.m4.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.4.m4.1c">a</annotation></semantics></math>. PQ first decomposes the vectors in the dataset into <math id="S2.SS1.p3.5.m5.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S2.SS1.p3.5.m5.1a"><mi id="S2.SS1.p3.5.m5.1.1" xref="S2.SS1.p3.5.m5.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.5.m5.1b"><ci id="S2.SS1.p3.5.m5.1.1.cmml" xref="S2.SS1.p3.5.m5.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.5.m5.1c">C</annotation></semantics></math> distinct <math id="S2.SS1.p3.6.m6.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.SS1.p3.6.m6.1a"><mi id="S2.SS1.p3.6.m6.1.1" xref="S2.SS1.p3.6.m6.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.6.m6.1b"><ci id="S2.SS1.p3.6.m6.1.1.cmml" xref="S2.SS1.p3.6.m6.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.6.m6.1c">V</annotation></semantics></math>-dimension sub-vectors, notated as <math id="S2.SS1.p3.7.m7.1" class="ltx_Math" alttext="\hat{A}^{c}\in\mathbb{R}^{\hat{N}\times V}" display="inline"><semantics id="S2.SS1.p3.7.m7.1a"><mrow id="S2.SS1.p3.7.m7.1.1" xref="S2.SS1.p3.7.m7.1.1.cmml"><msup id="S2.SS1.p3.7.m7.1.1.2" xref="S2.SS1.p3.7.m7.1.1.2.cmml"><mover accent="true" id="S2.SS1.p3.7.m7.1.1.2.2" xref="S2.SS1.p3.7.m7.1.1.2.2.cmml"><mi id="S2.SS1.p3.7.m7.1.1.2.2.2" xref="S2.SS1.p3.7.m7.1.1.2.2.2.cmml">A</mi><mo id="S2.SS1.p3.7.m7.1.1.2.2.1" xref="S2.SS1.p3.7.m7.1.1.2.2.1.cmml">^</mo></mover><mi id="S2.SS1.p3.7.m7.1.1.2.3" xref="S2.SS1.p3.7.m7.1.1.2.3.cmml">c</mi></msup><mo id="S2.SS1.p3.7.m7.1.1.1" xref="S2.SS1.p3.7.m7.1.1.1.cmml">∈</mo><msup id="S2.SS1.p3.7.m7.1.1.3" xref="S2.SS1.p3.7.m7.1.1.3.cmml"><mi id="S2.SS1.p3.7.m7.1.1.3.2" xref="S2.SS1.p3.7.m7.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p3.7.m7.1.1.3.3" xref="S2.SS1.p3.7.m7.1.1.3.3.cmml"><mover accent="true" id="S2.SS1.p3.7.m7.1.1.3.3.2" xref="S2.SS1.p3.7.m7.1.1.3.3.2.cmml"><mi id="S2.SS1.p3.7.m7.1.1.3.3.2.2" xref="S2.SS1.p3.7.m7.1.1.3.3.2.2.cmml">N</mi><mo id="S2.SS1.p3.7.m7.1.1.3.3.2.1" xref="S2.SS1.p3.7.m7.1.1.3.3.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p3.7.m7.1.1.3.3.1" xref="S2.SS1.p3.7.m7.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p3.7.m7.1.1.3.3.3" xref="S2.SS1.p3.7.m7.1.1.3.3.3.cmml">V</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.7.m7.1b"><apply id="S2.SS1.p3.7.m7.1.1.cmml" xref="S2.SS1.p3.7.m7.1.1"><in id="S2.SS1.p3.7.m7.1.1.1.cmml" xref="S2.SS1.p3.7.m7.1.1.1"></in><apply id="S2.SS1.p3.7.m7.1.1.2.cmml" xref="S2.SS1.p3.7.m7.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p3.7.m7.1.1.2.1.cmml" xref="S2.SS1.p3.7.m7.1.1.2">superscript</csymbol><apply id="S2.SS1.p3.7.m7.1.1.2.2.cmml" xref="S2.SS1.p3.7.m7.1.1.2.2"><ci id="S2.SS1.p3.7.m7.1.1.2.2.1.cmml" xref="S2.SS1.p3.7.m7.1.1.2.2.1">^</ci><ci id="S2.SS1.p3.7.m7.1.1.2.2.2.cmml" xref="S2.SS1.p3.7.m7.1.1.2.2.2">𝐴</ci></apply><ci id="S2.SS1.p3.7.m7.1.1.2.3.cmml" xref="S2.SS1.p3.7.m7.1.1.2.3">𝑐</ci></apply><apply id="S2.SS1.p3.7.m7.1.1.3.cmml" xref="S2.SS1.p3.7.m7.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p3.7.m7.1.1.3.1.cmml" xref="S2.SS1.p3.7.m7.1.1.3">superscript</csymbol><ci id="S2.SS1.p3.7.m7.1.1.3.2.cmml" xref="S2.SS1.p3.7.m7.1.1.3.2">ℝ</ci><apply id="S2.SS1.p3.7.m7.1.1.3.3.cmml" xref="S2.SS1.p3.7.m7.1.1.3.3"><times id="S2.SS1.p3.7.m7.1.1.3.3.1.cmml" xref="S2.SS1.p3.7.m7.1.1.3.3.1"></times><apply id="S2.SS1.p3.7.m7.1.1.3.3.2.cmml" xref="S2.SS1.p3.7.m7.1.1.3.3.2"><ci id="S2.SS1.p3.7.m7.1.1.3.3.2.1.cmml" xref="S2.SS1.p3.7.m7.1.1.3.3.2.1">^</ci><ci id="S2.SS1.p3.7.m7.1.1.3.3.2.2.cmml" xref="S2.SS1.p3.7.m7.1.1.3.3.2.2">𝑁</ci></apply><ci id="S2.SS1.p3.7.m7.1.1.3.3.3.cmml" xref="S2.SS1.p3.7.m7.1.1.3.3.3">𝑉</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.7.m7.1c">\hat{A}^{c}\in\mathbb{R}^{\hat{N}\times V}</annotation></semantics></math> (marked in different colors in Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>). To make optimal quantization, the centroid learning process is to find the <math id="S2.SS1.p3.8.m8.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.SS1.p3.8.m8.1a"><mi id="S2.SS1.p3.8.m8.1.1" xref="S2.SS1.p3.8.m8.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.8.m8.1b"><ci id="S2.SS1.p3.8.m8.1.1.cmml" xref="S2.SS1.p3.8.m8.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.8.m8.1c">K</annotation></semantics></math> centroids <math id="S2.SS1.p3.9.m9.1" class="ltx_Math" alttext="P^{c}" display="inline"><semantics id="S2.SS1.p3.9.m9.1a"><msup id="S2.SS1.p3.9.m9.1.1" xref="S2.SS1.p3.9.m9.1.1.cmml"><mi id="S2.SS1.p3.9.m9.1.1.2" xref="S2.SS1.p3.9.m9.1.1.2.cmml">P</mi><mi id="S2.SS1.p3.9.m9.1.1.3" xref="S2.SS1.p3.9.m9.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.9.m9.1b"><apply id="S2.SS1.p3.9.m9.1.1.cmml" xref="S2.SS1.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.9.m9.1.1.1.cmml" xref="S2.SS1.p3.9.m9.1.1">superscript</csymbol><ci id="S2.SS1.p3.9.m9.1.1.2.cmml" xref="S2.SS1.p3.9.m9.1.1.2">𝑃</ci><ci id="S2.SS1.p3.9.m9.1.1.3.cmml" xref="S2.SS1.p3.9.m9.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.9.m9.1c">P^{c}</annotation></semantics></math> (i.e., the <math id="S2.SS1.p3.10.m10.1" class="ltx_Math" alttext="c^{th}" display="inline"><semantics id="S2.SS1.p3.10.m10.1a"><msup id="S2.SS1.p3.10.m10.1.1" xref="S2.SS1.p3.10.m10.1.1.cmml"><mi id="S2.SS1.p3.10.m10.1.1.2" xref="S2.SS1.p3.10.m10.1.1.2.cmml">c</mi><mrow id="S2.SS1.p3.10.m10.1.1.3" xref="S2.SS1.p3.10.m10.1.1.3.cmml"><mi id="S2.SS1.p3.10.m10.1.1.3.2" xref="S2.SS1.p3.10.m10.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p3.10.m10.1.1.3.1" xref="S2.SS1.p3.10.m10.1.1.3.1.cmml">​</mo><mi id="S2.SS1.p3.10.m10.1.1.3.3" xref="S2.SS1.p3.10.m10.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.10.m10.1b"><apply id="S2.SS1.p3.10.m10.1.1.cmml" xref="S2.SS1.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.10.m10.1.1.1.cmml" xref="S2.SS1.p3.10.m10.1.1">superscript</csymbol><ci id="S2.SS1.p3.10.m10.1.1.2.cmml" xref="S2.SS1.p3.10.m10.1.1.2">𝑐</ci><apply id="S2.SS1.p3.10.m10.1.1.3.cmml" xref="S2.SS1.p3.10.m10.1.1.3"><times id="S2.SS1.p3.10.m10.1.1.3.1.cmml" xref="S2.SS1.p3.10.m10.1.1.3.1"></times><ci id="S2.SS1.p3.10.m10.1.1.3.2.cmml" xref="S2.SS1.p3.10.m10.1.1.3.2">𝑡</ci><ci id="S2.SS1.p3.10.m10.1.1.3.3.cmml" xref="S2.SS1.p3.10.m10.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.10.m10.1c">c^{th}</annotation></semantics></math> codebook) for <math id="S2.SS1.p3.11.m11.1" class="ltx_Math" alttext="\hat{A}^{c}" display="inline"><semantics id="S2.SS1.p3.11.m11.1a"><msup id="S2.SS1.p3.11.m11.1.1" xref="S2.SS1.p3.11.m11.1.1.cmml"><mover accent="true" id="S2.SS1.p3.11.m11.1.1.2" xref="S2.SS1.p3.11.m11.1.1.2.cmml"><mi id="S2.SS1.p3.11.m11.1.1.2.2" xref="S2.SS1.p3.11.m11.1.1.2.2.cmml">A</mi><mo id="S2.SS1.p3.11.m11.1.1.2.1" xref="S2.SS1.p3.11.m11.1.1.2.1.cmml">^</mo></mover><mi id="S2.SS1.p3.11.m11.1.1.3" xref="S2.SS1.p3.11.m11.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.11.m11.1b"><apply id="S2.SS1.p3.11.m11.1.1.cmml" xref="S2.SS1.p3.11.m11.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.11.m11.1.1.1.cmml" xref="S2.SS1.p3.11.m11.1.1">superscript</csymbol><apply id="S2.SS1.p3.11.m11.1.1.2.cmml" xref="S2.SS1.p3.11.m11.1.1.2"><ci id="S2.SS1.p3.11.m11.1.1.2.1.cmml" xref="S2.SS1.p3.11.m11.1.1.2.1">^</ci><ci id="S2.SS1.p3.11.m11.1.1.2.2.cmml" xref="S2.SS1.p3.11.m11.1.1.2.2">𝐴</ci></apply><ci id="S2.SS1.p3.11.m11.1.1.3.cmml" xref="S2.SS1.p3.11.m11.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.11.m11.1c">\hat{A}^{c}</annotation></semantics></math> by <math id="S2.SS1.p3.12.m12.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS1.p3.12.m12.1a"><mi id="S2.SS1.p3.12.m12.1.1" xref="S2.SS1.p3.12.m12.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.12.m12.1b"><ci id="S2.SS1.p3.12.m12.1.1.cmml" xref="S2.SS1.p3.12.m12.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.12.m12.1c">k</annotation></semantics></math>-means <cite class="ltx_cite ltx_citemacro_citep">(Lloyd, <a href="#bib.bib41" title="" class="ltx_ref">1982</a>)</cite>, which can minimize the distance sum of each sub-vector <math id="S2.SS1.p3.13.m13.1" class="ltx_Math" alttext="\hat{A}^{c}_{i}" display="inline"><semantics id="S2.SS1.p3.13.m13.1a"><msubsup id="S2.SS1.p3.13.m13.1.1" xref="S2.SS1.p3.13.m13.1.1.cmml"><mover accent="true" id="S2.SS1.p3.13.m13.1.1.2.2" xref="S2.SS1.p3.13.m13.1.1.2.2.cmml"><mi id="S2.SS1.p3.13.m13.1.1.2.2.2" xref="S2.SS1.p3.13.m13.1.1.2.2.2.cmml">A</mi><mo id="S2.SS1.p3.13.m13.1.1.2.2.1" xref="S2.SS1.p3.13.m13.1.1.2.2.1.cmml">^</mo></mover><mi id="S2.SS1.p3.13.m13.1.1.3" xref="S2.SS1.p3.13.m13.1.1.3.cmml">i</mi><mi id="S2.SS1.p3.13.m13.1.1.2.3" xref="S2.SS1.p3.13.m13.1.1.2.3.cmml">c</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.13.m13.1b"><apply id="S2.SS1.p3.13.m13.1.1.cmml" xref="S2.SS1.p3.13.m13.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.13.m13.1.1.1.cmml" xref="S2.SS1.p3.13.m13.1.1">subscript</csymbol><apply id="S2.SS1.p3.13.m13.1.1.2.cmml" xref="S2.SS1.p3.13.m13.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.13.m13.1.1.2.1.cmml" xref="S2.SS1.p3.13.m13.1.1">superscript</csymbol><apply id="S2.SS1.p3.13.m13.1.1.2.2.cmml" xref="S2.SS1.p3.13.m13.1.1.2.2"><ci id="S2.SS1.p3.13.m13.1.1.2.2.1.cmml" xref="S2.SS1.p3.13.m13.1.1.2.2.1">^</ci><ci id="S2.SS1.p3.13.m13.1.1.2.2.2.cmml" xref="S2.SS1.p3.13.m13.1.1.2.2.2">𝐴</ci></apply><ci id="S2.SS1.p3.13.m13.1.1.2.3.cmml" xref="S2.SS1.p3.13.m13.1.1.2.3">𝑐</ci></apply><ci id="S2.SS1.p3.13.m13.1.1.3.cmml" xref="S2.SS1.p3.13.m13.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.13.m13.1c">\hat{A}^{c}_{i}</annotation></semantics></math> and its nearest centroid <math id="S2.SS1.p3.14.m14.1" class="ltx_Math" alttext="P^{c}_{k}" display="inline"><semantics id="S2.SS1.p3.14.m14.1a"><msubsup id="S2.SS1.p3.14.m14.1.1" xref="S2.SS1.p3.14.m14.1.1.cmml"><mi id="S2.SS1.p3.14.m14.1.1.2.2" xref="S2.SS1.p3.14.m14.1.1.2.2.cmml">P</mi><mi id="S2.SS1.p3.14.m14.1.1.3" xref="S2.SS1.p3.14.m14.1.1.3.cmml">k</mi><mi id="S2.SS1.p3.14.m14.1.1.2.3" xref="S2.SS1.p3.14.m14.1.1.2.3.cmml">c</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.14.m14.1b"><apply id="S2.SS1.p3.14.m14.1.1.cmml" xref="S2.SS1.p3.14.m14.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.14.m14.1.1.1.cmml" xref="S2.SS1.p3.14.m14.1.1">subscript</csymbol><apply id="S2.SS1.p3.14.m14.1.1.2.cmml" xref="S2.SS1.p3.14.m14.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.14.m14.1.1.2.1.cmml" xref="S2.SS1.p3.14.m14.1.1">superscript</csymbol><ci id="S2.SS1.p3.14.m14.1.1.2.2.cmml" xref="S2.SS1.p3.14.m14.1.1.2.2">𝑃</ci><ci id="S2.SS1.p3.14.m14.1.1.2.3.cmml" xref="S2.SS1.p3.14.m14.1.1.2.3">𝑐</ci></apply><ci id="S2.SS1.p3.14.m14.1.1.3.cmml" xref="S2.SS1.p3.14.m14.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.14.m14.1c">P^{c}_{k}</annotation></semantics></math>, as shown in Eq. <a href="#S2.E1" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left"><span class="ltx_tag ltx_tag_equation ltx_align_left">(1)</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.1" class="ltx_Math" alttext="\underset{P}{\mathrm{arg\ min}}\sum_{c}\sum_{i}\left\|\hat{A}^{c}_{i}-P^{c}_{k}\right\|^{2}" display="block"><semantics id="S2.E1.m1.1a"><mrow id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml"><munder accentunder="true" id="S2.E1.m1.1.1.3" xref="S2.E1.m1.1.1.3.cmml"><mrow id="S2.E1.m1.1.1.3.2" xref="S2.E1.m1.1.1.3.2.cmml"><mi id="S2.E1.m1.1.1.3.2.2" xref="S2.E1.m1.1.1.3.2.2.cmml">arg</mi><mo lspace="0.500em" rspace="0em" id="S2.E1.m1.1.1.3.2.1" xref="S2.E1.m1.1.1.3.2.1.cmml">​</mo><mi id="S2.E1.m1.1.1.3.2.3" xref="S2.E1.m1.1.1.3.2.3.cmml">min</mi></mrow><mo id="S2.E1.m1.1.1.3.1" xref="S2.E1.m1.1.1.3.1.cmml">𝑃</mo></munder><mo lspace="0em" rspace="0em" id="S2.E1.m1.1.1.2" xref="S2.E1.m1.1.1.2.cmml">​</mo><mrow id="S2.E1.m1.1.1.1" xref="S2.E1.m1.1.1.1.cmml"><munder id="S2.E1.m1.1.1.1.2" xref="S2.E1.m1.1.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S2.E1.m1.1.1.1.2.2" xref="S2.E1.m1.1.1.1.2.2.cmml">∑</mo><mi id="S2.E1.m1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.2.3.cmml">c</mi></munder><mrow id="S2.E1.m1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.cmml"><munder id="S2.E1.m1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S2.E1.m1.1.1.1.1.2.2" xref="S2.E1.m1.1.1.1.1.2.2.cmml">∑</mo><mi id="S2.E1.m1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.2.3.cmml">i</mi></munder><msup id="S2.E1.m1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.cmml"><mrow id="S2.E1.m1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.2.cmml"><mo id="S2.E1.m1.1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S2.E1.m1.1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml"><msubsup id="S2.E1.m1.1.1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.cmml"><mover accent="true" id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.2.cmml">A</mi><mo id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.1.cmml">^</mo></mover><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.3.cmml">c</mi></msubsup><mo id="S2.E1.m1.1.1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S2.E1.m1.1.1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.3.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.3.cmml">k</mi><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.3.cmml">c</mi></msubsup></mrow><mo id="S2.E1.m1.1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.E1.m1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.3.cmml">2</mn></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1"><times id="S2.E1.m1.1.1.2.cmml" xref="S2.E1.m1.1.1.2"></times><apply id="S2.E1.m1.1.1.3.cmml" xref="S2.E1.m1.1.1.3"><ci id="S2.E1.m1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.3.1">𝑃</ci><apply id="S2.E1.m1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.3.2"><times id="S2.E1.m1.1.1.3.2.1.cmml" xref="S2.E1.m1.1.1.3.2.1"></times><ci id="S2.E1.m1.1.1.3.2.2.cmml" xref="S2.E1.m1.1.1.3.2.2">arg</ci><ci id="S2.E1.m1.1.1.3.2.3.cmml" xref="S2.E1.m1.1.1.3.2.3">min</ci></apply></apply><apply id="S2.E1.m1.1.1.1.cmml" xref="S2.E1.m1.1.1.1"><apply id="S2.E1.m1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.2">subscript</csymbol><sum id="S2.E1.m1.1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.2.2"></sum><ci id="S2.E1.m1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.2.3">𝑐</ci></apply><apply id="S2.E1.m1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1"><apply id="S2.E1.m1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2">subscript</csymbol><sum id="S2.E1.m1.1.1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2"></sum><ci id="S2.E1.m1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1">superscript</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.E1.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1"><minus id="S2.E1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.1"></minus><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2">subscript</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2"><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.1">^</ci><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.2.2">𝐴</ci></apply><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.2.3">𝑐</ci></apply><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.2.3">𝑐</ci></apply><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.3">𝑘</ci></apply></apply></apply><cn type="integer" id="S2.E1.m1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">\underset{P}{\mathrm{arg\ min}}\sum_{c}\sum_{i}\left\|\hat{A}^{c}_{i}-P^{c}_{k}\right\|^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.p4" class="ltx_para">
<p id="S2.SS1.p4.4" class="ltx_p"><span id="S2.SS1.p4.4.1" class="ltx_text ltx_font_bold">Sub-vector encoding</span> With the learned centroids <math id="S2.SS1.p4.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S2.SS1.p4.1.m1.1a"><mi id="S2.SS1.p4.1.m1.1.1" xref="S2.SS1.p4.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.1.m1.1b"><ci id="S2.SS1.p4.1.m1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.1.m1.1c">P</annotation></semantics></math>, for an input vector <math id="S2.SS1.p4.2.m2.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.SS1.p4.2.m2.1a"><mi id="S2.SS1.p4.2.m2.1.1" xref="S2.SS1.p4.2.m2.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.2.m2.1b"><ci id="S2.SS1.p4.2.m2.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.2.m2.1c">a</annotation></semantics></math>, PQ can encode it as the concatenation of the nearest centroids for each sub-vector. The encoding function for a sub-vector is shown in Eq. <a href="#S2.E2" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. By this vector decomposition method, the centroids can represent <math id="S2.SS1.p4.3.m3.1" class="ltx_Math" alttext="K^{C}" display="inline"><semantics id="S2.SS1.p4.3.m3.1a"><msup id="S2.SS1.p4.3.m3.1.1" xref="S2.SS1.p4.3.m3.1.1.cmml"><mi id="S2.SS1.p4.3.m3.1.1.2" xref="S2.SS1.p4.3.m3.1.1.2.cmml">K</mi><mi id="S2.SS1.p4.3.m3.1.1.3" xref="S2.SS1.p4.3.m3.1.1.3.cmml">C</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.3.m3.1b"><apply id="S2.SS1.p4.3.m3.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.3.m3.1.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1">superscript</csymbol><ci id="S2.SS1.p4.3.m3.1.1.2.cmml" xref="S2.SS1.p4.3.m3.1.1.2">𝐾</ci><ci id="S2.SS1.p4.3.m3.1.1.3.cmml" xref="S2.SS1.p4.3.m3.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.3.m3.1c">K^{C}</annotation></semantics></math> different vectors by only <math id="S2.SS1.p4.4.m4.1" class="ltx_Math" alttext="K\times C" display="inline"><semantics id="S2.SS1.p4.4.m4.1a"><mrow id="S2.SS1.p4.4.m4.1.1" xref="S2.SS1.p4.4.m4.1.1.cmml"><mi id="S2.SS1.p4.4.m4.1.1.2" xref="S2.SS1.p4.4.m4.1.1.2.cmml">K</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p4.4.m4.1.1.1" xref="S2.SS1.p4.4.m4.1.1.1.cmml">×</mo><mi id="S2.SS1.p4.4.m4.1.1.3" xref="S2.SS1.p4.4.m4.1.1.3.cmml">C</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.4.m4.1b"><apply id="S2.SS1.p4.4.m4.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1"><times id="S2.SS1.p4.4.m4.1.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1.1"></times><ci id="S2.SS1.p4.4.m4.1.1.2.cmml" xref="S2.SS1.p4.4.m4.1.1.2">𝐾</ci><ci id="S2.SS1.p4.4.m4.1.1.3.cmml" xref="S2.SS1.p4.4.m4.1.1.3">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.4.m4.1c">K\times C</annotation></semantics></math> memory cost.</p>
<table id="S2.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left"><span class="ltx_tag ltx_tag_equation ltx_align_left">(2)</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E2.m1.2" class="ltx_Math" alttext="g^{c}(a^{c})=\underset{k}{\mathrm{arg\ min}}\left\|a^{c}-P^{c}_{k}\right\|^{2}" display="block"><semantics id="S2.E2.m1.2a"><mrow id="S2.E2.m1.2.2" xref="S2.E2.m1.2.2.cmml"><mrow id="S2.E2.m1.1.1.1" xref="S2.E2.m1.1.1.1.cmml"><msup id="S2.E2.m1.1.1.1.3" xref="S2.E2.m1.1.1.1.3.cmml"><mi id="S2.E2.m1.1.1.1.3.2" xref="S2.E2.m1.1.1.1.3.2.cmml">g</mi><mi id="S2.E2.m1.1.1.1.3.3" xref="S2.E2.m1.1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S2.E2.m1.1.1.1.2" xref="S2.E2.m1.1.1.1.2.cmml">​</mo><mrow id="S2.E2.m1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2.m1.1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.1.1.cmml">(</mo><msup id="S2.E2.m1.1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m1.1.1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.1.1.2.cmml">a</mi><mi id="S2.E2.m1.1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S2.E2.m1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E2.m1.2.2.3" xref="S2.E2.m1.2.2.3.cmml">=</mo><mrow id="S2.E2.m1.2.2.2" xref="S2.E2.m1.2.2.2.cmml"><munder accentunder="true" id="S2.E2.m1.2.2.2.3" xref="S2.E2.m1.2.2.2.3.cmml"><mrow id="S2.E2.m1.2.2.2.3.2" xref="S2.E2.m1.2.2.2.3.2.cmml"><mi id="S2.E2.m1.2.2.2.3.2.2" xref="S2.E2.m1.2.2.2.3.2.2.cmml">arg</mi><mo lspace="0.500em" rspace="0em" id="S2.E2.m1.2.2.2.3.2.1" xref="S2.E2.m1.2.2.2.3.2.1.cmml">​</mo><mi id="S2.E2.m1.2.2.2.3.2.3" xref="S2.E2.m1.2.2.2.3.2.3.cmml">min</mi></mrow><mo id="S2.E2.m1.2.2.2.3.1" xref="S2.E2.m1.2.2.2.3.1.cmml">𝑘</mo></munder><mo lspace="0em" rspace="0em" id="S2.E2.m1.2.2.2.2" xref="S2.E2.m1.2.2.2.2.cmml">​</mo><msup id="S2.E2.m1.2.2.2.1" xref="S2.E2.m1.2.2.2.1.cmml"><mrow id="S2.E2.m1.2.2.2.1.1.1" xref="S2.E2.m1.2.2.2.1.1.2.cmml"><mo id="S2.E2.m1.2.2.2.1.1.1.2" xref="S2.E2.m1.2.2.2.1.1.2.1.cmml">‖</mo><mrow id="S2.E2.m1.2.2.2.1.1.1.1" xref="S2.E2.m1.2.2.2.1.1.1.1.cmml"><msup id="S2.E2.m1.2.2.2.1.1.1.1.2" xref="S2.E2.m1.2.2.2.1.1.1.1.2.cmml"><mi id="S2.E2.m1.2.2.2.1.1.1.1.2.2" xref="S2.E2.m1.2.2.2.1.1.1.1.2.2.cmml">a</mi><mi id="S2.E2.m1.2.2.2.1.1.1.1.2.3" xref="S2.E2.m1.2.2.2.1.1.1.1.2.3.cmml">c</mi></msup><mo id="S2.E2.m1.2.2.2.1.1.1.1.1" xref="S2.E2.m1.2.2.2.1.1.1.1.1.cmml">−</mo><msubsup id="S2.E2.m1.2.2.2.1.1.1.1.3" xref="S2.E2.m1.2.2.2.1.1.1.1.3.cmml"><mi id="S2.E2.m1.2.2.2.1.1.1.1.3.2.2" xref="S2.E2.m1.2.2.2.1.1.1.1.3.2.2.cmml">P</mi><mi id="S2.E2.m1.2.2.2.1.1.1.1.3.3" xref="S2.E2.m1.2.2.2.1.1.1.1.3.3.cmml">k</mi><mi id="S2.E2.m1.2.2.2.1.1.1.1.3.2.3" xref="S2.E2.m1.2.2.2.1.1.1.1.3.2.3.cmml">c</mi></msubsup></mrow><mo id="S2.E2.m1.2.2.2.1.1.1.3" xref="S2.E2.m1.2.2.2.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.E2.m1.2.2.2.1.3" xref="S2.E2.m1.2.2.2.1.3.cmml">2</mn></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.2b"><apply id="S2.E2.m1.2.2.cmml" xref="S2.E2.m1.2.2"><eq id="S2.E2.m1.2.2.3.cmml" xref="S2.E2.m1.2.2.3"></eq><apply id="S2.E2.m1.1.1.1.cmml" xref="S2.E2.m1.1.1.1"><times id="S2.E2.m1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.2"></times><apply id="S2.E2.m1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.3.1.cmml" xref="S2.E2.m1.1.1.1.3">superscript</csymbol><ci id="S2.E2.m1.1.1.1.3.2.cmml" xref="S2.E2.m1.1.1.1.3.2">𝑔</ci><ci id="S2.E2.m1.1.1.1.3.3.cmml" xref="S2.E2.m1.1.1.1.3.3">𝑐</ci></apply><apply id="S2.E2.m1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1">superscript</csymbol><ci id="S2.E2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2">𝑎</ci><ci id="S2.E2.m1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S2.E2.m1.2.2.2.cmml" xref="S2.E2.m1.2.2.2"><times id="S2.E2.m1.2.2.2.2.cmml" xref="S2.E2.m1.2.2.2.2"></times><apply id="S2.E2.m1.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.3"><ci id="S2.E2.m1.2.2.2.3.1.cmml" xref="S2.E2.m1.2.2.2.3.1">𝑘</ci><apply id="S2.E2.m1.2.2.2.3.2.cmml" xref="S2.E2.m1.2.2.2.3.2"><times id="S2.E2.m1.2.2.2.3.2.1.cmml" xref="S2.E2.m1.2.2.2.3.2.1"></times><ci id="S2.E2.m1.2.2.2.3.2.2.cmml" xref="S2.E2.m1.2.2.2.3.2.2">arg</ci><ci id="S2.E2.m1.2.2.2.3.2.3.cmml" xref="S2.E2.m1.2.2.2.3.2.3">min</ci></apply></apply><apply id="S2.E2.m1.2.2.2.1.cmml" xref="S2.E2.m1.2.2.2.1"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.1.2.cmml" xref="S2.E2.m1.2.2.2.1">superscript</csymbol><apply id="S2.E2.m1.2.2.2.1.1.2.cmml" xref="S2.E2.m1.2.2.2.1.1.1"><csymbol cd="latexml" id="S2.E2.m1.2.2.2.1.1.2.1.cmml" xref="S2.E2.m1.2.2.2.1.1.1.2">norm</csymbol><apply id="S2.E2.m1.2.2.2.1.1.1.1.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1"><minus id="S2.E2.m1.2.2.2.1.1.1.1.1.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.1"></minus><apply id="S2.E2.m1.2.2.2.1.1.1.1.2.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.1.1.1.1.2.1.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.2">superscript</csymbol><ci id="S2.E2.m1.2.2.2.1.1.1.1.2.2.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.2.2">𝑎</ci><ci id="S2.E2.m1.2.2.2.1.1.1.1.2.3.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.2.3">𝑐</ci></apply><apply id="S2.E2.m1.2.2.2.1.1.1.1.3.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.1.1.1.1.3.1.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3">subscript</csymbol><apply id="S2.E2.m1.2.2.2.1.1.1.1.3.2.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.1.1.1.1.3.2.1.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3">superscript</csymbol><ci id="S2.E2.m1.2.2.2.1.1.1.1.3.2.2.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3.2.2">𝑃</ci><ci id="S2.E2.m1.2.2.2.1.1.1.1.3.2.3.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3.2.3">𝑐</ci></apply><ci id="S2.E2.m1.2.2.2.1.1.1.1.3.3.cmml" xref="S2.E2.m1.2.2.2.1.1.1.1.3.3">𝑘</ci></apply></apply></apply><cn type="integer" id="S2.E2.m1.2.2.2.1.3.cmml" xref="S2.E2.m1.2.2.2.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.2c">g^{c}(a^{c})=\underset{k}{\mathrm{arg\ min}}\left\|a^{c}-P^{c}_{k}\right\|^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.p5" class="ltx_para">
<p id="S2.SS1.p5.3" class="ltx_p"><span id="S2.SS1.p5.3.1" class="ltx_text ltx_font_bold">Hashing for acceleration with bigger error </span> Learning centroids is a NP-hard problem. Vanilla PQ uses <math id="S2.SS1.p5.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS1.p5.1.m1.1a"><mi id="S2.SS1.p5.1.m1.1.1" xref="S2.SS1.p5.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.1.m1.1b"><ci id="S2.SS1.p5.1.m1.1.1.cmml" xref="S2.SS1.p5.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.1.m1.1c">k</annotation></semantics></math>-means to learn centroids and encode sub-vectors. <math id="S2.SS1.p5.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS1.p5.2.m2.1a"><mi id="S2.SS1.p5.2.m2.1.1" xref="S2.SS1.p5.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.2.m2.1b"><ci id="S2.SS1.p5.2.m2.1.1.cmml" xref="S2.SS1.p5.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.2.m2.1c">k</annotation></semantics></math>-means satisfies Lloyd optimal conditions <cite class="ltx_cite ltx_citemacro_citep">(Lloyd, <a href="#bib.bib41" title="" class="ltx_ref">1982</a>; Jegou et al<span class="ltx_text">.</span>, <a href="#bib.bib34" title="" class="ltx_ref">2010</a>)</cite> and can get a local optimal quantization error. However, the <math id="S2.SS1.p5.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS1.p5.3.m3.1a"><mi id="S2.SS1.p5.3.m3.1.1" xref="S2.SS1.p5.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.3.m3.1b"><ci id="S2.SS1.p5.3.m3.1.1.cmml" xref="S2.SS1.p5.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.3.m3.1c">k</annotation></semantics></math>-means encoding costs high to compute the Euclidean distance for each sub-vector with each centroid, as shown in Eq. <a href="#S2.E2" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S2.SS1.p6" class="ltx_para">
<p id="S2.SS1.p6.1" class="ltx_p">To reduce the encoding cost, some works propose hashing methods to encode sub-vectors  <cite class="ltx_cite ltx_citemacro_citep">(Blalock and Guttag, <a href="#bib.bib6" title="" class="ltx_ref">2021</a>; He et al<span class="ltx_text">.</span>, <a href="#bib.bib27" title="" class="ltx_ref">2013</a>)</cite>, but <em id="S2.SS1.p6.1.1" class="ltx_emph ltx_font_italic">at the cost of higher quantization error</em>. Hashing hashes a sub-vector to one of the <math id="S2.SS1.p6.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.SS1.p6.1.m1.1a"><mi id="S2.SS1.p6.1.m1.1.1" xref="S2.SS1.p6.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p6.1.m1.1b"><ci id="S2.SS1.p6.1.m1.1.1.cmml" xref="S2.SS1.p6.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p6.1.m1.1c">K</annotation></semantics></math> buckets. For example, <span id="S2.SS1.p6.1.2" class="ltx_text ltx_font_smallcaps">maddness</span> selects a 4-level balanced binary regression tree of the hashing function family, with each leaf as a hash bucket. A sub-vector is encoded by traversing the tree from the root, and moving to left or right child if the value of certain indices is above or below a threshold.</p>
</div>
<div id="S2.SS1.p7" class="ltx_para">
<p id="S2.SS1.p7.1" class="ltx_p">This paper will evaluate both distance-based encoding and hashing-based encoding methods.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2302.03213/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="236" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2. </span>Product Quantization for AMM. Each color marks a sub-vector dataset.</figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2. </span>PQ for AMM</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">PQ can be used for approximated matrix multiplication (AMM) <cite class="ltx_cite ltx_citemacro_citep">(Blalock and Guttag, <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>. The essence is to approximate the matrix multiplication by the centroids’ multiplication.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.12" class="ltx_p">To formulize it, for matrix multiplication <math id="S2.SS2.p2.1.m1.1" class="ltx_Math" alttext="A\times B^{T}" display="inline"><semantics id="S2.SS2.p2.1.m1.1a"><mrow id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml"><mi id="S2.SS2.p2.1.m1.1.1.2" xref="S2.SS2.p2.1.m1.1.1.2.cmml">A</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p2.1.m1.1.1.1" xref="S2.SS2.p2.1.m1.1.1.1.cmml">×</mo><msup id="S2.SS2.p2.1.m1.1.1.3" xref="S2.SS2.p2.1.m1.1.1.3.cmml"><mi id="S2.SS2.p2.1.m1.1.1.3.2" xref="S2.SS2.p2.1.m1.1.1.3.2.cmml">B</mi><mi id="S2.SS2.p2.1.m1.1.1.3.3" xref="S2.SS2.p2.1.m1.1.1.3.3.cmml">T</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.1b"><apply id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1"><times id="S2.SS2.p2.1.m1.1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1.1"></times><ci id="S2.SS2.p2.1.m1.1.1.2.cmml" xref="S2.SS2.p2.1.m1.1.1.2">𝐴</ci><apply id="S2.SS2.p2.1.m1.1.1.3.cmml" xref="S2.SS2.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.1.1.3.1.cmml" xref="S2.SS2.p2.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p2.1.m1.1.1.3.2.cmml" xref="S2.SS2.p2.1.m1.1.1.3.2">𝐵</ci><ci id="S2.SS2.p2.1.m1.1.1.3.3.cmml" xref="S2.SS2.p2.1.m1.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">A\times B^{T}</annotation></semantics></math>, <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><mi id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><ci id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">a</annotation></semantics></math> and <math id="S2.SS2.p2.3.m3.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS2.p2.3.m3.1a"><mi id="S2.SS2.p2.3.m3.1.1" xref="S2.SS2.p2.3.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.3.m3.1b"><ci id="S2.SS2.p2.3.m3.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.3.m3.1c">b</annotation></semantics></math> are the rows of <math id="S2.SS2.p2.4.m4.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p2.4.m4.1a"><mi id="S2.SS2.p2.4.m4.1.1" xref="S2.SS2.p2.4.m4.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.4.m4.1b"><ci id="S2.SS2.p2.4.m4.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.4.m4.1c">A</annotation></semantics></math> and <math id="S2.SS2.p2.5.m5.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.5.m5.1a"><mi id="S2.SS2.p2.5.m5.1.1" xref="S2.SS2.p2.5.m5.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.5.m5.1b"><ci id="S2.SS2.p2.5.m5.1.1.cmml" xref="S2.SS2.p2.5.m5.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.5.m5.1c">B</annotation></semantics></math> respectively. The centroid codebooks for <math id="S2.SS2.p2.6.m6.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p2.6.m6.1a"><mi id="S2.SS2.p2.6.m6.1.1" xref="S2.SS2.p2.6.m6.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.6.m6.1b"><ci id="S2.SS2.p2.6.m6.1.1.cmml" xref="S2.SS2.p2.6.m6.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.6.m6.1c">A</annotation></semantics></math> is <math id="S2.SS2.p2.7.m7.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S2.SS2.p2.7.m7.1a"><mi id="S2.SS2.p2.7.m7.1.1" xref="S2.SS2.p2.7.m7.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.7.m7.1b"><ci id="S2.SS2.p2.7.m7.1.1.cmml" xref="S2.SS2.p2.7.m7.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.7.m7.1c">P</annotation></semantics></math>. For a layer of DNN, <math id="S2.SS2.p2.8.m8.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p2.8.m8.1a"><mi id="S2.SS2.p2.8.m8.1.1" xref="S2.SS2.p2.8.m8.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.8.m8.1b"><ci id="S2.SS2.p2.8.m8.1.1.cmml" xref="S2.SS2.p2.8.m8.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.8.m8.1c">A</annotation></semantics></math> can be the input feature maps, and <math id="S2.SS2.p2.9.m9.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.9.m9.1a"><mi id="S2.SS2.p2.9.m9.1.1" xref="S2.SS2.p2.9.m9.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.9.m9.1b"><ci id="S2.SS2.p2.9.m9.1.1.cmml" xref="S2.SS2.p2.9.m9.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.9.m9.1c">B</annotation></semantics></math> can be the weights (note that convolution can be computed as matrix multiplication too). Since <math id="S2.SS2.p2.10.m10.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.10.m10.1a"><mi id="S2.SS2.p2.10.m10.1.1" xref="S2.SS2.p2.10.m10.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.10.m10.1b"><ci id="S2.SS2.p2.10.m10.1.1.cmml" xref="S2.SS2.p2.10.m10.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.10.m10.1c">B</annotation></semantics></math> is constant, the multiplication of all the centroids and <math id="S2.SS2.p2.11.m11.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.11.m11.1a"><mi id="S2.SS2.p2.11.m11.1.1" xref="S2.SS2.p2.11.m11.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.11.m11.1b"><ci id="S2.SS2.p2.11.m11.1.1.cmml" xref="S2.SS2.p2.11.m11.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.11.m11.1c">B</annotation></semantics></math> can be precomputed to construct a lookup table, as shown in Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>(b). The table construction function for <math id="S2.SS2.p2.12.m12.1" class="ltx_Math" alttext="b^{c}" display="inline"><semantics id="S2.SS2.p2.12.m12.1a"><msup id="S2.SS2.p2.12.m12.1.1" xref="S2.SS2.p2.12.m12.1.1.cmml"><mi id="S2.SS2.p2.12.m12.1.1.2" xref="S2.SS2.p2.12.m12.1.1.2.cmml">b</mi><mi id="S2.SS2.p2.12.m12.1.1.3" xref="S2.SS2.p2.12.m12.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.12.m12.1b"><apply id="S2.SS2.p2.12.m12.1.1.cmml" xref="S2.SS2.p2.12.m12.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.12.m12.1.1.1.cmml" xref="S2.SS2.p2.12.m12.1.1">superscript</csymbol><ci id="S2.SS2.p2.12.m12.1.1.2.cmml" xref="S2.SS2.p2.12.m12.1.1.2">𝑏</ci><ci id="S2.SS2.p2.12.m12.1.1.3.cmml" xref="S2.SS2.p2.12.m12.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.12.m12.1c">b^{c}</annotation></semantics></math> is shown in Eq. <a href="#S2.E3" title="In 2.2. PQ for AMM ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
<table id="S2.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left"><span class="ltx_tag ltx_tag_equation ltx_align_left">(3)</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E3.m1.39" class="ltx_Math" alttext="\begin{split}h^{c}(b^{c})=[P_{0}^{c}\cdot b^{c},P_{1}^{c}\cdot b^{c},\cdots,P_{K-1}^{c}\cdot b^{c}]\end{split}" display="block"><semantics id="S2.E3.m1.39a"><mtable displaystyle="true" id="S2.E3.m1.39.39.8" xref="S2.E3.m1.35.35.4.cmml"><mtr id="S2.E3.m1.39.39.8a" xref="S2.E3.m1.35.35.4.cmml"><mtd class="ltx_align_right" columnalign="right" id="S2.E3.m1.39.39.8b" xref="S2.E3.m1.35.35.4.cmml"><mrow id="S2.E3.m1.39.39.8.35.35.35" xref="S2.E3.m1.35.35.4.cmml"><mrow id="S2.E3.m1.36.36.5.32.32.32.32" xref="S2.E3.m1.35.35.4.cmml"><msup id="S2.E3.m1.36.36.5.32.32.32.32.3" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1" xref="S2.E3.m1.1.1.1.1.1.1.cmml">h</mi><mi id="S2.E3.m1.2.2.2.2.2.2.1" xref="S2.E3.m1.2.2.2.2.2.2.1.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S2.E3.m1.36.36.5.32.32.32.32.2" xref="S2.E3.m1.35.35.4.cmml">​</mo><mrow id="S2.E3.m1.36.36.5.32.32.32.32.1.1" xref="S2.E3.m1.35.35.4.cmml"><mo stretchy="false" id="S2.E3.m1.3.3.3.3.3.3" xref="S2.E3.m1.35.35.4.cmml">(</mo><msup id="S2.E3.m1.36.36.5.32.32.32.32.1.1.1" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.4.4.4.4.4.4" xref="S2.E3.m1.4.4.4.4.4.4.cmml">b</mi><mi id="S2.E3.m1.5.5.5.5.5.5.1" xref="S2.E3.m1.5.5.5.5.5.5.1.cmml">c</mi></msup><mo stretchy="false" id="S2.E3.m1.6.6.6.6.6.6" xref="S2.E3.m1.35.35.4.cmml">)</mo></mrow></mrow><mo id="S2.E3.m1.7.7.7.7.7.7" xref="S2.E3.m1.7.7.7.7.7.7.cmml">=</mo><mrow id="S2.E3.m1.39.39.8.35.35.35.35.3" xref="S2.E3.m1.35.35.4.cmml"><mo stretchy="false" id="S2.E3.m1.8.8.8.8.8.8" xref="S2.E3.m1.35.35.4.cmml">[</mo><mrow id="S2.E3.m1.37.37.6.33.33.33.33.1.1" xref="S2.E3.m1.35.35.4.cmml"><msubsup id="S2.E3.m1.37.37.6.33.33.33.33.1.1.1" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.9.9.9.9.9.9" xref="S2.E3.m1.9.9.9.9.9.9.cmml">P</mi><mn id="S2.E3.m1.10.10.10.10.10.10.1" xref="S2.E3.m1.10.10.10.10.10.10.1.cmml">0</mn><mi id="S2.E3.m1.11.11.11.11.11.11.1" xref="S2.E3.m1.11.11.11.11.11.11.1.cmml">c</mi></msubsup><mo lspace="0.222em" rspace="0.222em" id="S2.E3.m1.12.12.12.12.12.12" xref="S2.E3.m1.12.12.12.12.12.12.cmml">⋅</mo><msup id="S2.E3.m1.37.37.6.33.33.33.33.1.1.2" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.13.13.13.13.13.13" xref="S2.E3.m1.13.13.13.13.13.13.cmml">b</mi><mi id="S2.E3.m1.14.14.14.14.14.14.1" xref="S2.E3.m1.14.14.14.14.14.14.1.cmml">c</mi></msup></mrow><mo id="S2.E3.m1.15.15.15.15.15.15" xref="S2.E3.m1.35.35.4.cmml">,</mo><mrow id="S2.E3.m1.38.38.7.34.34.34.34.2.2" xref="S2.E3.m1.35.35.4.cmml"><msubsup id="S2.E3.m1.38.38.7.34.34.34.34.2.2.1" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.16.16.16.16.16.16" xref="S2.E3.m1.16.16.16.16.16.16.cmml">P</mi><mn id="S2.E3.m1.17.17.17.17.17.17.1" xref="S2.E3.m1.17.17.17.17.17.17.1.cmml">1</mn><mi id="S2.E3.m1.18.18.18.18.18.18.1" xref="S2.E3.m1.18.18.18.18.18.18.1.cmml">c</mi></msubsup><mo lspace="0.222em" rspace="0.222em" id="S2.E3.m1.19.19.19.19.19.19" xref="S2.E3.m1.19.19.19.19.19.19.cmml">⋅</mo><msup id="S2.E3.m1.38.38.7.34.34.34.34.2.2.2" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.20.20.20.20.20.20" xref="S2.E3.m1.20.20.20.20.20.20.cmml">b</mi><mi id="S2.E3.m1.21.21.21.21.21.21.1" xref="S2.E3.m1.21.21.21.21.21.21.1.cmml">c</mi></msup></mrow><mo id="S2.E3.m1.22.22.22.22.22.22" xref="S2.E3.m1.35.35.4.cmml">,</mo><mi mathvariant="normal" id="S2.E3.m1.23.23.23.23.23.23" xref="S2.E3.m1.23.23.23.23.23.23.cmml">⋯</mi><mo id="S2.E3.m1.24.24.24.24.24.24" xref="S2.E3.m1.35.35.4.cmml">,</mo><mrow id="S2.E3.m1.39.39.8.35.35.35.35.3.3" xref="S2.E3.m1.35.35.4.cmml"><msubsup id="S2.E3.m1.39.39.8.35.35.35.35.3.3.1" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.25.25.25.25.25.25" xref="S2.E3.m1.25.25.25.25.25.25.cmml">P</mi><mrow id="S2.E3.m1.26.26.26.26.26.26.1" xref="S2.E3.m1.26.26.26.26.26.26.1.cmml"><mi id="S2.E3.m1.26.26.26.26.26.26.1.2" xref="S2.E3.m1.26.26.26.26.26.26.1.2.cmml">K</mi><mo id="S2.E3.m1.26.26.26.26.26.26.1.1" xref="S2.E3.m1.26.26.26.26.26.26.1.1.cmml">−</mo><mn id="S2.E3.m1.26.26.26.26.26.26.1.3" xref="S2.E3.m1.26.26.26.26.26.26.1.3.cmml">1</mn></mrow><mi id="S2.E3.m1.27.27.27.27.27.27.1" xref="S2.E3.m1.27.27.27.27.27.27.1.cmml">c</mi></msubsup><mo lspace="0.222em" rspace="0.222em" id="S2.E3.m1.28.28.28.28.28.28" xref="S2.E3.m1.28.28.28.28.28.28.cmml">⋅</mo><msup id="S2.E3.m1.39.39.8.35.35.35.35.3.3.2" xref="S2.E3.m1.35.35.4.cmml"><mi id="S2.E3.m1.29.29.29.29.29.29" xref="S2.E3.m1.29.29.29.29.29.29.cmml">b</mi><mi id="S2.E3.m1.30.30.30.30.30.30.1" xref="S2.E3.m1.30.30.30.30.30.30.1.cmml">c</mi></msup></mrow><mo stretchy="false" id="S2.E3.m1.31.31.31.31.31.31" xref="S2.E3.m1.35.35.4.cmml">]</mo></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S2.E3.m1.39b"><apply id="S2.E3.m1.35.35.4.cmml" xref="S2.E3.m1.39.39.8"><eq id="S2.E3.m1.7.7.7.7.7.7.cmml" xref="S2.E3.m1.7.7.7.7.7.7"></eq><apply id="S2.E3.m1.32.32.1.1.cmml" xref="S2.E3.m1.39.39.8"><times id="S2.E3.m1.32.32.1.1.2.cmml" xref="S2.E3.m1.39.39.8"></times><apply id="S2.E3.m1.32.32.1.1.3.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.32.32.1.1.3.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1">ℎ</ci><ci id="S2.E3.m1.2.2.2.2.2.2.1.cmml" xref="S2.E3.m1.2.2.2.2.2.2.1">𝑐</ci></apply><apply id="S2.E3.m1.32.32.1.1.1.1.1.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.32.32.1.1.1.1.1.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><ci id="S2.E3.m1.4.4.4.4.4.4.cmml" xref="S2.E3.m1.4.4.4.4.4.4">𝑏</ci><ci id="S2.E3.m1.5.5.5.5.5.5.1.cmml" xref="S2.E3.m1.5.5.5.5.5.5.1">𝑐</ci></apply></apply><list id="S2.E3.m1.35.35.4.4.4.cmml" xref="S2.E3.m1.39.39.8"><apply id="S2.E3.m1.33.33.2.2.1.1.cmml" xref="S2.E3.m1.39.39.8"><ci id="S2.E3.m1.12.12.12.12.12.12.cmml" xref="S2.E3.m1.12.12.12.12.12.12">⋅</ci><apply id="S2.E3.m1.33.33.2.2.1.1.2.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.33.33.2.2.1.1.2.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><apply id="S2.E3.m1.33.33.2.2.1.1.2.2.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.33.33.2.2.1.1.2.2.1.cmml" xref="S2.E3.m1.39.39.8">subscript</csymbol><ci id="S2.E3.m1.9.9.9.9.9.9.cmml" xref="S2.E3.m1.9.9.9.9.9.9">𝑃</ci><cn type="integer" id="S2.E3.m1.10.10.10.10.10.10.1.cmml" xref="S2.E3.m1.10.10.10.10.10.10.1">0</cn></apply><ci id="S2.E3.m1.11.11.11.11.11.11.1.cmml" xref="S2.E3.m1.11.11.11.11.11.11.1">𝑐</ci></apply><apply id="S2.E3.m1.33.33.2.2.1.1.3.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.33.33.2.2.1.1.3.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><ci id="S2.E3.m1.13.13.13.13.13.13.cmml" xref="S2.E3.m1.13.13.13.13.13.13">𝑏</ci><ci id="S2.E3.m1.14.14.14.14.14.14.1.cmml" xref="S2.E3.m1.14.14.14.14.14.14.1">𝑐</ci></apply></apply><apply id="S2.E3.m1.34.34.3.3.2.2.cmml" xref="S2.E3.m1.39.39.8"><ci id="S2.E3.m1.19.19.19.19.19.19.cmml" xref="S2.E3.m1.19.19.19.19.19.19">⋅</ci><apply id="S2.E3.m1.34.34.3.3.2.2.2.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.34.34.3.3.2.2.2.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><apply id="S2.E3.m1.34.34.3.3.2.2.2.2.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.34.34.3.3.2.2.2.2.1.cmml" xref="S2.E3.m1.39.39.8">subscript</csymbol><ci id="S2.E3.m1.16.16.16.16.16.16.cmml" xref="S2.E3.m1.16.16.16.16.16.16">𝑃</ci><cn type="integer" id="S2.E3.m1.17.17.17.17.17.17.1.cmml" xref="S2.E3.m1.17.17.17.17.17.17.1">1</cn></apply><ci id="S2.E3.m1.18.18.18.18.18.18.1.cmml" xref="S2.E3.m1.18.18.18.18.18.18.1">𝑐</ci></apply><apply id="S2.E3.m1.34.34.3.3.2.2.3.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.34.34.3.3.2.2.3.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><ci id="S2.E3.m1.20.20.20.20.20.20.cmml" xref="S2.E3.m1.20.20.20.20.20.20">𝑏</ci><ci id="S2.E3.m1.21.21.21.21.21.21.1.cmml" xref="S2.E3.m1.21.21.21.21.21.21.1">𝑐</ci></apply></apply><ci id="S2.E3.m1.23.23.23.23.23.23.cmml" xref="S2.E3.m1.23.23.23.23.23.23">⋯</ci><apply id="S2.E3.m1.35.35.4.4.3.3.cmml" xref="S2.E3.m1.39.39.8"><ci id="S2.E3.m1.28.28.28.28.28.28.cmml" xref="S2.E3.m1.28.28.28.28.28.28">⋅</ci><apply id="S2.E3.m1.35.35.4.4.3.3.2.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.35.35.4.4.3.3.2.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><apply id="S2.E3.m1.35.35.4.4.3.3.2.2.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.35.35.4.4.3.3.2.2.1.cmml" xref="S2.E3.m1.39.39.8">subscript</csymbol><ci id="S2.E3.m1.25.25.25.25.25.25.cmml" xref="S2.E3.m1.25.25.25.25.25.25">𝑃</ci><apply id="S2.E3.m1.26.26.26.26.26.26.1.cmml" xref="S2.E3.m1.26.26.26.26.26.26.1"><minus id="S2.E3.m1.26.26.26.26.26.26.1.1.cmml" xref="S2.E3.m1.26.26.26.26.26.26.1.1"></minus><ci id="S2.E3.m1.26.26.26.26.26.26.1.2.cmml" xref="S2.E3.m1.26.26.26.26.26.26.1.2">𝐾</ci><cn type="integer" id="S2.E3.m1.26.26.26.26.26.26.1.3.cmml" xref="S2.E3.m1.26.26.26.26.26.26.1.3">1</cn></apply></apply><ci id="S2.E3.m1.27.27.27.27.27.27.1.cmml" xref="S2.E3.m1.27.27.27.27.27.27.1">𝑐</ci></apply><apply id="S2.E3.m1.35.35.4.4.3.3.3.cmml" xref="S2.E3.m1.39.39.8"><csymbol cd="ambiguous" id="S2.E3.m1.35.35.4.4.3.3.3.1.cmml" xref="S2.E3.m1.39.39.8">superscript</csymbol><ci id="S2.E3.m1.29.29.29.29.29.29.cmml" xref="S2.E3.m1.29.29.29.29.29.29">𝑏</ci><ci id="S2.E3.m1.30.30.30.30.30.30.1.cmml" xref="S2.E3.m1.30.30.30.30.30.30.1">𝑐</ci></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.39c">\begin{split}h^{c}(b^{c})=[P_{0}^{c}\cdot b^{c},P_{1}^{c}\cdot b^{c},\cdots,P_{K-1}^{c}\cdot b^{c}]\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.4" class="ltx_p">The matrix multiplication can then be approximated by looking up and aggregating the results of the nearest centroids in the precompute table, formulated in Eq. <a href="#S2.E4" title="In 2.2. PQ for AMM ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. Here considers the <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="g^{c}(a^{c})" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><mrow id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml"><msup id="S2.SS2.p3.1.m1.1.1.3" xref="S2.SS2.p3.1.m1.1.1.3.cmml"><mi id="S2.SS2.p3.1.m1.1.1.3.2" xref="S2.SS2.p3.1.m1.1.1.3.2.cmml">g</mi><mi id="S2.SS2.p3.1.m1.1.1.3.3" xref="S2.SS2.p3.1.m1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S2.SS2.p3.1.m1.1.1.2" xref="S2.SS2.p3.1.m1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p3.1.m1.1.1.1.1" xref="S2.SS2.p3.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p3.1.m1.1.1.1.1.2" xref="S2.SS2.p3.1.m1.1.1.1.1.1.cmml">(</mo><msup id="S2.SS2.p3.1.m1.1.1.1.1.1" xref="S2.SS2.p3.1.m1.1.1.1.1.1.cmml"><mi id="S2.SS2.p3.1.m1.1.1.1.1.1.2" xref="S2.SS2.p3.1.m1.1.1.1.1.1.2.cmml">a</mi><mi id="S2.SS2.p3.1.m1.1.1.1.1.1.3" xref="S2.SS2.p3.1.m1.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S2.SS2.p3.1.m1.1.1.1.1.3" xref="S2.SS2.p3.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><apply id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1"><times id="S2.SS2.p3.1.m1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.1.1.2"></times><apply id="S2.SS2.p3.1.m1.1.1.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.1.1.3.1.cmml" xref="S2.SS2.p3.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p3.1.m1.1.1.3.2.cmml" xref="S2.SS2.p3.1.m1.1.1.3.2">𝑔</ci><ci id="S2.SS2.p3.1.m1.1.1.3.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3.3">𝑐</ci></apply><apply id="S2.SS2.p3.1.m1.1.1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1.1.1">superscript</csymbol><ci id="S2.SS2.p3.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.1.1.1.1.1.2">𝑎</ci><ci id="S2.SS2.p3.1.m1.1.1.1.1.1.3.cmml" xref="S2.SS2.p3.1.m1.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">g^{c}(a^{c})</annotation></semantics></math> function with <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="onehot" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><mrow id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml"><mi id="S2.SS2.p3.2.m2.1.1.2" xref="S2.SS2.p3.2.m2.1.1.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.2.m2.1.1.1" xref="S2.SS2.p3.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.2.m2.1.1.3" xref="S2.SS2.p3.2.m2.1.1.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.2.m2.1.1.1a" xref="S2.SS2.p3.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.2.m2.1.1.4" xref="S2.SS2.p3.2.m2.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.2.m2.1.1.1b" xref="S2.SS2.p3.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.2.m2.1.1.5" xref="S2.SS2.p3.2.m2.1.1.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.2.m2.1.1.1c" xref="S2.SS2.p3.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.2.m2.1.1.6" xref="S2.SS2.p3.2.m2.1.1.6.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.2.m2.1.1.1d" xref="S2.SS2.p3.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.2.m2.1.1.7" xref="S2.SS2.p3.2.m2.1.1.7.cmml">t</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1"><times id="S2.SS2.p3.2.m2.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1.1"></times><ci id="S2.SS2.p3.2.m2.1.1.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2">𝑜</ci><ci id="S2.SS2.p3.2.m2.1.1.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3">𝑛</ci><ci id="S2.SS2.p3.2.m2.1.1.4.cmml" xref="S2.SS2.p3.2.m2.1.1.4">𝑒</ci><ci id="S2.SS2.p3.2.m2.1.1.5.cmml" xref="S2.SS2.p3.2.m2.1.1.5">ℎ</ci><ci id="S2.SS2.p3.2.m2.1.1.6.cmml" xref="S2.SS2.p3.2.m2.1.1.6">𝑜</ci><ci id="S2.SS2.p3.2.m2.1.1.7.cmml" xref="S2.SS2.p3.2.m2.1.1.7">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">onehot</annotation></semantics></math> representation for <math id="S2.SS2.p3.3.m3.1" class="ltx_Math" alttext="argmin" display="inline"><semantics id="S2.SS2.p3.3.m3.1a"><mrow id="S2.SS2.p3.3.m3.1.1" xref="S2.SS2.p3.3.m3.1.1.cmml"><mi id="S2.SS2.p3.3.m3.1.1.2" xref="S2.SS2.p3.3.m3.1.1.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.3.m3.1.1.1" xref="S2.SS2.p3.3.m3.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.3.m3.1.1.3" xref="S2.SS2.p3.3.m3.1.1.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.3.m3.1.1.1a" xref="S2.SS2.p3.3.m3.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.3.m3.1.1.4" xref="S2.SS2.p3.3.m3.1.1.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.3.m3.1.1.1b" xref="S2.SS2.p3.3.m3.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.3.m3.1.1.5" xref="S2.SS2.p3.3.m3.1.1.5.cmml">m</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.3.m3.1.1.1c" xref="S2.SS2.p3.3.m3.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.3.m3.1.1.6" xref="S2.SS2.p3.3.m3.1.1.6.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.3.m3.1.1.1d" xref="S2.SS2.p3.3.m3.1.1.1.cmml">​</mo><mi id="S2.SS2.p3.3.m3.1.1.7" xref="S2.SS2.p3.3.m3.1.1.7.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.m3.1b"><apply id="S2.SS2.p3.3.m3.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1"><times id="S2.SS2.p3.3.m3.1.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1.1"></times><ci id="S2.SS2.p3.3.m3.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1.2">𝑎</ci><ci id="S2.SS2.p3.3.m3.1.1.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3">𝑟</ci><ci id="S2.SS2.p3.3.m3.1.1.4.cmml" xref="S2.SS2.p3.3.m3.1.1.4">𝑔</ci><ci id="S2.SS2.p3.3.m3.1.1.5.cmml" xref="S2.SS2.p3.3.m3.1.1.5">𝑚</ci><ci id="S2.SS2.p3.3.m3.1.1.6.cmml" xref="S2.SS2.p3.3.m3.1.1.6">𝑖</ci><ci id="S2.SS2.p3.3.m3.1.1.7.cmml" xref="S2.SS2.p3.3.m3.1.1.7">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.m3.1c">argmin</annotation></semantics></math>, i.e., the nearest centroid is marked as 1 and others as 0, <math id="S2.SS2.p3.4.m4.9" class="ltx_Math" alttext="g^{c}(a^{c})=onehot(\underset{k}{\mathrm{arg\ min}}\left\|a^{c}-P^{c}_{k}\right\|^{2})=(0,...,0,1,0,...,0)" display="inline"><semantics id="S2.SS2.p3.4.m4.9a"><mrow id="S2.SS2.p3.4.m4.9.9" xref="S2.SS2.p3.4.m4.9.9.cmml"><mrow id="S2.SS2.p3.4.m4.8.8.1" xref="S2.SS2.p3.4.m4.8.8.1.cmml"><msup id="S2.SS2.p3.4.m4.8.8.1.3" xref="S2.SS2.p3.4.m4.8.8.1.3.cmml"><mi id="S2.SS2.p3.4.m4.8.8.1.3.2" xref="S2.SS2.p3.4.m4.8.8.1.3.2.cmml">g</mi><mi id="S2.SS2.p3.4.m4.8.8.1.3.3" xref="S2.SS2.p3.4.m4.8.8.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.8.8.1.2" xref="S2.SS2.p3.4.m4.8.8.1.2.cmml">​</mo><mrow id="S2.SS2.p3.4.m4.8.8.1.1.1" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p3.4.m4.8.8.1.1.1.2" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.cmml">(</mo><msup id="S2.SS2.p3.4.m4.8.8.1.1.1.1" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.cmml"><mi id="S2.SS2.p3.4.m4.8.8.1.1.1.1.2" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.2.cmml">a</mi><mi id="S2.SS2.p3.4.m4.8.8.1.1.1.1.3" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S2.SS2.p3.4.m4.8.8.1.1.1.3" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p3.4.m4.9.9.4" xref="S2.SS2.p3.4.m4.9.9.4.cmml">=</mo><mrow id="S2.SS2.p3.4.m4.9.9.2" xref="S2.SS2.p3.4.m4.9.9.2.cmml"><mi id="S2.SS2.p3.4.m4.9.9.2.3" xref="S2.SS2.p3.4.m4.9.9.2.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.2" xref="S2.SS2.p3.4.m4.9.9.2.2.cmml">​</mo><mi id="S2.SS2.p3.4.m4.9.9.2.4" xref="S2.SS2.p3.4.m4.9.9.2.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.2a" xref="S2.SS2.p3.4.m4.9.9.2.2.cmml">​</mo><mi id="S2.SS2.p3.4.m4.9.9.2.5" xref="S2.SS2.p3.4.m4.9.9.2.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.2b" xref="S2.SS2.p3.4.m4.9.9.2.2.cmml">​</mo><mi id="S2.SS2.p3.4.m4.9.9.2.6" xref="S2.SS2.p3.4.m4.9.9.2.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.2c" xref="S2.SS2.p3.4.m4.9.9.2.2.cmml">​</mo><mi id="S2.SS2.p3.4.m4.9.9.2.7" xref="S2.SS2.p3.4.m4.9.9.2.7.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.2d" xref="S2.SS2.p3.4.m4.9.9.2.2.cmml">​</mo><mi id="S2.SS2.p3.4.m4.9.9.2.8" xref="S2.SS2.p3.4.m4.9.9.2.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.2e" xref="S2.SS2.p3.4.m4.9.9.2.2.cmml">​</mo><mrow id="S2.SS2.p3.4.m4.9.9.2.1.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p3.4.m4.9.9.2.1.1.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.cmml">(</mo><mrow id="S2.SS2.p3.4.m4.9.9.2.1.1.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.cmml"><munder accentunder="true" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.cmml"><mrow id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.cmml"><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.2.cmml">arg</mi><mo lspace="0.500em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.1.cmml">​</mo><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.3.cmml">min</mi></mrow><mo id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.1.cmml">𝑘</mo></munder><mo lspace="0em" rspace="0em" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.2.cmml">​</mo><msup id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.cmml"><mrow id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.2.cmml"><mo id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.cmml"><msup id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.cmml"><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.2.cmml">a</mi><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.3.cmml">c</mi></msup><mo id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.1" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.cmml"><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.2" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.3.cmml">k</mi><mi id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.3.cmml">c</mi></msubsup></mrow><mo id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="S2.SS2.p3.4.m4.9.9.2.1.1.3" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p3.4.m4.9.9.5" xref="S2.SS2.p3.4.m4.9.9.5.cmml">=</mo><mrow id="S2.SS2.p3.4.m4.9.9.6.2" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml"><mo stretchy="false" id="S2.SS2.p3.4.m4.9.9.6.2.1" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">(</mo><mn id="S2.SS2.p3.4.m4.1.1" xref="S2.SS2.p3.4.m4.1.1.cmml">0</mn><mo id="S2.SS2.p3.4.m4.9.9.6.2.2" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">,</mo><mi mathvariant="normal" id="S2.SS2.p3.4.m4.2.2" xref="S2.SS2.p3.4.m4.2.2.cmml">…</mi><mo id="S2.SS2.p3.4.m4.9.9.6.2.3" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">,</mo><mn id="S2.SS2.p3.4.m4.3.3" xref="S2.SS2.p3.4.m4.3.3.cmml">0</mn><mo id="S2.SS2.p3.4.m4.9.9.6.2.4" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">,</mo><mn id="S2.SS2.p3.4.m4.4.4" xref="S2.SS2.p3.4.m4.4.4.cmml">1</mn><mo id="S2.SS2.p3.4.m4.9.9.6.2.5" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">,</mo><mn id="S2.SS2.p3.4.m4.5.5" xref="S2.SS2.p3.4.m4.5.5.cmml">0</mn><mo id="S2.SS2.p3.4.m4.9.9.6.2.6" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">,</mo><mi mathvariant="normal" id="S2.SS2.p3.4.m4.6.6" xref="S2.SS2.p3.4.m4.6.6.cmml">…</mi><mo id="S2.SS2.p3.4.m4.9.9.6.2.7" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">,</mo><mn id="S2.SS2.p3.4.m4.7.7" xref="S2.SS2.p3.4.m4.7.7.cmml">0</mn><mo stretchy="false" id="S2.SS2.p3.4.m4.9.9.6.2.8" xref="S2.SS2.p3.4.m4.9.9.6.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.4.m4.9b"><apply id="S2.SS2.p3.4.m4.9.9.cmml" xref="S2.SS2.p3.4.m4.9.9"><and id="S2.SS2.p3.4.m4.9.9a.cmml" xref="S2.SS2.p3.4.m4.9.9"></and><apply id="S2.SS2.p3.4.m4.9.9b.cmml" xref="S2.SS2.p3.4.m4.9.9"><eq id="S2.SS2.p3.4.m4.9.9.4.cmml" xref="S2.SS2.p3.4.m4.9.9.4"></eq><apply id="S2.SS2.p3.4.m4.8.8.1.cmml" xref="S2.SS2.p3.4.m4.8.8.1"><times id="S2.SS2.p3.4.m4.8.8.1.2.cmml" xref="S2.SS2.p3.4.m4.8.8.1.2"></times><apply id="S2.SS2.p3.4.m4.8.8.1.3.cmml" xref="S2.SS2.p3.4.m4.8.8.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.8.8.1.3.1.cmml" xref="S2.SS2.p3.4.m4.8.8.1.3">superscript</csymbol><ci id="S2.SS2.p3.4.m4.8.8.1.3.2.cmml" xref="S2.SS2.p3.4.m4.8.8.1.3.2">𝑔</ci><ci id="S2.SS2.p3.4.m4.8.8.1.3.3.cmml" xref="S2.SS2.p3.4.m4.8.8.1.3.3">𝑐</ci></apply><apply id="S2.SS2.p3.4.m4.8.8.1.1.1.1.cmml" xref="S2.SS2.p3.4.m4.8.8.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.8.8.1.1.1.1.1.cmml" xref="S2.SS2.p3.4.m4.8.8.1.1.1">superscript</csymbol><ci id="S2.SS2.p3.4.m4.8.8.1.1.1.1.2.cmml" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.2">𝑎</ci><ci id="S2.SS2.p3.4.m4.8.8.1.1.1.1.3.cmml" xref="S2.SS2.p3.4.m4.8.8.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S2.SS2.p3.4.m4.9.9.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2"><times id="S2.SS2.p3.4.m4.9.9.2.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.2"></times><ci id="S2.SS2.p3.4.m4.9.9.2.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.3">𝑜</ci><ci id="S2.SS2.p3.4.m4.9.9.2.4.cmml" xref="S2.SS2.p3.4.m4.9.9.2.4">𝑛</ci><ci id="S2.SS2.p3.4.m4.9.9.2.5.cmml" xref="S2.SS2.p3.4.m4.9.9.2.5">𝑒</ci><ci id="S2.SS2.p3.4.m4.9.9.2.6.cmml" xref="S2.SS2.p3.4.m4.9.9.2.6">ℎ</ci><ci id="S2.SS2.p3.4.m4.9.9.2.7.cmml" xref="S2.SS2.p3.4.m4.9.9.2.7">𝑜</ci><ci id="S2.SS2.p3.4.m4.9.9.2.8.cmml" xref="S2.SS2.p3.4.m4.9.9.2.8">𝑡</ci><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1"><times id="S2.SS2.p3.4.m4.9.9.2.1.1.1.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.2"></times><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3"><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.1">𝑘</ci><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2"><times id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.1"></times><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.2">arg</ci><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.3.2.3">min</ci></apply></apply><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1"><minus id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.1"></minus><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.2">𝑎</ci><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.2.3">𝑐</ci></apply><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.1.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.2.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.2.3">𝑐</ci></apply><ci id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.1.1.1.3.3">𝑘</ci></apply></apply></apply><cn type="integer" id="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.3.cmml" xref="S2.SS2.p3.4.m4.9.9.2.1.1.1.1.3">2</cn></apply></apply></apply></apply><apply id="S2.SS2.p3.4.m4.9.9c.cmml" xref="S2.SS2.p3.4.m4.9.9"><eq id="S2.SS2.p3.4.m4.9.9.5.cmml" xref="S2.SS2.p3.4.m4.9.9.5"></eq><share href="#S2.SS2.p3.4.m4.9.9.2.cmml" id="S2.SS2.p3.4.m4.9.9d.cmml" xref="S2.SS2.p3.4.m4.9.9"></share><vector id="S2.SS2.p3.4.m4.9.9.6.1.cmml" xref="S2.SS2.p3.4.m4.9.9.6.2"><cn type="integer" id="S2.SS2.p3.4.m4.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1">0</cn><ci id="S2.SS2.p3.4.m4.2.2.cmml" xref="S2.SS2.p3.4.m4.2.2">…</ci><cn type="integer" id="S2.SS2.p3.4.m4.3.3.cmml" xref="S2.SS2.p3.4.m4.3.3">0</cn><cn type="integer" id="S2.SS2.p3.4.m4.4.4.cmml" xref="S2.SS2.p3.4.m4.4.4">1</cn><cn type="integer" id="S2.SS2.p3.4.m4.5.5.cmml" xref="S2.SS2.p3.4.m4.5.5">0</cn><ci id="S2.SS2.p3.4.m4.6.6.cmml" xref="S2.SS2.p3.4.m4.6.6">…</ci><cn type="integer" id="S2.SS2.p3.4.m4.7.7.cmml" xref="S2.SS2.p3.4.m4.7.7">0</cn></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.4.m4.9c">g^{c}(a^{c})=onehot(\underset{k}{\mathrm{arg\ min}}\left\|a^{c}-P^{c}_{k}\right\|^{2})=(0,...,0,1,0,...,0)</annotation></semantics></math>.</p>
<table id="S2.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left"><span class="ltx_tag ltx_tag_equation ltx_align_left">(4)</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E4.m1.2" class="ltx_Math" alttext="a\cdot b=\sum_{c}{a^{c}}b^{c}\approx\sum_{c}{g^{c}(a^{c})}\cdot h^{c}(b^{c})" display="block"><semantics id="S2.E4.m1.2a"><mrow id="S2.E4.m1.2.2" xref="S2.E4.m1.2.2.cmml"><mrow id="S2.E4.m1.2.2.4" xref="S2.E4.m1.2.2.4.cmml"><mi id="S2.E4.m1.2.2.4.2" xref="S2.E4.m1.2.2.4.2.cmml">a</mi><mo lspace="0.222em" rspace="0.222em" id="S2.E4.m1.2.2.4.1" xref="S2.E4.m1.2.2.4.1.cmml">⋅</mo><mi id="S2.E4.m1.2.2.4.3" xref="S2.E4.m1.2.2.4.3.cmml">b</mi></mrow><mo rspace="0.111em" id="S2.E4.m1.2.2.5" xref="S2.E4.m1.2.2.5.cmml">=</mo><mrow id="S2.E4.m1.2.2.6" xref="S2.E4.m1.2.2.6.cmml"><munder id="S2.E4.m1.2.2.6.1" xref="S2.E4.m1.2.2.6.1.cmml"><mo movablelimits="false" id="S2.E4.m1.2.2.6.1.2" xref="S2.E4.m1.2.2.6.1.2.cmml">∑</mo><mi id="S2.E4.m1.2.2.6.1.3" xref="S2.E4.m1.2.2.6.1.3.cmml">c</mi></munder><mrow id="S2.E4.m1.2.2.6.2" xref="S2.E4.m1.2.2.6.2.cmml"><msup id="S2.E4.m1.2.2.6.2.2" xref="S2.E4.m1.2.2.6.2.2.cmml"><mi id="S2.E4.m1.2.2.6.2.2.2" xref="S2.E4.m1.2.2.6.2.2.2.cmml">a</mi><mi id="S2.E4.m1.2.2.6.2.2.3" xref="S2.E4.m1.2.2.6.2.2.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S2.E4.m1.2.2.6.2.1" xref="S2.E4.m1.2.2.6.2.1.cmml">​</mo><msup id="S2.E4.m1.2.2.6.2.3" xref="S2.E4.m1.2.2.6.2.3.cmml"><mi id="S2.E4.m1.2.2.6.2.3.2" xref="S2.E4.m1.2.2.6.2.3.2.cmml">b</mi><mi id="S2.E4.m1.2.2.6.2.3.3" xref="S2.E4.m1.2.2.6.2.3.3.cmml">c</mi></msup></mrow></mrow><mo rspace="0.111em" id="S2.E4.m1.2.2.7" xref="S2.E4.m1.2.2.7.cmml">≈</mo><mrow id="S2.E4.m1.2.2.2" xref="S2.E4.m1.2.2.2.cmml"><munder id="S2.E4.m1.2.2.2.3" xref="S2.E4.m1.2.2.2.3.cmml"><mo movablelimits="false" id="S2.E4.m1.2.2.2.3.2" xref="S2.E4.m1.2.2.2.3.2.cmml">∑</mo><mi id="S2.E4.m1.2.2.2.3.3" xref="S2.E4.m1.2.2.2.3.3.cmml">c</mi></munder><mrow id="S2.E4.m1.2.2.2.2" xref="S2.E4.m1.2.2.2.2.cmml"><mrow id="S2.E4.m1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.cmml"><mrow id="S2.E4.m1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.cmml"><msup id="S2.E4.m1.1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.3.2" xref="S2.E4.m1.1.1.1.1.1.1.3.2.cmml">g</mi><mi id="S2.E4.m1.1.1.1.1.1.1.3.3" xref="S2.E4.m1.1.1.1.1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S2.E4.m1.1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E4.m1.1.1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E4.m1.1.1.1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msup id="S2.E4.m1.1.1.1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.2.cmml">a</mi><mi id="S2.E4.m1.1.1.1.1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo rspace="0.055em" stretchy="false" id="S2.E4.m1.1.1.1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S2.E4.m1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.2.cmml">⋅</mo><msup id="S2.E4.m1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m1.1.1.1.1.1.3.2" xref="S2.E4.m1.1.1.1.1.1.3.2.cmml">h</mi><mi id="S2.E4.m1.1.1.1.1.1.3.3" xref="S2.E4.m1.1.1.1.1.1.3.3.cmml">c</mi></msup></mrow><mo lspace="0em" rspace="0em" id="S2.E4.m1.2.2.2.2.3" xref="S2.E4.m1.2.2.2.2.3.cmml">​</mo><mrow id="S2.E4.m1.2.2.2.2.2.1" xref="S2.E4.m1.2.2.2.2.2.1.1.cmml"><mo stretchy="false" id="S2.E4.m1.2.2.2.2.2.1.2" xref="S2.E4.m1.2.2.2.2.2.1.1.cmml">(</mo><msup id="S2.E4.m1.2.2.2.2.2.1.1" xref="S2.E4.m1.2.2.2.2.2.1.1.cmml"><mi id="S2.E4.m1.2.2.2.2.2.1.1.2" xref="S2.E4.m1.2.2.2.2.2.1.1.2.cmml">b</mi><mi id="S2.E4.m1.2.2.2.2.2.1.1.3" xref="S2.E4.m1.2.2.2.2.2.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S2.E4.m1.2.2.2.2.2.1.3" xref="S2.E4.m1.2.2.2.2.2.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E4.m1.2b"><apply id="S2.E4.m1.2.2.cmml" xref="S2.E4.m1.2.2"><and id="S2.E4.m1.2.2a.cmml" xref="S2.E4.m1.2.2"></and><apply id="S2.E4.m1.2.2b.cmml" xref="S2.E4.m1.2.2"><eq id="S2.E4.m1.2.2.5.cmml" xref="S2.E4.m1.2.2.5"></eq><apply id="S2.E4.m1.2.2.4.cmml" xref="S2.E4.m1.2.2.4"><ci id="S2.E4.m1.2.2.4.1.cmml" xref="S2.E4.m1.2.2.4.1">⋅</ci><ci id="S2.E4.m1.2.2.4.2.cmml" xref="S2.E4.m1.2.2.4.2">𝑎</ci><ci id="S2.E4.m1.2.2.4.3.cmml" xref="S2.E4.m1.2.2.4.3">𝑏</ci></apply><apply id="S2.E4.m1.2.2.6.cmml" xref="S2.E4.m1.2.2.6"><apply id="S2.E4.m1.2.2.6.1.cmml" xref="S2.E4.m1.2.2.6.1"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.6.1.1.cmml" xref="S2.E4.m1.2.2.6.1">subscript</csymbol><sum id="S2.E4.m1.2.2.6.1.2.cmml" xref="S2.E4.m1.2.2.6.1.2"></sum><ci id="S2.E4.m1.2.2.6.1.3.cmml" xref="S2.E4.m1.2.2.6.1.3">𝑐</ci></apply><apply id="S2.E4.m1.2.2.6.2.cmml" xref="S2.E4.m1.2.2.6.2"><times id="S2.E4.m1.2.2.6.2.1.cmml" xref="S2.E4.m1.2.2.6.2.1"></times><apply id="S2.E4.m1.2.2.6.2.2.cmml" xref="S2.E4.m1.2.2.6.2.2"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.6.2.2.1.cmml" xref="S2.E4.m1.2.2.6.2.2">superscript</csymbol><ci id="S2.E4.m1.2.2.6.2.2.2.cmml" xref="S2.E4.m1.2.2.6.2.2.2">𝑎</ci><ci id="S2.E4.m1.2.2.6.2.2.3.cmml" xref="S2.E4.m1.2.2.6.2.2.3">𝑐</ci></apply><apply id="S2.E4.m1.2.2.6.2.3.cmml" xref="S2.E4.m1.2.2.6.2.3"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.6.2.3.1.cmml" xref="S2.E4.m1.2.2.6.2.3">superscript</csymbol><ci id="S2.E4.m1.2.2.6.2.3.2.cmml" xref="S2.E4.m1.2.2.6.2.3.2">𝑏</ci><ci id="S2.E4.m1.2.2.6.2.3.3.cmml" xref="S2.E4.m1.2.2.6.2.3.3">𝑐</ci></apply></apply></apply></apply><apply id="S2.E4.m1.2.2c.cmml" xref="S2.E4.m1.2.2"><approx id="S2.E4.m1.2.2.7.cmml" xref="S2.E4.m1.2.2.7"></approx><share href="#S2.E4.m1.2.2.6.cmml" id="S2.E4.m1.2.2d.cmml" xref="S2.E4.m1.2.2"></share><apply id="S2.E4.m1.2.2.2.cmml" xref="S2.E4.m1.2.2.2"><apply id="S2.E4.m1.2.2.2.3.cmml" xref="S2.E4.m1.2.2.2.3"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.2.3.1.cmml" xref="S2.E4.m1.2.2.2.3">subscript</csymbol><sum id="S2.E4.m1.2.2.2.3.2.cmml" xref="S2.E4.m1.2.2.2.3.2"></sum><ci id="S2.E4.m1.2.2.2.3.3.cmml" xref="S2.E4.m1.2.2.2.3.3">𝑐</ci></apply><apply id="S2.E4.m1.2.2.2.2.cmml" xref="S2.E4.m1.2.2.2.2"><times id="S2.E4.m1.2.2.2.2.3.cmml" xref="S2.E4.m1.2.2.2.2.3"></times><apply id="S2.E4.m1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1"><ci id="S2.E4.m1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.1.1.1.1.1.2">⋅</ci><apply id="S2.E4.m1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1"><times id="S2.E4.m1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.2"></times><apply id="S2.E4.m1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E4.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.2">𝑔</ci><ci id="S2.E4.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.3">𝑐</ci></apply><apply id="S2.E4.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S2.E4.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.2">𝑎</ci><ci id="S2.E4.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S2.E4.m1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E4.m1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m1.1.1.1.1.1.3.2">ℎ</ci><ci id="S2.E4.m1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m1.1.1.1.1.1.3.3">𝑐</ci></apply></apply><apply id="S2.E4.m1.2.2.2.2.2.1.1.cmml" xref="S2.E4.m1.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.2.2.2.1.1.1.cmml" xref="S2.E4.m1.2.2.2.2.2.1">superscript</csymbol><ci id="S2.E4.m1.2.2.2.2.2.1.1.2.cmml" xref="S2.E4.m1.2.2.2.2.2.1.1.2">𝑏</ci><ci id="S2.E4.m1.2.2.2.2.2.1.1.3.cmml" xref="S2.E4.m1.2.2.2.2.2.1.1.3">𝑐</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m1.2c">a\cdot b=\sum_{c}{a^{c}}b^{c}\approx\sum_{c}{g^{c}(a^{c})}\cdot h^{c}(b^{c})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3. </span>Motivation: Poor results of PQ for DNN</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Since DNN models are composed of MM, a direct thought is that can we replace MM in a DNN model by the PQ-based AMM. However, results show that directly applying it to a DNN model leads to poor accuracy.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">Fig. <a href="#S2.F3" title="Figure 3 ‣ 2.3. Motivation: Poor results of PQ for DNN ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the accuracy of using PQ-based AMM for ResNet-20 on CIFAR-10, as well as the Mean Square Error (MSE) of the replaced model and the original model. We replace the MM from the last to the first layer by PQ-based AMM. Fig. <a href="#S2.F3.sf1" title="In Figure 3 ‣ 2.3. Motivation: Poor results of PQ for DNN ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3a</span></a> uses vanilla PQ with <math id="S2.SS3.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS3.p2.1.m1.1a"><mi id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><ci id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">k</annotation></semantics></math>-means for encoding, while Fig. <a href="#S2.F3.sf2" title="In Figure 3 ‣ 2.3. Motivation: Poor results of PQ for DNN ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3b</span></a> uses <span id="S2.SS3.p2.1.1" class="ltx_text ltx_font_smallcaps">maddness</span> with hashing for encoding.</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.1" class="ltx_p">Results show the accuracy keeps dropping while more layers are replaced by AMM, because the error of the AMMs is accumulated. As expected, vanilla PQ shows better results than <span id="S2.SS3.p3.1.1" class="ltx_text ltx_font_smallcaps">maddness</span>, since <math id="S2.SS3.p3.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS3.p3.1.m1.1a"><mi id="S2.SS3.p3.1.m1.1.1" xref="S2.SS3.p3.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.1.m1.1b"><ci id="S2.SS3.p3.1.m1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.1.m1.1c">k</annotation></semantics></math>-means introduces smaller quantization error than hashing. <span id="S2.SS3.p3.1.2" class="ltx_text ltx_font_smallcaps">maddness</span> can maintain accuracy when only the last layer is replaced.
This is consistent with the <span id="S2.SS3.p3.1.3" class="ltx_text ltx_font_smallcaps">maddness</span> paper, which only replaces the last layer, i.e., fully-connect, by AMM. However, if we replace the last two layers, the accuracy sharply drops by 30%, and finally ends in 10% accuracy. Vanilla PQ drops by 30% when the last six layers are replaced, and also ends in 10% accuracy.</p>
</div>
<figure id="S2.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F3.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2302.03213/assets/x3.png" id="S2.F3.sf1.g1" class="ltx_graphics ltx_img_landscape" width="106" height="44" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Vanilla PQ-based AMM</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F3.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2302.03213/assets/x4.png" id="S2.F3.sf2.g1" class="ltx_graphics ltx_img_landscape" width="106" height="44" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span><span id="S2.F3.sf2.2.1" class="ltx_text ltx_font_smallcaps">maddness</span> AMM</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3. </span>
Model accuracy keeps decreasing because MSE keeps increasing, while more MMs (from the last to the first) are replaced by PQ-based AMM, for ResNet20 on CIFAR-10. <span id="S2.F3.2.1" class="ltx_text ltx_font_smallcaps">maddness</span> leads to more accuracy loss because using hashing for encoding.</figcaption>
</figure>
<div id="S2.SS3.p4" class="ltx_para">
<p id="S2.SS3.p4.1" class="ltx_p"><span id="S2.SS3.p4.1.1" class="ltx_text ltx_font_bold">Reason for poor accuracy</span> We expose the key reason for the poor accuracy of PQ-based AMM is that <em id="S2.SS3.p4.1.2" class="ltx_emph ltx_font_italic">the optimization goal of PQ and DNN learning is different</em>. As shown in Eq. <a href="#S2.E1" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, the goal of PQ is to minimize quantization error, i.e., learn the centroids to minimize the distance of each sub-vector and its nearest centroid.
On the other hand, the learning goal of DNN is to minimize the final loss function, through backpropagation to iteratively adjust the model parameters of each layer. Without considering the loss function, when more layers use AMM, the approximation error gets accumulated, as shown in Fig. <a href="#S2.F3" title="Figure 3 ‣ 2.3. Motivation: Poor results of PQ for DNN ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. To get better accuracy, it is necessary to learn the centroids by the DNN training process.</p>
</div>
<div id="S2.SS3.p5" class="ltx_para">
<p id="S2.SS3.p5.1" class="ltx_p">However, the challenge is that PQ centroid learning and encoding functions shown in Eqs. <a href="#S2.E1" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and  <a href="#S2.E2" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> are not differentiable, and cannot use backpropagation to calculate gradient.
This paper thus proposes the soft-PQ technique, which empowers the centroid learning for a DNN by backpropagation and gradient descent.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3. </span>Differentiable Centroid Learning for DNN</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">As exposed above, the key factor to empower table lookup for DNNs is to learn the centroids of each layer through backpropagation to minimize the model loss. However, the main challenge is that the <code id="S3.p1.1.1" class="ltx_verbatim ltx_font_typewriter">argmin</code> function in vanilla PQ, according to Eq. <a href="#S2.E2" title="In 2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, is not differentiable. We therefore propose the differentiable centroid learning for DNN. The learning integrates three methods to adapt three levels of approximation.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">Firstly, we propose soft-PQ method, which leverages the continuous and differentiable approximation to <code id="S3.p2.1.1" class="ltx_verbatim ltx_font_typewriter">argmax</code> function, i.e., <code id="S3.p2.1.2" class="ltx_verbatim ltx_font_typewriter">softmax</code> <cite class="ltx_cite ltx_citemacro_citep">(Jang et al<span class="ltx_text">.</span>, <a href="#bib.bib33" title="" class="ltx_ref">2016</a>)</cite>, for backpropagation. We approximate the <math id="S3.p2.1.m1.1" class="ltx_Math" alttext="onehot" display="inline"><semantics id="S3.p2.1.m1.1a"><mrow id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml"><mi id="S3.p2.1.m1.1.1.2" xref="S3.p2.1.m1.1.1.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.1" xref="S3.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.3" xref="S3.p2.1.m1.1.1.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.1a" xref="S3.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.4" xref="S3.p2.1.m1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.1b" xref="S3.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.5" xref="S3.p2.1.m1.1.1.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.1c" xref="S3.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.6" xref="S3.p2.1.m1.1.1.6.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.1d" xref="S3.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.7" xref="S3.p2.1.m1.1.1.7.cmml">t</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.1b"><apply id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1"><times id="S3.p2.1.m1.1.1.1.cmml" xref="S3.p2.1.m1.1.1.1"></times><ci id="S3.p2.1.m1.1.1.2.cmml" xref="S3.p2.1.m1.1.1.2">𝑜</ci><ci id="S3.p2.1.m1.1.1.3.cmml" xref="S3.p2.1.m1.1.1.3">𝑛</ci><ci id="S3.p2.1.m1.1.1.4.cmml" xref="S3.p2.1.m1.1.1.4">𝑒</ci><ci id="S3.p2.1.m1.1.1.5.cmml" xref="S3.p2.1.m1.1.1.5">ℎ</ci><ci id="S3.p2.1.m1.1.1.6.cmml" xref="S3.p2.1.m1.1.1.6">𝑜</ci><ci id="S3.p2.1.m1.1.1.7.cmml" xref="S3.p2.1.m1.1.1.7">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.1c">onehot</annotation></semantics></math> centroid result as the weighted sum of all centroid results (Sec. <a href="#S3.SS1" title="3.1. Backpropagation through soft-PQ ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>).
Prior work has used <code id="S3.p2.1.3" class="ltx_verbatim ltx_font_typewriter">softmax</code> for differentiable PQ, but only for input embedding compression<cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib9" title="" class="ltx_ref">2020a</a>)</cite>. Our work is the first to enable soft-PQ for end-to-end DNNs.</p>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">Secondly, the use of <code id="S3.p3.1.1" class="ltx_verbatim ltx_font_typewriter">softmax</code> introduces another level of approximation to using <code id="S3.p3.1.2" class="ltx_verbatim ltx_font_typewriter">argmin</code>, which could also lead to reduced model accuracy. We propose a learnable temperature method to address this challenge to adjust the approximation error of <code id="S3.p3.1.3" class="ltx_verbatim ltx_font_typewriter">softmax</code> with <code id="S3.p3.1.4" class="ltx_verbatim ltx_font_typewriter">argmin</code> for each layer.
The temperature of each layer’s <code id="S3.p3.1.5" class="ltx_verbatim ltx_font_typewriter">softmax</code> can be learned through backpropagation (Sec. <a href="#S3.SS2" title="3.2. Learned temperature ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>).</p>
</div>
<div id="S3.p4" class="ltx_para">
<p id="S3.p4.1" class="ltx_p">Thirdly, to reduce memory cost of the lookup tables, LUT-NN applies scalar quantization on the lookup tables. However, this also introduces a level of approximation. Similarly, we propose quantization-aware training to adapt this approximation during centroid learning (Sec. <a href="#S3.SS3" title="3.3. Scalar quantized lookup table ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>).</p>
</div>
<figure id="S3.F4" class="ltx_figure"><img src="/html/2302.03213/assets/x5.png" id="S3.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="346" height="314" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4. </span>

Soft PQ learns centroids. Forward pass uses <math id="S3.F4.6.m1.1" class="ltx_Math" alttext="\mathrm{argmin}" display="inline"><semantics id="S3.F4.6.m1.1b"><mi id="S3.F4.6.m1.1.1" xref="S3.F4.6.m1.1.1.cmml">argmin</mi><annotation-xml encoding="MathML-Content" id="S3.F4.6.m1.1c"><ci id="S3.F4.6.m1.1.1.cmml" xref="S3.F4.6.m1.1.1">argmin</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.6.m1.1d">\mathrm{argmin}</annotation></semantics></math> as the encoding function, and backward uses <math id="S3.F4.7.m2.1" class="ltx_Math" alttext="\mathrm{softmax}" display="inline"><semantics id="S3.F4.7.m2.1b"><mi id="S3.F4.7.m2.1.1" xref="S3.F4.7.m2.1.1.cmml">softmax</mi><annotation-xml encoding="MathML-Content" id="S3.F4.7.m2.1c"><ci id="S3.F4.7.m2.1.1.cmml" xref="S3.F4.7.m2.1.1">softmax</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.7.m2.1d">\mathrm{softmax}</annotation></semantics></math>. Each color marks a sub-vector dataset using one codebook. <math id="S3.F4.8.m3.1" class="ltx_Math" alttext="P^{c}_{k}" display="inline"><semantics id="S3.F4.8.m3.1b"><msubsup id="S3.F4.8.m3.1.1" xref="S3.F4.8.m3.1.1.cmml"><mi id="S3.F4.8.m3.1.1.2.2" xref="S3.F4.8.m3.1.1.2.2.cmml">P</mi><mi id="S3.F4.8.m3.1.1.3" xref="S3.F4.8.m3.1.1.3.cmml">k</mi><mi id="S3.F4.8.m3.1.1.2.3" xref="S3.F4.8.m3.1.1.2.3.cmml">c</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.F4.8.m3.1c"><apply id="S3.F4.8.m3.1.1.cmml" xref="S3.F4.8.m3.1.1"><csymbol cd="ambiguous" id="S3.F4.8.m3.1.1.1.cmml" xref="S3.F4.8.m3.1.1">subscript</csymbol><apply id="S3.F4.8.m3.1.1.2.cmml" xref="S3.F4.8.m3.1.1"><csymbol cd="ambiguous" id="S3.F4.8.m3.1.1.2.1.cmml" xref="S3.F4.8.m3.1.1">superscript</csymbol><ci id="S3.F4.8.m3.1.1.2.2.cmml" xref="S3.F4.8.m3.1.1.2.2">𝑃</ci><ci id="S3.F4.8.m3.1.1.2.3.cmml" xref="S3.F4.8.m3.1.1.2.3">𝑐</ci></apply><ci id="S3.F4.8.m3.1.1.3.cmml" xref="S3.F4.8.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.8.m3.1d">P^{c}_{k}</annotation></semantics></math> is the <math id="S3.F4.9.m4.1" class="ltx_Math" alttext="k^{th}" display="inline"><semantics id="S3.F4.9.m4.1b"><msup id="S3.F4.9.m4.1.1" xref="S3.F4.9.m4.1.1.cmml"><mi id="S3.F4.9.m4.1.1.2" xref="S3.F4.9.m4.1.1.2.cmml">k</mi><mrow id="S3.F4.9.m4.1.1.3" xref="S3.F4.9.m4.1.1.3.cmml"><mi id="S3.F4.9.m4.1.1.3.2" xref="S3.F4.9.m4.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.F4.9.m4.1.1.3.1" xref="S3.F4.9.m4.1.1.3.1.cmml">​</mo><mi id="S3.F4.9.m4.1.1.3.3" xref="S3.F4.9.m4.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.F4.9.m4.1c"><apply id="S3.F4.9.m4.1.1.cmml" xref="S3.F4.9.m4.1.1"><csymbol cd="ambiguous" id="S3.F4.9.m4.1.1.1.cmml" xref="S3.F4.9.m4.1.1">superscript</csymbol><ci id="S3.F4.9.m4.1.1.2.cmml" xref="S3.F4.9.m4.1.1.2">𝑘</ci><apply id="S3.F4.9.m4.1.1.3.cmml" xref="S3.F4.9.m4.1.1.3"><times id="S3.F4.9.m4.1.1.3.1.cmml" xref="S3.F4.9.m4.1.1.3.1"></times><ci id="S3.F4.9.m4.1.1.3.2.cmml" xref="S3.F4.9.m4.1.1.3.2">𝑡</ci><ci id="S3.F4.9.m4.1.1.3.3.cmml" xref="S3.F4.9.m4.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.9.m4.1d">k^{th}</annotation></semantics></math> centroid for sub-vector <math id="S3.F4.10.m5.1" class="ltx_Math" alttext="a^{c}" display="inline"><semantics id="S3.F4.10.m5.1b"><msup id="S3.F4.10.m5.1.1" xref="S3.F4.10.m5.1.1.cmml"><mi id="S3.F4.10.m5.1.1.2" xref="S3.F4.10.m5.1.1.2.cmml">a</mi><mi id="S3.F4.10.m5.1.1.3" xref="S3.F4.10.m5.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S3.F4.10.m5.1c"><apply id="S3.F4.10.m5.1.1.cmml" xref="S3.F4.10.m5.1.1"><csymbol cd="ambiguous" id="S3.F4.10.m5.1.1.1.cmml" xref="S3.F4.10.m5.1.1">superscript</csymbol><ci id="S3.F4.10.m5.1.1.2.cmml" xref="S3.F4.10.m5.1.1.2">𝑎</ci><ci id="S3.F4.10.m5.1.1.3.cmml" xref="S3.F4.10.m5.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.10.m5.1d">a^{c}</annotation></semantics></math>.
</figcaption>
</figure>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1. </span>Backpropagation through soft-PQ</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.4" class="ltx_p">As discussed in Sec.<a href="#S2.SS1" title="2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>, vanilla PQ employs <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">k</annotation></semantics></math>-means to learn centroids from the dataset, and the encoding function <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="g^{c}(a^{c})" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mrow id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><msup id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.3.2.cmml">g</mi><mi id="S3.SS1.p1.2.m2.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p1.2.m2.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.2.cmml">a</mi><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><times id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2"></times><apply id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.3">superscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.3.2">𝑔</ci><ci id="S3.SS1.p1.2.m2.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3.3">𝑐</ci></apply><apply id="S3.SS1.p1.2.m2.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.2">𝑎</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">g^{c}(a^{c})</annotation></semantics></math> utilizes <code id="S3.SS1.p1.4.1" class="ltx_verbatim ltx_font_typewriter">argmin</code> to encode a sub-vector as the nearest centroid, represented by a <code id="S3.SS1.p1.4.2" class="ltx_verbatim ltx_font_typewriter">onehot</code> vector <math id="S3.SS1.p1.3.m3.5" class="ltx_Math" alttext="(0,...,1,...,0)" display="inline"><semantics id="S3.SS1.p1.3.m3.5a"><mrow id="S3.SS1.p1.3.m3.5.6.2" xref="S3.SS1.p1.3.m3.5.6.1.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.5.6.2.1" xref="S3.SS1.p1.3.m3.5.6.1.cmml">(</mo><mn id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">0</mn><mo id="S3.SS1.p1.3.m3.5.6.2.2" xref="S3.SS1.p1.3.m3.5.6.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.3.m3.2.2" xref="S3.SS1.p1.3.m3.2.2.cmml">…</mi><mo id="S3.SS1.p1.3.m3.5.6.2.3" xref="S3.SS1.p1.3.m3.5.6.1.cmml">,</mo><mn id="S3.SS1.p1.3.m3.3.3" xref="S3.SS1.p1.3.m3.3.3.cmml">1</mn><mo id="S3.SS1.p1.3.m3.5.6.2.4" xref="S3.SS1.p1.3.m3.5.6.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.3.m3.4.4" xref="S3.SS1.p1.3.m3.4.4.cmml">…</mi><mo id="S3.SS1.p1.3.m3.5.6.2.5" xref="S3.SS1.p1.3.m3.5.6.1.cmml">,</mo><mn id="S3.SS1.p1.3.m3.5.5" xref="S3.SS1.p1.3.m3.5.5.cmml">0</mn><mo stretchy="false" id="S3.SS1.p1.3.m3.5.6.2.6" xref="S3.SS1.p1.3.m3.5.6.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.5b"><vector id="S3.SS1.p1.3.m3.5.6.1.cmml" xref="S3.SS1.p1.3.m3.5.6.2"><cn type="integer" id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">0</cn><ci id="S3.SS1.p1.3.m3.2.2.cmml" xref="S3.SS1.p1.3.m3.2.2">…</ci><cn type="integer" id="S3.SS1.p1.3.m3.3.3.cmml" xref="S3.SS1.p1.3.m3.3.3">1</cn><ci id="S3.SS1.p1.3.m3.4.4.cmml" xref="S3.SS1.p1.3.m3.4.4">…</ci><cn type="integer" id="S3.SS1.p1.3.m3.5.5.cmml" xref="S3.SS1.p1.3.m3.5.5">0</cn></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.5c">(0,...,1,...,0)</annotation></semantics></math> as shown in Fig.<a href="#S3.F4" title="Figure 4 ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. The sub-vector AMM result can be read directly from the lookup table by <math id="S3.SS1.p1.4.m4.2" class="ltx_Math" alttext="g^{c}(a^{c})\cdot h^{c}(a^{c})" display="inline"><semantics id="S3.SS1.p1.4.m4.2a"><mrow id="S3.SS1.p1.4.m4.2.2" xref="S3.SS1.p1.4.m4.2.2.cmml"><mrow id="S3.SS1.p1.4.m4.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.cmml"><mrow id="S3.SS1.p1.4.m4.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.cmml"><msup id="S3.SS1.p1.4.m4.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.3.2" xref="S3.SS1.p1.4.m4.1.1.1.1.3.2.cmml">g</mi><mi id="S3.SS1.p1.4.m4.1.1.1.1.3.3" xref="S3.SS1.p1.4.m4.1.1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.4.m4.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.2.cmml">a</mi><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo rspace="0.055em" stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S3.SS1.p1.4.m4.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.2.cmml">⋅</mo><msup id="S3.SS1.p1.4.m4.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.3.2" xref="S3.SS1.p1.4.m4.1.1.1.3.2.cmml">h</mi><mi id="S3.SS1.p1.4.m4.1.1.1.3.3" xref="S3.SS1.p1.4.m4.1.1.1.3.3.cmml">c</mi></msup></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.2.2.3" xref="S3.SS1.p1.4.m4.2.2.3.cmml">​</mo><mrow id="S3.SS1.p1.4.m4.2.2.2.1" xref="S3.SS1.p1.4.m4.2.2.2.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.2.2.2.1.2" xref="S3.SS1.p1.4.m4.2.2.2.1.1.cmml">(</mo><msup id="S3.SS1.p1.4.m4.2.2.2.1.1" xref="S3.SS1.p1.4.m4.2.2.2.1.1.cmml"><mi id="S3.SS1.p1.4.m4.2.2.2.1.1.2" xref="S3.SS1.p1.4.m4.2.2.2.1.1.2.cmml">a</mi><mi id="S3.SS1.p1.4.m4.2.2.2.1.1.3" xref="S3.SS1.p1.4.m4.2.2.2.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS1.p1.4.m4.2.2.2.1.3" xref="S3.SS1.p1.4.m4.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.2b"><apply id="S3.SS1.p1.4.m4.2.2.cmml" xref="S3.SS1.p1.4.m4.2.2"><times id="S3.SS1.p1.4.m4.2.2.3.cmml" xref="S3.SS1.p1.4.m4.2.2.3"></times><apply id="S3.SS1.p1.4.m4.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1"><ci id="S3.SS1.p1.4.m4.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.2">⋅</ci><apply id="S3.SS1.p1.4.m4.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1"><times id="S3.SS1.p1.4.m4.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.2"></times><apply id="S3.SS1.p1.4.m4.1.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.3.2">𝑔</ci><ci id="S3.SS1.p1.4.m4.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.3.3">𝑐</ci></apply><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.2">𝑎</ci><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S3.SS1.p1.4.m4.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.3.2">ℎ</ci><ci id="S3.SS1.p1.4.m4.1.1.1.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.3.3">𝑐</ci></apply></apply><apply id="S3.SS1.p1.4.m4.2.2.2.1.1.cmml" xref="S3.SS1.p1.4.m4.2.2.2.1"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.2.2.2.1.1.1.cmml" xref="S3.SS1.p1.4.m4.2.2.2.1">superscript</csymbol><ci id="S3.SS1.p1.4.m4.2.2.2.1.1.2.cmml" xref="S3.SS1.p1.4.m4.2.2.2.1.1.2">𝑎</ci><ci id="S3.SS1.p1.4.m4.2.2.2.1.1.3.cmml" xref="S3.SS1.p1.4.m4.2.2.2.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.2c">g^{c}(a^{c})\cdot h^{c}(a^{c})</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">However, to apply PQ to the entire DNN model and minimize the model loss, the centroids for each layer must be learned from backpropagation and gradient descent. We utilize the smooth <code id="S3.SS1.p2.1.1" class="ltx_verbatim ltx_font_typewriter">argmax</code> function, i.e., <code id="S3.SS1.p2.1.2" class="ltx_verbatim ltx_font_typewriter">softmax</code>, as the encoding function for backpropagation, as shown in Eq.<a href="#S3.E5" title="In 3.1. Backpropagation through soft-PQ ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>. Here, <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">t</annotation></semantics></math> represents the <code id="S3.SS1.p2.1.3" class="ltx_verbatim ltx_font_typewriter">temperature</code> hyperparameter, which will be further discussed in Sec. <a href="#S3.SS2" title="3.2. Learned temperature ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left"><span class="ltx_tag ltx_tag_equation ltx_align_left">(5)</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.2" class="ltx_Math" alttext="\tilde{g}^{c}(a^{c})=\mathrm{softmax}(-\left\|a^{c}-P^{c}_{k}\right\|^{2}/\ t)" display="block"><semantics id="S3.E5.m1.2a"><mrow id="S3.E5.m1.2.2" xref="S3.E5.m1.2.2.cmml"><mrow id="S3.E5.m1.1.1.1" xref="S3.E5.m1.1.1.1.cmml"><msup id="S3.E5.m1.1.1.1.3" xref="S3.E5.m1.1.1.1.3.cmml"><mover accent="true" id="S3.E5.m1.1.1.1.3.2" xref="S3.E5.m1.1.1.1.3.2.cmml"><mi id="S3.E5.m1.1.1.1.3.2.2" xref="S3.E5.m1.1.1.1.3.2.2.cmml">g</mi><mo id="S3.E5.m1.1.1.1.3.2.1" xref="S3.E5.m1.1.1.1.3.2.1.cmml">~</mo></mover><mi id="S3.E5.m1.1.1.1.3.3" xref="S3.E5.m1.1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.E5.m1.1.1.1.2" xref="S3.E5.m1.1.1.1.2.cmml">​</mo><mrow id="S3.E5.m1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E5.m1.1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.1.1.cmml">(</mo><msup id="S3.E5.m1.1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.cmml"><mi id="S3.E5.m1.1.1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.1.1.2.cmml">a</mi><mi id="S3.E5.m1.1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.E5.m1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E5.m1.2.2.3" xref="S3.E5.m1.2.2.3.cmml">=</mo><mrow id="S3.E5.m1.2.2.2" xref="S3.E5.m1.2.2.2.cmml"><mi id="S3.E5.m1.2.2.2.3" xref="S3.E5.m1.2.2.2.3.cmml">softmax</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.2.2.2.2" xref="S3.E5.m1.2.2.2.2.cmml">​</mo><mrow id="S3.E5.m1.2.2.2.1.1" xref="S3.E5.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E5.m1.2.2.2.1.1.2" xref="S3.E5.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="S3.E5.m1.2.2.2.1.1.1" xref="S3.E5.m1.2.2.2.1.1.1.cmml"><mo id="S3.E5.m1.2.2.2.1.1.1a" xref="S3.E5.m1.2.2.2.1.1.1.cmml">−</mo><mrow id="S3.E5.m1.2.2.2.1.1.1.1" xref="S3.E5.m1.2.2.2.1.1.1.1.cmml"><msup id="S3.E5.m1.2.2.2.1.1.1.1.1" xref="S3.E5.m1.2.2.2.1.1.1.1.1.cmml"><mrow id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.2.cmml"><mo id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.2" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.cmml"><msup id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.2" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.2.cmml">a</mi><mi id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.3" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.3.cmml">c</mi></msup><mo id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.3" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.3.cmml">k</mi><mi id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3.cmml">c</mi></msubsup></mrow><mo id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.3" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S3.E5.m1.2.2.2.1.1.1.1.1.3" xref="S3.E5.m1.2.2.2.1.1.1.1.1.3.cmml">2</mn></msup><mo rspace="0.722em" id="S3.E5.m1.2.2.2.1.1.1.1.2" xref="S3.E5.m1.2.2.2.1.1.1.1.2.cmml">/</mo><mi id="S3.E5.m1.2.2.2.1.1.1.1.3" xref="S3.E5.m1.2.2.2.1.1.1.1.3.cmml">t</mi></mrow></mrow><mo stretchy="false" id="S3.E5.m1.2.2.2.1.1.3" xref="S3.E5.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.2b"><apply id="S3.E5.m1.2.2.cmml" xref="S3.E5.m1.2.2"><eq id="S3.E5.m1.2.2.3.cmml" xref="S3.E5.m1.2.2.3"></eq><apply id="S3.E5.m1.1.1.1.cmml" xref="S3.E5.m1.1.1.1"><times id="S3.E5.m1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.2"></times><apply id="S3.E5.m1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.1.3.1.cmml" xref="S3.E5.m1.1.1.1.3">superscript</csymbol><apply id="S3.E5.m1.1.1.1.3.2.cmml" xref="S3.E5.m1.1.1.1.3.2"><ci id="S3.E5.m1.1.1.1.3.2.1.cmml" xref="S3.E5.m1.1.1.1.3.2.1">~</ci><ci id="S3.E5.m1.1.1.1.3.2.2.cmml" xref="S3.E5.m1.1.1.1.3.2.2">𝑔</ci></apply><ci id="S3.E5.m1.1.1.1.3.3.cmml" xref="S3.E5.m1.1.1.1.3.3">𝑐</ci></apply><apply id="S3.E5.m1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1">superscript</csymbol><ci id="S3.E5.m1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.1.1.2">𝑎</ci><ci id="S3.E5.m1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S3.E5.m1.2.2.2.cmml" xref="S3.E5.m1.2.2.2"><times id="S3.E5.m1.2.2.2.2.cmml" xref="S3.E5.m1.2.2.2.2"></times><ci id="S3.E5.m1.2.2.2.3.cmml" xref="S3.E5.m1.2.2.2.3">softmax</ci><apply id="S3.E5.m1.2.2.2.1.1.1.cmml" xref="S3.E5.m1.2.2.2.1.1"><minus id="S3.E5.m1.2.2.2.1.1.1.2.cmml" xref="S3.E5.m1.2.2.2.1.1"></minus><apply id="S3.E5.m1.2.2.2.1.1.1.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1"><divide id="S3.E5.m1.2.2.2.1.1.1.1.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.2"></divide><apply id="S3.E5.m1.2.2.2.1.1.1.1.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.2.1.1.1.1.1.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1">superscript</csymbol><apply id="S3.E5.m1.2.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E5.m1.2.2.2.1.1.1.1.1.1.2.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1"><minus id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.2">𝑎</ci><ci id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.2.3">𝑐</ci></apply><apply id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3">𝑐</ci></apply><ci id="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.1.1.1.3.3">𝑘</ci></apply></apply></apply><cn type="integer" id="S3.E5.m1.2.2.2.1.1.1.1.1.3.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.1.3">2</cn></apply><ci id="S3.E5.m1.2.2.2.1.1.1.1.3.cmml" xref="S3.E5.m1.2.2.2.1.1.1.1.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.2c">\tilde{g}^{c}(a^{c})=\mathrm{softmax}(-\left\|a^{c}-P^{c}_{k}\right\|^{2}/\ t)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<p id="S3.SS1.p4.6" class="ltx_p">For a codebook with <math id="S3.SS1.p4.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS1.p4.1.m1.1a"><mi id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.1b"><ci id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.1c">K</annotation></semantics></math> centroids, the <code id="S3.SS1.p4.6.1" class="ltx_verbatim ltx_font_typewriter">softmax</code> function takes a vector of <math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><mi id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><ci id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">K</annotation></semantics></math> distance results between the sub-vector <math id="S3.SS1.p4.3.m3.1" class="ltx_Math" alttext="a^{c}" display="inline"><semantics id="S3.SS1.p4.3.m3.1a"><msup id="S3.SS1.p4.3.m3.1.1" xref="S3.SS1.p4.3.m3.1.1.cmml"><mi id="S3.SS1.p4.3.m3.1.1.2" xref="S3.SS1.p4.3.m3.1.1.2.cmml">a</mi><mi id="S3.SS1.p4.3.m3.1.1.3" xref="S3.SS1.p4.3.m3.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.3.m3.1b"><apply id="S3.SS1.p4.3.m3.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.3.m3.1.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1">superscript</csymbol><ci id="S3.SS1.p4.3.m3.1.1.2.cmml" xref="S3.SS1.p4.3.m3.1.1.2">𝑎</ci><ci id="S3.SS1.p4.3.m3.1.1.3.cmml" xref="S3.SS1.p4.3.m3.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.3.m3.1c">a^{c}</annotation></semantics></math> and each centroid <math id="S3.SS1.p4.4.m4.1" class="ltx_Math" alttext="P^{c}_{k}" display="inline"><semantics id="S3.SS1.p4.4.m4.1a"><msubsup id="S3.SS1.p4.4.m4.1.1" xref="S3.SS1.p4.4.m4.1.1.cmml"><mi id="S3.SS1.p4.4.m4.1.1.2.2" xref="S3.SS1.p4.4.m4.1.1.2.2.cmml">P</mi><mi id="S3.SS1.p4.4.m4.1.1.3" xref="S3.SS1.p4.4.m4.1.1.3.cmml">k</mi><mi id="S3.SS1.p4.4.m4.1.1.2.3" xref="S3.SS1.p4.4.m4.1.1.2.3.cmml">c</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.4.m4.1b"><apply id="S3.SS1.p4.4.m4.1.1.cmml" xref="S3.SS1.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.4.m4.1.1.1.cmml" xref="S3.SS1.p4.4.m4.1.1">subscript</csymbol><apply id="S3.SS1.p4.4.m4.1.1.2.cmml" xref="S3.SS1.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.4.m4.1.1.2.1.cmml" xref="S3.SS1.p4.4.m4.1.1">superscript</csymbol><ci id="S3.SS1.p4.4.m4.1.1.2.2.cmml" xref="S3.SS1.p4.4.m4.1.1.2.2">𝑃</ci><ci id="S3.SS1.p4.4.m4.1.1.2.3.cmml" xref="S3.SS1.p4.4.m4.1.1.2.3">𝑐</ci></apply><ci id="S3.SS1.p4.4.m4.1.1.3.cmml" xref="S3.SS1.p4.4.m4.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.4.m4.1c">P^{c}_{k}</annotation></semantics></math> as input.
It normalizes the input to a probability distribution that adds up to 1. According to the definition of <code id="S3.SS1.p4.6.2" class="ltx_verbatim ltx_font_typewriter">softmax</code>, each probability is proportional to the exponent of the distance, i.e., <math id="S3.SS1.p4.5.m5.1" class="ltx_Math" alttext="exp({-\left\|a^{c}-P^{c}_{k}\right\|^{2}}/{t})" display="inline"><semantics id="S3.SS1.p4.5.m5.1a"><mrow id="S3.SS1.p4.5.m5.1.1" xref="S3.SS1.p4.5.m5.1.1.cmml"><mi id="S3.SS1.p4.5.m5.1.1.3" xref="S3.SS1.p4.5.m5.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.5.m5.1.1.2" xref="S3.SS1.p4.5.m5.1.1.2.cmml">​</mo><mi id="S3.SS1.p4.5.m5.1.1.4" xref="S3.SS1.p4.5.m5.1.1.4.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.5.m5.1.1.2a" xref="S3.SS1.p4.5.m5.1.1.2.cmml">​</mo><mi id="S3.SS1.p4.5.m5.1.1.5" xref="S3.SS1.p4.5.m5.1.1.5.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.5.m5.1.1.2b" xref="S3.SS1.p4.5.m5.1.1.2.cmml">​</mo><mrow id="S3.SS1.p4.5.m5.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.5.m5.1.1.1.1.2" xref="S3.SS1.p4.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p4.5.m5.1.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.cmml"><mo id="S3.SS1.p4.5.m5.1.1.1.1.1a" xref="S3.SS1.p4.5.m5.1.1.1.1.1.cmml">−</mo><mrow id="S3.SS1.p4.5.m5.1.1.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.cmml"><msup id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.cmml"><mrow id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.2.cmml"><mo id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><msup id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2.cmml">a</mi><mi id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3.cmml">c</mi></msup><mo id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.2" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.3.cmml">k</mi><mi id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.3.cmml">c</mi></msubsup></mrow><mo id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.3.cmml">2</mn></msup><mo id="S3.SS1.p4.5.m5.1.1.1.1.1.1.2" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.2.cmml">/</mo><mi id="S3.SS1.p4.5.m5.1.1.1.1.1.1.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.3.cmml">t</mi></mrow></mrow><mo stretchy="false" id="S3.SS1.p4.5.m5.1.1.1.1.3" xref="S3.SS1.p4.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.5.m5.1b"><apply id="S3.SS1.p4.5.m5.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1"><times id="S3.SS1.p4.5.m5.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.2"></times><ci id="S3.SS1.p4.5.m5.1.1.3.cmml" xref="S3.SS1.p4.5.m5.1.1.3">𝑒</ci><ci id="S3.SS1.p4.5.m5.1.1.4.cmml" xref="S3.SS1.p4.5.m5.1.1.4">𝑥</ci><ci id="S3.SS1.p4.5.m5.1.1.5.cmml" xref="S3.SS1.p4.5.m5.1.1.5">𝑝</ci><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1"><minus id="S3.SS1.p4.5.m5.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1"></minus><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1"><divide id="S3.SS1.p4.5.m5.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.2"></divide><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1"><minus id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2">𝑎</ci><ci id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3">𝑐</ci></apply><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.2.3">𝑐</ci></apply><ci id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.3">𝑘</ci></apply></apply></apply><cn type="integer" id="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.1.3">2</cn></apply><ci id="S3.SS1.p4.5.m5.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p4.5.m5.1.1.1.1.1.1.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.5.m5.1c">exp({-\left\|a^{c}-P^{c}_{k}\right\|^{2}}/{t})</annotation></semantics></math>.
The closer the centroid is to the sub-vector, the higher the probability will be.
The encoding for a sub-vector is transformed from a deterministic <code id="S3.SS1.p4.6.3" class="ltx_verbatim ltx_font_typewriter">onehot</code> vector into a probability vector.
For example, as illustrated in Fig. <a href="#S3.F4" title="Figure 4 ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, the output of the <code id="S3.SS1.p4.6.4" class="ltx_verbatim ltx_font_typewriter">softmax</code> encoding function is <math id="S3.SS1.p4.6.m6.4" class="ltx_Math" alttext="(0.12,...,0.64,...)" display="inline"><semantics id="S3.SS1.p4.6.m6.4a"><mrow id="S3.SS1.p4.6.m6.4.5.2" xref="S3.SS1.p4.6.m6.4.5.1.cmml"><mo stretchy="false" id="S3.SS1.p4.6.m6.4.5.2.1" xref="S3.SS1.p4.6.m6.4.5.1.cmml">(</mo><mn id="S3.SS1.p4.6.m6.1.1" xref="S3.SS1.p4.6.m6.1.1.cmml">0.12</mn><mo id="S3.SS1.p4.6.m6.4.5.2.2" xref="S3.SS1.p4.6.m6.4.5.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p4.6.m6.2.2" xref="S3.SS1.p4.6.m6.2.2.cmml">…</mi><mo id="S3.SS1.p4.6.m6.4.5.2.3" xref="S3.SS1.p4.6.m6.4.5.1.cmml">,</mo><mn id="S3.SS1.p4.6.m6.3.3" xref="S3.SS1.p4.6.m6.3.3.cmml">0.64</mn><mo id="S3.SS1.p4.6.m6.4.5.2.4" xref="S3.SS1.p4.6.m6.4.5.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p4.6.m6.4.4" xref="S3.SS1.p4.6.m6.4.4.cmml">…</mi><mo stretchy="false" id="S3.SS1.p4.6.m6.4.5.2.5" xref="S3.SS1.p4.6.m6.4.5.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.6.m6.4b"><vector id="S3.SS1.p4.6.m6.4.5.1.cmml" xref="S3.SS1.p4.6.m6.4.5.2"><cn type="float" id="S3.SS1.p4.6.m6.1.1.cmml" xref="S3.SS1.p4.6.m6.1.1">0.12</cn><ci id="S3.SS1.p4.6.m6.2.2.cmml" xref="S3.SS1.p4.6.m6.2.2">…</ci><cn type="float" id="S3.SS1.p4.6.m6.3.3.cmml" xref="S3.SS1.p4.6.m6.3.3">0.64</cn><ci id="S3.SS1.p4.6.m6.4.4.cmml" xref="S3.SS1.p4.6.m6.4.4">…</ci></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.6.m6.4c">(0.12,...,0.64,...)</annotation></semantics></math>, where 0.64 is the probability of the nearest centroid. The sub-vector AMM result is then obtained as the dot product of the probability vector and the lookup table entries.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para">
<p id="S3.SS1.p5.1" class="ltx_p"><span id="S3.SS1.p5.1.1" class="ltx_text ltx_font_bold">Soft-PQ centroid learning:</span> Using <code id="S3.SS1.p5.1.2" class="ltx_verbatim ltx_font_typewriter">softmax</code>, the centroid learning process of our soft-PQ for the entire model is illustrated in Fig.<a href="#S3.F4" title="Figure 4 ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
During the forward pass, the onehot <code id="S3.SS1.p5.1.3" class="ltx_verbatim ltx_font_typewriter">argmin</code> function is utilized to calculate the model output and loss, as model inference will also use <code id="S3.SS1.p5.1.4" class="ltx_verbatim ltx_font_typewriter">argmin</code> for simplicity.
The backward pass utilizes <code id="S3.SS1.p5.1.5" class="ltx_verbatim ltx_font_typewriter">softmax</code> as the encoding function to calculate gradients, adjust centroids via gradient descent, and rebuild lookup tables with the updated centroids for the next training iteration.
Based on Eq.<a href="#S2.E4" title="In 2.2. PQ for AMM ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, the sub-vector AMM in soft-PQ is formulated as Eq. <a href="#S3.E6" title="In 3.1. Backpropagation through soft-PQ ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_left"><span class="ltx_tag ltx_tag_equation ltx_align_left">(6)</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.3" class="ltx_Math" alttext="a^{c}b^{c}=\tilde{g}^{c}(a^{c})\cdot h^{c}(b^{c})-\mathrm{sg}(\tilde{g}^{c}(a^{c})\cdot h^{c}(b^{c})-g^{c}(a^{c})\cdot h^{c}(b^{c}))" display="block"><semantics id="S3.E6.m1.3a"><mrow id="S3.E6.m1.3.3" xref="S3.E6.m1.3.3.cmml"><mrow id="S3.E6.m1.3.3.5" xref="S3.E6.m1.3.3.5.cmml"><msup id="S3.E6.m1.3.3.5.2" xref="S3.E6.m1.3.3.5.2.cmml"><mi id="S3.E6.m1.3.3.5.2.2" xref="S3.E6.m1.3.3.5.2.2.cmml">a</mi><mi id="S3.E6.m1.3.3.5.2.3" xref="S3.E6.m1.3.3.5.2.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.3.3.5.1" xref="S3.E6.m1.3.3.5.1.cmml">​</mo><msup id="S3.E6.m1.3.3.5.3" xref="S3.E6.m1.3.3.5.3.cmml"><mi id="S3.E6.m1.3.3.5.3.2" xref="S3.E6.m1.3.3.5.3.2.cmml">b</mi><mi id="S3.E6.m1.3.3.5.3.3" xref="S3.E6.m1.3.3.5.3.3.cmml">c</mi></msup></mrow><mo id="S3.E6.m1.3.3.4" xref="S3.E6.m1.3.3.4.cmml">=</mo><mrow id="S3.E6.m1.3.3.3" xref="S3.E6.m1.3.3.3.cmml"><mrow id="S3.E6.m1.2.2.2.2" xref="S3.E6.m1.2.2.2.2.cmml"><mrow id="S3.E6.m1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.cmml"><msup id="S3.E6.m1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E6.m1.1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.3.2.2" xref="S3.E6.m1.1.1.1.1.1.1.3.2.2.cmml">g</mi><mo id="S3.E6.m1.1.1.1.1.1.1.3.2.1" xref="S3.E6.m1.1.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mi id="S3.E6.m1.1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msup id="S3.E6.m1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.cmml">a</mi><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo rspace="0.055em" stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S3.E6.m1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.2.cmml">⋅</mo><msup id="S3.E6.m1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.3.cmml"><mi id="S3.E6.m1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.3.2.cmml">h</mi><mi id="S3.E6.m1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.3.3.cmml">c</mi></msup></mrow><mo lspace="0em" rspace="0em" id="S3.E6.m1.2.2.2.2.3" xref="S3.E6.m1.2.2.2.2.3.cmml">​</mo><mrow id="S3.E6.m1.2.2.2.2.2.1" xref="S3.E6.m1.2.2.2.2.2.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.2.2.2.2.2.1.2" xref="S3.E6.m1.2.2.2.2.2.1.1.cmml">(</mo><msup id="S3.E6.m1.2.2.2.2.2.1.1" xref="S3.E6.m1.2.2.2.2.2.1.1.cmml"><mi id="S3.E6.m1.2.2.2.2.2.1.1.2" xref="S3.E6.m1.2.2.2.2.2.1.1.2.cmml">b</mi><mi id="S3.E6.m1.2.2.2.2.2.1.1.3" xref="S3.E6.m1.2.2.2.2.2.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.E6.m1.2.2.2.2.2.1.3" xref="S3.E6.m1.2.2.2.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E6.m1.3.3.3.4" xref="S3.E6.m1.3.3.3.4.cmml">−</mo><mrow id="S3.E6.m1.3.3.3.3" xref="S3.E6.m1.3.3.3.3.cmml"><mi id="S3.E6.m1.3.3.3.3.3" xref="S3.E6.m1.3.3.3.3.3.cmml">sg</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.3.3.3.3.2" xref="S3.E6.m1.3.3.3.3.2.cmml">​</mo><mrow id="S3.E6.m1.3.3.3.3.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.cmml">(</mo><mrow id="S3.E6.m1.3.3.3.3.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.cmml"><mrow id="S3.E6.m1.3.3.3.3.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.2.cmml"><mrow id="S3.E6.m1.3.3.3.3.1.1.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.cmml"><msup id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.2.cmml">g</mi><mo id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.1" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mi id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.3" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.cmml">(</mo><msup id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.2.cmml">a</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo rspace="0.055em" stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.2.cmml">⋅</mo><msup id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.2" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.2.cmml">h</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.3" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.3.cmml">c</mi></msup></mrow><mo lspace="0em" rspace="0em" id="S3.E6.m1.3.3.3.3.1.1.1.2.3" xref="S3.E6.m1.3.3.3.3.1.1.1.2.3.cmml">​</mo><mrow id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.cmml">(</mo><msup id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.2.cmml">b</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E6.m1.3.3.3.3.1.1.1.5" xref="S3.E6.m1.3.3.3.3.1.1.1.5.cmml">−</mo><mrow id="S3.E6.m1.3.3.3.3.1.1.1.4" xref="S3.E6.m1.3.3.3.3.1.1.1.4.cmml"><mrow id="S3.E6.m1.3.3.3.3.1.1.1.3.1" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.cmml"><mrow id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.cmml"><msup id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.2" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.2.cmml">g</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.3" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.2.cmml">​</mo><mrow id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.cmml">(</mo><msup id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.2.cmml">a</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.3.cmml">c</mi></msup><mo rspace="0.055em" stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.2.cmml">⋅</mo><msup id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.2" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.2.cmml">h</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.3" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.3.cmml">c</mi></msup></mrow><mo lspace="0em" rspace="0em" id="S3.E6.m1.3.3.3.3.1.1.1.4.3" xref="S3.E6.m1.3.3.3.3.1.1.1.4.3.cmml">​</mo><mrow id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.cmml">(</mo><msup id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.cmml"><mi id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.2" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.2.cmml">b</mi><mi id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E6.m1.3.3.3.3.1.1.3" xref="S3.E6.m1.3.3.3.3.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.3b"><apply id="S3.E6.m1.3.3.cmml" xref="S3.E6.m1.3.3"><eq id="S3.E6.m1.3.3.4.cmml" xref="S3.E6.m1.3.3.4"></eq><apply id="S3.E6.m1.3.3.5.cmml" xref="S3.E6.m1.3.3.5"><times id="S3.E6.m1.3.3.5.1.cmml" xref="S3.E6.m1.3.3.5.1"></times><apply id="S3.E6.m1.3.3.5.2.cmml" xref="S3.E6.m1.3.3.5.2"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.5.2.1.cmml" xref="S3.E6.m1.3.3.5.2">superscript</csymbol><ci id="S3.E6.m1.3.3.5.2.2.cmml" xref="S3.E6.m1.3.3.5.2.2">𝑎</ci><ci id="S3.E6.m1.3.3.5.2.3.cmml" xref="S3.E6.m1.3.3.5.2.3">𝑐</ci></apply><apply id="S3.E6.m1.3.3.5.3.cmml" xref="S3.E6.m1.3.3.5.3"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.5.3.1.cmml" xref="S3.E6.m1.3.3.5.3">superscript</csymbol><ci id="S3.E6.m1.3.3.5.3.2.cmml" xref="S3.E6.m1.3.3.5.3.2">𝑏</ci><ci id="S3.E6.m1.3.3.5.3.3.cmml" xref="S3.E6.m1.3.3.5.3.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.cmml" xref="S3.E6.m1.3.3.3"><minus id="S3.E6.m1.3.3.3.4.cmml" xref="S3.E6.m1.3.3.3.4"></minus><apply id="S3.E6.m1.2.2.2.2.cmml" xref="S3.E6.m1.2.2.2.2"><times id="S3.E6.m1.2.2.2.2.3.cmml" xref="S3.E6.m1.2.2.2.2.3"></times><apply id="S3.E6.m1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1"><ci id="S3.E6.m1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.2">⋅</ci><apply id="S3.E6.m1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><times id="S3.E6.m1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2"></times><apply id="S3.E6.m1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3">superscript</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2"><ci id="S3.E6.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2.1">~</ci><ci id="S3.E6.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2.2">𝑔</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3">𝑐</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2">𝑎</ci><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.3.2">ℎ</ci><ci id="S3.E6.m1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.3.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.2.2.2.2.2.1.1.cmml" xref="S3.E6.m1.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S3.E6.m1.2.2.2.2.2.1.1.1.cmml" xref="S3.E6.m1.2.2.2.2.2.1">superscript</csymbol><ci id="S3.E6.m1.2.2.2.2.2.1.1.2.cmml" xref="S3.E6.m1.2.2.2.2.2.1.1.2">𝑏</ci><ci id="S3.E6.m1.2.2.2.2.2.1.1.3.cmml" xref="S3.E6.m1.2.2.2.2.2.1.1.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.3.cmml" xref="S3.E6.m1.3.3.3.3"><times id="S3.E6.m1.3.3.3.3.2.cmml" xref="S3.E6.m1.3.3.3.3.2"></times><ci id="S3.E6.m1.3.3.3.3.3.cmml" xref="S3.E6.m1.3.3.3.3.3">sg</ci><apply id="S3.E6.m1.3.3.3.3.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1"><minus id="S3.E6.m1.3.3.3.3.1.1.1.5.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.5"></minus><apply id="S3.E6.m1.3.3.3.3.1.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.2"><times id="S3.E6.m1.3.3.3.3.1.1.1.2.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.2.3"></times><apply id="S3.E6.m1.3.3.3.3.1.1.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1"><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.2">⋅</ci><apply id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1"><times id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.2"></times><apply id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3">superscript</csymbol><apply id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2"><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.1">~</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.2.2">𝑔</ci></apply><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.3.3">𝑐</ci></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.2">𝑎</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.2">ℎ</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.1.1.3.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.2">𝑏</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.2.2.1.1.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.4.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.4"><times id="S3.E6.m1.3.3.3.3.1.1.1.4.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.4.3"></times><apply id="S3.E6.m1.3.3.3.3.1.1.1.3.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1"><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.2">⋅</ci><apply id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1"><times id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.2"></times><apply id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.2">𝑔</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.3.3">𝑐</ci></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.2">𝑎</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.1.1.1.1.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.2">ℎ</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.3.1.3.3">𝑐</ci></apply></apply><apply id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1"><csymbol cd="ambiguous" id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.1.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1">superscript</csymbol><ci id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.2.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.2">𝑏</ci><ci id="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.3.cmml" xref="S3.E6.m1.3.3.3.3.1.1.1.4.2.1.1.3">𝑐</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.3c">a^{c}b^{c}=\tilde{g}^{c}(a^{c})\cdot h^{c}(b^{c})-\mathrm{sg}(\tilde{g}^{c}(a^{c})\cdot h^{c}(b^{c})-g^{c}(a^{c})\cdot h^{c}(b^{c}))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS1.p6" class="ltx_para">
<p id="S3.SS1.p6.3" class="ltx_p">Here, <math id="S3.SS1.p6.1.m1.1" class="ltx_Math" alttext="sg" display="inline"><semantics id="S3.SS1.p6.1.m1.1a"><mrow id="S3.SS1.p6.1.m1.1.1" xref="S3.SS1.p6.1.m1.1.1.cmml"><mi id="S3.SS1.p6.1.m1.1.1.2" xref="S3.SS1.p6.1.m1.1.1.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p6.1.m1.1.1.1" xref="S3.SS1.p6.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS1.p6.1.m1.1.1.3" xref="S3.SS1.p6.1.m1.1.1.3.cmml">g</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.1.m1.1b"><apply id="S3.SS1.p6.1.m1.1.1.cmml" xref="S3.SS1.p6.1.m1.1.1"><times id="S3.SS1.p6.1.m1.1.1.1.cmml" xref="S3.SS1.p6.1.m1.1.1.1"></times><ci id="S3.SS1.p6.1.m1.1.1.2.cmml" xref="S3.SS1.p6.1.m1.1.1.2">𝑠</ci><ci id="S3.SS1.p6.1.m1.1.1.3.cmml" xref="S3.SS1.p6.1.m1.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.1.m1.1c">sg</annotation></semantics></math> represents the <em id="S3.SS1.p6.3.1" class="ltx_emph ltx_font_italic">stop gradient</em> operator. It serves as an identity function during the forward pass to enable the use of <math id="S3.SS1.p6.2.m2.1" class="ltx_Math" alttext="g^{c}(a^{c})" display="inline"><semantics id="S3.SS1.p6.2.m2.1a"><mrow id="S3.SS1.p6.2.m2.1.1" xref="S3.SS1.p6.2.m2.1.1.cmml"><msup id="S3.SS1.p6.2.m2.1.1.3" xref="S3.SS1.p6.2.m2.1.1.3.cmml"><mi id="S3.SS1.p6.2.m2.1.1.3.2" xref="S3.SS1.p6.2.m2.1.1.3.2.cmml">g</mi><mi id="S3.SS1.p6.2.m2.1.1.3.3" xref="S3.SS1.p6.2.m2.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p6.2.m2.1.1.2" xref="S3.SS1.p6.2.m2.1.1.2.cmml">​</mo><mrow id="S3.SS1.p6.2.m2.1.1.1.1" xref="S3.SS1.p6.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p6.2.m2.1.1.1.1.2" xref="S3.SS1.p6.2.m2.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p6.2.m2.1.1.1.1.1" xref="S3.SS1.p6.2.m2.1.1.1.1.1.cmml"><mi id="S3.SS1.p6.2.m2.1.1.1.1.1.2" xref="S3.SS1.p6.2.m2.1.1.1.1.1.2.cmml">a</mi><mi id="S3.SS1.p6.2.m2.1.1.1.1.1.3" xref="S3.SS1.p6.2.m2.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS1.p6.2.m2.1.1.1.1.3" xref="S3.SS1.p6.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.2.m2.1b"><apply id="S3.SS1.p6.2.m2.1.1.cmml" xref="S3.SS1.p6.2.m2.1.1"><times id="S3.SS1.p6.2.m2.1.1.2.cmml" xref="S3.SS1.p6.2.m2.1.1.2"></times><apply id="S3.SS1.p6.2.m2.1.1.3.cmml" xref="S3.SS1.p6.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p6.2.m2.1.1.3.1.cmml" xref="S3.SS1.p6.2.m2.1.1.3">superscript</csymbol><ci id="S3.SS1.p6.2.m2.1.1.3.2.cmml" xref="S3.SS1.p6.2.m2.1.1.3.2">𝑔</ci><ci id="S3.SS1.p6.2.m2.1.1.3.3.cmml" xref="S3.SS1.p6.2.m2.1.1.3.3">𝑐</ci></apply><apply id="S3.SS1.p6.2.m2.1.1.1.1.1.cmml" xref="S3.SS1.p6.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p6.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS1.p6.2.m2.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p6.2.m2.1.1.1.1.1.2.cmml" xref="S3.SS1.p6.2.m2.1.1.1.1.1.2">𝑎</ci><ci id="S3.SS1.p6.2.m2.1.1.1.1.1.3.cmml" xref="S3.SS1.p6.2.m2.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.2.m2.1c">g^{c}(a^{c})</annotation></semantics></math> encoding in <code id="S3.SS1.p6.3.2" class="ltx_verbatim ltx_font_typewriter">argmin</code>. During the backward pass, it drops gradients inside it to enable <math id="S3.SS1.p6.3.m3.1" class="ltx_Math" alttext="\tilde{g}^{c}(a^{c})" display="inline"><semantics id="S3.SS1.p6.3.m3.1a"><mrow id="S3.SS1.p6.3.m3.1.1" xref="S3.SS1.p6.3.m3.1.1.cmml"><msup id="S3.SS1.p6.3.m3.1.1.3" xref="S3.SS1.p6.3.m3.1.1.3.cmml"><mover accent="true" id="S3.SS1.p6.3.m3.1.1.3.2" xref="S3.SS1.p6.3.m3.1.1.3.2.cmml"><mi id="S3.SS1.p6.3.m3.1.1.3.2.2" xref="S3.SS1.p6.3.m3.1.1.3.2.2.cmml">g</mi><mo id="S3.SS1.p6.3.m3.1.1.3.2.1" xref="S3.SS1.p6.3.m3.1.1.3.2.1.cmml">~</mo></mover><mi id="S3.SS1.p6.3.m3.1.1.3.3" xref="S3.SS1.p6.3.m3.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p6.3.m3.1.1.2" xref="S3.SS1.p6.3.m3.1.1.2.cmml">​</mo><mrow id="S3.SS1.p6.3.m3.1.1.1.1" xref="S3.SS1.p6.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p6.3.m3.1.1.1.1.2" xref="S3.SS1.p6.3.m3.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p6.3.m3.1.1.1.1.1" xref="S3.SS1.p6.3.m3.1.1.1.1.1.cmml"><mi id="S3.SS1.p6.3.m3.1.1.1.1.1.2" xref="S3.SS1.p6.3.m3.1.1.1.1.1.2.cmml">a</mi><mi id="S3.SS1.p6.3.m3.1.1.1.1.1.3" xref="S3.SS1.p6.3.m3.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS1.p6.3.m3.1.1.1.1.3" xref="S3.SS1.p6.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.3.m3.1b"><apply id="S3.SS1.p6.3.m3.1.1.cmml" xref="S3.SS1.p6.3.m3.1.1"><times id="S3.SS1.p6.3.m3.1.1.2.cmml" xref="S3.SS1.p6.3.m3.1.1.2"></times><apply id="S3.SS1.p6.3.m3.1.1.3.cmml" xref="S3.SS1.p6.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p6.3.m3.1.1.3.1.cmml" xref="S3.SS1.p6.3.m3.1.1.3">superscript</csymbol><apply id="S3.SS1.p6.3.m3.1.1.3.2.cmml" xref="S3.SS1.p6.3.m3.1.1.3.2"><ci id="S3.SS1.p6.3.m3.1.1.3.2.1.cmml" xref="S3.SS1.p6.3.m3.1.1.3.2.1">~</ci><ci id="S3.SS1.p6.3.m3.1.1.3.2.2.cmml" xref="S3.SS1.p6.3.m3.1.1.3.2.2">𝑔</ci></apply><ci id="S3.SS1.p6.3.m3.1.1.3.3.cmml" xref="S3.SS1.p6.3.m3.1.1.3.3">𝑐</ci></apply><apply id="S3.SS1.p6.3.m3.1.1.1.1.1.cmml" xref="S3.SS1.p6.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p6.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS1.p6.3.m3.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p6.3.m3.1.1.1.1.1.2.cmml" xref="S3.SS1.p6.3.m3.1.1.1.1.1.2">𝑎</ci><ci id="S3.SS1.p6.3.m3.1.1.1.1.1.3.cmml" xref="S3.SS1.p6.3.m3.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.3.m3.1c">\tilde{g}^{c}(a^{c})</annotation></semantics></math> to generate gradients via <code id="S3.SS1.p6.3.3" class="ltx_verbatim ltx_font_typewriter">softmax</code>.</p>
</div>
<div id="S3.SS1.p7" class="ltx_para">
<p id="S3.SS1.p7.1" class="ltx_p">The initial value is critical for learning convergence and accuracy. Therefore, we use the centroids learned by <math id="S3.SS1.p7.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.p7.1.m1.1a"><mi id="S3.SS1.p7.1.m1.1.1" xref="S3.SS1.p7.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.1.m1.1b"><ci id="S3.SS1.p7.1.m1.1.1.cmml" xref="S3.SS1.p7.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.1.m1.1c">k</annotation></semantics></math>-means from vanilla PQ to initialize the centroids and lookup tables.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2. </span>Learned temperature</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.7" class="ltx_p">The <code id="S3.SS2.p1.7.1" class="ltx_verbatim ltx_font_typewriter">temperature</code> hyperparameter <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">t</annotation></semantics></math> of <code id="S3.SS2.p1.7.2" class="ltx_verbatim ltx_font_typewriter">softmax</code> <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al<span class="ltx_text">.</span>, <a href="#bib.bib29" title="" class="ltx_ref">2015</a>)</cite> controls the approximation error of <code id="S3.SS2.p1.7.3" class="ltx_verbatim ltx_font_typewriter">softmax</code> to <code id="S3.SS2.p1.7.4" class="ltx_verbatim ltx_font_typewriter">argmax</code>. Shown in Fig. <a href="#S3.F5.sf1" title="In Figure 5 ‣ 3.2. Learned temperature ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5a</span></a>, as <math id="S3.SS2.p1.2.m2.3" class="ltx_Math" alttext="softmax(x)_{i}=\frac{exp(x_{i}/t)}{\sum_{k=1}^{K}exp(x_{k}/t)}" display="inline"><semantics id="S3.SS2.p1.2.m2.3a"><mrow id="S3.SS2.p1.2.m2.3.4" xref="S3.SS2.p1.2.m2.3.4.cmml"><mrow id="S3.SS2.p1.2.m2.3.4.2" xref="S3.SS2.p1.2.m2.3.4.2.cmml"><mi id="S3.SS2.p1.2.m2.3.4.2.2" xref="S3.SS2.p1.2.m2.3.4.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.4.2.3" xref="S3.SS2.p1.2.m2.3.4.2.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1a" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.4.2.4" xref="S3.SS2.p1.2.m2.3.4.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1b" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.4.2.5" xref="S3.SS2.p1.2.m2.3.4.2.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1c" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.4.2.6" xref="S3.SS2.p1.2.m2.3.4.2.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1d" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.4.2.7" xref="S3.SS2.p1.2.m2.3.4.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1e" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.4.2.8" xref="S3.SS2.p1.2.m2.3.4.2.8.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.4.2.1f" xref="S3.SS2.p1.2.m2.3.4.2.1.cmml">​</mo><msub id="S3.SS2.p1.2.m2.3.4.2.9" xref="S3.SS2.p1.2.m2.3.4.2.9.cmml"><mrow id="S3.SS2.p1.2.m2.3.4.2.9.2.2" xref="S3.SS2.p1.2.m2.3.4.2.9.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.3.4.2.9.2.2.1" xref="S3.SS2.p1.2.m2.3.4.2.9.cmml">(</mo><mi id="S3.SS2.p1.2.m2.3.3" xref="S3.SS2.p1.2.m2.3.3.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.2.m2.3.4.2.9.2.2.2" xref="S3.SS2.p1.2.m2.3.4.2.9.cmml">)</mo></mrow><mi id="S3.SS2.p1.2.m2.3.4.2.9.3" xref="S3.SS2.p1.2.m2.3.4.2.9.3.cmml">i</mi></msub></mrow><mo id="S3.SS2.p1.2.m2.3.4.1" xref="S3.SS2.p1.2.m2.3.4.1.cmml">=</mo><mfrac id="S3.SS2.p1.2.m2.2.2" xref="S3.SS2.p1.2.m2.2.2.cmml"><mrow id="S3.SS2.p1.2.m2.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.cmml"><mi id="S3.SS2.p1.2.m2.1.1.1.3" xref="S3.SS2.p1.2.m2.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.2" xref="S3.SS2.p1.2.m2.1.1.1.2.cmml">​</mo><mi id="S3.SS2.p1.2.m2.1.1.1.4" xref="S3.SS2.p1.2.m2.1.1.1.4.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.2a" xref="S3.SS2.p1.2.m2.1.1.1.2.cmml">​</mo><mi id="S3.SS2.p1.2.m2.1.1.1.5" xref="S3.SS2.p1.2.m2.1.1.1.5.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.2b" xref="S3.SS2.p1.2.m2.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.1.1.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.1.1.1.1.1.2" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.2.m2.1.1.1.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml"><msub id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.2" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.3" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS2.p1.2.m2.1.1.1.1.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.1.cmml">/</mo><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.1.3" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.3.cmml">t</mi></mrow><mo stretchy="false" id="S3.SS2.p1.2.m2.1.1.1.1.1.3" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S3.SS2.p1.2.m2.2.2.2" xref="S3.SS2.p1.2.m2.2.2.2.cmml"><mstyle displaystyle="false" id="S3.SS2.p1.2.m2.2.2.2.2" xref="S3.SS2.p1.2.m2.2.2.2.2.cmml"><msubsup id="S3.SS2.p1.2.m2.2.2.2.2a" xref="S3.SS2.p1.2.m2.2.2.2.2.cmml"><mo id="S3.SS2.p1.2.m2.2.2.2.2.2.2" xref="S3.SS2.p1.2.m2.2.2.2.2.2.2.cmml">∑</mo><mrow id="S3.SS2.p1.2.m2.2.2.2.2.2.3" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.cmml"><mi id="S3.SS2.p1.2.m2.2.2.2.2.2.3.2" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.2.cmml">k</mi><mo id="S3.SS2.p1.2.m2.2.2.2.2.2.3.1" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.1.cmml">=</mo><mn id="S3.SS2.p1.2.m2.2.2.2.2.2.3.3" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p1.2.m2.2.2.2.2.3" xref="S3.SS2.p1.2.m2.2.2.2.2.3.cmml">K</mi></msubsup></mstyle><mrow id="S3.SS2.p1.2.m2.2.2.2.1" xref="S3.SS2.p1.2.m2.2.2.2.1.cmml"><mi id="S3.SS2.p1.2.m2.2.2.2.1.3" xref="S3.SS2.p1.2.m2.2.2.2.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.2.2.2.1.2" xref="S3.SS2.p1.2.m2.2.2.2.1.2.cmml">​</mo><mi id="S3.SS2.p1.2.m2.2.2.2.1.4" xref="S3.SS2.p1.2.m2.2.2.2.1.4.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.2.2.2.1.2a" xref="S3.SS2.p1.2.m2.2.2.2.1.2.cmml">​</mo><mi id="S3.SS2.p1.2.m2.2.2.2.1.5" xref="S3.SS2.p1.2.m2.2.2.2.1.5.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.2.2.2.1.2b" xref="S3.SS2.p1.2.m2.2.2.2.1.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.2.2.2.1.1.1" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.2.2.2.1.1.1.2" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.cmml"><msub id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.cmml"><mi id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.2" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.3" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.3.cmml">k</mi></msub><mo id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.1" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.1.cmml">/</mo><mi id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.3" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.3.cmml">t</mi></mrow><mo stretchy="false" id="S3.SS2.p1.2.m2.2.2.2.1.1.1.3" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.3b"><apply id="S3.SS2.p1.2.m2.3.4.cmml" xref="S3.SS2.p1.2.m2.3.4"><eq id="S3.SS2.p1.2.m2.3.4.1.cmml" xref="S3.SS2.p1.2.m2.3.4.1"></eq><apply id="S3.SS2.p1.2.m2.3.4.2.cmml" xref="S3.SS2.p1.2.m2.3.4.2"><times id="S3.SS2.p1.2.m2.3.4.2.1.cmml" xref="S3.SS2.p1.2.m2.3.4.2.1"></times><ci id="S3.SS2.p1.2.m2.3.4.2.2.cmml" xref="S3.SS2.p1.2.m2.3.4.2.2">𝑠</ci><ci id="S3.SS2.p1.2.m2.3.4.2.3.cmml" xref="S3.SS2.p1.2.m2.3.4.2.3">𝑜</ci><ci id="S3.SS2.p1.2.m2.3.4.2.4.cmml" xref="S3.SS2.p1.2.m2.3.4.2.4">𝑓</ci><ci id="S3.SS2.p1.2.m2.3.4.2.5.cmml" xref="S3.SS2.p1.2.m2.3.4.2.5">𝑡</ci><ci id="S3.SS2.p1.2.m2.3.4.2.6.cmml" xref="S3.SS2.p1.2.m2.3.4.2.6">𝑚</ci><ci id="S3.SS2.p1.2.m2.3.4.2.7.cmml" xref="S3.SS2.p1.2.m2.3.4.2.7">𝑎</ci><ci id="S3.SS2.p1.2.m2.3.4.2.8.cmml" xref="S3.SS2.p1.2.m2.3.4.2.8">𝑥</ci><apply id="S3.SS2.p1.2.m2.3.4.2.9.cmml" xref="S3.SS2.p1.2.m2.3.4.2.9"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.3.4.2.9.1.cmml" xref="S3.SS2.p1.2.m2.3.4.2.9">subscript</csymbol><ci id="S3.SS2.p1.2.m2.3.3.cmml" xref="S3.SS2.p1.2.m2.3.3">𝑥</ci><ci id="S3.SS2.p1.2.m2.3.4.2.9.3.cmml" xref="S3.SS2.p1.2.m2.3.4.2.9.3">𝑖</ci></apply></apply><apply id="S3.SS2.p1.2.m2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2"><divide id="S3.SS2.p1.2.m2.2.2.3.cmml" xref="S3.SS2.p1.2.m2.2.2"></divide><apply id="S3.SS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1"><times id="S3.SS2.p1.2.m2.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.1.1.1.2"></times><ci id="S3.SS2.p1.2.m2.1.1.1.3.cmml" xref="S3.SS2.p1.2.m2.1.1.1.3">𝑒</ci><ci id="S3.SS2.p1.2.m2.1.1.1.4.cmml" xref="S3.SS2.p1.2.m2.1.1.1.4">𝑥</ci><ci id="S3.SS2.p1.2.m2.1.1.1.5.cmml" xref="S3.SS2.p1.2.m2.1.1.1.5">𝑝</ci><apply id="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1"><divide id="S3.SS2.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.1"></divide><apply id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.2.3">𝑖</ci></apply><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.3">𝑡</ci></apply></apply><apply id="S3.SS2.p1.2.m2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2"><apply id="S3.SS2.p1.2.m2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.2.2.2.2.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2">superscript</csymbol><apply id="S3.SS2.p1.2.m2.2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.2.2.2.2.2.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2">subscript</csymbol><sum id="S3.SS2.p1.2.m2.2.2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2.2.2"></sum><apply id="S3.SS2.p1.2.m2.2.2.2.2.2.3.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3"><eq id="S3.SS2.p1.2.m2.2.2.2.2.2.3.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.1"></eq><ci id="S3.SS2.p1.2.m2.2.2.2.2.2.3.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.2">𝑘</ci><cn type="integer" id="S3.SS2.p1.2.m2.2.2.2.2.2.3.3.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2.2.3.3">1</cn></apply></apply><ci id="S3.SS2.p1.2.m2.2.2.2.2.3.cmml" xref="S3.SS2.p1.2.m2.2.2.2.2.3">𝐾</ci></apply><apply id="S3.SS2.p1.2.m2.2.2.2.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1"><times id="S3.SS2.p1.2.m2.2.2.2.1.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.2"></times><ci id="S3.SS2.p1.2.m2.2.2.2.1.3.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.3">𝑒</ci><ci id="S3.SS2.p1.2.m2.2.2.2.1.4.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.4">𝑥</ci><ci id="S3.SS2.p1.2.m2.2.2.2.1.5.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.5">𝑝</ci><apply id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1"><divide id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.1"></divide><apply id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.1.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.3.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.2.3">𝑘</ci></apply><ci id="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.3.cmml" xref="S3.SS2.p1.2.m2.2.2.2.1.1.1.1.3">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.3c">softmax(x)_{i}=\frac{exp(x_{i}/t)}{\sum_{k=1}^{K}exp(x_{k}/t)}</annotation></semantics></math>, when <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="t\rightarrow\infty" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mrow id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.p1.3.m3.1.1.2" xref="S3.SS2.p1.3.m3.1.1.2.cmml">t</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.1.1.1" xref="S3.SS2.p1.3.m3.1.1.1.cmml">→</mo><mi mathvariant="normal" id="S3.SS2.p1.3.m3.1.1.3" xref="S3.SS2.p1.3.m3.1.1.3.cmml">∞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1"><ci id="S3.SS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1.1">→</ci><ci id="S3.SS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2">𝑡</ci><infinity id="S3.SS2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.p1.3.m3.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">t\rightarrow\infty</annotation></semantics></math>, <math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="softmax(x)_{i}\rightarrow\frac{1}{K}" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><mrow id="S3.SS2.p1.4.m4.1.2" xref="S3.SS2.p1.4.m4.1.2.cmml"><mrow id="S3.SS2.p1.4.m4.1.2.2" xref="S3.SS2.p1.4.m4.1.2.2.cmml"><mi id="S3.SS2.p1.4.m4.1.2.2.2" xref="S3.SS2.p1.4.m4.1.2.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.2.2.3" xref="S3.SS2.p1.4.m4.1.2.2.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1a" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.2.2.4" xref="S3.SS2.p1.4.m4.1.2.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1b" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.2.2.5" xref="S3.SS2.p1.4.m4.1.2.2.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1c" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.2.2.6" xref="S3.SS2.p1.4.m4.1.2.2.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1d" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.2.2.7" xref="S3.SS2.p1.4.m4.1.2.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1e" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.2.2.8" xref="S3.SS2.p1.4.m4.1.2.2.8.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.2.2.1f" xref="S3.SS2.p1.4.m4.1.2.2.1.cmml">​</mo><msub id="S3.SS2.p1.4.m4.1.2.2.9" xref="S3.SS2.p1.4.m4.1.2.2.9.cmml"><mrow id="S3.SS2.p1.4.m4.1.2.2.9.2.2" xref="S3.SS2.p1.4.m4.1.2.2.9.cmml"><mo stretchy="false" id="S3.SS2.p1.4.m4.1.2.2.9.2.2.1" xref="S3.SS2.p1.4.m4.1.2.2.9.cmml">(</mo><mi id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.4.m4.1.2.2.9.2.2.2" xref="S3.SS2.p1.4.m4.1.2.2.9.cmml">)</mo></mrow><mi id="S3.SS2.p1.4.m4.1.2.2.9.3" xref="S3.SS2.p1.4.m4.1.2.2.9.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.SS2.p1.4.m4.1.2.1" xref="S3.SS2.p1.4.m4.1.2.1.cmml">→</mo><mfrac id="S3.SS2.p1.4.m4.1.2.3" xref="S3.SS2.p1.4.m4.1.2.3.cmml"><mn id="S3.SS2.p1.4.m4.1.2.3.2" xref="S3.SS2.p1.4.m4.1.2.3.2.cmml">1</mn><mi id="S3.SS2.p1.4.m4.1.2.3.3" xref="S3.SS2.p1.4.m4.1.2.3.3.cmml">K</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><apply id="S3.SS2.p1.4.m4.1.2.cmml" xref="S3.SS2.p1.4.m4.1.2"><ci id="S3.SS2.p1.4.m4.1.2.1.cmml" xref="S3.SS2.p1.4.m4.1.2.1">→</ci><apply id="S3.SS2.p1.4.m4.1.2.2.cmml" xref="S3.SS2.p1.4.m4.1.2.2"><times id="S3.SS2.p1.4.m4.1.2.2.1.cmml" xref="S3.SS2.p1.4.m4.1.2.2.1"></times><ci id="S3.SS2.p1.4.m4.1.2.2.2.cmml" xref="S3.SS2.p1.4.m4.1.2.2.2">𝑠</ci><ci id="S3.SS2.p1.4.m4.1.2.2.3.cmml" xref="S3.SS2.p1.4.m4.1.2.2.3">𝑜</ci><ci id="S3.SS2.p1.4.m4.1.2.2.4.cmml" xref="S3.SS2.p1.4.m4.1.2.2.4">𝑓</ci><ci id="S3.SS2.p1.4.m4.1.2.2.5.cmml" xref="S3.SS2.p1.4.m4.1.2.2.5">𝑡</ci><ci id="S3.SS2.p1.4.m4.1.2.2.6.cmml" xref="S3.SS2.p1.4.m4.1.2.2.6">𝑚</ci><ci id="S3.SS2.p1.4.m4.1.2.2.7.cmml" xref="S3.SS2.p1.4.m4.1.2.2.7">𝑎</ci><ci id="S3.SS2.p1.4.m4.1.2.2.8.cmml" xref="S3.SS2.p1.4.m4.1.2.2.8">𝑥</ci><apply id="S3.SS2.p1.4.m4.1.2.2.9.cmml" xref="S3.SS2.p1.4.m4.1.2.2.9"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.1.2.2.9.1.cmml" xref="S3.SS2.p1.4.m4.1.2.2.9">subscript</csymbol><ci id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">𝑥</ci><ci id="S3.SS2.p1.4.m4.1.2.2.9.3.cmml" xref="S3.SS2.p1.4.m4.1.2.2.9.3">𝑖</ci></apply></apply><apply id="S3.SS2.p1.4.m4.1.2.3.cmml" xref="S3.SS2.p1.4.m4.1.2.3"><divide id="S3.SS2.p1.4.m4.1.2.3.1.cmml" xref="S3.SS2.p1.4.m4.1.2.3"></divide><cn type="integer" id="S3.SS2.p1.4.m4.1.2.3.2.cmml" xref="S3.SS2.p1.4.m4.1.2.3.2">1</cn><ci id="S3.SS2.p1.4.m4.1.2.3.3.cmml" xref="S3.SS2.p1.4.m4.1.2.3.3">𝐾</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">softmax(x)_{i}\rightarrow\frac{1}{K}</annotation></semantics></math>, i.e., the output probability distribution approaches uniform distribution. When <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="t\rightarrow 0" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><mrow id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml"><mi id="S3.SS2.p1.5.m5.1.1.2" xref="S3.SS2.p1.5.m5.1.1.2.cmml">t</mi><mo stretchy="false" id="S3.SS2.p1.5.m5.1.1.1" xref="S3.SS2.p1.5.m5.1.1.1.cmml">→</mo><mn id="S3.SS2.p1.5.m5.1.1.3" xref="S3.SS2.p1.5.m5.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><apply id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1"><ci id="S3.SS2.p1.5.m5.1.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1">→</ci><ci id="S3.SS2.p1.5.m5.1.1.2.cmml" xref="S3.SS2.p1.5.m5.1.1.2">𝑡</ci><cn type="integer" id="S3.SS2.p1.5.m5.1.1.3.cmml" xref="S3.SS2.p1.5.m5.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">t\rightarrow 0</annotation></semantics></math>, <math id="S3.SS2.p1.6.m6.3" class="ltx_Math" alttext="softmax(x)\rightarrow onehot(argmax(x))" display="inline"><semantics id="S3.SS2.p1.6.m6.3a"><mrow id="S3.SS2.p1.6.m6.3.3" xref="S3.SS2.p1.6.m6.3.3.cmml"><mrow id="S3.SS2.p1.6.m6.3.3.3" xref="S3.SS2.p1.6.m6.3.3.3.cmml"><mi id="S3.SS2.p1.6.m6.3.3.3.2" xref="S3.SS2.p1.6.m6.3.3.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.3.3" xref="S3.SS2.p1.6.m6.3.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1a" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.3.4" xref="S3.SS2.p1.6.m6.3.3.3.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1b" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.3.5" xref="S3.SS2.p1.6.m6.3.3.3.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1c" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.3.6" xref="S3.SS2.p1.6.m6.3.3.3.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1d" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.3.7" xref="S3.SS2.p1.6.m6.3.3.3.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1e" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.3.8" xref="S3.SS2.p1.6.m6.3.3.3.8.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.3.1f" xref="S3.SS2.p1.6.m6.3.3.3.1.cmml">​</mo><mrow id="S3.SS2.p1.6.m6.3.3.3.9.2" xref="S3.SS2.p1.6.m6.3.3.3.cmml"><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.3.9.2.1" xref="S3.SS2.p1.6.m6.3.3.3.cmml">(</mo><mi id="S3.SS2.p1.6.m6.1.1" xref="S3.SS2.p1.6.m6.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.3.9.2.2" xref="S3.SS2.p1.6.m6.3.3.3.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.2" xref="S3.SS2.p1.6.m6.3.3.2.cmml">→</mo><mrow id="S3.SS2.p1.6.m6.3.3.1" xref="S3.SS2.p1.6.m6.3.3.1.cmml"><mi id="S3.SS2.p1.6.m6.3.3.1.3" xref="S3.SS2.p1.6.m6.3.3.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.2" xref="S3.SS2.p1.6.m6.3.3.1.2.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.4" xref="S3.SS2.p1.6.m6.3.3.1.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.2a" xref="S3.SS2.p1.6.m6.3.3.1.2.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.5" xref="S3.SS2.p1.6.m6.3.3.1.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.2b" xref="S3.SS2.p1.6.m6.3.3.1.2.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.6" xref="S3.SS2.p1.6.m6.3.3.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.2c" xref="S3.SS2.p1.6.m6.3.3.1.2.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.7" xref="S3.SS2.p1.6.m6.3.3.1.7.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.2d" xref="S3.SS2.p1.6.m6.3.3.1.2.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.8" xref="S3.SS2.p1.6.m6.3.3.1.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.2e" xref="S3.SS2.p1.6.m6.3.3.1.2.cmml">​</mo><mrow id="S3.SS2.p1.6.m6.3.3.1.1.1" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.1.1.1.2" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.6.m6.3.3.1.1.1.1" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml"><mi id="S3.SS2.p1.6.m6.3.3.1.1.1.1.2" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.1.1.1.3" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1a" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.1.1.1.4" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1b" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.1.1.1.5" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.5.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1c" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.1.1.1.6" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.6.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1d" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.3.3.1.1.1.1.7" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.7.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1e" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml">​</mo><mrow id="S3.SS2.p1.6.m6.3.3.1.1.1.1.8.2" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.8.2.1" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml">(</mo><mi id="S3.SS2.p1.6.m6.2.2" xref="S3.SS2.p1.6.m6.2.2.cmml">x</mi><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.1.1.1.1.8.2.2" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.6.m6.3.3.1.1.1.3" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m6.3b"><apply id="S3.SS2.p1.6.m6.3.3.cmml" xref="S3.SS2.p1.6.m6.3.3"><ci id="S3.SS2.p1.6.m6.3.3.2.cmml" xref="S3.SS2.p1.6.m6.3.3.2">→</ci><apply id="S3.SS2.p1.6.m6.3.3.3.cmml" xref="S3.SS2.p1.6.m6.3.3.3"><times id="S3.SS2.p1.6.m6.3.3.3.1.cmml" xref="S3.SS2.p1.6.m6.3.3.3.1"></times><ci id="S3.SS2.p1.6.m6.3.3.3.2.cmml" xref="S3.SS2.p1.6.m6.3.3.3.2">𝑠</ci><ci id="S3.SS2.p1.6.m6.3.3.3.3.cmml" xref="S3.SS2.p1.6.m6.3.3.3.3">𝑜</ci><ci id="S3.SS2.p1.6.m6.3.3.3.4.cmml" xref="S3.SS2.p1.6.m6.3.3.3.4">𝑓</ci><ci id="S3.SS2.p1.6.m6.3.3.3.5.cmml" xref="S3.SS2.p1.6.m6.3.3.3.5">𝑡</ci><ci id="S3.SS2.p1.6.m6.3.3.3.6.cmml" xref="S3.SS2.p1.6.m6.3.3.3.6">𝑚</ci><ci id="S3.SS2.p1.6.m6.3.3.3.7.cmml" xref="S3.SS2.p1.6.m6.3.3.3.7">𝑎</ci><ci id="S3.SS2.p1.6.m6.3.3.3.8.cmml" xref="S3.SS2.p1.6.m6.3.3.3.8">𝑥</ci><ci id="S3.SS2.p1.6.m6.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1">𝑥</ci></apply><apply id="S3.SS2.p1.6.m6.3.3.1.cmml" xref="S3.SS2.p1.6.m6.3.3.1"><times id="S3.SS2.p1.6.m6.3.3.1.2.cmml" xref="S3.SS2.p1.6.m6.3.3.1.2"></times><ci id="S3.SS2.p1.6.m6.3.3.1.3.cmml" xref="S3.SS2.p1.6.m6.3.3.1.3">𝑜</ci><ci id="S3.SS2.p1.6.m6.3.3.1.4.cmml" xref="S3.SS2.p1.6.m6.3.3.1.4">𝑛</ci><ci id="S3.SS2.p1.6.m6.3.3.1.5.cmml" xref="S3.SS2.p1.6.m6.3.3.1.5">𝑒</ci><ci id="S3.SS2.p1.6.m6.3.3.1.6.cmml" xref="S3.SS2.p1.6.m6.3.3.1.6">ℎ</ci><ci id="S3.SS2.p1.6.m6.3.3.1.7.cmml" xref="S3.SS2.p1.6.m6.3.3.1.7">𝑜</ci><ci id="S3.SS2.p1.6.m6.3.3.1.8.cmml" xref="S3.SS2.p1.6.m6.3.3.1.8">𝑡</ci><apply id="S3.SS2.p1.6.m6.3.3.1.1.1.1.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1"><times id="S3.SS2.p1.6.m6.3.3.1.1.1.1.1.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.1"></times><ci id="S3.SS2.p1.6.m6.3.3.1.1.1.1.2.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.2">𝑎</ci><ci id="S3.SS2.p1.6.m6.3.3.1.1.1.1.3.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.3">𝑟</ci><ci id="S3.SS2.p1.6.m6.3.3.1.1.1.1.4.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.4">𝑔</ci><ci id="S3.SS2.p1.6.m6.3.3.1.1.1.1.5.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.5">𝑚</ci><ci id="S3.SS2.p1.6.m6.3.3.1.1.1.1.6.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.6">𝑎</ci><ci id="S3.SS2.p1.6.m6.3.3.1.1.1.1.7.cmml" xref="S3.SS2.p1.6.m6.3.3.1.1.1.1.7">𝑥</ci><ci id="S3.SS2.p1.6.m6.2.2.cmml" xref="S3.SS2.p1.6.m6.2.2">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m6.3c">softmax(x)\rightarrow onehot(argmax(x))</annotation></semantics></math>, i.e., the probability of the largest <math id="S3.SS2.p1.7.m7.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS2.p1.7.m7.1a"><mi id="S3.SS2.p1.7.m7.1.1" xref="S3.SS2.p1.7.m7.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m7.1b"><ci id="S3.SS2.p1.7.m7.1.1.cmml" xref="S3.SS2.p1.7.m7.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m7.1c">x</annotation></semantics></math> approaching 1 and the others is 0.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">Therefore, there is a tradeoff between small and large temperature. For small temperature, <code id="S3.SS2.p2.1.1" class="ltx_verbatim ltx_font_typewriter">softmax</code> is close to <code id="S3.SS2.p2.1.2" class="ltx_verbatim ltx_font_typewriter">onehot</code> <code id="S3.SS2.p2.1.3" class="ltx_verbatim ltx_font_typewriter">argmax</code>, but the training is difficult since the variance of gradients is large. For larger temperature, the approximation error is increased, but the variance of gradients is smaller.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.2" class="ltx_p">Works before normally set <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mi id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">t</annotation></semantics></math> as a fixed value (mostly 1), or anneal it from a large number to a small one during training <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al<span class="ltx_text">.</span>, <a href="#bib.bib29" title="" class="ltx_ref">2015</a>; Jang et al<span class="ltx_text">.</span>, <a href="#bib.bib33" title="" class="ltx_ref">2016</a>)</cite>, but never analyze how to set it reasonably. This is because currently for a DNN model, <code id="S3.SS2.p3.2.1" class="ltx_verbatim ltx_font_typewriter">softmax</code> is only used by the output layer to produce class probability, or used by the input layer to produce symbol embedding. The approximation error barely impacts model accuracy. However, our soft-PQ employs <code id="S3.SS2.p3.2.2" class="ltx_verbatim ltx_font_typewriter">softmax</code> in every layer. The accumulated error can decrease accuracy without proper <math id="S3.SS2.p3.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS2.p3.2.m2.1a"><mi id="S3.SS2.p3.2.m2.1.1" xref="S3.SS2.p3.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m2.1b"><ci id="S3.SS2.p3.2.m2.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m2.1c">t</annotation></semantics></math> settings.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.2" class="ltx_p">We thus propose to learn the temperature for each layer, also during backpropagation while centroid learning. Fig. <a href="#S3.F5.sf2" title="In Figure 5 ‣ 3.2. Learned temperature ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5b</span></a> shows the learned <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><mi id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><ci id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">t</annotation></semantics></math> for each layer for ResNet18. The value for each layer is different, and thus not practical to tune by hand.
According to the CIFAR10 accuracy experiments,
training with the learned temperature technique only spends <math id="S3.SS2.p4.2.m2.1" class="ltx_Math" alttext="\frac{1}{10}" display="inline"><semantics id="S3.SS2.p4.2.m2.1a"><mfrac id="S3.SS2.p4.2.m2.1.1" xref="S3.SS2.p4.2.m2.1.1.cmml"><mn id="S3.SS2.p4.2.m2.1.1.2" xref="S3.SS2.p4.2.m2.1.1.2.cmml">1</mn><mn id="S3.SS2.p4.2.m2.1.1.3" xref="S3.SS2.p4.2.m2.1.1.3.cmml">10</mn></mfrac><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.1b"><apply id="S3.SS2.p4.2.m2.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"><divide id="S3.SS2.p4.2.m2.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"></divide><cn type="integer" id="S3.SS2.p4.2.m2.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.2">1</cn><cn type="integer" id="S3.SS2.p4.2.m2.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">\frac{1}{10}</annotation></semantics></math> iterations of training with temperature setting to 1 to achieve 85% accuracy.
Detailed results will show in Sec. <a href="#S6" title="6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<figure id="S3.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S3.F5.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2302.03213/assets/x6.png" id="S3.F5.sf1.g1" class="ltx_graphics ltx_img_landscape" width="115" height="92" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S3.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2302.03213/assets/x7.png" id="S3.F5.sf2.g1" class="ltx_graphics ltx_img_landscape" width="120" height="96" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5. </span>
(a) The output probability distribution of <math id="S3.F5.6.m1.1" class="ltx_Math" alttext="\mathrm{softmax}(\frac{x}{t})" display="inline"><semantics id="S3.F5.6.m1.1b"><mrow id="S3.F5.6.m1.1.2" xref="S3.F5.6.m1.1.2.cmml"><mi id="S3.F5.6.m1.1.2.2" xref="S3.F5.6.m1.1.2.2.cmml">softmax</mi><mo lspace="0em" rspace="0em" id="S3.F5.6.m1.1.2.1" xref="S3.F5.6.m1.1.2.1.cmml">​</mo><mrow id="S3.F5.6.m1.1.2.3.2" xref="S3.F5.6.m1.1.1.cmml"><mo stretchy="false" id="S3.F5.6.m1.1.2.3.2.1" xref="S3.F5.6.m1.1.1.cmml">(</mo><mfrac id="S3.F5.6.m1.1.1" xref="S3.F5.6.m1.1.1.cmml"><mi id="S3.F5.6.m1.1.1.2" xref="S3.F5.6.m1.1.1.2.cmml">x</mi><mi id="S3.F5.6.m1.1.1.3" xref="S3.F5.6.m1.1.1.3.cmml">t</mi></mfrac><mo stretchy="false" id="S3.F5.6.m1.1.2.3.2.2" xref="S3.F5.6.m1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F5.6.m1.1c"><apply id="S3.F5.6.m1.1.2.cmml" xref="S3.F5.6.m1.1.2"><times id="S3.F5.6.m1.1.2.1.cmml" xref="S3.F5.6.m1.1.2.1"></times><ci id="S3.F5.6.m1.1.2.2.cmml" xref="S3.F5.6.m1.1.2.2">softmax</ci><apply id="S3.F5.6.m1.1.1.cmml" xref="S3.F5.6.m1.1.2.3.2"><divide id="S3.F5.6.m1.1.1.1.cmml" xref="S3.F5.6.m1.1.2.3.2"></divide><ci id="S3.F5.6.m1.1.1.2.cmml" xref="S3.F5.6.m1.1.1.2">𝑥</ci><ci id="S3.F5.6.m1.1.1.3.cmml" xref="S3.F5.6.m1.1.1.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.6.m1.1d">\mathrm{softmax}(\frac{x}{t})</annotation></semantics></math> at different temperature, for four centroids as an example. <math id="S3.F5.7.m2.1" class="ltx_Math" alttext="t\rightarrow\infty" display="inline"><semantics id="S3.F5.7.m2.1b"><mrow id="S3.F5.7.m2.1.1" xref="S3.F5.7.m2.1.1.cmml"><mi id="S3.F5.7.m2.1.1.2" xref="S3.F5.7.m2.1.1.2.cmml">t</mi><mo stretchy="false" id="S3.F5.7.m2.1.1.1" xref="S3.F5.7.m2.1.1.1.cmml">→</mo><mi mathvariant="normal" id="S3.F5.7.m2.1.1.3" xref="S3.F5.7.m2.1.1.3.cmml">∞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.F5.7.m2.1c"><apply id="S3.F5.7.m2.1.1.cmml" xref="S3.F5.7.m2.1.1"><ci id="S3.F5.7.m2.1.1.1.cmml" xref="S3.F5.7.m2.1.1.1">→</ci><ci id="S3.F5.7.m2.1.1.2.cmml" xref="S3.F5.7.m2.1.1.2">𝑡</ci><infinity id="S3.F5.7.m2.1.1.3.cmml" xref="S3.F5.7.m2.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.7.m2.1d">t\rightarrow\infty</annotation></semantics></math> approaches uniform distribution, and <math id="S3.F5.8.m3.1" class="ltx_Math" alttext="t\rightarrow 0" display="inline"><semantics id="S3.F5.8.m3.1b"><mrow id="S3.F5.8.m3.1.1" xref="S3.F5.8.m3.1.1.cmml"><mi id="S3.F5.8.m3.1.1.2" xref="S3.F5.8.m3.1.1.2.cmml">t</mi><mo stretchy="false" id="S3.F5.8.m3.1.1.1" xref="S3.F5.8.m3.1.1.1.cmml">→</mo><mn id="S3.F5.8.m3.1.1.3" xref="S3.F5.8.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F5.8.m3.1c"><apply id="S3.F5.8.m3.1.1.cmml" xref="S3.F5.8.m3.1.1"><ci id="S3.F5.8.m3.1.1.1.cmml" xref="S3.F5.8.m3.1.1.1">→</ci><ci id="S3.F5.8.m3.1.1.2.cmml" xref="S3.F5.8.m3.1.1.2">𝑡</ci><cn type="integer" id="S3.F5.8.m3.1.1.3.cmml" xref="S3.F5.8.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.8.m3.1d">t\rightarrow 0</annotation></semantics></math> approaches <math id="S3.F5.9.m4.1" class="ltx_Math" alttext="\mathrm{argmax}" display="inline"><semantics id="S3.F5.9.m4.1b"><mi id="S3.F5.9.m4.1.1" xref="S3.F5.9.m4.1.1.cmml">argmax</mi><annotation-xml encoding="MathML-Content" id="S3.F5.9.m4.1c"><ci id="S3.F5.9.m4.1.1.cmml" xref="S3.F5.9.m4.1.1">argmax</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.9.m4.1d">\mathrm{argmax}</annotation></semantics></math>. (b) The learned <math id="S3.F5.10.m5.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.F5.10.m5.1b"><mi id="S3.F5.10.m5.1.1" xref="S3.F5.10.m5.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.F5.10.m5.1c"><ci id="S3.F5.10.m5.1.1.cmml" xref="S3.F5.10.m5.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.10.m5.1d">t</annotation></semantics></math> for each layer of ResNet18.
</figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3. </span>Scalar quantized lookup table</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.9" class="ltx_p">Lookup tables are the main disk and memory cost. We reduce the table size by scalar quantization (e.g., FP32 to INT8). We leverage the classic range-based linear quantization. The formula is <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="r=s(q-z)" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mrow id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml"><mi id="S3.SS3.p1.1.m1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.3.cmml">r</mi><mo id="S3.SS3.p1.1.m1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.2.cmml">=</mo><mrow id="S3.SS3.p1.1.m1.1.1.1" xref="S3.SS3.p1.1.m1.1.1.1.cmml"><mi id="S3.SS3.p1.1.m1.1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p1.1.m1.1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p1.1.m1.1.1.1.1.1" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p1.1.m1.1.1.1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS3.p1.1.m1.1.1.1.1.1.1" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p1.1.m1.1.1.1.1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.2.cmml">q</mi><mo id="S3.SS3.p1.1.m1.1.1.1.1.1.1.1" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.1.cmml">−</mo><mi id="S3.SS3.p1.1.m1.1.1.1.1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.3.cmml">z</mi></mrow><mo stretchy="false" id="S3.SS3.p1.1.m1.1.1.1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><apply id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1"><eq id="S3.SS3.p1.1.m1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1.2"></eq><ci id="S3.SS3.p1.1.m1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.3">𝑟</ci><apply id="S3.SS3.p1.1.m1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1.1"><times id="S3.SS3.p1.1.m1.1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1.1.2"></times><ci id="S3.SS3.p1.1.m1.1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.1.3">𝑠</ci><apply id="S3.SS3.p1.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1.1.1.1"><minus id="S3.SS3.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.1"></minus><ci id="S3.SS3.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.2">𝑞</ci><ci id="S3.SS3.p1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.1.1.1.1.3">𝑧</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">r=s(q-z)</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Jacob et al<span class="ltx_text">.</span>, <a href="#bib.bib30" title="" class="ltx_ref">2018</a>)</cite>, in which <math id="S3.SS3.p1.2.m2.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS3.p1.2.m2.1a"><mi id="S3.SS3.p1.2.m2.1.1" xref="S3.SS3.p1.2.m2.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.2.m2.1b"><ci id="S3.SS3.p1.2.m2.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.2.m2.1c">r</annotation></semantics></math> is the real value, <math id="S3.SS3.p1.3.m3.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS3.p1.3.m3.1a"><mi id="S3.SS3.p1.3.m3.1.1" xref="S3.SS3.p1.3.m3.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.3.m3.1b"><ci id="S3.SS3.p1.3.m3.1.1.cmml" xref="S3.SS3.p1.3.m3.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.3.m3.1c">s</annotation></semantics></math> is the scaling factor, <math id="S3.SS3.p1.4.m4.1" class="ltx_Math" alttext="q" display="inline"><semantics id="S3.SS3.p1.4.m4.1a"><mi id="S3.SS3.p1.4.m4.1.1" xref="S3.SS3.p1.4.m4.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.4.m4.1b"><ci id="S3.SS3.p1.4.m4.1.1.cmml" xref="S3.SS3.p1.4.m4.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.4.m4.1c">q</annotation></semantics></math> is the quantized value, and <math id="S3.SS3.p1.5.m5.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS3.p1.5.m5.1a"><mi id="S3.SS3.p1.5.m5.1.1" xref="S3.SS3.p1.5.m5.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.5.m5.1b"><ci id="S3.SS3.p1.5.m5.1.1.cmml" xref="S3.SS3.p1.5.m5.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.5.m5.1c">z</annotation></semantics></math> is the zero point.
We use symmetric quantization, so <math id="S3.SS3.p1.6.m6.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS3.p1.6.m6.1a"><mi id="S3.SS3.p1.6.m6.1.1" xref="S3.SS3.p1.6.m6.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.6.m6.1b"><ci id="S3.SS3.p1.6.m6.1.1.cmml" xref="S3.SS3.p1.6.m6.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.6.m6.1c">z</annotation></semantics></math> is forced to be 0, and the quantized range is <math id="S3.SS3.p1.7.m7.2" class="ltx_Math" alttext="[-2^{n-1},2^{n-1}-1]" display="inline"><semantics id="S3.SS3.p1.7.m7.2a"><mrow id="S3.SS3.p1.7.m7.2.2.2" xref="S3.SS3.p1.7.m7.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.p1.7.m7.2.2.2.3" xref="S3.SS3.p1.7.m7.2.2.3.cmml">[</mo><mrow id="S3.SS3.p1.7.m7.1.1.1.1" xref="S3.SS3.p1.7.m7.1.1.1.1.cmml"><mo id="S3.SS3.p1.7.m7.1.1.1.1a" xref="S3.SS3.p1.7.m7.1.1.1.1.cmml">−</mo><msup id="S3.SS3.p1.7.m7.1.1.1.1.2" xref="S3.SS3.p1.7.m7.1.1.1.1.2.cmml"><mn id="S3.SS3.p1.7.m7.1.1.1.1.2.2" xref="S3.SS3.p1.7.m7.1.1.1.1.2.2.cmml">2</mn><mrow id="S3.SS3.p1.7.m7.1.1.1.1.2.3" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.cmml"><mi id="S3.SS3.p1.7.m7.1.1.1.1.2.3.2" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.2.cmml">n</mi><mo id="S3.SS3.p1.7.m7.1.1.1.1.2.3.1" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.1.cmml">−</mo><mn id="S3.SS3.p1.7.m7.1.1.1.1.2.3.3" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.3.cmml">1</mn></mrow></msup></mrow><mo id="S3.SS3.p1.7.m7.2.2.2.4" xref="S3.SS3.p1.7.m7.2.2.3.cmml">,</mo><mrow id="S3.SS3.p1.7.m7.2.2.2.2" xref="S3.SS3.p1.7.m7.2.2.2.2.cmml"><msup id="S3.SS3.p1.7.m7.2.2.2.2.2" xref="S3.SS3.p1.7.m7.2.2.2.2.2.cmml"><mn id="S3.SS3.p1.7.m7.2.2.2.2.2.2" xref="S3.SS3.p1.7.m7.2.2.2.2.2.2.cmml">2</mn><mrow id="S3.SS3.p1.7.m7.2.2.2.2.2.3" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.cmml"><mi id="S3.SS3.p1.7.m7.2.2.2.2.2.3.2" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.2.cmml">n</mi><mo id="S3.SS3.p1.7.m7.2.2.2.2.2.3.1" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.1.cmml">−</mo><mn id="S3.SS3.p1.7.m7.2.2.2.2.2.3.3" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.3.cmml">1</mn></mrow></msup><mo id="S3.SS3.p1.7.m7.2.2.2.2.1" xref="S3.SS3.p1.7.m7.2.2.2.2.1.cmml">−</mo><mn id="S3.SS3.p1.7.m7.2.2.2.2.3" xref="S3.SS3.p1.7.m7.2.2.2.2.3.cmml">1</mn></mrow><mo stretchy="false" id="S3.SS3.p1.7.m7.2.2.2.5" xref="S3.SS3.p1.7.m7.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.7.m7.2b"><interval closure="closed" id="S3.SS3.p1.7.m7.2.2.3.cmml" xref="S3.SS3.p1.7.m7.2.2.2"><apply id="S3.SS3.p1.7.m7.1.1.1.1.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1"><minus id="S3.SS3.p1.7.m7.1.1.1.1.1.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1"></minus><apply id="S3.SS3.p1.7.m7.1.1.1.1.2.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p1.7.m7.1.1.1.1.2.1.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2">superscript</csymbol><cn type="integer" id="S3.SS3.p1.7.m7.1.1.1.1.2.2.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2.2">2</cn><apply id="S3.SS3.p1.7.m7.1.1.1.1.2.3.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3"><minus id="S3.SS3.p1.7.m7.1.1.1.1.2.3.1.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.1"></minus><ci id="S3.SS3.p1.7.m7.1.1.1.1.2.3.2.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.2">𝑛</ci><cn type="integer" id="S3.SS3.p1.7.m7.1.1.1.1.2.3.3.cmml" xref="S3.SS3.p1.7.m7.1.1.1.1.2.3.3">1</cn></apply></apply></apply><apply id="S3.SS3.p1.7.m7.2.2.2.2.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2"><minus id="S3.SS3.p1.7.m7.2.2.2.2.1.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.1"></minus><apply id="S3.SS3.p1.7.m7.2.2.2.2.2.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p1.7.m7.2.2.2.2.2.1.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2">superscript</csymbol><cn type="integer" id="S3.SS3.p1.7.m7.2.2.2.2.2.2.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2.2">2</cn><apply id="S3.SS3.p1.7.m7.2.2.2.2.2.3.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3"><minus id="S3.SS3.p1.7.m7.2.2.2.2.2.3.1.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.1"></minus><ci id="S3.SS3.p1.7.m7.2.2.2.2.2.3.2.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.2">𝑛</ci><cn type="integer" id="S3.SS3.p1.7.m7.2.2.2.2.2.3.3.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.2.3.3">1</cn></apply></apply><cn type="integer" id="S3.SS3.p1.7.m7.2.2.2.2.3.cmml" xref="S3.SS3.p1.7.m7.2.2.2.2.3">1</cn></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.7.m7.2c">[-2^{n-1},2^{n-1}-1]</annotation></semantics></math>. The scaling factor <math id="S3.SS3.p1.8.m8.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS3.p1.8.m8.1a"><mi id="S3.SS3.p1.8.m8.1.1" xref="S3.SS3.p1.8.m8.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.8.m8.1b"><ci id="S3.SS3.p1.8.m8.1.1.cmml" xref="S3.SS3.p1.8.m8.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.8.m8.1c">s</annotation></semantics></math> is calculated as the max absolute value in the table divided by half of the range, i.e., <math id="S3.SS3.p1.9.m9.1" class="ltx_Math" alttext="s=\frac{\text{max(|value|)}}{2^{n-1}-1}" display="inline"><semantics id="S3.SS3.p1.9.m9.1a"><mrow id="S3.SS3.p1.9.m9.1.1" xref="S3.SS3.p1.9.m9.1.1.cmml"><mi id="S3.SS3.p1.9.m9.1.1.2" xref="S3.SS3.p1.9.m9.1.1.2.cmml">s</mi><mo id="S3.SS3.p1.9.m9.1.1.1" xref="S3.SS3.p1.9.m9.1.1.1.cmml">=</mo><mfrac id="S3.SS3.p1.9.m9.1.1.3" xref="S3.SS3.p1.9.m9.1.1.3.cmml"><mtext id="S3.SS3.p1.9.m9.1.1.3.2" xref="S3.SS3.p1.9.m9.1.1.3.2a.cmml">max(—value—)</mtext><mrow id="S3.SS3.p1.9.m9.1.1.3.3" xref="S3.SS3.p1.9.m9.1.1.3.3.cmml"><msup id="S3.SS3.p1.9.m9.1.1.3.3.2" xref="S3.SS3.p1.9.m9.1.1.3.3.2.cmml"><mn id="S3.SS3.p1.9.m9.1.1.3.3.2.2" xref="S3.SS3.p1.9.m9.1.1.3.3.2.2.cmml">2</mn><mrow id="S3.SS3.p1.9.m9.1.1.3.3.2.3" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.cmml"><mi id="S3.SS3.p1.9.m9.1.1.3.3.2.3.2" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.2.cmml">n</mi><mo id="S3.SS3.p1.9.m9.1.1.3.3.2.3.1" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.1.cmml">−</mo><mn id="S3.SS3.p1.9.m9.1.1.3.3.2.3.3" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.3.cmml">1</mn></mrow></msup><mo id="S3.SS3.p1.9.m9.1.1.3.3.1" xref="S3.SS3.p1.9.m9.1.1.3.3.1.cmml">−</mo><mn id="S3.SS3.p1.9.m9.1.1.3.3.3" xref="S3.SS3.p1.9.m9.1.1.3.3.3.cmml">1</mn></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.9.m9.1b"><apply id="S3.SS3.p1.9.m9.1.1.cmml" xref="S3.SS3.p1.9.m9.1.1"><eq id="S3.SS3.p1.9.m9.1.1.1.cmml" xref="S3.SS3.p1.9.m9.1.1.1"></eq><ci id="S3.SS3.p1.9.m9.1.1.2.cmml" xref="S3.SS3.p1.9.m9.1.1.2">𝑠</ci><apply id="S3.SS3.p1.9.m9.1.1.3.cmml" xref="S3.SS3.p1.9.m9.1.1.3"><divide id="S3.SS3.p1.9.m9.1.1.3.1.cmml" xref="S3.SS3.p1.9.m9.1.1.3"></divide><ci id="S3.SS3.p1.9.m9.1.1.3.2a.cmml" xref="S3.SS3.p1.9.m9.1.1.3.2"><mtext mathsize="70%" id="S3.SS3.p1.9.m9.1.1.3.2.cmml" xref="S3.SS3.p1.9.m9.1.1.3.2">max(—value—)</mtext></ci><apply id="S3.SS3.p1.9.m9.1.1.3.3.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3"><minus id="S3.SS3.p1.9.m9.1.1.3.3.1.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.1"></minus><apply id="S3.SS3.p1.9.m9.1.1.3.3.2.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS3.p1.9.m9.1.1.3.3.2.1.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2">superscript</csymbol><cn type="integer" id="S3.SS3.p1.9.m9.1.1.3.3.2.2.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2.2">2</cn><apply id="S3.SS3.p1.9.m9.1.1.3.3.2.3.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3"><minus id="S3.SS3.p1.9.m9.1.1.3.3.2.3.1.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.1"></minus><ci id="S3.SS3.p1.9.m9.1.1.3.3.2.3.2.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.2">𝑛</ci><cn type="integer" id="S3.SS3.p1.9.m9.1.1.3.3.2.3.3.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.2.3.3">1</cn></apply></apply><cn type="integer" id="S3.SS3.p1.9.m9.1.1.3.3.3.cmml" xref="S3.SS3.p1.9.m9.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.9.m9.1c">s=\frac{\text{max(|value|)}}{2^{n-1}-1}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">Quantized lookup tables introduce another level of approximation. Similar to <code id="S3.SS3.p2.1.1" class="ltx_verbatim ltx_font_typewriter">temperature</code>, we thus quantize the tables during centroid learning, to minimize the loss function. Inspired by Jacob <span id="S3.SS3.p2.1.2" class="ltx_text"><em id="S3.SS3.p2.1.2.1" class="ltx_emph ltx_font_italic">et al.</em></span> <cite class="ltx_cite ltx_citemacro_citep">(Jacob et al<span class="ltx_text">.</span>, <a href="#bib.bib30" title="" class="ltx_ref">2018</a>)</cite>, the backpropagation uses lookup tables in real values, so that they can be adjusted in small
amounts. The forward pass uses quantized lookup tables as in the inference to calculate the loss. Results show that by this learning method, the quantized lookup table has little impact on the model accuracy.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4. </span>cost reduction of LUT-NN</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">Before introducing the inference design of LUT-NN, this section analyzes the theoretical FLOPs and model size reduction by LUT-NN.
The two primary factors of LUT-NN are the number of centroids and the length of sub-vectors, representing a tradeoff between cost and accuracy.
We conduct the analysis with these two factors.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p">According to the output formula Eq. <a href="#S2.E4" title="In 2.2. PQ for AMM ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> of a PQ-based AMM in LUT-NN, the main cost is from the encoding function <math id="S4.p2.1.m1.1" class="ltx_Math" alttext="g^{c}(a^{c})" display="inline"><semantics id="S4.p2.1.m1.1a"><mrow id="S4.p2.1.m1.1.1" xref="S4.p2.1.m1.1.1.cmml"><msup id="S4.p2.1.m1.1.1.3" xref="S4.p2.1.m1.1.1.3.cmml"><mi id="S4.p2.1.m1.1.1.3.2" xref="S4.p2.1.m1.1.1.3.2.cmml">g</mi><mi id="S4.p2.1.m1.1.1.3.3" xref="S4.p2.1.m1.1.1.3.3.cmml">c</mi></msup><mo lspace="0em" rspace="0em" id="S4.p2.1.m1.1.1.2" xref="S4.p2.1.m1.1.1.2.cmml">​</mo><mrow id="S4.p2.1.m1.1.1.1.1" xref="S4.p2.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p2.1.m1.1.1.1.1.2" xref="S4.p2.1.m1.1.1.1.1.1.cmml">(</mo><msup id="S4.p2.1.m1.1.1.1.1.1" xref="S4.p2.1.m1.1.1.1.1.1.cmml"><mi id="S4.p2.1.m1.1.1.1.1.1.2" xref="S4.p2.1.m1.1.1.1.1.1.2.cmml">a</mi><mi id="S4.p2.1.m1.1.1.1.1.1.3" xref="S4.p2.1.m1.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S4.p2.1.m1.1.1.1.1.3" xref="S4.p2.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.1.m1.1b"><apply id="S4.p2.1.m1.1.1.cmml" xref="S4.p2.1.m1.1.1"><times id="S4.p2.1.m1.1.1.2.cmml" xref="S4.p2.1.m1.1.1.2"></times><apply id="S4.p2.1.m1.1.1.3.cmml" xref="S4.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.p2.1.m1.1.1.3.1.cmml" xref="S4.p2.1.m1.1.1.3">superscript</csymbol><ci id="S4.p2.1.m1.1.1.3.2.cmml" xref="S4.p2.1.m1.1.1.3.2">𝑔</ci><ci id="S4.p2.1.m1.1.1.3.3.cmml" xref="S4.p2.1.m1.1.1.3.3">𝑐</ci></apply><apply id="S4.p2.1.m1.1.1.1.1.1.cmml" xref="S4.p2.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S4.p2.1.m1.1.1.1.1.1.1.cmml" xref="S4.p2.1.m1.1.1.1.1">superscript</csymbol><ci id="S4.p2.1.m1.1.1.1.1.1.2.cmml" xref="S4.p2.1.m1.1.1.1.1.1.2">𝑎</ci><ci id="S4.p2.1.m1.1.1.1.1.1.3.cmml" xref="S4.p2.1.m1.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.1.m1.1c">g^{c}(a^{c})</annotation></semantics></math>, which calculates the Euclidean distance of the sub-vector with each centroid. After that, the cost is from table lookup with the encoding result (i.e., index of the closest centroid), and the result aggregation of sub-vectors. For file size, the major cost is from the lookup tables, which saves the dot product result of each centroid and the according sub-vectors in the weight matrix. The size of codebooks is relatively small, since the sub-vectors on the same column share one codebook.</p>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.6" class="ltx_p">Therefore, we analyze the FLOPs of encoding, table lookup and aggregation, as well as the size of lookup tables as the cost of a LUT-NN AMM, to compare it with normal MM in Table <a href="#S4.T1" title="Table 1 ‣ 4. cost reduction of LUT-NN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Since convolution can be transformed to MM by im2col, its cost also follows these formulas. For a convolution, <math id="S4.p3.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p3.1.m1.1a"><mi id="S4.p3.1.m1.1.1" xref="S4.p3.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p3.1.m1.1b"><ci id="S4.p3.1.m1.1.1.cmml" xref="S4.p3.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.1.m1.1c">M</annotation></semantics></math> is the number of output channels, <math id="S4.p3.2.m2.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S4.p3.2.m2.1a"><mi id="S4.p3.2.m2.1.1" xref="S4.p3.2.m2.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S4.p3.2.m2.1b"><ci id="S4.p3.2.m2.1.1.cmml" xref="S4.p3.2.m2.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.2.m2.1c">D</annotation></semantics></math> is the number of input channels <math id="S4.p3.3.m3.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.p3.3.m3.1a"><mo id="S4.p3.3.m3.1.1" xref="S4.p3.3.m3.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.p3.3.m3.1b"><times id="S4.p3.3.m3.1.1.cmml" xref="S4.p3.3.m3.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.3.m3.1c">\times</annotation></semantics></math> filter size<sup id="S4.p3.6.1" class="ltx_sup">2</sup>, and <math id="S4.p3.5.m5.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S4.p3.5.m5.1a"><mi id="S4.p3.5.m5.1.1" xref="S4.p3.5.m5.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S4.p3.5.m5.1b"><ci id="S4.p3.5.m5.1.1.cmml" xref="S4.p3.5.m5.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.5.m5.1c">N</annotation></semantics></math> is height <math id="S4.p3.6.m6.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.p3.6.m6.1a"><mo id="S4.p3.6.m6.1.1" xref="S4.p3.6.m6.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.p3.6.m6.1b"><times id="S4.p3.6.m6.1.1.cmml" xref="S4.p3.6.m6.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.6.m6.1c">\times</annotation></semantics></math> width.</p>
</div>
<div id="S4.p4" class="ltx_para">
<p id="S4.p4.7" class="ltx_p">The number of centroids <math id="S4.p4.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.p4.1.m1.1a"><mi id="S4.p4.1.m1.1.1" xref="S4.p4.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.p4.1.m1.1b"><ci id="S4.p4.1.m1.1.1.cmml" xref="S4.p4.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.1.m1.1c">K</annotation></semantics></math> and the sub-vector length <math id="S4.p4.2.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.p4.2.m2.1a"><mi id="S4.p4.2.m2.1.1" xref="S4.p4.2.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.p4.2.m2.1b"><ci id="S4.p4.2.m2.1.1.cmml" xref="S4.p4.2.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.2.m2.1c">V</annotation></semantics></math> are two hyperparameters of LUT-NN. They are tradeoffs between accuracy and cost (refer to Sec. <a href="#S6.SS3" title="6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.3</span></a>).
The more centroids <math id="S4.p4.3.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.p4.3.m3.1a"><mi id="S4.p4.3.m3.1.1" xref="S4.p4.3.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.p4.3.m3.1b"><ci id="S4.p4.3.m3.1.1.cmml" xref="S4.p4.3.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.3.m3.1c">K</annotation></semantics></math> and shorter sub-vector <math id="S4.p4.4.m4.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.p4.4.m4.1a"><mi id="S4.p4.4.m4.1.1" xref="S4.p4.4.m4.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.p4.4.m4.1b"><ci id="S4.p4.4.m4.1.1.cmml" xref="S4.p4.4.m4.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.4.m4.1c">V</annotation></semantics></math> may lead to higher accuracy, but will increase the cost of LUT-NN. Table <a href="#S4.T2" title="Table 2 ‣ 4. cost reduction of LUT-NN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows the calculated GFLOPs and model size by the formulas in Table <a href="#S4.T1" title="Table 1 ‣ 4. cost reduction of LUT-NN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> for different models, using typical <math id="S4.p4.5.m5.2" class="ltx_Math" alttext="(K,V)" display="inline"><semantics id="S4.p4.5.m5.2a"><mrow id="S4.p4.5.m5.2.3.2" xref="S4.p4.5.m5.2.3.1.cmml"><mo stretchy="false" id="S4.p4.5.m5.2.3.2.1" xref="S4.p4.5.m5.2.3.1.cmml">(</mo><mi id="S4.p4.5.m5.1.1" xref="S4.p4.5.m5.1.1.cmml">K</mi><mo id="S4.p4.5.m5.2.3.2.2" xref="S4.p4.5.m5.2.3.1.cmml">,</mo><mi id="S4.p4.5.m5.2.2" xref="S4.p4.5.m5.2.2.cmml">V</mi><mo stretchy="false" id="S4.p4.5.m5.2.3.2.3" xref="S4.p4.5.m5.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.5.m5.2b"><interval closure="open" id="S4.p4.5.m5.2.3.1.cmml" xref="S4.p4.5.m5.2.3.2"><ci id="S4.p4.5.m5.1.1.cmml" xref="S4.p4.5.m5.1.1">𝐾</ci><ci id="S4.p4.5.m5.2.2.cmml" xref="S4.p4.5.m5.2.2">𝑉</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.5.m5.2c">(K,V)</annotation></semantics></math> settings in LUT-NN. These typical settings can achieve comparable accuracy with the original model, and also align with the SIMD width for high performance. Similar to other hyperparameters in DNN training, <math id="S4.p4.6.m6.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.p4.6.m6.1a"><mi id="S4.p4.6.m6.1.1" xref="S4.p4.6.m6.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.p4.6.m6.1b"><ci id="S4.p4.6.m6.1.1.cmml" xref="S4.p4.6.m6.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.6.m6.1c">K</annotation></semantics></math> and <math id="S4.p4.7.m7.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.p4.7.m7.1a"><mi id="S4.p4.7.m7.1.1" xref="S4.p4.7.m7.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.p4.7.m7.1b"><ci id="S4.p4.7.m7.1.1.cmml" xref="S4.p4.7.m7.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.7.m7.1c">V</annotation></semantics></math> can be set by grid search, evolutionary search, or other popular searching methods.</p>
</div>
<div id="S4.p5" class="ltx_para">
<p id="S4.p5.7" class="ltx_p">It is clear that LUT-NN can achieve both computation and model size saving. The FLOPs saving is because <math id="S4.p5.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.p5.1.m1.1a"><mi id="S4.p5.1.m1.1.1" xref="S4.p5.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.p5.1.m1.1b"><ci id="S4.p5.1.m1.1.1.cmml" xref="S4.p5.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.1.m1.1c">K</annotation></semantics></math> is normally smaller than <math id="S4.p5.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p5.2.m2.1a"><mi id="S4.p5.2.m2.1.1" xref="S4.p5.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p5.2.m2.1b"><ci id="S4.p5.2.m2.1.1.cmml" xref="S4.p5.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.2.m2.1c">M</annotation></semantics></math>. For example, the number of output channels i.e., <math id="S4.p5.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p5.3.m3.1a"><mi id="S4.p5.3.m3.1.1" xref="S4.p5.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p5.3.m3.1b"><ci id="S4.p5.3.m3.1.1.cmml" xref="S4.p5.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.3.m3.1c">M</annotation></semantics></math>, for ResNet50 is normally 128, 256, or 512, so the FLOPs can be reduced by <math id="S4.p5.4.m4.1" class="ltx_math_unparsed" alttext="4\times" display="inline"><semantics id="S4.p5.4.m4.1a"><mrow id="S4.p5.4.m4.1b"><mn id="S4.p5.4.m4.1.1">4</mn><mo lspace="0.222em" id="S4.p5.4.m4.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S4.p5.4.m4.1c">4\times</annotation></semantics></math> when <math id="S4.p5.5.m5.1" class="ltx_Math" alttext="K=8" display="inline"><semantics id="S4.p5.5.m5.1a"><mrow id="S4.p5.5.m5.1.1" xref="S4.p5.5.m5.1.1.cmml"><mi id="S4.p5.5.m5.1.1.2" xref="S4.p5.5.m5.1.1.2.cmml">K</mi><mo id="S4.p5.5.m5.1.1.1" xref="S4.p5.5.m5.1.1.1.cmml">=</mo><mn id="S4.p5.5.m5.1.1.3" xref="S4.p5.5.m5.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.5.m5.1b"><apply id="S4.p5.5.m5.1.1.cmml" xref="S4.p5.5.m5.1.1"><eq id="S4.p5.5.m5.1.1.1.cmml" xref="S4.p5.5.m5.1.1.1"></eq><ci id="S4.p5.5.m5.1.1.2.cmml" xref="S4.p5.5.m5.1.1.2">𝐾</ci><cn type="integer" id="S4.p5.5.m5.1.1.3.cmml" xref="S4.p5.5.m5.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.5.m5.1c">K=8</annotation></semantics></math>. For ResNet20, the number of output channel is 16, 32, and 64, so the FLOPs is reduced by <math id="S4.p5.6.m6.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S4.p5.6.m6.1a"><mrow id="S4.p5.6.m6.1b"><mn id="S4.p5.6.m6.1.1">2</mn><mo lspace="0.222em" id="S4.p5.6.m6.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S4.p5.6.m6.1c">2\times</annotation></semantics></math> when <math id="S4.p5.7.m7.1" class="ltx_Math" alttext="K=8" display="inline"><semantics id="S4.p5.7.m7.1a"><mrow id="S4.p5.7.m7.1.1" xref="S4.p5.7.m7.1.1.cmml"><mi id="S4.p5.7.m7.1.1.2" xref="S4.p5.7.m7.1.1.2.cmml">K</mi><mo id="S4.p5.7.m7.1.1.1" xref="S4.p5.7.m7.1.1.1.cmml">=</mo><mn id="S4.p5.7.m7.1.1.3" xref="S4.p5.7.m7.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.7.m7.1b"><apply id="S4.p5.7.m7.1.1.cmml" xref="S4.p5.7.m7.1.1"><eq id="S4.p5.7.m7.1.1.1.cmml" xref="S4.p5.7.m7.1.1.1"></eq><ci id="S4.p5.7.m7.1.1.2.cmml" xref="S4.p5.7.m7.1.1.2">𝐾</ci><cn type="integer" id="S4.p5.7.m7.1.1.3.cmml" xref="S4.p5.7.m7.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.7.m7.1c">K=8</annotation></semantics></math>.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<table id="S4.T1.10" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T1.1.1" class="ltx_tr">
<th id="S4.T1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" colspan="3">
<math id="S4.T1.1.1.1.m1.1" class="ltx_Math" alttext="A\in\mathbb{R}^{N\times D}" display="inline"><semantics id="S4.T1.1.1.1.m1.1a"><mrow id="S4.T1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.m1.1.1.cmml"><mi id="S4.T1.1.1.1.m1.1.1.2" xref="S4.T1.1.1.1.m1.1.1.2.cmml">A</mi><mo id="S4.T1.1.1.1.m1.1.1.1" xref="S4.T1.1.1.1.m1.1.1.1.cmml">∈</mo><msup id="S4.T1.1.1.1.m1.1.1.3" xref="S4.T1.1.1.1.m1.1.1.3.cmml"><mi id="S4.T1.1.1.1.m1.1.1.3.2" xref="S4.T1.1.1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S4.T1.1.1.1.m1.1.1.3.3" xref="S4.T1.1.1.1.m1.1.1.3.3.cmml"><mi id="S4.T1.1.1.1.m1.1.1.3.3.2" xref="S4.T1.1.1.1.m1.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.1.1.1.m1.1.1.3.3.1" xref="S4.T1.1.1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.T1.1.1.1.m1.1.1.3.3.3" xref="S4.T1.1.1.1.m1.1.1.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.m1.1b"><apply id="S4.T1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.m1.1.1"><in id="S4.T1.1.1.1.m1.1.1.1.cmml" xref="S4.T1.1.1.1.m1.1.1.1"></in><ci id="S4.T1.1.1.1.m1.1.1.2.cmml" xref="S4.T1.1.1.1.m1.1.1.2">𝐴</ci><apply id="S4.T1.1.1.1.m1.1.1.3.cmml" xref="S4.T1.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T1.1.1.1.m1.1.1.3.1.cmml" xref="S4.T1.1.1.1.m1.1.1.3">superscript</csymbol><ci id="S4.T1.1.1.1.m1.1.1.3.2.cmml" xref="S4.T1.1.1.1.m1.1.1.3.2">ℝ</ci><apply id="S4.T1.1.1.1.m1.1.1.3.3.cmml" xref="S4.T1.1.1.1.m1.1.1.3.3"><times id="S4.T1.1.1.1.m1.1.1.3.3.1.cmml" xref="S4.T1.1.1.1.m1.1.1.3.3.1"></times><ci id="S4.T1.1.1.1.m1.1.1.3.3.2.cmml" xref="S4.T1.1.1.1.m1.1.1.3.3.2">𝑁</ci><ci id="S4.T1.1.1.1.m1.1.1.3.3.3.cmml" xref="S4.T1.1.1.1.m1.1.1.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.m1.1c">A\in\mathbb{R}^{N\times D}</annotation></semantics></math>: input matrix</th>
</tr>
<tr id="S4.T1.2.2" class="ltx_tr">
<th id="S4.T1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" colspan="3">
<math id="S4.T1.2.2.1.m1.1" class="ltx_Math" alttext="B\in\mathbb{R}^{D\times M}" display="inline"><semantics id="S4.T1.2.2.1.m1.1a"><mrow id="S4.T1.2.2.1.m1.1.1" xref="S4.T1.2.2.1.m1.1.1.cmml"><mi id="S4.T1.2.2.1.m1.1.1.2" xref="S4.T1.2.2.1.m1.1.1.2.cmml">B</mi><mo id="S4.T1.2.2.1.m1.1.1.1" xref="S4.T1.2.2.1.m1.1.1.1.cmml">∈</mo><msup id="S4.T1.2.2.1.m1.1.1.3" xref="S4.T1.2.2.1.m1.1.1.3.cmml"><mi id="S4.T1.2.2.1.m1.1.1.3.2" xref="S4.T1.2.2.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S4.T1.2.2.1.m1.1.1.3.3" xref="S4.T1.2.2.1.m1.1.1.3.3.cmml"><mi id="S4.T1.2.2.1.m1.1.1.3.3.2" xref="S4.T1.2.2.1.m1.1.1.3.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.2.2.1.m1.1.1.3.3.1" xref="S4.T1.2.2.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.T1.2.2.1.m1.1.1.3.3.3" xref="S4.T1.2.2.1.m1.1.1.3.3.3.cmml">M</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.1.m1.1b"><apply id="S4.T1.2.2.1.m1.1.1.cmml" xref="S4.T1.2.2.1.m1.1.1"><in id="S4.T1.2.2.1.m1.1.1.1.cmml" xref="S4.T1.2.2.1.m1.1.1.1"></in><ci id="S4.T1.2.2.1.m1.1.1.2.cmml" xref="S4.T1.2.2.1.m1.1.1.2">𝐵</ci><apply id="S4.T1.2.2.1.m1.1.1.3.cmml" xref="S4.T1.2.2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T1.2.2.1.m1.1.1.3.1.cmml" xref="S4.T1.2.2.1.m1.1.1.3">superscript</csymbol><ci id="S4.T1.2.2.1.m1.1.1.3.2.cmml" xref="S4.T1.2.2.1.m1.1.1.3.2">ℝ</ci><apply id="S4.T1.2.2.1.m1.1.1.3.3.cmml" xref="S4.T1.2.2.1.m1.1.1.3.3"><times id="S4.T1.2.2.1.m1.1.1.3.3.1.cmml" xref="S4.T1.2.2.1.m1.1.1.3.3.1"></times><ci id="S4.T1.2.2.1.m1.1.1.3.3.2.cmml" xref="S4.T1.2.2.1.m1.1.1.3.3.2">𝐷</ci><ci id="S4.T1.2.2.1.m1.1.1.3.3.3.cmml" xref="S4.T1.2.2.1.m1.1.1.3.3.3">𝑀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.1.m1.1c">B\in\mathbb{R}^{D\times M}</annotation></semantics></math>: weight matrix</th>
</tr>
<tr id="S4.T1.4.4" class="ltx_tr">
<th id="S4.T1.4.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" colspan="3">
<math id="S4.T1.3.3.1.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.T1.3.3.1.m1.1a"><mi id="S4.T1.3.3.1.m1.1.1" xref="S4.T1.3.3.1.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.1.m1.1b"><ci id="S4.T1.3.3.1.m1.1.1.cmml" xref="S4.T1.3.3.1.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.1.m1.1c">V</annotation></semantics></math>: the length of a sub-vector <math id="S4.T1.4.4.2.m2.1" class="ltx_Math" alttext="a^{c}" display="inline"><semantics id="S4.T1.4.4.2.m2.1a"><msup id="S4.T1.4.4.2.m2.1.1" xref="S4.T1.4.4.2.m2.1.1.cmml"><mi id="S4.T1.4.4.2.m2.1.1.2" xref="S4.T1.4.4.2.m2.1.1.2.cmml">a</mi><mi id="S4.T1.4.4.2.m2.1.1.3" xref="S4.T1.4.4.2.m2.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.2.m2.1b"><apply id="S4.T1.4.4.2.m2.1.1.cmml" xref="S4.T1.4.4.2.m2.1.1"><csymbol cd="ambiguous" id="S4.T1.4.4.2.m2.1.1.1.cmml" xref="S4.T1.4.4.2.m2.1.1">superscript</csymbol><ci id="S4.T1.4.4.2.m2.1.1.2.cmml" xref="S4.T1.4.4.2.m2.1.1.2">𝑎</ci><ci id="S4.T1.4.4.2.m2.1.1.3.cmml" xref="S4.T1.4.4.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.2.m2.1c">a^{c}</annotation></semantics></math>
</th>
</tr>
<tr id="S4.T1.6.6" class="ltx_tr">
<th id="S4.T1.6.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" colspan="3">
<math id="S4.T1.5.5.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.T1.5.5.1.m1.1a"><mi id="S4.T1.5.5.1.m1.1.1" xref="S4.T1.5.5.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.1.m1.1b"><ci id="S4.T1.5.5.1.m1.1.1.cmml" xref="S4.T1.5.5.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.1.m1.1c">K</annotation></semantics></math>: the number of centroids in a codebook for <math id="S4.T1.6.6.2.m2.1" class="ltx_Math" alttext="a^{c}" display="inline"><semantics id="S4.T1.6.6.2.m2.1a"><msup id="S4.T1.6.6.2.m2.1.1" xref="S4.T1.6.6.2.m2.1.1.cmml"><mi id="S4.T1.6.6.2.m2.1.1.2" xref="S4.T1.6.6.2.m2.1.1.2.cmml">a</mi><mi id="S4.T1.6.6.2.m2.1.1.3" xref="S4.T1.6.6.2.m2.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S4.T1.6.6.2.m2.1b"><apply id="S4.T1.6.6.2.m2.1.1.cmml" xref="S4.T1.6.6.2.m2.1.1"><csymbol cd="ambiguous" id="S4.T1.6.6.2.m2.1.1.1.cmml" xref="S4.T1.6.6.2.m2.1.1">superscript</csymbol><ci id="S4.T1.6.6.2.m2.1.1.2.cmml" xref="S4.T1.6.6.2.m2.1.1.2">𝑎</ci><ci id="S4.T1.6.6.2.m2.1.1.3.cmml" xref="S4.T1.6.6.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.6.2.m2.1c">a^{c}</annotation></semantics></math>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T1.10.11.1" class="ltx_tr">
<th id="S4.T1.10.11.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t"></th>
<td id="S4.T1.10.11.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S4.T1.10.11.1.2.1" class="ltx_text ltx_font_bold">Ours</span></td>
<td id="S4.T1.10.11.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S4.T1.10.11.1.3.1" class="ltx_text ltx_font_bold">MM</span></td>
</tr>
<tr id="S4.T1.8.8" class="ltx_tr">
<th id="S4.T1.8.8.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t"><span id="S4.T1.8.8.3.1" class="ltx_text ltx_font_bold">FLOPs</span></th>
<td id="S4.T1.7.7.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S4.T1.7.7.1.m1.1" class="ltx_Math" alttext="N\cdot D\cdot K+N\cdot M\cdot D/V" display="inline"><semantics id="S4.T1.7.7.1.m1.1a"><mrow id="S4.T1.7.7.1.m1.1.1" xref="S4.T1.7.7.1.m1.1.1.cmml"><mrow id="S4.T1.7.7.1.m1.1.1.2" xref="S4.T1.7.7.1.m1.1.1.2.cmml"><mi id="S4.T1.7.7.1.m1.1.1.2.2" xref="S4.T1.7.7.1.m1.1.1.2.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.7.7.1.m1.1.1.2.1" xref="S4.T1.7.7.1.m1.1.1.2.1.cmml">⋅</mo><mi id="S4.T1.7.7.1.m1.1.1.2.3" xref="S4.T1.7.7.1.m1.1.1.2.3.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.7.7.1.m1.1.1.2.1a" xref="S4.T1.7.7.1.m1.1.1.2.1.cmml">⋅</mo><mi id="S4.T1.7.7.1.m1.1.1.2.4" xref="S4.T1.7.7.1.m1.1.1.2.4.cmml">K</mi></mrow><mo id="S4.T1.7.7.1.m1.1.1.1" xref="S4.T1.7.7.1.m1.1.1.1.cmml">+</mo><mrow id="S4.T1.7.7.1.m1.1.1.3" xref="S4.T1.7.7.1.m1.1.1.3.cmml"><mrow id="S4.T1.7.7.1.m1.1.1.3.2" xref="S4.T1.7.7.1.m1.1.1.3.2.cmml"><mi id="S4.T1.7.7.1.m1.1.1.3.2.2" xref="S4.T1.7.7.1.m1.1.1.3.2.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.7.7.1.m1.1.1.3.2.1" xref="S4.T1.7.7.1.m1.1.1.3.2.1.cmml">⋅</mo><mi id="S4.T1.7.7.1.m1.1.1.3.2.3" xref="S4.T1.7.7.1.m1.1.1.3.2.3.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.7.7.1.m1.1.1.3.2.1a" xref="S4.T1.7.7.1.m1.1.1.3.2.1.cmml">⋅</mo><mi id="S4.T1.7.7.1.m1.1.1.3.2.4" xref="S4.T1.7.7.1.m1.1.1.3.2.4.cmml">D</mi></mrow><mo id="S4.T1.7.7.1.m1.1.1.3.1" xref="S4.T1.7.7.1.m1.1.1.3.1.cmml">/</mo><mi id="S4.T1.7.7.1.m1.1.1.3.3" xref="S4.T1.7.7.1.m1.1.1.3.3.cmml">V</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.7.7.1.m1.1b"><apply id="S4.T1.7.7.1.m1.1.1.cmml" xref="S4.T1.7.7.1.m1.1.1"><plus id="S4.T1.7.7.1.m1.1.1.1.cmml" xref="S4.T1.7.7.1.m1.1.1.1"></plus><apply id="S4.T1.7.7.1.m1.1.1.2.cmml" xref="S4.T1.7.7.1.m1.1.1.2"><ci id="S4.T1.7.7.1.m1.1.1.2.1.cmml" xref="S4.T1.7.7.1.m1.1.1.2.1">⋅</ci><ci id="S4.T1.7.7.1.m1.1.1.2.2.cmml" xref="S4.T1.7.7.1.m1.1.1.2.2">𝑁</ci><ci id="S4.T1.7.7.1.m1.1.1.2.3.cmml" xref="S4.T1.7.7.1.m1.1.1.2.3">𝐷</ci><ci id="S4.T1.7.7.1.m1.1.1.2.4.cmml" xref="S4.T1.7.7.1.m1.1.1.2.4">𝐾</ci></apply><apply id="S4.T1.7.7.1.m1.1.1.3.cmml" xref="S4.T1.7.7.1.m1.1.1.3"><divide id="S4.T1.7.7.1.m1.1.1.3.1.cmml" xref="S4.T1.7.7.1.m1.1.1.3.1"></divide><apply id="S4.T1.7.7.1.m1.1.1.3.2.cmml" xref="S4.T1.7.7.1.m1.1.1.3.2"><ci id="S4.T1.7.7.1.m1.1.1.3.2.1.cmml" xref="S4.T1.7.7.1.m1.1.1.3.2.1">⋅</ci><ci id="S4.T1.7.7.1.m1.1.1.3.2.2.cmml" xref="S4.T1.7.7.1.m1.1.1.3.2.2">𝑁</ci><ci id="S4.T1.7.7.1.m1.1.1.3.2.3.cmml" xref="S4.T1.7.7.1.m1.1.1.3.2.3">𝑀</ci><ci id="S4.T1.7.7.1.m1.1.1.3.2.4.cmml" xref="S4.T1.7.7.1.m1.1.1.3.2.4">𝐷</ci></apply><ci id="S4.T1.7.7.1.m1.1.1.3.3.cmml" xref="S4.T1.7.7.1.m1.1.1.3.3">𝑉</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.7.1.m1.1c">N\cdot D\cdot K+N\cdot M\cdot D/V</annotation></semantics></math></td>
<td id="S4.T1.8.8.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S4.T1.8.8.2.m1.1" class="ltx_Math" alttext="N\cdot D\cdot M" display="inline"><semantics id="S4.T1.8.8.2.m1.1a"><mrow id="S4.T1.8.8.2.m1.1.1" xref="S4.T1.8.8.2.m1.1.1.cmml"><mi id="S4.T1.8.8.2.m1.1.1.2" xref="S4.T1.8.8.2.m1.1.1.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.8.8.2.m1.1.1.1" xref="S4.T1.8.8.2.m1.1.1.1.cmml">⋅</mo><mi id="S4.T1.8.8.2.m1.1.1.3" xref="S4.T1.8.8.2.m1.1.1.3.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.8.8.2.m1.1.1.1a" xref="S4.T1.8.8.2.m1.1.1.1.cmml">⋅</mo><mi id="S4.T1.8.8.2.m1.1.1.4" xref="S4.T1.8.8.2.m1.1.1.4.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.8.8.2.m1.1b"><apply id="S4.T1.8.8.2.m1.1.1.cmml" xref="S4.T1.8.8.2.m1.1.1"><ci id="S4.T1.8.8.2.m1.1.1.1.cmml" xref="S4.T1.8.8.2.m1.1.1.1">⋅</ci><ci id="S4.T1.8.8.2.m1.1.1.2.cmml" xref="S4.T1.8.8.2.m1.1.1.2">𝑁</ci><ci id="S4.T1.8.8.2.m1.1.1.3.cmml" xref="S4.T1.8.8.2.m1.1.1.3">𝐷</ci><ci id="S4.T1.8.8.2.m1.1.1.4.cmml" xref="S4.T1.8.8.2.m1.1.1.4">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.8.2.m1.1c">N\cdot D\cdot M</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.10.10" class="ltx_tr">
<th id="S4.T1.10.10.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r ltx_border_t"><span id="S4.T1.10.10.3.1" class="ltx_text ltx_font_bold">Size</span></th>
<td id="S4.T1.9.9.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><math id="S4.T1.9.9.1.m1.1" class="ltx_Math" alttext="4\cdot D\cdot K+D\cdot M\cdot K/\,V" display="inline"><semantics id="S4.T1.9.9.1.m1.1a"><mrow id="S4.T1.9.9.1.m1.1.1" xref="S4.T1.9.9.1.m1.1.1.cmml"><mrow id="S4.T1.9.9.1.m1.1.1.2" xref="S4.T1.9.9.1.m1.1.1.2.cmml"><mn id="S4.T1.9.9.1.m1.1.1.2.2" xref="S4.T1.9.9.1.m1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="S4.T1.9.9.1.m1.1.1.2.1" xref="S4.T1.9.9.1.m1.1.1.2.1.cmml">⋅</mo><mi id="S4.T1.9.9.1.m1.1.1.2.3" xref="S4.T1.9.9.1.m1.1.1.2.3.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.9.9.1.m1.1.1.2.1a" xref="S4.T1.9.9.1.m1.1.1.2.1.cmml">⋅</mo><mi id="S4.T1.9.9.1.m1.1.1.2.4" xref="S4.T1.9.9.1.m1.1.1.2.4.cmml">K</mi></mrow><mo id="S4.T1.9.9.1.m1.1.1.1" xref="S4.T1.9.9.1.m1.1.1.1.cmml">+</mo><mrow id="S4.T1.9.9.1.m1.1.1.3" xref="S4.T1.9.9.1.m1.1.1.3.cmml"><mrow id="S4.T1.9.9.1.m1.1.1.3.2" xref="S4.T1.9.9.1.m1.1.1.3.2.cmml"><mi id="S4.T1.9.9.1.m1.1.1.3.2.2" xref="S4.T1.9.9.1.m1.1.1.3.2.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.9.9.1.m1.1.1.3.2.1" xref="S4.T1.9.9.1.m1.1.1.3.2.1.cmml">⋅</mo><mi id="S4.T1.9.9.1.m1.1.1.3.2.3" xref="S4.T1.9.9.1.m1.1.1.3.2.3.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.9.9.1.m1.1.1.3.2.1a" xref="S4.T1.9.9.1.m1.1.1.3.2.1.cmml">⋅</mo><mi id="S4.T1.9.9.1.m1.1.1.3.2.4" xref="S4.T1.9.9.1.m1.1.1.3.2.4.cmml">K</mi></mrow><mo rspace="0.392em" id="S4.T1.9.9.1.m1.1.1.3.1" xref="S4.T1.9.9.1.m1.1.1.3.1.cmml">/</mo><mi id="S4.T1.9.9.1.m1.1.1.3.3" xref="S4.T1.9.9.1.m1.1.1.3.3.cmml">V</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.9.9.1.m1.1b"><apply id="S4.T1.9.9.1.m1.1.1.cmml" xref="S4.T1.9.9.1.m1.1.1"><plus id="S4.T1.9.9.1.m1.1.1.1.cmml" xref="S4.T1.9.9.1.m1.1.1.1"></plus><apply id="S4.T1.9.9.1.m1.1.1.2.cmml" xref="S4.T1.9.9.1.m1.1.1.2"><ci id="S4.T1.9.9.1.m1.1.1.2.1.cmml" xref="S4.T1.9.9.1.m1.1.1.2.1">⋅</ci><cn type="integer" id="S4.T1.9.9.1.m1.1.1.2.2.cmml" xref="S4.T1.9.9.1.m1.1.1.2.2">4</cn><ci id="S4.T1.9.9.1.m1.1.1.2.3.cmml" xref="S4.T1.9.9.1.m1.1.1.2.3">𝐷</ci><ci id="S4.T1.9.9.1.m1.1.1.2.4.cmml" xref="S4.T1.9.9.1.m1.1.1.2.4">𝐾</ci></apply><apply id="S4.T1.9.9.1.m1.1.1.3.cmml" xref="S4.T1.9.9.1.m1.1.1.3"><divide id="S4.T1.9.9.1.m1.1.1.3.1.cmml" xref="S4.T1.9.9.1.m1.1.1.3.1"></divide><apply id="S4.T1.9.9.1.m1.1.1.3.2.cmml" xref="S4.T1.9.9.1.m1.1.1.3.2"><ci id="S4.T1.9.9.1.m1.1.1.3.2.1.cmml" xref="S4.T1.9.9.1.m1.1.1.3.2.1">⋅</ci><ci id="S4.T1.9.9.1.m1.1.1.3.2.2.cmml" xref="S4.T1.9.9.1.m1.1.1.3.2.2">𝐷</ci><ci id="S4.T1.9.9.1.m1.1.1.3.2.3.cmml" xref="S4.T1.9.9.1.m1.1.1.3.2.3">𝑀</ci><ci id="S4.T1.9.9.1.m1.1.1.3.2.4.cmml" xref="S4.T1.9.9.1.m1.1.1.3.2.4">𝐾</ci></apply><ci id="S4.T1.9.9.1.m1.1.1.3.3.cmml" xref="S4.T1.9.9.1.m1.1.1.3.3">𝑉</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.9.1.m1.1c">4\cdot D\cdot K+D\cdot M\cdot K/\,V</annotation></semantics></math></td>
<td id="S4.T1.10.10.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><math id="S4.T1.10.10.2.m1.1" class="ltx_Math" alttext="4\cdot D\cdot M" display="inline"><semantics id="S4.T1.10.10.2.m1.1a"><mrow id="S4.T1.10.10.2.m1.1.1" xref="S4.T1.10.10.2.m1.1.1.cmml"><mn id="S4.T1.10.10.2.m1.1.1.2" xref="S4.T1.10.10.2.m1.1.1.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="S4.T1.10.10.2.m1.1.1.1" xref="S4.T1.10.10.2.m1.1.1.1.cmml">⋅</mo><mi id="S4.T1.10.10.2.m1.1.1.3" xref="S4.T1.10.10.2.m1.1.1.3.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T1.10.10.2.m1.1.1.1a" xref="S4.T1.10.10.2.m1.1.1.1.cmml">⋅</mo><mi id="S4.T1.10.10.2.m1.1.1.4" xref="S4.T1.10.10.2.m1.1.1.4.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.10.10.2.m1.1b"><apply id="S4.T1.10.10.2.m1.1.1.cmml" xref="S4.T1.10.10.2.m1.1.1"><ci id="S4.T1.10.10.2.m1.1.1.1.cmml" xref="S4.T1.10.10.2.m1.1.1.1">⋅</ci><cn type="integer" id="S4.T1.10.10.2.m1.1.1.2.cmml" xref="S4.T1.10.10.2.m1.1.1.2">4</cn><ci id="S4.T1.10.10.2.m1.1.1.3.cmml" xref="S4.T1.10.10.2.m1.1.1.3">𝐷</ci><ci id="S4.T1.10.10.2.m1.1.1.4.cmml" xref="S4.T1.10.10.2.m1.1.1.4">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.10.2.m1.1c">4\cdot D\cdot M</annotation></semantics></math></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1. </span>FLOPs and disk size of a LUT-NN AMM compared to MM (FP32).</figcaption>
</figure>
<figure id="S4.T2" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S4.T2.5" class="ltx_tabular ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.5.1.1" class="ltx_tr">
<th id="S4.T2.5.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_l ltx_border_r ltx_border_t" rowspan="2"><span id="S4.T2.5.1.1.1.1" class="ltx_text">Model</span></th>
<th id="S4.T2.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="3">GFLOPs</th>
</tr>
<tr id="S4.T2.5.2.2" class="ltx_tr">
<th id="S4.T2.5.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">original</th>
<th id="S4.T2.5.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(8, 9)</th>
<th id="S4.T2.5.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">(16, 9)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.5.3.1" class="ltx_tr">
<th id="S4.T2.5.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">ResNet18 (CIFAR10)</th>
<td id="S4.T2.5.3.1.2" class="ltx_td ltx_align_center ltx_border_t">0.555</td>
<td id="S4.T2.5.3.1.3" class="ltx_td ltx_align_center ltx_border_t">0.098</td>
<td id="S4.T2.5.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.132</td>
</tr>
<tr id="S4.T2.5.4.2" class="ltx_tr">
<th id="S4.T2.5.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">SENet18 (CIFAR10)</th>
<td id="S4.T2.5.4.2.2" class="ltx_td ltx_align_center">0.556</td>
<td id="S4.T2.5.4.2.3" class="ltx_td ltx_align_center">0.098</td>
<td id="S4.T2.5.4.2.4" class="ltx_td ltx_align_center ltx_border_r">0.131</td>
</tr>
<tr id="S4.T2.5.5.3" class="ltx_tr">
<th id="S4.T2.5.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">VGG11 (CIFAR10)</th>
<td id="S4.T2.5.5.3.2" class="ltx_td ltx_align_center">0.606</td>
<td id="S4.T2.5.5.3.3" class="ltx_td ltx_align_center">0.085</td>
<td id="S4.T2.5.5.3.4" class="ltx_td ltx_align_center ltx_border_r">0.102</td>
</tr>
<tr id="S4.T2.5.6.4" class="ltx_tr">
<th id="S4.T2.5.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">ResNet18</th>
<td id="S4.T2.5.6.4.2" class="ltx_td ltx_align_center">1.814</td>
<td id="S4.T2.5.6.4.3" class="ltx_td ltx_align_center">0.412</td>
<td id="S4.T2.5.6.4.4" class="ltx_td ltx_align_center ltx_border_r">0.515</td>
</tr>
<tr id="S4.T2.5.7.5" class="ltx_tr">
<th id="S4.T2.5.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">SENet18</th>
<td id="S4.T2.5.7.5.2" class="ltx_td ltx_align_center">1.814</td>
<td id="S4.T2.5.7.5.3" class="ltx_td ltx_align_center">0.412</td>
<td id="S4.T2.5.7.5.4" class="ltx_td ltx_align_center ltx_border_r">0.515</td>
</tr>
<tr id="S4.T2.5.8.6" class="ltx_tr">
<th id="S4.T2.5.8.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">VGG11</th>
<td id="S4.T2.5.8.6.2" class="ltx_td ltx_align_center">7.609</td>
<td id="S4.T2.5.8.6.3" class="ltx_td ltx_align_center">1.132</td>
<td id="S4.T2.5.8.6.4" class="ltx_td ltx_align_center ltx_border_r">1.334</td>
</tr>
<tr id="S4.T2.5.9.7" class="ltx_tr">
<th id="S4.T2.5.9.7.1" class="ltx_td ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t"></th>
<td id="S4.T2.5.9.7.2" class="ltx_td ltx_align_center ltx_border_t">original</td>
<td id="S4.T2.5.9.7.3" class="ltx_td ltx_align_center ltx_border_t">(16, 32)</td>
<td id="S4.T2.5.9.7.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">(16, 16)</td>
</tr>
<tr id="S4.T2.5.10.8" class="ltx_tr">
<th id="S4.T2.5.10.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r ltx_border_t">BERT</th>
<td id="S4.T2.5.10.8.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">2.759</td>
<td id="S4.T2.5.10.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">0.169</td>
<td id="S4.T2.5.10.8.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">0.254</td>
</tr>
</tbody>
</table>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S4.T2.6" class="ltx_tabular ltx_centering ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.6.1.1" class="ltx_tr">
<th id="S4.T2.6.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_l ltx_border_r ltx_border_t" rowspan="2"><span id="S4.T2.6.1.1.1.1" class="ltx_text">Model</span></th>
<th id="S4.T2.6.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="3">Disk Size (MB)</th>
</tr>
<tr id="S4.T2.6.2.2" class="ltx_tr">
<th id="S4.T2.6.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">original</th>
<th id="S4.T2.6.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(8, 9)</th>
<th id="S4.T2.6.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">(16, 9)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.6.3.1" class="ltx_tr">
<th id="S4.T2.6.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">ResNet18 (CIFAR10)</th>
<td id="S4.T2.6.3.1.2" class="ltx_td ltx_align_center ltx_border_t">42.59</td>
<td id="S4.T2.6.3.1.3" class="ltx_td ltx_align_center ltx_border_t">12.55</td>
<td id="S4.T2.6.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">23.13</td>
</tr>
<tr id="S4.T2.6.4.2" class="ltx_tr">
<th id="S4.T2.6.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">SENet18 (CIFAR10)</th>
<td id="S4.T2.6.4.2.2" class="ltx_td ltx_align_center">42.92</td>
<td id="S4.T2.6.4.2.3" class="ltx_td ltx_align_center">10.94</td>
<td id="S4.T2.6.4.2.4" class="ltx_td ltx_align_center ltx_border_r">21.53</td>
</tr>
<tr id="S4.T2.6.5.3" class="ltx_tr">
<th id="S4.T2.6.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">VGG11 (CIFAR10)</th>
<td id="S4.T2.6.5.3.2" class="ltx_td ltx_align_center">35.18</td>
<td id="S4.T2.6.5.3.3" class="ltx_td ltx_align_center">8.45</td>
<td id="S4.T2.6.5.3.4" class="ltx_td ltx_align_center ltx_border_r">16.88</td>
</tr>
<tr id="S4.T2.6.6.4" class="ltx_tr">
<th id="S4.T2.6.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">ResNet18</th>
<td id="S4.T2.6.6.4.2" class="ltx_td ltx_align_center">44.55</td>
<td id="S4.T2.6.6.4.3" class="ltx_td ltx_align_center">12.57</td>
<td id="S4.T2.6.6.4.4" class="ltx_td ltx_align_center ltx_border_r">23.16</td>
</tr>
<tr id="S4.T2.6.7.5" class="ltx_tr">
<th id="S4.T2.6.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">SENet18</th>
<td id="S4.T2.6.7.5.2" class="ltx_td ltx_align_center">44.88</td>
<td id="S4.T2.6.7.5.3" class="ltx_td ltx_align_center">12.91</td>
<td id="S4.T2.6.7.5.4" class="ltx_td ltx_align_center ltx_border_r">23.49</td>
</tr>
<tr id="S4.T2.6.8.6" class="ltx_tr">
<th id="S4.T2.6.8.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_l ltx_border_r">VGG11</th>
<td id="S4.T2.6.8.6.2" class="ltx_td ltx_align_center">37.12</td>
<td id="S4.T2.6.8.6.3" class="ltx_td ltx_align_center">10.39</td>
<td id="S4.T2.6.8.6.4" class="ltx_td ltx_align_center ltx_border_r">18.82</td>
</tr>
<tr id="S4.T2.6.9.7" class="ltx_tr">
<th id="S4.T2.6.9.7.1" class="ltx_td ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t"></th>
<td id="S4.T2.6.9.7.2" class="ltx_td ltx_align_center ltx_border_t">original</td>
<td id="S4.T2.6.9.7.3" class="ltx_td ltx_align_center ltx_border_t">(16, 32)</td>
<td id="S4.T2.6.9.7.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">(16, 16)</td>
</tr>
<tr id="S4.T2.6.10.8" class="ltx_tr">
<th id="S4.T2.6.10.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r ltx_border_t">BERT</th>
<td id="S4.T2.6.10.8.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">162.26</td>
<td id="S4.T2.6.10.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">23.05</td>
<td id="S4.T2.6.10.8.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">43.30</td>
</tr>
</tbody>
</table>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2. </span>

Theoretical GFLOPs and model size of typical (<math id="S4.T2.3.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.T2.3.m1.1b"><mi id="S4.T2.3.m1.1.1" xref="S4.T2.3.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.T2.3.m1.1c"><ci id="S4.T2.3.m1.1.1.cmml" xref="S4.T2.3.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.m1.1d">K</annotation></semantics></math>, <math id="S4.T2.4.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.T2.4.m2.1b"><mi id="S4.T2.4.m2.1.1" xref="S4.T2.4.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.T2.4.m2.1c"><ci id="S4.T2.4.m2.1.1.cmml" xref="S4.T2.4.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.4.m2.1d">V</annotation></semantics></math>), calculated by formulas in Table <a href="#S4.T1" title="Table 1 ‣ 4. cost reduction of LUT-NN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
</figcaption>
</figure>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5. </span>Table lookup Inference Design</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this section, we present the design and optimization of the inference system that supports LUT-NN.
Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> depicts the LUT-NN model inference architecture, comprising the Closest Centroid Search stage and the Table Read and Accumulation stage.
In the Closest Centroid Search stage, LUT-NN first computes the distance between input tensors and centroids and searches the nearest centroids for each input tensor.
In the Table Read and Accumulation stage, LUT-NN reads the pre-computed results from the lookup table and accumulate results.
LUT-NN inference design fully takes advantage of the features of current hardware architectures to optimize the design and implementation of Table Lookup.
These optimizations improve inference by utilizing memory hierarchy, utilizing shuffle instructions, and utilizing mixed-precision accumulation instructions.</p>
</div>
<figure id="S5.F6" class="ltx_figure"><img src="/html/2302.03213/assets/x8.png" id="S5.F6.g1" class="ltx_graphics ltx_centering ltx_img_square" width="369" height="322" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6. </span>Table Lookup Inference Design</figcaption>
</figure>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1. </span>Closest centroid search</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">The centroid search takes the most computation costs in LUT-NN, and efficient centroid search is critical for performance efficiency compared to conventional computational methods.
The <span id="S5.SS1.p1.1.1" class="ltx_text ltx_font_bold">Closest Centroid Search</span> stage first calculates the distance between input tensors and centroids.
We can represent this calculation as matrix multiplications of input tensors and codebook matrices.
Then, it searches for the nearest centroids for each input sub-vector.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.7" class="ltx_p">However, the design of centroid search in LUT-NN presents challenges that involve leveraging the features of existing hardware architectures.
Firstly, LUT-NN distance computation is irregular-shaped (tall-and-skinny) and difficult to optimize using BLAS libraries.
For example, the XNNPACK library achieves only <math id="S5.SS1.p2.1.m1.1" class="ltx_Math" alttext="23.0" display="inline"><semantics id="S5.SS1.p2.1.m1.1a"><mn id="S5.SS1.p2.1.m1.1.1" xref="S5.SS1.p2.1.m1.1.1.cmml">23.0</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.1.m1.1b"><cn type="float" id="S5.SS1.p2.1.m1.1.1.cmml" xref="S5.SS1.p2.1.m1.1.1">23.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.1.m1.1c">23.0</annotation></semantics></math> GFLOP/s on Pixel 6 for the second layer of LUT-NN based ResNet18, which accounts for only <math id="S5.SS1.p2.2.m2.1" class="ltx_Math" alttext="25.7\%" display="inline"><semantics id="S5.SS1.p2.2.m2.1a"><mrow id="S5.SS1.p2.2.m2.1.1" xref="S5.SS1.p2.2.m2.1.1.cmml"><mn id="S5.SS1.p2.2.m2.1.1.2" xref="S5.SS1.p2.2.m2.1.1.2.cmml">25.7</mn><mo id="S5.SS1.p2.2.m2.1.1.1" xref="S5.SS1.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.2.m2.1b"><apply id="S5.SS1.p2.2.m2.1.1.cmml" xref="S5.SS1.p2.2.m2.1.1"><csymbol cd="latexml" id="S5.SS1.p2.2.m2.1.1.1.cmml" xref="S5.SS1.p2.2.m2.1.1.1">percent</csymbol><cn type="float" id="S5.SS1.p2.2.m2.1.1.2.cmml" xref="S5.SS1.p2.2.m2.1.1.2">25.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.2.m2.1c">25.7\%</annotation></semantics></math> of peak performance.
The height of the input tensor (<math id="S5.SS1.p2.3.m3.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S5.SS1.p2.3.m3.1a"><mi id="S5.SS1.p2.3.m3.1.1" xref="S5.SS1.p2.3.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.3.m3.1b"><ci id="S5.SS1.p2.3.m3.1.1.cmml" xref="S5.SS1.p2.3.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.3.m3.1c">N</annotation></semantics></math>) is usually much larger than the length of the sub-vector (<math id="S5.SS1.p2.4.m4.1" class="ltx_Math" alttext="N\gg V" display="inline"><semantics id="S5.SS1.p2.4.m4.1a"><mrow id="S5.SS1.p2.4.m4.1.1" xref="S5.SS1.p2.4.m4.1.1.cmml"><mi id="S5.SS1.p2.4.m4.1.1.2" xref="S5.SS1.p2.4.m4.1.1.2.cmml">N</mi><mo id="S5.SS1.p2.4.m4.1.1.1" xref="S5.SS1.p2.4.m4.1.1.1.cmml">≫</mo><mi id="S5.SS1.p2.4.m4.1.1.3" xref="S5.SS1.p2.4.m4.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.4.m4.1b"><apply id="S5.SS1.p2.4.m4.1.1.cmml" xref="S5.SS1.p2.4.m4.1.1"><csymbol cd="latexml" id="S5.SS1.p2.4.m4.1.1.1.cmml" xref="S5.SS1.p2.4.m4.1.1.1">much-greater-than</csymbol><ci id="S5.SS1.p2.4.m4.1.1.2.cmml" xref="S5.SS1.p2.4.m4.1.1.2">𝑁</ci><ci id="S5.SS1.p2.4.m4.1.1.3.cmml" xref="S5.SS1.p2.4.m4.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.4.m4.1c">N\gg V</annotation></semantics></math>) and the number of centroids (<math id="S5.SS1.p2.5.m5.1" class="ltx_Math" alttext="N\gg K" display="inline"><semantics id="S5.SS1.p2.5.m5.1a"><mrow id="S5.SS1.p2.5.m5.1.1" xref="S5.SS1.p2.5.m5.1.1.cmml"><mi id="S5.SS1.p2.5.m5.1.1.2" xref="S5.SS1.p2.5.m5.1.1.2.cmml">N</mi><mo id="S5.SS1.p2.5.m5.1.1.1" xref="S5.SS1.p2.5.m5.1.1.1.cmml">≫</mo><mi id="S5.SS1.p2.5.m5.1.1.3" xref="S5.SS1.p2.5.m5.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.5.m5.1b"><apply id="S5.SS1.p2.5.m5.1.1.cmml" xref="S5.SS1.p2.5.m5.1.1"><csymbol cd="latexml" id="S5.SS1.p2.5.m5.1.1.1.cmml" xref="S5.SS1.p2.5.m5.1.1.1">much-greater-than</csymbol><ci id="S5.SS1.p2.5.m5.1.1.2.cmml" xref="S5.SS1.p2.5.m5.1.1.2">𝑁</ci><ci id="S5.SS1.p2.5.m5.1.1.3.cmml" xref="S5.SS1.p2.5.m5.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.5.m5.1c">N\gg K</annotation></semantics></math>) on each codebook.
Therefore, the operation intensity of the distance calculation can be approximated by <math id="S5.SS1.p2.6.m6.1" class="ltx_Math" alttext="\frac{2NVK}{NV+KV+NK}\approx\frac{2}{1/K+1/V}\text{FLOP/Byte}" display="inline"><semantics id="S5.SS1.p2.6.m6.1a"><mrow id="S5.SS1.p2.6.m6.1.1" xref="S5.SS1.p2.6.m6.1.1.cmml"><mfrac id="S5.SS1.p2.6.m6.1.1.2" xref="S5.SS1.p2.6.m6.1.1.2.cmml"><mrow id="S5.SS1.p2.6.m6.1.1.2.2" xref="S5.SS1.p2.6.m6.1.1.2.2.cmml"><mn id="S5.SS1.p2.6.m6.1.1.2.2.2" xref="S5.SS1.p2.6.m6.1.1.2.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.2.2.1" xref="S5.SS1.p2.6.m6.1.1.2.2.1.cmml">​</mo><mi id="S5.SS1.p2.6.m6.1.1.2.2.3" xref="S5.SS1.p2.6.m6.1.1.2.2.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.2.2.1a" xref="S5.SS1.p2.6.m6.1.1.2.2.1.cmml">​</mo><mi id="S5.SS1.p2.6.m6.1.1.2.2.4" xref="S5.SS1.p2.6.m6.1.1.2.2.4.cmml">V</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.2.2.1b" xref="S5.SS1.p2.6.m6.1.1.2.2.1.cmml">​</mo><mi id="S5.SS1.p2.6.m6.1.1.2.2.5" xref="S5.SS1.p2.6.m6.1.1.2.2.5.cmml">K</mi></mrow><mrow id="S5.SS1.p2.6.m6.1.1.2.3" xref="S5.SS1.p2.6.m6.1.1.2.3.cmml"><mrow id="S5.SS1.p2.6.m6.1.1.2.3.2" xref="S5.SS1.p2.6.m6.1.1.2.3.2.cmml"><mi id="S5.SS1.p2.6.m6.1.1.2.3.2.2" xref="S5.SS1.p2.6.m6.1.1.2.3.2.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.2.3.2.1" xref="S5.SS1.p2.6.m6.1.1.2.3.2.1.cmml">​</mo><mi id="S5.SS1.p2.6.m6.1.1.2.3.2.3" xref="S5.SS1.p2.6.m6.1.1.2.3.2.3.cmml">V</mi></mrow><mo id="S5.SS1.p2.6.m6.1.1.2.3.1" xref="S5.SS1.p2.6.m6.1.1.2.3.1.cmml">+</mo><mrow id="S5.SS1.p2.6.m6.1.1.2.3.3" xref="S5.SS1.p2.6.m6.1.1.2.3.3.cmml"><mi id="S5.SS1.p2.6.m6.1.1.2.3.3.2" xref="S5.SS1.p2.6.m6.1.1.2.3.3.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.2.3.3.1" xref="S5.SS1.p2.6.m6.1.1.2.3.3.1.cmml">​</mo><mi id="S5.SS1.p2.6.m6.1.1.2.3.3.3" xref="S5.SS1.p2.6.m6.1.1.2.3.3.3.cmml">V</mi></mrow><mo id="S5.SS1.p2.6.m6.1.1.2.3.1a" xref="S5.SS1.p2.6.m6.1.1.2.3.1.cmml">+</mo><mrow id="S5.SS1.p2.6.m6.1.1.2.3.4" xref="S5.SS1.p2.6.m6.1.1.2.3.4.cmml"><mi id="S5.SS1.p2.6.m6.1.1.2.3.4.2" xref="S5.SS1.p2.6.m6.1.1.2.3.4.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.2.3.4.1" xref="S5.SS1.p2.6.m6.1.1.2.3.4.1.cmml">​</mo><mi id="S5.SS1.p2.6.m6.1.1.2.3.4.3" xref="S5.SS1.p2.6.m6.1.1.2.3.4.3.cmml">K</mi></mrow></mrow></mfrac><mo id="S5.SS1.p2.6.m6.1.1.1" xref="S5.SS1.p2.6.m6.1.1.1.cmml">≈</mo><mrow id="S5.SS1.p2.6.m6.1.1.3" xref="S5.SS1.p2.6.m6.1.1.3.cmml"><mfrac id="S5.SS1.p2.6.m6.1.1.3.2" xref="S5.SS1.p2.6.m6.1.1.3.2.cmml"><mn id="S5.SS1.p2.6.m6.1.1.3.2.2" xref="S5.SS1.p2.6.m6.1.1.3.2.2.cmml">2</mn><mrow id="S5.SS1.p2.6.m6.1.1.3.2.3" xref="S5.SS1.p2.6.m6.1.1.3.2.3.cmml"><mrow id="S5.SS1.p2.6.m6.1.1.3.2.3.2" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.cmml"><mn id="S5.SS1.p2.6.m6.1.1.3.2.3.2.2" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.2.cmml">1</mn><mo id="S5.SS1.p2.6.m6.1.1.3.2.3.2.1" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.1.cmml">/</mo><mi id="S5.SS1.p2.6.m6.1.1.3.2.3.2.3" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.3.cmml">K</mi></mrow><mo id="S5.SS1.p2.6.m6.1.1.3.2.3.1" xref="S5.SS1.p2.6.m6.1.1.3.2.3.1.cmml">+</mo><mrow id="S5.SS1.p2.6.m6.1.1.3.2.3.3" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.cmml"><mn id="S5.SS1.p2.6.m6.1.1.3.2.3.3.2" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.2.cmml">1</mn><mo id="S5.SS1.p2.6.m6.1.1.3.2.3.3.1" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.1.cmml">/</mo><mi id="S5.SS1.p2.6.m6.1.1.3.2.3.3.3" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.3.cmml">V</mi></mrow></mrow></mfrac><mo lspace="0em" rspace="0em" id="S5.SS1.p2.6.m6.1.1.3.1" xref="S5.SS1.p2.6.m6.1.1.3.1.cmml">​</mo><mtext id="S5.SS1.p2.6.m6.1.1.3.3" xref="S5.SS1.p2.6.m6.1.1.3.3a.cmml">FLOP/Byte</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.6.m6.1b"><apply id="S5.SS1.p2.6.m6.1.1.cmml" xref="S5.SS1.p2.6.m6.1.1"><approx id="S5.SS1.p2.6.m6.1.1.1.cmml" xref="S5.SS1.p2.6.m6.1.1.1"></approx><apply id="S5.SS1.p2.6.m6.1.1.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2"><divide id="S5.SS1.p2.6.m6.1.1.2.1.cmml" xref="S5.SS1.p2.6.m6.1.1.2"></divide><apply id="S5.SS1.p2.6.m6.1.1.2.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2.2"><times id="S5.SS1.p2.6.m6.1.1.2.2.1.cmml" xref="S5.SS1.p2.6.m6.1.1.2.2.1"></times><cn type="integer" id="S5.SS1.p2.6.m6.1.1.2.2.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2.2.2">2</cn><ci id="S5.SS1.p2.6.m6.1.1.2.2.3.cmml" xref="S5.SS1.p2.6.m6.1.1.2.2.3">𝑁</ci><ci id="S5.SS1.p2.6.m6.1.1.2.2.4.cmml" xref="S5.SS1.p2.6.m6.1.1.2.2.4">𝑉</ci><ci id="S5.SS1.p2.6.m6.1.1.2.2.5.cmml" xref="S5.SS1.p2.6.m6.1.1.2.2.5">𝐾</ci></apply><apply id="S5.SS1.p2.6.m6.1.1.2.3.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3"><plus id="S5.SS1.p2.6.m6.1.1.2.3.1.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.1"></plus><apply id="S5.SS1.p2.6.m6.1.1.2.3.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.2"><times id="S5.SS1.p2.6.m6.1.1.2.3.2.1.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.2.1"></times><ci id="S5.SS1.p2.6.m6.1.1.2.3.2.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.2.2">𝑁</ci><ci id="S5.SS1.p2.6.m6.1.1.2.3.2.3.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.2.3">𝑉</ci></apply><apply id="S5.SS1.p2.6.m6.1.1.2.3.3.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.3"><times id="S5.SS1.p2.6.m6.1.1.2.3.3.1.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.3.1"></times><ci id="S5.SS1.p2.6.m6.1.1.2.3.3.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.3.2">𝐾</ci><ci id="S5.SS1.p2.6.m6.1.1.2.3.3.3.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.3.3">𝑉</ci></apply><apply id="S5.SS1.p2.6.m6.1.1.2.3.4.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.4"><times id="S5.SS1.p2.6.m6.1.1.2.3.4.1.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.4.1"></times><ci id="S5.SS1.p2.6.m6.1.1.2.3.4.2.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.4.2">𝑁</ci><ci id="S5.SS1.p2.6.m6.1.1.2.3.4.3.cmml" xref="S5.SS1.p2.6.m6.1.1.2.3.4.3">𝐾</ci></apply></apply></apply><apply id="S5.SS1.p2.6.m6.1.1.3.cmml" xref="S5.SS1.p2.6.m6.1.1.3"><times id="S5.SS1.p2.6.m6.1.1.3.1.cmml" xref="S5.SS1.p2.6.m6.1.1.3.1"></times><apply id="S5.SS1.p2.6.m6.1.1.3.2.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2"><divide id="S5.SS1.p2.6.m6.1.1.3.2.1.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2"></divide><cn type="integer" id="S5.SS1.p2.6.m6.1.1.3.2.2.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.2">2</cn><apply id="S5.SS1.p2.6.m6.1.1.3.2.3.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3"><plus id="S5.SS1.p2.6.m6.1.1.3.2.3.1.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.1"></plus><apply id="S5.SS1.p2.6.m6.1.1.3.2.3.2.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2"><divide id="S5.SS1.p2.6.m6.1.1.3.2.3.2.1.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.1"></divide><cn type="integer" id="S5.SS1.p2.6.m6.1.1.3.2.3.2.2.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.2">1</cn><ci id="S5.SS1.p2.6.m6.1.1.3.2.3.2.3.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.2.3">𝐾</ci></apply><apply id="S5.SS1.p2.6.m6.1.1.3.2.3.3.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3"><divide id="S5.SS1.p2.6.m6.1.1.3.2.3.3.1.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.1"></divide><cn type="integer" id="S5.SS1.p2.6.m6.1.1.3.2.3.3.2.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.2">1</cn><ci id="S5.SS1.p2.6.m6.1.1.3.2.3.3.3.cmml" xref="S5.SS1.p2.6.m6.1.1.3.2.3.3.3">𝑉</ci></apply></apply></apply><ci id="S5.SS1.p2.6.m6.1.1.3.3a.cmml" xref="S5.SS1.p2.6.m6.1.1.3.3"><mtext id="S5.SS1.p2.6.m6.1.1.3.3.cmml" xref="S5.SS1.p2.6.m6.1.1.3.3">FLOP/Byte</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.6.m6.1c">\frac{2NVK}{NV+KV+NK}\approx\frac{2}{1/K+1/V}\text{FLOP/Byte}</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Wang et al<span class="ltx_text">.</span>, <a href="#bib.bib56" title="" class="ltx_ref">2021</a>)</cite>.
Since the length of the sub-vector and the number of centroids are small in LUT-NN, the operation intensity <math id="S5.SS1.p2.7.m7.1" class="ltx_Math" alttext="\frac{2}{1/K+1/V}" display="inline"><semantics id="S5.SS1.p2.7.m7.1a"><mfrac id="S5.SS1.p2.7.m7.1.1" xref="S5.SS1.p2.7.m7.1.1.cmml"><mn id="S5.SS1.p2.7.m7.1.1.2" xref="S5.SS1.p2.7.m7.1.1.2.cmml">2</mn><mrow id="S5.SS1.p2.7.m7.1.1.3" xref="S5.SS1.p2.7.m7.1.1.3.cmml"><mrow id="S5.SS1.p2.7.m7.1.1.3.2" xref="S5.SS1.p2.7.m7.1.1.3.2.cmml"><mn id="S5.SS1.p2.7.m7.1.1.3.2.2" xref="S5.SS1.p2.7.m7.1.1.3.2.2.cmml">1</mn><mo id="S5.SS1.p2.7.m7.1.1.3.2.1" xref="S5.SS1.p2.7.m7.1.1.3.2.1.cmml">/</mo><mi id="S5.SS1.p2.7.m7.1.1.3.2.3" xref="S5.SS1.p2.7.m7.1.1.3.2.3.cmml">K</mi></mrow><mo id="S5.SS1.p2.7.m7.1.1.3.1" xref="S5.SS1.p2.7.m7.1.1.3.1.cmml">+</mo><mrow id="S5.SS1.p2.7.m7.1.1.3.3" xref="S5.SS1.p2.7.m7.1.1.3.3.cmml"><mn id="S5.SS1.p2.7.m7.1.1.3.3.2" xref="S5.SS1.p2.7.m7.1.1.3.3.2.cmml">1</mn><mo id="S5.SS1.p2.7.m7.1.1.3.3.1" xref="S5.SS1.p2.7.m7.1.1.3.3.1.cmml">/</mo><mi id="S5.SS1.p2.7.m7.1.1.3.3.3" xref="S5.SS1.p2.7.m7.1.1.3.3.3.cmml">V</mi></mrow></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.7.m7.1b"><apply id="S5.SS1.p2.7.m7.1.1.cmml" xref="S5.SS1.p2.7.m7.1.1"><divide id="S5.SS1.p2.7.m7.1.1.1.cmml" xref="S5.SS1.p2.7.m7.1.1"></divide><cn type="integer" id="S5.SS1.p2.7.m7.1.1.2.cmml" xref="S5.SS1.p2.7.m7.1.1.2">2</cn><apply id="S5.SS1.p2.7.m7.1.1.3.cmml" xref="S5.SS1.p2.7.m7.1.1.3"><plus id="S5.SS1.p2.7.m7.1.1.3.1.cmml" xref="S5.SS1.p2.7.m7.1.1.3.1"></plus><apply id="S5.SS1.p2.7.m7.1.1.3.2.cmml" xref="S5.SS1.p2.7.m7.1.1.3.2"><divide id="S5.SS1.p2.7.m7.1.1.3.2.1.cmml" xref="S5.SS1.p2.7.m7.1.1.3.2.1"></divide><cn type="integer" id="S5.SS1.p2.7.m7.1.1.3.2.2.cmml" xref="S5.SS1.p2.7.m7.1.1.3.2.2">1</cn><ci id="S5.SS1.p2.7.m7.1.1.3.2.3.cmml" xref="S5.SS1.p2.7.m7.1.1.3.2.3">𝐾</ci></apply><apply id="S5.SS1.p2.7.m7.1.1.3.3.cmml" xref="S5.SS1.p2.7.m7.1.1.3.3"><divide id="S5.SS1.p2.7.m7.1.1.3.3.1.cmml" xref="S5.SS1.p2.7.m7.1.1.3.3.1"></divide><cn type="integer" id="S5.SS1.p2.7.m7.1.1.3.3.2.cmml" xref="S5.SS1.p2.7.m7.1.1.3.3.2">1</cn><ci id="S5.SS1.p2.7.m7.1.1.3.3.3.cmml" xref="S5.SS1.p2.7.m7.1.1.3.3.3">𝑉</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.7.m7.1c">\frac{2}{1/K+1/V}</annotation></semantics></math> is also small.
The distance computation becomes a memory-intensive matrix multiplication.</p>
</div>
<div id="S5.SS1.p3" class="ltx_para">
<p id="S5.SS1.p3.2" class="ltx_p">Therefore, we focus primarily on optimizing memory access for centroid distance computations (❶ in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) to leverage memory hierarchy in current hardware.
To optimize memory access for centroid distance computations (❶ in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>), we reduce memory access overhead by keeping frequently accessed data in registers and caches as much as possible.
So we design a <em id="S5.SS1.p3.2.1" class="ltx_emph ltx_font_italic">centroid-stationary computation</em> scheme to reside centroid matrices in registers and reorder centroid matrix loads in the inner loop to keep them in the cache as long as possible.
The centroid-stationary computation keeps <math id="S5.SS1.p3.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S5.SS1.p3.1.m1.1a"><mi id="S5.SS1.p3.1.m1.1.1" xref="S5.SS1.p3.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p3.1.m1.1b"><ci id="S5.SS1.p3.1.m1.1.1.cmml" xref="S5.SS1.p3.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p3.1.m1.1c">K</annotation></semantics></math> centroids in the cache for each codebook, which only requires reading these centroids once from DRAM.
Consequently, it only requires reading an <math id="S5.SS1.p3.2.m2.1" class="ltx_Math" alttext="N\cdot V" display="inline"><semantics id="S5.SS1.p3.2.m2.1a"><mrow id="S5.SS1.p3.2.m2.1.1" xref="S5.SS1.p3.2.m2.1.1.cmml"><mi id="S5.SS1.p3.2.m2.1.1.2" xref="S5.SS1.p3.2.m2.1.1.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS1.p3.2.m2.1.1.1" xref="S5.SS1.p3.2.m2.1.1.1.cmml">⋅</mo><mi id="S5.SS1.p3.2.m2.1.1.3" xref="S5.SS1.p3.2.m2.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p3.2.m2.1b"><apply id="S5.SS1.p3.2.m2.1.1.cmml" xref="S5.SS1.p3.2.m2.1.1"><ci id="S5.SS1.p3.2.m2.1.1.1.cmml" xref="S5.SS1.p3.2.m2.1.1.1">⋅</ci><ci id="S5.SS1.p3.2.m2.1.1.2.cmml" xref="S5.SS1.p3.2.m2.1.1.2">𝑁</ci><ci id="S5.SS1.p3.2.m2.1.1.3.cmml" xref="S5.SS1.p3.2.m2.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p3.2.m2.1c">N\cdot V</annotation></semantics></math> input tensor from DRAM once, reducing memory bandwidth costs and improving performance.</p>
</div>
<div id="S5.SS1.p4" class="ltx_para">
<p id="S5.SS1.p4.1" class="ltx_p">Second, after computing centroid distances, the Closest Centroid Search stage must identify the nearest centroid for each input sub-vector and generate the centroid index (❷ in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).
It can be represented by an <code id="S5.SS1.p4.1.1" class="ltx_verbatim ltx_font_typewriter">argmin</code> function, which finds an index with the shortest distance.
However, searching for the nearest centroid for each input sub-vector is a data-dependent operation.
To find the closest one, we must compare each distance sequentially, which is RAW (Read After Write) dependent and hard to be parallelized on CPUs.</p>
</div>
<div id="S5.SS1.p5" class="ltx_para">
<p id="S5.SS1.p5.1" class="ltx_p">We propose<em id="S5.SS1.p5.1.1" class="ltx_emph ltx_font_italic">intra-codebook parallelism</em> to optimize the Closest Centroid Search stage.
Intra-codebook parallelism searches the nearest centroid for the input sub-vector on a codebook in parallel.
We slice a codebook into multiple sub-codebooks and compare each distance between sub-vector and centroids in each sub-codebook.
Then, we interleave each sub-codebook’s execution, and avoid data dependency in closest centroid search.
We merge the compared distances by reduction and find the index corresponding to the closest centroid.
It leverages instruction-level parallelism to improve hardware utilization and performance.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2. </span>Table read and accumulation</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">LUT-NN obtains the indices of the nearest centroids for the input sub-vectors after closest centroid search, and leverages <span id="S5.SS2.p1.1.1" class="ltx_text ltx_font_bold">Table Read and Accumulation</span> stage to compute the final results.
This stage first reads the pre-computed results from the corresponding lookup table through indices (❸ in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) and completes the computation by accumulation operation (❹ in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).
For example, convolution operators directly read out filter’s outputs from lookup tables and accumulate each input channel’s result for output channels.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.3" class="ltx_p">However, Table Read and Accumulation stage introduces additional overhead in inference.
We enhance inference efficiency by skillfully utilizing widely available and supported instructions in commercial hardware.
First, table read is difficult to parallelize and introduces additional indirect memory accesses, which exaggerate the memory overhead on lookup tables.
Since we quantize lookup table into INT8 in Sec. <a href="#S3.SS3" title="3.3. Scalar quantized lookup table ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>, we leverage widely supported SIMD <em id="S5.SS2.p2.3.1" class="ltx_emph ltx_font_italic">shuffle</em> instructions (x86: <code id="S5.SS2.p2.3.2" class="ltx_verbatim ltx_font_typewriter">pshfb</code> and ARM: <code id="S5.SS2.p2.3.3" class="ltx_verbatim ltx_font_typewriter">tbl</code>) to achieve parallel and efficient table read.
We demonstrate the implementation of table read using shuffle instructions in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
The shuffle instruction <cite class="ltx_cite ltx_citemacro_citep">(Blalock and Guttag, <a href="#bib.bib7" title="" class="ltx_ref">2017</a>)</cite> permutes each byte of a vector based on an index vector and stores the shuffled bytes in the result vector register in each clock cycle.
On 128-bit wide SIMD, a vectorized table read instruction handles 16 sub-vector lookups (<math id="S5.SS2.p2.1.m1.1" class="ltx_Math" alttext="128/8=16" display="inline"><semantics id="S5.SS2.p2.1.m1.1a"><mrow id="S5.SS2.p2.1.m1.1.1" xref="S5.SS2.p2.1.m1.1.1.cmml"><mrow id="S5.SS2.p2.1.m1.1.1.2" xref="S5.SS2.p2.1.m1.1.1.2.cmml"><mn id="S5.SS2.p2.1.m1.1.1.2.2" xref="S5.SS2.p2.1.m1.1.1.2.2.cmml">128</mn><mo id="S5.SS2.p2.1.m1.1.1.2.1" xref="S5.SS2.p2.1.m1.1.1.2.1.cmml">/</mo><mn id="S5.SS2.p2.1.m1.1.1.2.3" xref="S5.SS2.p2.1.m1.1.1.2.3.cmml">8</mn></mrow><mo id="S5.SS2.p2.1.m1.1.1.1" xref="S5.SS2.p2.1.m1.1.1.1.cmml">=</mo><mn id="S5.SS2.p2.1.m1.1.1.3" xref="S5.SS2.p2.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.1.m1.1b"><apply id="S5.SS2.p2.1.m1.1.1.cmml" xref="S5.SS2.p2.1.m1.1.1"><eq id="S5.SS2.p2.1.m1.1.1.1.cmml" xref="S5.SS2.p2.1.m1.1.1.1"></eq><apply id="S5.SS2.p2.1.m1.1.1.2.cmml" xref="S5.SS2.p2.1.m1.1.1.2"><divide id="S5.SS2.p2.1.m1.1.1.2.1.cmml" xref="S5.SS2.p2.1.m1.1.1.2.1"></divide><cn type="integer" id="S5.SS2.p2.1.m1.1.1.2.2.cmml" xref="S5.SS2.p2.1.m1.1.1.2.2">128</cn><cn type="integer" id="S5.SS2.p2.1.m1.1.1.2.3.cmml" xref="S5.SS2.p2.1.m1.1.1.2.3">8</cn></apply><cn type="integer" id="S5.SS2.p2.1.m1.1.1.3.cmml" xref="S5.SS2.p2.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.1.m1.1c">128/8=16</annotation></semantics></math>) on <math id="S5.SS2.p2.2.m2.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S5.SS2.p2.2.m2.1a"><mn id="S5.SS2.p2.2.m2.1.1" xref="S5.SS2.p2.2.m2.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.2.m2.1b"><cn type="integer" id="S5.SS2.p2.2.m2.1.1.cmml" xref="S5.SS2.p2.2.m2.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.2.m2.1c">16</annotation></semantics></math> results (<math id="S5.SS2.p2.3.m3.1" class="ltx_Math" alttext="128/8=16" display="inline"><semantics id="S5.SS2.p2.3.m3.1a"><mrow id="S5.SS2.p2.3.m3.1.1" xref="S5.SS2.p2.3.m3.1.1.cmml"><mrow id="S5.SS2.p2.3.m3.1.1.2" xref="S5.SS2.p2.3.m3.1.1.2.cmml"><mn id="S5.SS2.p2.3.m3.1.1.2.2" xref="S5.SS2.p2.3.m3.1.1.2.2.cmml">128</mn><mo id="S5.SS2.p2.3.m3.1.1.2.1" xref="S5.SS2.p2.3.m3.1.1.2.1.cmml">/</mo><mn id="S5.SS2.p2.3.m3.1.1.2.3" xref="S5.SS2.p2.3.m3.1.1.2.3.cmml">8</mn></mrow><mo id="S5.SS2.p2.3.m3.1.1.1" xref="S5.SS2.p2.3.m3.1.1.1.cmml">=</mo><mn id="S5.SS2.p2.3.m3.1.1.3" xref="S5.SS2.p2.3.m3.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.3.m3.1b"><apply id="S5.SS2.p2.3.m3.1.1.cmml" xref="S5.SS2.p2.3.m3.1.1"><eq id="S5.SS2.p2.3.m3.1.1.1.cmml" xref="S5.SS2.p2.3.m3.1.1.1"></eq><apply id="S5.SS2.p2.3.m3.1.1.2.cmml" xref="S5.SS2.p2.3.m3.1.1.2"><divide id="S5.SS2.p2.3.m3.1.1.2.1.cmml" xref="S5.SS2.p2.3.m3.1.1.2.1"></divide><cn type="integer" id="S5.SS2.p2.3.m3.1.1.2.2.cmml" xref="S5.SS2.p2.3.m3.1.1.2.2">128</cn><cn type="integer" id="S5.SS2.p2.3.m3.1.1.2.3.cmml" xref="S5.SS2.p2.3.m3.1.1.2.3">8</cn></apply><cn type="integer" id="S5.SS2.p2.3.m3.1.1.3.cmml" xref="S5.SS2.p2.3.m3.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.3.m3.1c">128/8=16</annotation></semantics></math>) simultaneously, greatly simplifying table read and reducing overheads.</p>
</div>
<div id="S5.SS2.p3" class="ltx_para">
<p id="S5.SS2.p3.5" class="ltx_p">Second, the accumulation still has comparable computation costs to the entire process.
For example, when a codebook handles <math id="S5.SS2.p3.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S5.SS2.p3.1.m1.1a"><mi id="S5.SS2.p3.1.m1.1.1" xref="S5.SS2.p3.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.1.m1.1b"><ci id="S5.SS2.p3.1.m1.1.1.cmml" xref="S5.SS2.p3.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.1.m1.1c">N</annotation></semantics></math> index lookups on a <math id="S5.SS2.p3.2.m2.1" class="ltx_Math" alttext="K\cdot M" display="inline"><semantics id="S5.SS2.p3.2.m2.1a"><mrow id="S5.SS2.p3.2.m2.1.1" xref="S5.SS2.p3.2.m2.1.1.cmml"><mi id="S5.SS2.p3.2.m2.1.1.2" xref="S5.SS2.p3.2.m2.1.1.2.cmml">K</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS2.p3.2.m2.1.1.1" xref="S5.SS2.p3.2.m2.1.1.1.cmml">⋅</mo><mi id="S5.SS2.p3.2.m2.1.1.3" xref="S5.SS2.p3.2.m2.1.1.3.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.2.m2.1b"><apply id="S5.SS2.p3.2.m2.1.1.cmml" xref="S5.SS2.p3.2.m2.1.1"><ci id="S5.SS2.p3.2.m2.1.1.1.cmml" xref="S5.SS2.p3.2.m2.1.1.1">⋅</ci><ci id="S5.SS2.p3.2.m2.1.1.2.cmml" xref="S5.SS2.p3.2.m2.1.1.2">𝐾</ci><ci id="S5.SS2.p3.2.m2.1.1.3.cmml" xref="S5.SS2.p3.2.m2.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.2.m2.1c">K\cdot M</annotation></semantics></math> lookup table, it costs <math id="S5.SS2.p3.3.m3.1" class="ltx_Math" alttext="N\cdot M" display="inline"><semantics id="S5.SS2.p3.3.m3.1a"><mrow id="S5.SS2.p3.3.m3.1.1" xref="S5.SS2.p3.3.m3.1.1.cmml"><mi id="S5.SS2.p3.3.m3.1.1.2" xref="S5.SS2.p3.3.m3.1.1.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS2.p3.3.m3.1.1.1" xref="S5.SS2.p3.3.m3.1.1.1.cmml">⋅</mo><mi id="S5.SS2.p3.3.m3.1.1.3" xref="S5.SS2.p3.3.m3.1.1.3.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.3.m3.1b"><apply id="S5.SS2.p3.3.m3.1.1.cmml" xref="S5.SS2.p3.3.m3.1.1"><ci id="S5.SS2.p3.3.m3.1.1.1.cmml" xref="S5.SS2.p3.3.m3.1.1.1">⋅</ci><ci id="S5.SS2.p3.3.m3.1.1.2.cmml" xref="S5.SS2.p3.3.m3.1.1.2">𝑁</ci><ci id="S5.SS2.p3.3.m3.1.1.3.cmml" xref="S5.SS2.p3.3.m3.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.3.m3.1c">N\cdot M</annotation></semantics></math> table reads (<math id="S5.SS2.p3.4.m4.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S5.SS2.p3.4.m4.1a"><mrow id="S5.SS2.p3.4.m4.1.1" xref="S5.SS2.p3.4.m4.1.1.cmml"><mi id="S5.SS2.p3.4.m4.1.1.2" xref="S5.SS2.p3.4.m4.1.1.2.cmml">K</mi><mo id="S5.SS2.p3.4.m4.1.1.1" xref="S5.SS2.p3.4.m4.1.1.1.cmml">=</mo><mn id="S5.SS2.p3.4.m4.1.1.3" xref="S5.SS2.p3.4.m4.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.4.m4.1b"><apply id="S5.SS2.p3.4.m4.1.1.cmml" xref="S5.SS2.p3.4.m4.1.1"><eq id="S5.SS2.p3.4.m4.1.1.1.cmml" xref="S5.SS2.p3.4.m4.1.1.1"></eq><ci id="S5.SS2.p3.4.m4.1.1.2.cmml" xref="S5.SS2.p3.4.m4.1.1.2">𝐾</ci><cn type="integer" id="S5.SS2.p3.4.m4.1.1.3.cmml" xref="S5.SS2.p3.4.m4.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.4.m4.1c">K=16</annotation></semantics></math>) and <math id="S5.SS2.p3.5.m5.1" class="ltx_Math" alttext="N\cdot M" display="inline"><semantics id="S5.SS2.p3.5.m5.1a"><mrow id="S5.SS2.p3.5.m5.1.1" xref="S5.SS2.p3.5.m5.1.1.cmml"><mi id="S5.SS2.p3.5.m5.1.1.2" xref="S5.SS2.p3.5.m5.1.1.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS2.p3.5.m5.1.1.1" xref="S5.SS2.p3.5.m5.1.1.1.cmml">⋅</mo><mi id="S5.SS2.p3.5.m5.1.1.3" xref="S5.SS2.p3.5.m5.1.1.3.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.5.m5.1b"><apply id="S5.SS2.p3.5.m5.1.1.cmml" xref="S5.SS2.p3.5.m5.1.1"><ci id="S5.SS2.p3.5.m5.1.1.1.cmml" xref="S5.SS2.p3.5.m5.1.1.1">⋅</ci><ci id="S5.SS2.p3.5.m5.1.1.2.cmml" xref="S5.SS2.p3.5.m5.1.1.2">𝑁</ci><ci id="S5.SS2.p3.5.m5.1.1.3.cmml" xref="S5.SS2.p3.5.m5.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.5.m5.1c">N\cdot M</annotation></semantics></math> accumulation adds.
Therefore, the vectorized lookup table leaves accumulation operations as the performance bottleneck of table reads.
The number of parallel processing units within a SIMD instruction is called SIMD lanes.
We typically set the number of centroids to 16 (K=16) to maximize the utilization of all SIMD instruction lanes.
Since a higher number of SIMD lanes leads to higher throughput on the same width of SIMD instruction (e.g., the vectorized INT16 add instruction has the half throughput of INT8 on a 128-bit SIMD),
we maximize accumulation throughput by mixed precision accumulation. It first accumulates results in INT16 to utilize more SIMD lanes and then gathers INT16 to INT32 to avoid overflow.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3. </span>Utilization of memory hierarchy</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">As LUT-NN is composed of many memory accesses, we explain more details for its memory hierachy usage.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.3" class="ltx_p">The frequent-accessed codebook (for centroids) is kept in the inner loop during distance calculation, so that the hardware cache can leaverage the locality to keep the codebook in cache.
The size of a codebook for one layer is normally small, in KB level (calculated by sub-vector length <math id="S5.SS3.p2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS3.p2.1.m1.1a"><mo id="S5.SS3.p2.1.m1.1.1" xref="S5.SS3.p2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.1.m1.1b"><times id="S5.SS3.p2.1.m1.1.1.cmml" xref="S5.SS3.p2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.1.m1.1c">\times</annotation></semantics></math> number of centroids <math id="S5.SS3.p2.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS3.p2.2.m2.1a"><mo id="S5.SS3.p2.2.m2.1.1" xref="S5.SS3.p2.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.2.m2.1b"><times id="S5.SS3.p2.2.m2.1.1.cmml" xref="S5.SS3.p2.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.2.m2.1c">\times</annotation></semantics></math> number of codebooks <math id="S5.SS3.p2.3.m3.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS3.p2.3.m3.1a"><mo id="S5.SS3.p2.3.m3.1.1" xref="S5.SS3.p2.3.m3.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.3.m3.1b"><times id="S5.SS3.p2.3.m3.1.1.cmml" xref="S5.SS3.p2.3.m3.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.3.m3.1c">\times</annotation></semantics></math> number of bytes). For example, the largest codebook in our tested models is 288 KB. It can be held in L2 cache on many mobile devices.</p>
</div>
<div id="S5.SS3.p3" class="ltx_para">
<p id="S5.SS3.p3.1" class="ltx_p">The lookup table (for precomputed results) is larger, in MB level. It can be held in L3 cache. Even if it is larger than L3 cache, the memory access overhead could still be hidden in our implementation. LUT-NN uses Shuffle instructions to parallelize table lookup, turning the pattern of reading index numbers and lookup tables from random to sequential. The memory access is thus predictable and easy to prefetch. Furthermore, we interleave computation and memory access instructions within the kernel, which can overlap computation and memory access time.</p>
</div>
<figure id="S5.T3" class="ltx_table">
<table id="S5.T3.6" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T3.6.7.1" class="ltx_tr">
<th id="S5.T3.6.7.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t">Dataset</th>
<th id="S5.T3.6.7.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">Model</th>
<th id="S5.T3.6.7.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T3.6.7.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.6.7.1.3.1.1" class="ltx_tr">
<td id="S5.T3.6.7.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Centroid</td>
</tr>
<tr id="S5.T3.6.7.1.3.1.2" class="ltx_tr">
<td id="S5.T3.6.7.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Learning Rate</td>
</tr>
</table>
</th>
<th id="S5.T3.6.7.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T3.6.7.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.6.7.1.4.1.1" class="ltx_tr">
<td id="S5.T3.6.7.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Temperature</td>
</tr>
<tr id="S5.T3.6.7.1.4.1.2" class="ltx_tr">
<td id="S5.T3.6.7.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Learning Rate</td>
</tr>
</table>
</th>
<th id="S5.T3.6.7.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T3.6.7.1.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.6.7.1.5.1.1" class="ltx_tr">
<td id="S5.T3.6.7.1.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Weight</td>
</tr>
<tr id="S5.T3.6.7.1.5.1.2" class="ltx_tr">
<td id="S5.T3.6.7.1.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Decay</td>
</tr>
</table>
</th>
<th id="S5.T3.6.7.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T3.6.7.1.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.6.7.1.6.1.1" class="ltx_tr">
<td id="S5.T3.6.7.1.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Batch</td>
</tr>
<tr id="S5.T3.6.7.1.6.1.2" class="ltx_tr">
<td id="S5.T3.6.7.1.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Size</td>
</tr>
</table>
</th>
<th id="S5.T3.6.7.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t"># of Epochs</th>
<th id="S5.T3.6.7.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">Optimizer</th>
<th id="S5.T3.6.7.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T3.6.7.1.9.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.6.7.1.9.1.1" class="ltx_tr">
<td id="S5.T3.6.7.1.9.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">LR</td>
</tr>
<tr id="S5.T3.6.7.1.9.1.2" class="ltx_tr">
<td id="S5.T3.6.7.1.9.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Scheduler</td>
</tr>
</table>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T3.2.2" class="ltx_tr">
<td id="S5.T3.2.2.3" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">CIFAR10</td>
<td id="S5.T3.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="6"><span id="S5.T3.2.2.4.1" class="ltx_text">
<span id="S5.T3.2.2.4.1.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T3.2.2.4.1.1.1" class="ltx_tr">
<span id="S5.T3.2.2.4.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">ResNet18</span></span>
<span id="S5.T3.2.2.4.1.1.2" class="ltx_tr">
<span id="S5.T3.2.2.4.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">SENet18</span></span>
<span id="S5.T3.2.2.4.1.1.3" class="ltx_tr">
<span id="S5.T3.2.2.4.1.1.3.1" class="ltx_td ltx_nopad_r ltx_align_center">VGG11</span></span>
</span></span></td>
<td id="S5.T3.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="2"><span id="S5.T3.1.1.1.1" class="ltx_text"><math id="S5.T3.1.1.1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-3}" display="inline"><semantics id="S5.T3.1.1.1.1.m1.1a"><mrow id="S5.T3.1.1.1.1.m1.1.1" xref="S5.T3.1.1.1.1.m1.1.1.cmml"><mn id="S5.T3.1.1.1.1.m1.1.1.2" xref="S5.T3.1.1.1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.1.1.1.1.m1.1.1.1" xref="S5.T3.1.1.1.1.m1.1.1.1.cmml">×</mo><msup id="S5.T3.1.1.1.1.m1.1.1.3" xref="S5.T3.1.1.1.1.m1.1.1.3.cmml"><mn id="S5.T3.1.1.1.1.m1.1.1.3.2" xref="S5.T3.1.1.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S5.T3.1.1.1.1.m1.1.1.3.3" xref="S5.T3.1.1.1.1.m1.1.1.3.3.cmml"><mo id="S5.T3.1.1.1.1.m1.1.1.3.3a" xref="S5.T3.1.1.1.1.m1.1.1.3.3.cmml">−</mo><mn id="S5.T3.1.1.1.1.m1.1.1.3.3.2" xref="S5.T3.1.1.1.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.1.1.1.1.m1.1b"><apply id="S5.T3.1.1.1.1.m1.1.1.cmml" xref="S5.T3.1.1.1.1.m1.1.1"><times id="S5.T3.1.1.1.1.m1.1.1.1.cmml" xref="S5.T3.1.1.1.1.m1.1.1.1"></times><cn type="integer" id="S5.T3.1.1.1.1.m1.1.1.2.cmml" xref="S5.T3.1.1.1.1.m1.1.1.2">1</cn><apply id="S5.T3.1.1.1.1.m1.1.1.3.cmml" xref="S5.T3.1.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S5.T3.1.1.1.1.m1.1.1.3.1.cmml" xref="S5.T3.1.1.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.1.1.1.1.m1.1.1.3.2.cmml" xref="S5.T3.1.1.1.1.m1.1.1.3.2">10</cn><apply id="S5.T3.1.1.1.1.m1.1.1.3.3.cmml" xref="S5.T3.1.1.1.1.m1.1.1.3.3"><minus id="S5.T3.1.1.1.1.m1.1.1.3.3.1.cmml" xref="S5.T3.1.1.1.1.m1.1.1.3.3"></minus><cn type="integer" id="S5.T3.1.1.1.1.m1.1.1.3.3.2.cmml" xref="S5.T3.1.1.1.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.1.1.1.1.m1.1c">1\times 10^{-3}</annotation></semantics></math></span></td>
<td id="S5.T3.2.2.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t" rowspan="7"><span id="S5.T3.2.2.2.1" class="ltx_text"><math id="S5.T3.2.2.2.1.m1.1" class="ltx_Math" alttext="1\times 10^{-1}" display="inline"><semantics id="S5.T3.2.2.2.1.m1.1a"><mrow id="S5.T3.2.2.2.1.m1.1.1" xref="S5.T3.2.2.2.1.m1.1.1.cmml"><mn id="S5.T3.2.2.2.1.m1.1.1.2" xref="S5.T3.2.2.2.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.2.2.2.1.m1.1.1.1" xref="S5.T3.2.2.2.1.m1.1.1.1.cmml">×</mo><msup id="S5.T3.2.2.2.1.m1.1.1.3" xref="S5.T3.2.2.2.1.m1.1.1.3.cmml"><mn id="S5.T3.2.2.2.1.m1.1.1.3.2" xref="S5.T3.2.2.2.1.m1.1.1.3.2.cmml">10</mn><mrow id="S5.T3.2.2.2.1.m1.1.1.3.3" xref="S5.T3.2.2.2.1.m1.1.1.3.3.cmml"><mo id="S5.T3.2.2.2.1.m1.1.1.3.3a" xref="S5.T3.2.2.2.1.m1.1.1.3.3.cmml">−</mo><mn id="S5.T3.2.2.2.1.m1.1.1.3.3.2" xref="S5.T3.2.2.2.1.m1.1.1.3.3.2.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.2.2.2.1.m1.1b"><apply id="S5.T3.2.2.2.1.m1.1.1.cmml" xref="S5.T3.2.2.2.1.m1.1.1"><times id="S5.T3.2.2.2.1.m1.1.1.1.cmml" xref="S5.T3.2.2.2.1.m1.1.1.1"></times><cn type="integer" id="S5.T3.2.2.2.1.m1.1.1.2.cmml" xref="S5.T3.2.2.2.1.m1.1.1.2">1</cn><apply id="S5.T3.2.2.2.1.m1.1.1.3.cmml" xref="S5.T3.2.2.2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S5.T3.2.2.2.1.m1.1.1.3.1.cmml" xref="S5.T3.2.2.2.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.2.2.2.1.m1.1.1.3.2.cmml" xref="S5.T3.2.2.2.1.m1.1.1.3.2">10</cn><apply id="S5.T3.2.2.2.1.m1.1.1.3.3.cmml" xref="S5.T3.2.2.2.1.m1.1.1.3.3"><minus id="S5.T3.2.2.2.1.m1.1.1.3.3.1.cmml" xref="S5.T3.2.2.2.1.m1.1.1.3.3"></minus><cn type="integer" id="S5.T3.2.2.2.1.m1.1.1.3.3.2.cmml" xref="S5.T3.2.2.2.1.m1.1.1.3.3.2">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.2.2.2.1.m1.1c">1\times 10^{-1}</annotation></semantics></math></span></td>
<td id="S5.T3.2.2.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="6"><span id="S5.T3.2.2.5.1" class="ltx_text">0</span></td>
<td id="S5.T3.2.2.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="6"><span id="S5.T3.2.2.6.1" class="ltx_text">256</span></td>
<td id="S5.T3.2.2.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="3"><span id="S5.T3.2.2.7.1" class="ltx_text">200</span></td>
<td id="S5.T3.2.2.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="6"><span id="S5.T3.2.2.8.1" class="ltx_text">Adam</span></td>
<td id="S5.T3.2.2.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="6"><span id="S5.T3.2.2.9.1" class="ltx_text">
<span id="S5.T3.2.2.9.1.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T3.2.2.9.1.1.1" class="ltx_tr">
<span id="S5.T3.2.2.9.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Cosine</span></span>
<span id="S5.T3.2.2.9.1.1.2" class="ltx_tr">
<span id="S5.T3.2.2.9.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Annealing</span></span>
</span></span></td>
</tr>
<tr id="S5.T3.6.8.1" class="ltx_tr">
<td id="S5.T3.6.8.1.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">GRSRB</td>
</tr>
<tr id="S5.T3.3.3" class="ltx_tr">
<td id="S5.T3.3.3.2" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">Speech Commands</td>
<td id="S5.T3.3.3.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="2"><span id="S5.T3.3.3.1.1" class="ltx_text"><math id="S5.T3.3.3.1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-4}" display="inline"><semantics id="S5.T3.3.3.1.1.m1.1a"><mrow id="S5.T3.3.3.1.1.m1.1.1" xref="S5.T3.3.3.1.1.m1.1.1.cmml"><mn id="S5.T3.3.3.1.1.m1.1.1.2" xref="S5.T3.3.3.1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.3.3.1.1.m1.1.1.1" xref="S5.T3.3.3.1.1.m1.1.1.1.cmml">×</mo><msup id="S5.T3.3.3.1.1.m1.1.1.3" xref="S5.T3.3.3.1.1.m1.1.1.3.cmml"><mn id="S5.T3.3.3.1.1.m1.1.1.3.2" xref="S5.T3.3.3.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S5.T3.3.3.1.1.m1.1.1.3.3" xref="S5.T3.3.3.1.1.m1.1.1.3.3.cmml"><mo id="S5.T3.3.3.1.1.m1.1.1.3.3a" xref="S5.T3.3.3.1.1.m1.1.1.3.3.cmml">−</mo><mn id="S5.T3.3.3.1.1.m1.1.1.3.3.2" xref="S5.T3.3.3.1.1.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.3.3.1.1.m1.1b"><apply id="S5.T3.3.3.1.1.m1.1.1.cmml" xref="S5.T3.3.3.1.1.m1.1.1"><times id="S5.T3.3.3.1.1.m1.1.1.1.cmml" xref="S5.T3.3.3.1.1.m1.1.1.1"></times><cn type="integer" id="S5.T3.3.3.1.1.m1.1.1.2.cmml" xref="S5.T3.3.3.1.1.m1.1.1.2">1</cn><apply id="S5.T3.3.3.1.1.m1.1.1.3.cmml" xref="S5.T3.3.3.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S5.T3.3.3.1.1.m1.1.1.3.1.cmml" xref="S5.T3.3.3.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.3.3.1.1.m1.1.1.3.2.cmml" xref="S5.T3.3.3.1.1.m1.1.1.3.2">10</cn><apply id="S5.T3.3.3.1.1.m1.1.1.3.3.cmml" xref="S5.T3.3.3.1.1.m1.1.1.3.3"><minus id="S5.T3.3.3.1.1.m1.1.1.3.3.1.cmml" xref="S5.T3.3.3.1.1.m1.1.1.3.3"></minus><cn type="integer" id="S5.T3.3.3.1.1.m1.1.1.3.3.2.cmml" xref="S5.T3.3.3.1.1.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.3.3.1.1.m1.1c">1\times 10^{-4}</annotation></semantics></math></span></td>
</tr>
<tr id="S5.T3.6.9.2" class="ltx_tr">
<td id="S5.T3.6.9.2.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">UTKFace</td>
<td id="S5.T3.6.9.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">150</td>
</tr>
<tr id="S5.T3.4.4" class="ltx_tr">
<td id="S5.T3.4.4.2" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">SVHN</td>
<td id="S5.T3.4.4.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="2"><span id="S5.T3.4.4.1.1" class="ltx_text"><math id="S5.T3.4.4.1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-3}" display="inline"><semantics id="S5.T3.4.4.1.1.m1.1a"><mrow id="S5.T3.4.4.1.1.m1.1.1" xref="S5.T3.4.4.1.1.m1.1.1.cmml"><mn id="S5.T3.4.4.1.1.m1.1.1.2" xref="S5.T3.4.4.1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.4.4.1.1.m1.1.1.1" xref="S5.T3.4.4.1.1.m1.1.1.1.cmml">×</mo><msup id="S5.T3.4.4.1.1.m1.1.1.3" xref="S5.T3.4.4.1.1.m1.1.1.3.cmml"><mn id="S5.T3.4.4.1.1.m1.1.1.3.2" xref="S5.T3.4.4.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S5.T3.4.4.1.1.m1.1.1.3.3" xref="S5.T3.4.4.1.1.m1.1.1.3.3.cmml"><mo id="S5.T3.4.4.1.1.m1.1.1.3.3a" xref="S5.T3.4.4.1.1.m1.1.1.3.3.cmml">−</mo><mn id="S5.T3.4.4.1.1.m1.1.1.3.3.2" xref="S5.T3.4.4.1.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.4.4.1.1.m1.1b"><apply id="S5.T3.4.4.1.1.m1.1.1.cmml" xref="S5.T3.4.4.1.1.m1.1.1"><times id="S5.T3.4.4.1.1.m1.1.1.1.cmml" xref="S5.T3.4.4.1.1.m1.1.1.1"></times><cn type="integer" id="S5.T3.4.4.1.1.m1.1.1.2.cmml" xref="S5.T3.4.4.1.1.m1.1.1.2">1</cn><apply id="S5.T3.4.4.1.1.m1.1.1.3.cmml" xref="S5.T3.4.4.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S5.T3.4.4.1.1.m1.1.1.3.1.cmml" xref="S5.T3.4.4.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.4.4.1.1.m1.1.1.3.2.cmml" xref="S5.T3.4.4.1.1.m1.1.1.3.2">10</cn><apply id="S5.T3.4.4.1.1.m1.1.1.3.3.cmml" xref="S5.T3.4.4.1.1.m1.1.1.3.3"><minus id="S5.T3.4.4.1.1.m1.1.1.3.3.1.cmml" xref="S5.T3.4.4.1.1.m1.1.1.3.3"></minus><cn type="integer" id="S5.T3.4.4.1.1.m1.1.1.3.3.2.cmml" xref="S5.T3.4.4.1.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.4.4.1.1.m1.1c">1\times 10^{-3}</annotation></semantics></math></span></td>
<td id="S5.T3.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">200</td>
</tr>
<tr id="S5.T3.6.10.3" class="ltx_tr">
<td id="S5.T3.6.10.3.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">ImageNet</td>
<td id="S5.T3.6.10.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">150</td>
</tr>
<tr id="S5.T3.6.6" class="ltx_tr">
<td id="S5.T3.6.6.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t">GLUE</td>
<td id="S5.T3.6.6.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">BERT</td>
<td id="S5.T3.5.5.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><math id="S5.T3.5.5.1.m1.4" class="ltx_Math" alttext="\{5,4,3,2\}\times 10^{-5}" display="inline"><semantics id="S5.T3.5.5.1.m1.4a"><mrow id="S5.T3.5.5.1.m1.4.5" xref="S5.T3.5.5.1.m1.4.5.cmml"><mrow id="S5.T3.5.5.1.m1.4.5.2.2" xref="S5.T3.5.5.1.m1.4.5.2.1.cmml"><mo stretchy="false" id="S5.T3.5.5.1.m1.4.5.2.2.1" xref="S5.T3.5.5.1.m1.4.5.2.1.cmml">{</mo><mn id="S5.T3.5.5.1.m1.1.1" xref="S5.T3.5.5.1.m1.1.1.cmml">5</mn><mo id="S5.T3.5.5.1.m1.4.5.2.2.2" xref="S5.T3.5.5.1.m1.4.5.2.1.cmml">,</mo><mn id="S5.T3.5.5.1.m1.2.2" xref="S5.T3.5.5.1.m1.2.2.cmml">4</mn><mo id="S5.T3.5.5.1.m1.4.5.2.2.3" xref="S5.T3.5.5.1.m1.4.5.2.1.cmml">,</mo><mn id="S5.T3.5.5.1.m1.3.3" xref="S5.T3.5.5.1.m1.3.3.cmml">3</mn><mo id="S5.T3.5.5.1.m1.4.5.2.2.4" xref="S5.T3.5.5.1.m1.4.5.2.1.cmml">,</mo><mn id="S5.T3.5.5.1.m1.4.4" xref="S5.T3.5.5.1.m1.4.4.cmml">2</mn><mo rspace="0.055em" stretchy="false" id="S5.T3.5.5.1.m1.4.5.2.2.5" xref="S5.T3.5.5.1.m1.4.5.2.1.cmml">}</mo></mrow><mo rspace="0.222em" id="S5.T3.5.5.1.m1.4.5.1" xref="S5.T3.5.5.1.m1.4.5.1.cmml">×</mo><msup id="S5.T3.5.5.1.m1.4.5.3" xref="S5.T3.5.5.1.m1.4.5.3.cmml"><mn id="S5.T3.5.5.1.m1.4.5.3.2" xref="S5.T3.5.5.1.m1.4.5.3.2.cmml">10</mn><mrow id="S5.T3.5.5.1.m1.4.5.3.3" xref="S5.T3.5.5.1.m1.4.5.3.3.cmml"><mo id="S5.T3.5.5.1.m1.4.5.3.3a" xref="S5.T3.5.5.1.m1.4.5.3.3.cmml">−</mo><mn id="S5.T3.5.5.1.m1.4.5.3.3.2" xref="S5.T3.5.5.1.m1.4.5.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.5.5.1.m1.4b"><apply id="S5.T3.5.5.1.m1.4.5.cmml" xref="S5.T3.5.5.1.m1.4.5"><times id="S5.T3.5.5.1.m1.4.5.1.cmml" xref="S5.T3.5.5.1.m1.4.5.1"></times><set id="S5.T3.5.5.1.m1.4.5.2.1.cmml" xref="S5.T3.5.5.1.m1.4.5.2.2"><cn type="integer" id="S5.T3.5.5.1.m1.1.1.cmml" xref="S5.T3.5.5.1.m1.1.1">5</cn><cn type="integer" id="S5.T3.5.5.1.m1.2.2.cmml" xref="S5.T3.5.5.1.m1.2.2">4</cn><cn type="integer" id="S5.T3.5.5.1.m1.3.3.cmml" xref="S5.T3.5.5.1.m1.3.3">3</cn><cn type="integer" id="S5.T3.5.5.1.m1.4.4.cmml" xref="S5.T3.5.5.1.m1.4.4">2</cn></set><apply id="S5.T3.5.5.1.m1.4.5.3.cmml" xref="S5.T3.5.5.1.m1.4.5.3"><csymbol cd="ambiguous" id="S5.T3.5.5.1.m1.4.5.3.1.cmml" xref="S5.T3.5.5.1.m1.4.5.3">superscript</csymbol><cn type="integer" id="S5.T3.5.5.1.m1.4.5.3.2.cmml" xref="S5.T3.5.5.1.m1.4.5.3.2">10</cn><apply id="S5.T3.5.5.1.m1.4.5.3.3.cmml" xref="S5.T3.5.5.1.m1.4.5.3.3"><minus id="S5.T3.5.5.1.m1.4.5.3.3.1.cmml" xref="S5.T3.5.5.1.m1.4.5.3.3"></minus><cn type="integer" id="S5.T3.5.5.1.m1.4.5.3.3.2.cmml" xref="S5.T3.5.5.1.m1.4.5.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.5.5.1.m1.4c">\{5,4,3,2\}\times 10^{-5}</annotation></semantics></math></td>
<td id="S5.T3.6.6.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><math id="S5.T3.6.6.2.m1.1" class="ltx_Math" alttext="1\times 10^{-2}" display="inline"><semantics id="S5.T3.6.6.2.m1.1a"><mrow id="S5.T3.6.6.2.m1.1.1" xref="S5.T3.6.6.2.m1.1.1.cmml"><mn id="S5.T3.6.6.2.m1.1.1.2" xref="S5.T3.6.6.2.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.6.6.2.m1.1.1.1" xref="S5.T3.6.6.2.m1.1.1.1.cmml">×</mo><msup id="S5.T3.6.6.2.m1.1.1.3" xref="S5.T3.6.6.2.m1.1.1.3.cmml"><mn id="S5.T3.6.6.2.m1.1.1.3.2" xref="S5.T3.6.6.2.m1.1.1.3.2.cmml">10</mn><mrow id="S5.T3.6.6.2.m1.1.1.3.3" xref="S5.T3.6.6.2.m1.1.1.3.3.cmml"><mo id="S5.T3.6.6.2.m1.1.1.3.3a" xref="S5.T3.6.6.2.m1.1.1.3.3.cmml">−</mo><mn id="S5.T3.6.6.2.m1.1.1.3.3.2" xref="S5.T3.6.6.2.m1.1.1.3.3.2.cmml">2</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.6.6.2.m1.1b"><apply id="S5.T3.6.6.2.m1.1.1.cmml" xref="S5.T3.6.6.2.m1.1.1"><times id="S5.T3.6.6.2.m1.1.1.1.cmml" xref="S5.T3.6.6.2.m1.1.1.1"></times><cn type="integer" id="S5.T3.6.6.2.m1.1.1.2.cmml" xref="S5.T3.6.6.2.m1.1.1.2">1</cn><apply id="S5.T3.6.6.2.m1.1.1.3.cmml" xref="S5.T3.6.6.2.m1.1.1.3"><csymbol cd="ambiguous" id="S5.T3.6.6.2.m1.1.1.3.1.cmml" xref="S5.T3.6.6.2.m1.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.6.6.2.m1.1.1.3.2.cmml" xref="S5.T3.6.6.2.m1.1.1.3.2">10</cn><apply id="S5.T3.6.6.2.m1.1.1.3.3.cmml" xref="S5.T3.6.6.2.m1.1.1.3.3"><minus id="S5.T3.6.6.2.m1.1.1.3.3.1.cmml" xref="S5.T3.6.6.2.m1.1.1.3.3"></minus><cn type="integer" id="S5.T3.6.6.2.m1.1.1.3.3.2.cmml" xref="S5.T3.6.6.2.m1.1.1.3.3.2">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.6.6.2.m1.1c">1\times 10^{-2}</annotation></semantics></math></td>
<td id="S5.T3.6.6.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">32</td>
<td id="S5.T3.6.6.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">3</td>
<td id="S5.T3.6.6.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">AdamW</td>
<td id="S5.T3.6.6.8" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">Constant</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3. </span>

Soft-PQ training settings for centroid and temperature learning.
We choose the best fine-tuning learning rate among {<math id="S5.T3.11.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="S5.T3.11.m1.1b"><mrow id="S5.T3.11.m1.1.1" xref="S5.T3.11.m1.1.1.cmml"><mn id="S5.T3.11.m1.1.1.2" xref="S5.T3.11.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.11.m1.1.1.1" xref="S5.T3.11.m1.1.1.1.cmml">×</mo><msup id="S5.T3.11.m1.1.1.3" xref="S5.T3.11.m1.1.1.3.cmml"><mn id="S5.T3.11.m1.1.1.3.2" xref="S5.T3.11.m1.1.1.3.2.cmml">10</mn><mrow id="S5.T3.11.m1.1.1.3.3" xref="S5.T3.11.m1.1.1.3.3.cmml"><mo id="S5.T3.11.m1.1.1.3.3b" xref="S5.T3.11.m1.1.1.3.3.cmml">−</mo><mn id="S5.T3.11.m1.1.1.3.3.2" xref="S5.T3.11.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.11.m1.1c"><apply id="S5.T3.11.m1.1.1.cmml" xref="S5.T3.11.m1.1.1"><times id="S5.T3.11.m1.1.1.1.cmml" xref="S5.T3.11.m1.1.1.1"></times><cn type="integer" id="S5.T3.11.m1.1.1.2.cmml" xref="S5.T3.11.m1.1.1.2">5</cn><apply id="S5.T3.11.m1.1.1.3.cmml" xref="S5.T3.11.m1.1.1.3"><csymbol cd="ambiguous" id="S5.T3.11.m1.1.1.3.1.cmml" xref="S5.T3.11.m1.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.11.m1.1.1.3.2.cmml" xref="S5.T3.11.m1.1.1.3.2">10</cn><apply id="S5.T3.11.m1.1.1.3.3.cmml" xref="S5.T3.11.m1.1.1.3.3"><minus id="S5.T3.11.m1.1.1.3.3.1.cmml" xref="S5.T3.11.m1.1.1.3.3"></minus><cn type="integer" id="S5.T3.11.m1.1.1.3.3.2.cmml" xref="S5.T3.11.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.11.m1.1d">5\times 10^{-5}</annotation></semantics></math>, <math id="S5.T3.12.m2.1" class="ltx_Math" alttext="4\times 10^{-5}" display="inline"><semantics id="S5.T3.12.m2.1b"><mrow id="S5.T3.12.m2.1.1" xref="S5.T3.12.m2.1.1.cmml"><mn id="S5.T3.12.m2.1.1.2" xref="S5.T3.12.m2.1.1.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.12.m2.1.1.1" xref="S5.T3.12.m2.1.1.1.cmml">×</mo><msup id="S5.T3.12.m2.1.1.3" xref="S5.T3.12.m2.1.1.3.cmml"><mn id="S5.T3.12.m2.1.1.3.2" xref="S5.T3.12.m2.1.1.3.2.cmml">10</mn><mrow id="S5.T3.12.m2.1.1.3.3" xref="S5.T3.12.m2.1.1.3.3.cmml"><mo id="S5.T3.12.m2.1.1.3.3b" xref="S5.T3.12.m2.1.1.3.3.cmml">−</mo><mn id="S5.T3.12.m2.1.1.3.3.2" xref="S5.T3.12.m2.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.12.m2.1c"><apply id="S5.T3.12.m2.1.1.cmml" xref="S5.T3.12.m2.1.1"><times id="S5.T3.12.m2.1.1.1.cmml" xref="S5.T3.12.m2.1.1.1"></times><cn type="integer" id="S5.T3.12.m2.1.1.2.cmml" xref="S5.T3.12.m2.1.1.2">4</cn><apply id="S5.T3.12.m2.1.1.3.cmml" xref="S5.T3.12.m2.1.1.3"><csymbol cd="ambiguous" id="S5.T3.12.m2.1.1.3.1.cmml" xref="S5.T3.12.m2.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.12.m2.1.1.3.2.cmml" xref="S5.T3.12.m2.1.1.3.2">10</cn><apply id="S5.T3.12.m2.1.1.3.3.cmml" xref="S5.T3.12.m2.1.1.3.3"><minus id="S5.T3.12.m2.1.1.3.3.1.cmml" xref="S5.T3.12.m2.1.1.3.3"></minus><cn type="integer" id="S5.T3.12.m2.1.1.3.3.2.cmml" xref="S5.T3.12.m2.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.12.m2.1d">4\times 10^{-5}</annotation></semantics></math>, <math id="S5.T3.13.m3.1" class="ltx_Math" alttext="3\times 10^{-5}" display="inline"><semantics id="S5.T3.13.m3.1b"><mrow id="S5.T3.13.m3.1.1" xref="S5.T3.13.m3.1.1.cmml"><mn id="S5.T3.13.m3.1.1.2" xref="S5.T3.13.m3.1.1.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.13.m3.1.1.1" xref="S5.T3.13.m3.1.1.1.cmml">×</mo><msup id="S5.T3.13.m3.1.1.3" xref="S5.T3.13.m3.1.1.3.cmml"><mn id="S5.T3.13.m3.1.1.3.2" xref="S5.T3.13.m3.1.1.3.2.cmml">10</mn><mrow id="S5.T3.13.m3.1.1.3.3" xref="S5.T3.13.m3.1.1.3.3.cmml"><mo id="S5.T3.13.m3.1.1.3.3b" xref="S5.T3.13.m3.1.1.3.3.cmml">−</mo><mn id="S5.T3.13.m3.1.1.3.3.2" xref="S5.T3.13.m3.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.13.m3.1c"><apply id="S5.T3.13.m3.1.1.cmml" xref="S5.T3.13.m3.1.1"><times id="S5.T3.13.m3.1.1.1.cmml" xref="S5.T3.13.m3.1.1.1"></times><cn type="integer" id="S5.T3.13.m3.1.1.2.cmml" xref="S5.T3.13.m3.1.1.2">3</cn><apply id="S5.T3.13.m3.1.1.3.cmml" xref="S5.T3.13.m3.1.1.3"><csymbol cd="ambiguous" id="S5.T3.13.m3.1.1.3.1.cmml" xref="S5.T3.13.m3.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.13.m3.1.1.3.2.cmml" xref="S5.T3.13.m3.1.1.3.2">10</cn><apply id="S5.T3.13.m3.1.1.3.3.cmml" xref="S5.T3.13.m3.1.1.3.3"><minus id="S5.T3.13.m3.1.1.3.3.1.cmml" xref="S5.T3.13.m3.1.1.3.3"></minus><cn type="integer" id="S5.T3.13.m3.1.1.3.3.2.cmml" xref="S5.T3.13.m3.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.13.m3.1d">3\times 10^{-5}</annotation></semantics></math>, <math id="S5.T3.14.m4.1" class="ltx_Math" alttext="2\times 10^{-5}" display="inline"><semantics id="S5.T3.14.m4.1b"><mrow id="S5.T3.14.m4.1.1" xref="S5.T3.14.m4.1.1.cmml"><mn id="S5.T3.14.m4.1.1.2" xref="S5.T3.14.m4.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S5.T3.14.m4.1.1.1" xref="S5.T3.14.m4.1.1.1.cmml">×</mo><msup id="S5.T3.14.m4.1.1.3" xref="S5.T3.14.m4.1.1.3.cmml"><mn id="S5.T3.14.m4.1.1.3.2" xref="S5.T3.14.m4.1.1.3.2.cmml">10</mn><mrow id="S5.T3.14.m4.1.1.3.3" xref="S5.T3.14.m4.1.1.3.3.cmml"><mo id="S5.T3.14.m4.1.1.3.3b" xref="S5.T3.14.m4.1.1.3.3.cmml">−</mo><mn id="S5.T3.14.m4.1.1.3.3.2" xref="S5.T3.14.m4.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S5.T3.14.m4.1c"><apply id="S5.T3.14.m4.1.1.cmml" xref="S5.T3.14.m4.1.1"><times id="S5.T3.14.m4.1.1.1.cmml" xref="S5.T3.14.m4.1.1.1"></times><cn type="integer" id="S5.T3.14.m4.1.1.2.cmml" xref="S5.T3.14.m4.1.1.2">2</cn><apply id="S5.T3.14.m4.1.1.3.cmml" xref="S5.T3.14.m4.1.1.3"><csymbol cd="ambiguous" id="S5.T3.14.m4.1.1.3.1.cmml" xref="S5.T3.14.m4.1.1.3">superscript</csymbol><cn type="integer" id="S5.T3.14.m4.1.1.3.2.cmml" xref="S5.T3.14.m4.1.1.3.2">10</cn><apply id="S5.T3.14.m4.1.1.3.3.cmml" xref="S5.T3.14.m4.1.1.3.3"><minus id="S5.T3.14.m4.1.1.3.3.1.cmml" xref="S5.T3.14.m4.1.1.3.3"></minus><cn type="integer" id="S5.T3.14.m4.1.1.3.3.2.cmml" xref="S5.T3.14.m4.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.14.m4.1d">2\times 10^{-5}</annotation></semantics></math>} based on the accuracy of the validation set for BERT, following <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al<span class="ltx_text">.</span>, <a href="#bib.bib16" title="" class="ltx_ref">2018</a>)</cite>.
</figcaption>
</figure>
<figure id="S5.T4" class="ltx_table">
<table id="S5.T4.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T4.1.1.1" class="ltx_tr">
<td id="S5.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">Model</td>
<td id="S5.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="3">ResNet18</td>
<td id="S5.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="3">SENet18</td>
<td id="S5.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="3">VGG11</td>
</tr>
<tr id="S5.T4.1.2.2" class="ltx_tr">
<td id="S5.T4.1.2.2.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">Dataset</td>
<td id="S5.T4.1.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">LUT-NN</td>
<td id="S5.T4.1.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">MADDNESS</td>
<td id="S5.T4.1.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">baseline</td>
<td id="S5.T4.1.2.2.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">LUT-NN</td>
<td id="S5.T4.1.2.2.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">MADDNESS</td>
<td id="S5.T4.1.2.2.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">baseline</td>
<td id="S5.T4.1.2.2.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">LUT-NN</td>
<td id="S5.T4.1.2.2.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">MADDNESS</td>
<td id="S5.T4.1.2.2.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">baseline</td>
</tr>
<tr id="S5.T4.1.3.3" class="ltx_tr">
<td id="S5.T4.1.3.3.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">CIFAR10</td>
<td id="S5.T4.1.3.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">94.40</td>
<td id="S5.T4.1.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">10.01</td>
<td id="S5.T4.1.3.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">95.26</td>
<td id="S5.T4.1.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">94.48</td>
<td id="S5.T4.1.3.3.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">10.65</td>
<td id="S5.T4.1.3.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">95.47</td>
<td id="S5.T4.1.3.3.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">93.89</td>
<td id="S5.T4.1.3.3.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">22.87</td>
<td id="S5.T4.1.3.3.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">95.04</td>
</tr>
<tr id="S5.T4.1.4.4" class="ltx_tr">
<td id="S5.T4.1.4.4.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">GTSRB</td>
<td id="S5.T4.1.4.4.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">98.73</td>
<td id="S5.T4.1.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">4.53</td>
<td id="S5.T4.1.4.4.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">98.80</td>
<td id="S5.T4.1.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">98.36</td>
<td id="S5.T4.1.4.4.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.68</td>
<td id="S5.T4.1.4.4.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">98.84</td>
<td id="S5.T4.1.4.4.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">98.55</td>
<td id="S5.T4.1.4.4.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.70</td>
<td id="S5.T4.1.4.4.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">99.22</td>
</tr>
<tr id="S5.T4.1.5.5" class="ltx_tr">
<td id="S5.T4.1.5.5.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">
<table id="S5.T4.1.5.5.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T4.1.5.5.1.1.1" class="ltx_tr">
<td id="S5.T4.1.5.5.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Speech</td>
</tr>
<tr id="S5.T4.1.5.5.1.1.2" class="ltx_tr">
<td id="S5.T4.1.5.5.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Commands</td>
</tr>
</table>
</td>
<td id="S5.T4.1.5.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">93.70</td>
<td id="S5.T4.1.5.5.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.49</td>
<td id="S5.T4.1.5.5.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">91.72</td>
<td id="S5.T4.1.5.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">93.04</td>
<td id="S5.T4.1.5.5.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.49</td>
<td id="S5.T4.1.5.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">94.36</td>
<td id="S5.T4.1.5.5.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">93.38</td>
<td id="S5.T4.1.5.5.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.49</td>
<td id="S5.T4.1.5.5.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">93.11</td>
</tr>
<tr id="S5.T4.1.6.6" class="ltx_tr">
<td id="S5.T4.1.6.6.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">SVHN</td>
<td id="S5.T4.1.6.6.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.00</td>
<td id="S5.T4.1.6.6.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">20.68</td>
<td id="S5.T4.1.6.6.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.67</td>
<td id="S5.T4.1.6.6.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.22</td>
<td id="S5.T4.1.6.6.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">20.12</td>
<td id="S5.T4.1.6.6.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.60</td>
<td id="S5.T4.1.6.6.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.23</td>
<td id="S5.T4.1.6.6.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">29.97</td>
<td id="S5.T4.1.6.6.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">96.62</td>
</tr>
<tr id="S5.T4.1.7.7" class="ltx_tr">
<td id="S5.T4.1.7.7.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">UTKFace</td>
<td id="S5.T4.1.7.7.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">4.91</td>
<td id="S5.T4.1.7.7.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">10.51</td>
<td id="S5.T4.1.7.7.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.57</td>
<td id="S5.T4.1.7.7.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">4.74</td>
<td id="S5.T4.1.7.7.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">11.02</td>
<td id="S5.T4.1.7.7.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.46</td>
<td id="S5.T4.1.7.7.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.69</td>
<td id="S5.T4.1.7.7.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">24.57</td>
<td id="S5.T4.1.7.7.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.85</td>
</tr>
<tr id="S5.T4.1.8.8" class="ltx_tr">
<td id="S5.T4.1.8.8.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t">ImageNet</td>
<td id="S5.T4.1.8.8.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">67.38</td>
<td id="S5.T4.1.8.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">0.10</td>
<td id="S5.T4.1.8.8.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">69.76</td>
<td id="S5.T4.1.8.8.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">68.21</td>
<td id="S5.T4.1.8.8.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">0.17</td>
<td id="S5.T4.1.8.8.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">70.63</td>
<td id="S5.T4.1.8.8.8" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">68.04</td>
<td id="S5.T4.1.8.8.9" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">0.16</td>
<td id="S5.T4.1.8.8.10" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">68.33</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4. </span>
LUT-NN achieves comparable accuracy
with the original models, significantly outperforming <span id="S5.T4.3.1" class="ltx_text ltx_font_smallcaps">Maddness</span>. UTKFace is the lower the better. Others are the higher the better. </figcaption>
</figure>
<figure id="S5.T5" class="ltx_table">
<table id="S5.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T5.1.1.1" class="ltx_tr">
<th id="S5.T5.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t" rowspan="2"><span id="S5.T5.1.1.1.1.1" class="ltx_text">Dataset Task</span></th>
<th id="S5.T5.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T5.1.1.1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.1.2.1.1" class="ltx_tr">
<td id="S5.T5.1.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Single</td>
</tr>
<tr id="S5.T5.1.1.1.2.1.2" class="ltx_tr">
<td id="S5.T5.1.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Sentence</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T5.1.1.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.1.3.1.1" class="ltx_tr">
<td id="S5.T5.1.1.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Similarity and</td>
</tr>
<tr id="S5.T5.1.1.1.3.1.2" class="ltx_tr">
<td id="S5.T5.1.1.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Paraphrase</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="2">
<table id="S5.T5.1.1.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.1.4.1.1" class="ltx_tr">
<td id="S5.T5.1.1.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Natural Language</td>
</tr>
<tr id="S5.T5.1.1.1.4.1.2" class="ltx_tr">
<td id="S5.T5.1.1.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Inference</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.1.5" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t"></th>
</tr>
<tr id="S5.T5.1.2.2" class="ltx_tr">
<td id="S5.T5.1.2.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">SST-2</td>
<td id="S5.T5.1.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">QQP</td>
<td id="S5.T5.1.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">QNLI</td>
<td id="S5.T5.1.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">RTE</td>
<td id="S5.T5.1.2.2.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Average</td>
</tr>
<tr id="S5.T5.1.3.3" class="ltx_tr">
<td id="S5.T5.1.3.3.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">Training Dataset Size</td>
<td id="S5.T5.1.3.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">67k</td>
<td id="S5.T5.1.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">364k</td>
<td id="S5.T5.1.3.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">105k</td>
<td id="S5.T5.1.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.5k</td>
<td id="S5.T5.1.3.3.6" class="ltx_td ltx_border_r ltx_border_t"></td>
</tr>
<tr id="S5.T5.1.4.4" class="ltx_tr">
<td id="S5.T5.1.4.4.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">Test Dataset Size</td>
<td id="S5.T5.1.4.4.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.8k</td>
<td id="S5.T5.1.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">391k</td>
<td id="S5.T5.1.4.4.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.4k</td>
<td id="S5.T5.1.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">3k</td>
<td id="S5.T5.1.4.4.6" class="ltx_td ltx_border_r ltx_border_t"></td>
</tr>
<tr id="S5.T5.1.5.5" class="ltx_tr">
<td id="S5.T5.1.5.5.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">BERT base (%)</td>
<td id="S5.T5.1.5.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">93.5</td>
<td id="S5.T5.1.5.5.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">71.2</td>
<td id="S5.T5.1.5.5.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">90.5</td>
<td id="S5.T5.1.5.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">66.4</td>
<td id="S5.T5.1.5.5.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">80.4</td>
</tr>
<tr id="S5.T5.1.6.6" class="ltx_tr">
<td id="S5.T5.1.6.6.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t">LUT-NN (%)</td>
<td id="S5.T5.1.6.6.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">92.4</td>
<td id="S5.T5.1.6.6.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">69.6</td>
<td id="S5.T5.1.6.6.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">87.4</td>
<td id="S5.T5.1.6.6.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">64.7</td>
<td id="S5.T5.1.6.6.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">78.5</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5. </span>
LUT-NN achieves comparable accuracy to BERT-base on NLP tasks selected from GLUE.

</figcaption>
</figure>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6. </span>Evaluation</h2>

<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1. </span>Experiment methodologies and settings</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.14" class="ltx_p"><span id="S6.SS1.p1.14.1" class="ltx_text ltx_font_bold">Dataset and Models</span>
Our experiment tests LUT-NN’s effectiveness on image recognition, speech recognition, and NLP tasks.
For image recognition and speech recognition, we evaluate VGG, ResNet, and SENet family models on different datasets, including CIFAR-10 <cite class="ltx_cite ltx_citemacro_citep">(Krizhevsky, <a href="#bib.bib38" title="" class="ltx_ref">2009</a>)</cite>, GTSRB (German Traffic Sign Recognition Benchmark) <cite class="ltx_cite ltx_citemacro_citep">(Stallkamp et al<span class="ltx_text">.</span>, <a href="#bib.bib51" title="" class="ltx_ref">2011</a>)</cite>, Google Speech Command <cite class="ltx_cite ltx_citemacro_citep">(Warden, <a href="#bib.bib57" title="" class="ltx_ref">2018</a>)</cite>, SVHN (Street View House Numbers) <cite class="ltx_cite ltx_citemacro_citep">(Netzer et al<span class="ltx_text">.</span>, <a href="#bib.bib48" title="" class="ltx_ref">2011</a>)</cite>, UTKFace <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al<span class="ltx_text">.</span>, <a href="#bib.bib61" title="" class="ltx_ref">2017</a>)</cite>, and the large-scale complex ImageNet <cite class="ltx_cite ltx_citemacro_citep">(Deng et al<span class="ltx_text">.</span>, <a href="#bib.bib13" title="" class="ltx_ref">2009</a>)</cite> dataset.
The CIFAR-10 dataset consists of <math id="S6.SS1.p1.1.m1.1" class="ltx_Math" alttext="60K" display="inline"><semantics id="S6.SS1.p1.1.m1.1a"><mrow id="S6.SS1.p1.1.m1.1.1" xref="S6.SS1.p1.1.m1.1.1.cmml"><mn id="S6.SS1.p1.1.m1.1.1.2" xref="S6.SS1.p1.1.m1.1.1.2.cmml">60</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.1.m1.1.1.1" xref="S6.SS1.p1.1.m1.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.1.m1.1.1.3" xref="S6.SS1.p1.1.m1.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.1.m1.1b"><apply id="S6.SS1.p1.1.m1.1.1.cmml" xref="S6.SS1.p1.1.m1.1.1"><times id="S6.SS1.p1.1.m1.1.1.1.cmml" xref="S6.SS1.p1.1.m1.1.1.1"></times><cn type="integer" id="S6.SS1.p1.1.m1.1.1.2.cmml" xref="S6.SS1.p1.1.m1.1.1.2">60</cn><ci id="S6.SS1.p1.1.m1.1.1.3.cmml" xref="S6.SS1.p1.1.m1.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.1.m1.1c">60K</annotation></semantics></math> images in <math id="S6.SS1.p1.2.m2.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S6.SS1.p1.2.m2.1a"><mn id="S6.SS1.p1.2.m2.1.1" xref="S6.SS1.p1.2.m2.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.2.m2.1b"><cn type="integer" id="S6.SS1.p1.2.m2.1.1.cmml" xref="S6.SS1.p1.2.m2.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.2.m2.1c">10</annotation></semantics></math> classes, with <math id="S6.SS1.p1.3.m3.1" class="ltx_Math" alttext="50K" display="inline"><semantics id="S6.SS1.p1.3.m3.1a"><mrow id="S6.SS1.p1.3.m3.1.1" xref="S6.SS1.p1.3.m3.1.1.cmml"><mn id="S6.SS1.p1.3.m3.1.1.2" xref="S6.SS1.p1.3.m3.1.1.2.cmml">50</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.3.m3.1.1.1" xref="S6.SS1.p1.3.m3.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.3.m3.1.1.3" xref="S6.SS1.p1.3.m3.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.3.m3.1b"><apply id="S6.SS1.p1.3.m3.1.1.cmml" xref="S6.SS1.p1.3.m3.1.1"><times id="S6.SS1.p1.3.m3.1.1.1.cmml" xref="S6.SS1.p1.3.m3.1.1.1"></times><cn type="integer" id="S6.SS1.p1.3.m3.1.1.2.cmml" xref="S6.SS1.p1.3.m3.1.1.2">50</cn><ci id="S6.SS1.p1.3.m3.1.1.3.cmml" xref="S6.SS1.p1.3.m3.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.3.m3.1c">50K</annotation></semantics></math> for training and <math id="S6.SS1.p1.4.m4.1" class="ltx_Math" alttext="10K" display="inline"><semantics id="S6.SS1.p1.4.m4.1a"><mrow id="S6.SS1.p1.4.m4.1.1" xref="S6.SS1.p1.4.m4.1.1.cmml"><mn id="S6.SS1.p1.4.m4.1.1.2" xref="S6.SS1.p1.4.m4.1.1.2.cmml">10</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.4.m4.1.1.1" xref="S6.SS1.p1.4.m4.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.4.m4.1.1.3" xref="S6.SS1.p1.4.m4.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.4.m4.1b"><apply id="S6.SS1.p1.4.m4.1.1.cmml" xref="S6.SS1.p1.4.m4.1.1"><times id="S6.SS1.p1.4.m4.1.1.1.cmml" xref="S6.SS1.p1.4.m4.1.1.1"></times><cn type="integer" id="S6.SS1.p1.4.m4.1.1.2.cmml" xref="S6.SS1.p1.4.m4.1.1.2">10</cn><ci id="S6.SS1.p1.4.m4.1.1.3.cmml" xref="S6.SS1.p1.4.m4.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.4.m4.1c">10K</annotation></semantics></math> for validation.
The GTSRB dataset has <math id="S6.SS1.p1.5.m5.1" class="ltx_Math" alttext="43" display="inline"><semantics id="S6.SS1.p1.5.m5.1a"><mn id="S6.SS1.p1.5.m5.1.1" xref="S6.SS1.p1.5.m5.1.1.cmml">43</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.5.m5.1b"><cn type="integer" id="S6.SS1.p1.5.m5.1.1.cmml" xref="S6.SS1.p1.5.m5.1.1">43</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.5.m5.1c">43</annotation></semantics></math> classes of traffic signs, with <math id="S6.SS1.p1.6.m6.1" class="ltx_Math" alttext="39K" display="inline"><semantics id="S6.SS1.p1.6.m6.1a"><mrow id="S6.SS1.p1.6.m6.1.1" xref="S6.SS1.p1.6.m6.1.1.cmml"><mn id="S6.SS1.p1.6.m6.1.1.2" xref="S6.SS1.p1.6.m6.1.1.2.cmml">39</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.6.m6.1.1.1" xref="S6.SS1.p1.6.m6.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.6.m6.1.1.3" xref="S6.SS1.p1.6.m6.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.6.m6.1b"><apply id="S6.SS1.p1.6.m6.1.1.cmml" xref="S6.SS1.p1.6.m6.1.1"><times id="S6.SS1.p1.6.m6.1.1.1.cmml" xref="S6.SS1.p1.6.m6.1.1.1"></times><cn type="integer" id="S6.SS1.p1.6.m6.1.1.2.cmml" xref="S6.SS1.p1.6.m6.1.1.2">39</cn><ci id="S6.SS1.p1.6.m6.1.1.3.cmml" xref="S6.SS1.p1.6.m6.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.6.m6.1c">39K</annotation></semantics></math> training images and <math id="S6.SS1.p1.7.m7.1" class="ltx_Math" alttext="12K" display="inline"><semantics id="S6.SS1.p1.7.m7.1a"><mrow id="S6.SS1.p1.7.m7.1.1" xref="S6.SS1.p1.7.m7.1.1.cmml"><mn id="S6.SS1.p1.7.m7.1.1.2" xref="S6.SS1.p1.7.m7.1.1.2.cmml">12</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.7.m7.1.1.1" xref="S6.SS1.p1.7.m7.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.7.m7.1.1.3" xref="S6.SS1.p1.7.m7.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.7.m7.1b"><apply id="S6.SS1.p1.7.m7.1.1.cmml" xref="S6.SS1.p1.7.m7.1.1"><times id="S6.SS1.p1.7.m7.1.1.1.cmml" xref="S6.SS1.p1.7.m7.1.1.1"></times><cn type="integer" id="S6.SS1.p1.7.m7.1.1.2.cmml" xref="S6.SS1.p1.7.m7.1.1.2">12</cn><ci id="S6.SS1.p1.7.m7.1.1.3.cmml" xref="S6.SS1.p1.7.m7.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.7.m7.1c">12K</annotation></semantics></math> test images.
Google Speech Command is an audio dataset containing <math id="S6.SS1.p1.8.m8.1" class="ltx_Math" alttext="65K" display="inline"><semantics id="S6.SS1.p1.8.m8.1a"><mrow id="S6.SS1.p1.8.m8.1.1" xref="S6.SS1.p1.8.m8.1.1.cmml"><mn id="S6.SS1.p1.8.m8.1.1.2" xref="S6.SS1.p1.8.m8.1.1.2.cmml">65</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.8.m8.1.1.1" xref="S6.SS1.p1.8.m8.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.8.m8.1.1.3" xref="S6.SS1.p1.8.m8.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.8.m8.1b"><apply id="S6.SS1.p1.8.m8.1.1.cmml" xref="S6.SS1.p1.8.m8.1.1"><times id="S6.SS1.p1.8.m8.1.1.1.cmml" xref="S6.SS1.p1.8.m8.1.1.1"></times><cn type="integer" id="S6.SS1.p1.8.m8.1.1.2.cmml" xref="S6.SS1.p1.8.m8.1.1.2">65</cn><ci id="S6.SS1.p1.8.m8.1.1.3.cmml" xref="S6.SS1.p1.8.m8.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.8.m8.1c">65K</annotation></semantics></math> recordings of <math id="S6.SS1.p1.9.m9.1" class="ltx_Math" alttext="30" display="inline"><semantics id="S6.SS1.p1.9.m9.1a"><mn id="S6.SS1.p1.9.m9.1.1" xref="S6.SS1.p1.9.m9.1.1.cmml">30</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.9.m9.1b"><cn type="integer" id="S6.SS1.p1.9.m9.1.1.cmml" xref="S6.SS1.p1.9.m9.1.1">30</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.9.m9.1c">30</annotation></semantics></math> words, designed for keyword spotting on edge devices.
SVHN is a dataset including <math id="S6.SS1.p1.10.m10.1" class="ltx_Math" alttext="600K" display="inline"><semantics id="S6.SS1.p1.10.m10.1a"><mrow id="S6.SS1.p1.10.m10.1.1" xref="S6.SS1.p1.10.m10.1.1.cmml"><mn id="S6.SS1.p1.10.m10.1.1.2" xref="S6.SS1.p1.10.m10.1.1.2.cmml">600</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.10.m10.1.1.1" xref="S6.SS1.p1.10.m10.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.10.m10.1.1.3" xref="S6.SS1.p1.10.m10.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.10.m10.1b"><apply id="S6.SS1.p1.10.m10.1.1.cmml" xref="S6.SS1.p1.10.m10.1.1"><times id="S6.SS1.p1.10.m10.1.1.1.cmml" xref="S6.SS1.p1.10.m10.1.1.1"></times><cn type="integer" id="S6.SS1.p1.10.m10.1.1.2.cmml" xref="S6.SS1.p1.10.m10.1.1.2">600</cn><ci id="S6.SS1.p1.10.m10.1.1.3.cmml" xref="S6.SS1.p1.10.m10.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.10.m10.1c">600K</annotation></semantics></math> real-world images of house numbers for <math id="S6.SS1.p1.11.m11.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S6.SS1.p1.11.m11.1a"><mn id="S6.SS1.p1.11.m11.1.1" xref="S6.SS1.p1.11.m11.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.11.m11.1b"><cn type="integer" id="S6.SS1.p1.11.m11.1.1.cmml" xref="S6.SS1.p1.11.m11.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.11.m11.1c">10</annotation></semantics></math> classes.
UTKFace dataset consists of <math id="S6.SS1.p1.12.m12.1" class="ltx_Math" alttext="20K" display="inline"><semantics id="S6.SS1.p1.12.m12.1a"><mrow id="S6.SS1.p1.12.m12.1.1" xref="S6.SS1.p1.12.m12.1.1.cmml"><mn id="S6.SS1.p1.12.m12.1.1.2" xref="S6.SS1.p1.12.m12.1.1.2.cmml">20</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.12.m12.1.1.1" xref="S6.SS1.p1.12.m12.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.12.m12.1.1.3" xref="S6.SS1.p1.12.m12.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.12.m12.1b"><apply id="S6.SS1.p1.12.m12.1.1.cmml" xref="S6.SS1.p1.12.m12.1.1"><times id="S6.SS1.p1.12.m12.1.1.1.cmml" xref="S6.SS1.p1.12.m12.1.1.1"></times><cn type="integer" id="S6.SS1.p1.12.m12.1.1.2.cmml" xref="S6.SS1.p1.12.m12.1.1.2">20</cn><ci id="S6.SS1.p1.12.m12.1.1.3.cmml" xref="S6.SS1.p1.12.m12.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.12.m12.1c">20K</annotation></semantics></math> face images labeled by age, gender, and ethnicity. We use the age prediction task to test the regression ability of LUT-NN. We will report Mean Average Error (MAE) for this task.
ImageNet dataset has <math id="S6.SS1.p1.13.m13.1" class="ltx_Math" alttext="1.28M" display="inline"><semantics id="S6.SS1.p1.13.m13.1a"><mrow id="S6.SS1.p1.13.m13.1.1" xref="S6.SS1.p1.13.m13.1.1.cmml"><mn id="S6.SS1.p1.13.m13.1.1.2" xref="S6.SS1.p1.13.m13.1.1.2.cmml">1.28</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.13.m13.1.1.1" xref="S6.SS1.p1.13.m13.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.13.m13.1.1.3" xref="S6.SS1.p1.13.m13.1.1.3.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.13.m13.1b"><apply id="S6.SS1.p1.13.m13.1.1.cmml" xref="S6.SS1.p1.13.m13.1.1"><times id="S6.SS1.p1.13.m13.1.1.1.cmml" xref="S6.SS1.p1.13.m13.1.1.1"></times><cn type="float" id="S6.SS1.p1.13.m13.1.1.2.cmml" xref="S6.SS1.p1.13.m13.1.1.2">1.28</cn><ci id="S6.SS1.p1.13.m13.1.1.3.cmml" xref="S6.SS1.p1.13.m13.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.13.m13.1c">1.28M</annotation></semantics></math> images for training and <math id="S6.SS1.p1.14.m14.1" class="ltx_Math" alttext="50K" display="inline"><semantics id="S6.SS1.p1.14.m14.1a"><mrow id="S6.SS1.p1.14.m14.1.1" xref="S6.SS1.p1.14.m14.1.1.cmml"><mn id="S6.SS1.p1.14.m14.1.1.2" xref="S6.SS1.p1.14.m14.1.1.2.cmml">50</mn><mo lspace="0em" rspace="0em" id="S6.SS1.p1.14.m14.1.1.1" xref="S6.SS1.p1.14.m14.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.14.m14.1.1.3" xref="S6.SS1.p1.14.m14.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.14.m14.1b"><apply id="S6.SS1.p1.14.m14.1.1.cmml" xref="S6.SS1.p1.14.m14.1.1"><times id="S6.SS1.p1.14.m14.1.1.1.cmml" xref="S6.SS1.p1.14.m14.1.1.1"></times><cn type="integer" id="S6.SS1.p1.14.m14.1.1.2.cmml" xref="S6.SS1.p1.14.m14.1.1.2">50</cn><ci id="S6.SS1.p1.14.m14.1.1.3.cmml" xref="S6.SS1.p1.14.m14.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.14.m14.1c">50K</annotation></semantics></math> for validation.</p>
</div>
<div id="S6.SS1.p2" class="ltx_para">
<p id="S6.SS1.p2.1" class="ltx_p">On the large ImageNet dataset, ResNet18 and SENet18 models use 7x7 convolution for the first layer and 3x3 max-pooling, while other datasets use 3x3 convolution for the first layer and no max-pooling.
The final three dense layers of the original VGG models are replaced by an average pooling layer and a dense layer to reduce model size for mobile deployment, following the general practice for VGG deployment <cite class="ltx_cite ltx_citemacro_citep">(Wang et al<span class="ltx_text">.</span>, <a href="#bib.bib56" title="" class="ltx_ref">2021</a>)</cite>. Except for ImageNet, the first max pooling layer is removed for VGG. In evaluation results, the models for ImageNet use the model names, and the models for other datasets are marked as ”Model (CIFAR10)” to differentiate.</p>
</div>
<div id="S6.SS1.p3" class="ltx_para">
<p id="S6.SS1.p3.1" class="ltx_p">To evaluate LUT-NN for NLP tasks, we used BERT models and the tasks from GLUE benchmark <cite class="ltx_cite ltx_citemacro_citep">(Wang et al<span class="ltx_text">.</span>, <a href="#bib.bib54" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
<div id="S6.SS1.p4" class="ltx_para ltx_noindent">
<p id="S6.SS1.p4.2" class="ltx_p"><span id="S6.SS1.p4.2.1" class="ltx_text ltx_font_bold">Soft-PQ training setting</span>
Table <a href="#S5.T3" title="Table 3 ‣ 5.3. Utilization of memory hierarchy ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> lists the detailed training hyperparameter settings.
Except for the initial learning rate, we followed standard practices for all other training procedures <cite class="ltx_cite ltx_citemacro_citep">(He et al<span class="ltx_text">.</span>, <a href="#bib.bib28" title="" class="ltx_ref">2016</a>; Devlin et al<span class="ltx_text">.</span>, <a href="#bib.bib16" title="" class="ltx_ref">2018</a>)</cite>.
As the learned temperature requires a larger learning rate to converge quickly to a sub-optimal solution, we use different learning rates for the centroid and temperature learning.
Prior to soft-PQ training, we initialize centroids using <math id="S6.SS1.p4.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S6.SS1.p4.1.m1.1a"><mi id="S6.SS1.p4.1.m1.1.1" xref="S6.SS1.p4.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p4.1.m1.1b"><ci id="S6.SS1.p4.1.m1.1.1.cmml" xref="S6.SS1.p4.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p4.1.m1.1c">k</annotation></semantics></math>-means clustering.
Specifically, we apply the original model inference to a randomly sampled sub-dataset (consisting of 1024 training samples) and collect each operator’s inputs.
We then utilize the <math id="S6.SS1.p4.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S6.SS1.p4.2.m2.1a"><mi id="S6.SS1.p4.2.m2.1.1" xref="S6.SS1.p4.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p4.2.m2.1b"><ci id="S6.SS1.p4.2.m2.1.1.cmml" xref="S6.SS1.p4.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p4.2.m2.1c">k</annotation></semantics></math>-means algorithm to cluster each operator’s inputs and obtain the initial centroids.</p>
</div>
<div id="S6.SS1.p5" class="ltx_para ltx_noindent">
<p id="S6.SS1.p5.8" class="ltx_p"><span id="S6.SS1.p5.8.1" class="ltx_text ltx_font_bold">LUT-NN system settings</span> In order to balance accuracy and cost, we can specify various settings for LUT-NN, including the number of centroids in a codebook (<math id="S6.SS1.p5.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S6.SS1.p5.1.m1.1a"><mi id="S6.SS1.p5.1.m1.1.1" xref="S6.SS1.p5.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.1.m1.1b"><ci id="S6.SS1.p5.1.m1.1.1.cmml" xref="S6.SS1.p5.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.1.m1.1c">K</annotation></semantics></math>), the length of the sub-vector (<math id="S6.SS1.p5.2.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S6.SS1.p5.2.m2.1a"><mi id="S6.SS1.p5.2.m2.1.1" xref="S6.SS1.p5.2.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.2.m2.1b"><ci id="S6.SS1.p5.2.m2.1.1.cmml" xref="S6.SS1.p5.2.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.2.m2.1c">V</annotation></semantics></math>) (as shown in Table <a href="#S4.T1" title="Table 1 ‣ 4. cost reduction of LUT-NN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), and the operators to be replaced by table lookup.
The default setting for <math id="S6.SS1.p5.3.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S6.SS1.p5.3.m3.1a"><mi id="S6.SS1.p5.3.m3.1.1" xref="S6.SS1.p5.3.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.3.m3.1b"><ci id="S6.SS1.p5.3.m3.1.1.cmml" xref="S6.SS1.p5.3.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.3.m3.1c">K</annotation></semantics></math> and <math id="S6.SS1.p5.4.m4.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S6.SS1.p5.4.m4.1a"><mi id="S6.SS1.p5.4.m4.1.1" xref="S6.SS1.p5.4.m4.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.4.m4.1b"><ci id="S6.SS1.p5.4.m4.1.1.cmml" xref="S6.SS1.p5.4.m4.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.4.m4.1c">V</annotation></semantics></math> is to align with both the feature size and the length of SIMD instructions.
Specifically, we set <math id="S6.SS1.p5.5.m5.4" class="ltx_Math" alttext="(K,V)=(16,9)" display="inline"><semantics id="S6.SS1.p5.5.m5.4a"><mrow id="S6.SS1.p5.5.m5.4.5" xref="S6.SS1.p5.5.m5.4.5.cmml"><mrow id="S6.SS1.p5.5.m5.4.5.2.2" xref="S6.SS1.p5.5.m5.4.5.2.1.cmml"><mo stretchy="false" id="S6.SS1.p5.5.m5.4.5.2.2.1" xref="S6.SS1.p5.5.m5.4.5.2.1.cmml">(</mo><mi id="S6.SS1.p5.5.m5.1.1" xref="S6.SS1.p5.5.m5.1.1.cmml">K</mi><mo id="S6.SS1.p5.5.m5.4.5.2.2.2" xref="S6.SS1.p5.5.m5.4.5.2.1.cmml">,</mo><mi id="S6.SS1.p5.5.m5.2.2" xref="S6.SS1.p5.5.m5.2.2.cmml">V</mi><mo stretchy="false" id="S6.SS1.p5.5.m5.4.5.2.2.3" xref="S6.SS1.p5.5.m5.4.5.2.1.cmml">)</mo></mrow><mo id="S6.SS1.p5.5.m5.4.5.1" xref="S6.SS1.p5.5.m5.4.5.1.cmml">=</mo><mrow id="S6.SS1.p5.5.m5.4.5.3.2" xref="S6.SS1.p5.5.m5.4.5.3.1.cmml"><mo stretchy="false" id="S6.SS1.p5.5.m5.4.5.3.2.1" xref="S6.SS1.p5.5.m5.4.5.3.1.cmml">(</mo><mn id="S6.SS1.p5.5.m5.3.3" xref="S6.SS1.p5.5.m5.3.3.cmml">16</mn><mo id="S6.SS1.p5.5.m5.4.5.3.2.2" xref="S6.SS1.p5.5.m5.4.5.3.1.cmml">,</mo><mn id="S6.SS1.p5.5.m5.4.4" xref="S6.SS1.p5.5.m5.4.4.cmml">9</mn><mo stretchy="false" id="S6.SS1.p5.5.m5.4.5.3.2.3" xref="S6.SS1.p5.5.m5.4.5.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.5.m5.4b"><apply id="S6.SS1.p5.5.m5.4.5.cmml" xref="S6.SS1.p5.5.m5.4.5"><eq id="S6.SS1.p5.5.m5.4.5.1.cmml" xref="S6.SS1.p5.5.m5.4.5.1"></eq><interval closure="open" id="S6.SS1.p5.5.m5.4.5.2.1.cmml" xref="S6.SS1.p5.5.m5.4.5.2.2"><ci id="S6.SS1.p5.5.m5.1.1.cmml" xref="S6.SS1.p5.5.m5.1.1">𝐾</ci><ci id="S6.SS1.p5.5.m5.2.2.cmml" xref="S6.SS1.p5.5.m5.2.2">𝑉</ci></interval><interval closure="open" id="S6.SS1.p5.5.m5.4.5.3.1.cmml" xref="S6.SS1.p5.5.m5.4.5.3.2"><cn type="integer" id="S6.SS1.p5.5.m5.3.3.cmml" xref="S6.SS1.p5.5.m5.3.3">16</cn><cn type="integer" id="S6.SS1.p5.5.m5.4.4.cmml" xref="S6.SS1.p5.5.m5.4.4">9</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.5.m5.4c">(K,V)=(16,9)</annotation></semantics></math> for <math id="S6.SS1.p5.6.m6.1" class="ltx_Math" alttext="3\times 3" display="inline"><semantics id="S6.SS1.p5.6.m6.1a"><mrow id="S6.SS1.p5.6.m6.1.1" xref="S6.SS1.p5.6.m6.1.1.cmml"><mn id="S6.SS1.p5.6.m6.1.1.2" xref="S6.SS1.p5.6.m6.1.1.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS1.p5.6.m6.1.1.1" xref="S6.SS1.p5.6.m6.1.1.1.cmml">×</mo><mn id="S6.SS1.p5.6.m6.1.1.3" xref="S6.SS1.p5.6.m6.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.6.m6.1b"><apply id="S6.SS1.p5.6.m6.1.1.cmml" xref="S6.SS1.p5.6.m6.1.1"><times id="S6.SS1.p5.6.m6.1.1.1.cmml" xref="S6.SS1.p5.6.m6.1.1.1"></times><cn type="integer" id="S6.SS1.p5.6.m6.1.1.2.cmml" xref="S6.SS1.p5.6.m6.1.1.2">3</cn><cn type="integer" id="S6.SS1.p5.6.m6.1.1.3.cmml" xref="S6.SS1.p5.6.m6.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.6.m6.1c">3\times 3</annotation></semantics></math> convolution, <math id="S6.SS1.p5.7.m7.4" class="ltx_Math" alttext="(K,V)=(16,4)" display="inline"><semantics id="S6.SS1.p5.7.m7.4a"><mrow id="S6.SS1.p5.7.m7.4.5" xref="S6.SS1.p5.7.m7.4.5.cmml"><mrow id="S6.SS1.p5.7.m7.4.5.2.2" xref="S6.SS1.p5.7.m7.4.5.2.1.cmml"><mo stretchy="false" id="S6.SS1.p5.7.m7.4.5.2.2.1" xref="S6.SS1.p5.7.m7.4.5.2.1.cmml">(</mo><mi id="S6.SS1.p5.7.m7.1.1" xref="S6.SS1.p5.7.m7.1.1.cmml">K</mi><mo id="S6.SS1.p5.7.m7.4.5.2.2.2" xref="S6.SS1.p5.7.m7.4.5.2.1.cmml">,</mo><mi id="S6.SS1.p5.7.m7.2.2" xref="S6.SS1.p5.7.m7.2.2.cmml">V</mi><mo stretchy="false" id="S6.SS1.p5.7.m7.4.5.2.2.3" xref="S6.SS1.p5.7.m7.4.5.2.1.cmml">)</mo></mrow><mo id="S6.SS1.p5.7.m7.4.5.1" xref="S6.SS1.p5.7.m7.4.5.1.cmml">=</mo><mrow id="S6.SS1.p5.7.m7.4.5.3.2" xref="S6.SS1.p5.7.m7.4.5.3.1.cmml"><mo stretchy="false" id="S6.SS1.p5.7.m7.4.5.3.2.1" xref="S6.SS1.p5.7.m7.4.5.3.1.cmml">(</mo><mn id="S6.SS1.p5.7.m7.3.3" xref="S6.SS1.p5.7.m7.3.3.cmml">16</mn><mo id="S6.SS1.p5.7.m7.4.5.3.2.2" xref="S6.SS1.p5.7.m7.4.5.3.1.cmml">,</mo><mn id="S6.SS1.p5.7.m7.4.4" xref="S6.SS1.p5.7.m7.4.4.cmml">4</mn><mo stretchy="false" id="S6.SS1.p5.7.m7.4.5.3.2.3" xref="S6.SS1.p5.7.m7.4.5.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.7.m7.4b"><apply id="S6.SS1.p5.7.m7.4.5.cmml" xref="S6.SS1.p5.7.m7.4.5"><eq id="S6.SS1.p5.7.m7.4.5.1.cmml" xref="S6.SS1.p5.7.m7.4.5.1"></eq><interval closure="open" id="S6.SS1.p5.7.m7.4.5.2.1.cmml" xref="S6.SS1.p5.7.m7.4.5.2.2"><ci id="S6.SS1.p5.7.m7.1.1.cmml" xref="S6.SS1.p5.7.m7.1.1">𝐾</ci><ci id="S6.SS1.p5.7.m7.2.2.cmml" xref="S6.SS1.p5.7.m7.2.2">𝑉</ci></interval><interval closure="open" id="S6.SS1.p5.7.m7.4.5.3.1.cmml" xref="S6.SS1.p5.7.m7.4.5.3.2"><cn type="integer" id="S6.SS1.p5.7.m7.3.3.cmml" xref="S6.SS1.p5.7.m7.3.3">16</cn><cn type="integer" id="S6.SS1.p5.7.m7.4.4.cmml" xref="S6.SS1.p5.7.m7.4.4">4</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.7.m7.4c">(K,V)=(16,4)</annotation></semantics></math> for <math id="S6.SS1.p5.8.m8.1" class="ltx_Math" alttext="1\times 1" display="inline"><semantics id="S6.SS1.p5.8.m8.1a"><mrow id="S6.SS1.p5.8.m8.1.1" xref="S6.SS1.p5.8.m8.1.1.cmml"><mn id="S6.SS1.p5.8.m8.1.1.2" xref="S6.SS1.p5.8.m8.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS1.p5.8.m8.1.1.1" xref="S6.SS1.p5.8.m8.1.1.1.cmml">×</mo><mn id="S6.SS1.p5.8.m8.1.1.3" xref="S6.SS1.p5.8.m8.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p5.8.m8.1b"><apply id="S6.SS1.p5.8.m8.1.1.cmml" xref="S6.SS1.p5.8.m8.1.1"><times id="S6.SS1.p5.8.m8.1.1.1.cmml" xref="S6.SS1.p5.8.m8.1.1.1"></times><cn type="integer" id="S6.SS1.p5.8.m8.1.1.2.cmml" xref="S6.SS1.p5.8.m8.1.1.2">1</cn><cn type="integer" id="S6.SS1.p5.8.m8.1.1.3.cmml" xref="S6.SS1.p5.8.m8.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p5.8.m8.1c">1\times 1</annotation></semantics></math> convolution, We will also evaluate other settings in Sec. <a href="#S6.SS3" title="6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.3</span></a> ablation study.</p>
</div>
<div id="S6.SS1.p6" class="ltx_para">
<p id="S6.SS1.p6.1" class="ltx_p">For operators to replace, we replace all convolution operators for CNN models by table lookup except the first one. This is because as explained by Zhou <span id="S6.SS1.p6.1.1" class="ltx_text"><em id="S6.SS1.p6.1.1.1" class="ltx_emph ltx_font_italic">et al.</em></span> <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al<span class="ltx_text">.</span>, <a href="#bib.bib62" title="" class="ltx_ref">2016a</a>)</cite>, the layer interacted with the model input is very sensitive to accuracy. We also observe this phenomenon. Replacing the first layer by table-lookup can lead to obvious accuracy drop. This issue is even more serious for BERT. Replacing the first two layers results in 80% accuracy loss (refer to Fig. <a href="#S6.F13" title="Figure 13 ‣ 6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>).
Therefore, we do not replace these layers by table lookup. For BERT, other than explicitly stated, we replace the fully connected operators of the last 6 layers. The results with different replaced layers for BERT will be shown in Section <a href="#S6.SS3" title="6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.3</span></a> ablation study.</p>
</div>
<div id="S6.SS1.p7" class="ltx_para ltx_noindent">
<p id="S6.SS1.p7.4" class="ltx_p"><span id="S6.SS1.p7.4.1" class="ltx_text ltx_font_bold">Evaluation platforms</span>
LUT-NN is implemented on ARM Neon and Intel SIMD ISA (Instruction Set Architecture), and can run in a single or multi threads. We evaluate LUT-NN on two mobile devices Google Pixel 4 and 6, which are equipped with Cortex-A76 (<math id="S6.SS1.p7.1.m1.1" class="ltx_Math" alttext="2.42" display="inline"><semantics id="S6.SS1.p7.1.m1.1a"><mn id="S6.SS1.p7.1.m1.1.1" xref="S6.SS1.p7.1.m1.1.1.cmml">2.42</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p7.1.m1.1b"><cn type="float" id="S6.SS1.p7.1.m1.1.1.cmml" xref="S6.SS1.p7.1.m1.1.1">2.42</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p7.1.m1.1c">2.42</annotation></semantics></math> GHz) and Cortex-X1 (<math id="S6.SS1.p7.2.m2.1" class="ltx_Math" alttext="2.80" display="inline"><semantics id="S6.SS1.p7.2.m2.1a"><mn id="S6.SS1.p7.2.m2.1.1" xref="S6.SS1.p7.2.m2.1.1.cmml">2.80</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p7.2.m2.1b"><cn type="float" id="S6.SS1.p7.2.m2.1.1.cmml" xref="S6.SS1.p7.2.m2.1.1">2.80</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p7.2.m2.1c">2.80</annotation></semantics></math> GHz), respectively.
In addition, we evaluate LUT-NN on a desktop CPU (Intel Core i7-4790) running at <math id="S6.SS1.p7.3.m3.1" class="ltx_Math" alttext="3.60" display="inline"><semantics id="S6.SS1.p7.3.m3.1a"><mn id="S6.SS1.p7.3.m3.1.1" xref="S6.SS1.p7.3.m3.1.1.cmml">3.60</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p7.3.m3.1b"><cn type="float" id="S6.SS1.p7.3.m3.1.1.cmml" xref="S6.SS1.p7.3.m3.1.1">3.60</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p7.3.m3.1c">3.60</annotation></semantics></math> GHz and a server CPU (Xeon Silver 4210) running at <math id="S6.SS1.p7.4.m4.1" class="ltx_Math" alttext="2.20" display="inline"><semantics id="S6.SS1.p7.4.m4.1a"><mn id="S6.SS1.p7.4.m4.1.1" xref="S6.SS1.p7.4.m4.1.1.cmml">2.20</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p7.4.m4.1b"><cn type="float" id="S6.SS1.p7.4.m4.1.1.cmml" xref="S6.SS1.p7.4.m4.1.1">2.20</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p7.4.m4.1c">2.20</annotation></semantics></math> GHz.</p>
</div>
<div id="S6.SS1.p8" class="ltx_para ltx_noindent">
<p id="S6.SS1.p8.1" class="ltx_p"><span id="S6.SS1.p8.1.1" class="ltx_text ltx_font_bold">Comparison baselines</span> The accuracy baselines are (1) original models and (2) <span id="S6.SS1.p8.1.2" class="ltx_text ltx_font_smallcaps">Maddness</span> <cite class="ltx_cite ltx_citemacro_citep">(Blalock and Guttag, <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>, respectively. The inference performance baselines are ONNX Runtime (ORT) v1.12.1 <cite class="ltx_cite ltx_citemacro_citep">(Microsoft, <a href="#bib.bib47" title="" class="ltx_ref">2019</a>)</cite> and TVM v0.9.0 <cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib10" title="" class="ltx_ref">2018</a>)</cite>, as the SOTA hand-written performance, and the auto-tuned performance upper bound for linear computation opeartors. We tune the TVM baselines using AutoScheduler and AutoTVM in <math id="S6.SS1.p8.1.m1.1" class="ltx_Math" alttext="1500" display="inline"><semantics id="S6.SS1.p8.1.m1.1a"><mn id="S6.SS1.p8.1.m1.1.1" xref="S6.SS1.p8.1.m1.1.1.cmml">1500</mn><annotation-xml encoding="MathML-Content" id="S6.SS1.p8.1.m1.1b"><cn type="integer" id="S6.SS1.p8.1.m1.1.1.cmml" xref="S6.SS1.p8.1.m1.1.1">1500</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p8.1.m1.1c">1500</annotation></semantics></math> iterations.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2. </span>Accuracy, latency, memory and power</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.8" class="ltx_p"><span id="S6.SS2.p1.8.1" class="ltx_text ltx_font_bold">Accuracy</span>
Table <a href="#S5.T4" title="Table 4 ‣ 5.3. Utilization of memory hierarchy ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and <a href="#S5.T5" title="Table 5 ‣ 5.3. Utilization of memory hierarchy ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> summarize the accuracy comparison. Remarkably, to compare with directly applying <span id="S6.SS2.p1.8.2" class="ltx_text ltx_font_smallcaps">maddness</span>, LUT-NN can improve accuracy by 66% to 92%, achieving similar-level of accuracy with the original models. LUT-NN behaves particularly well on GTSRB (<math id="S6.SS2.p1.1.m1.1" class="ltx_Math" alttext="\leq 0.67\%" display="inline"><semantics id="S6.SS2.p1.1.m1.1a"><mrow id="S6.SS2.p1.1.m1.1.1" xref="S6.SS2.p1.1.m1.1.1.cmml"><mi id="S6.SS2.p1.1.m1.1.1.2" xref="S6.SS2.p1.1.m1.1.1.2.cmml"></mi><mo id="S6.SS2.p1.1.m1.1.1.1" xref="S6.SS2.p1.1.m1.1.1.1.cmml">≤</mo><mrow id="S6.SS2.p1.1.m1.1.1.3" xref="S6.SS2.p1.1.m1.1.1.3.cmml"><mn id="S6.SS2.p1.1.m1.1.1.3.2" xref="S6.SS2.p1.1.m1.1.1.3.2.cmml">0.67</mn><mo id="S6.SS2.p1.1.m1.1.1.3.1" xref="S6.SS2.p1.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.1.m1.1b"><apply id="S6.SS2.p1.1.m1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1"><leq id="S6.SS2.p1.1.m1.1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1.1"></leq><csymbol cd="latexml" id="S6.SS2.p1.1.m1.1.1.2.cmml" xref="S6.SS2.p1.1.m1.1.1.2">absent</csymbol><apply id="S6.SS2.p1.1.m1.1.1.3.cmml" xref="S6.SS2.p1.1.m1.1.1.3"><csymbol cd="latexml" id="S6.SS2.p1.1.m1.1.1.3.1.cmml" xref="S6.SS2.p1.1.m1.1.1.3.1">percent</csymbol><cn type="float" id="S6.SS2.p1.1.m1.1.1.3.2.cmml" xref="S6.SS2.p1.1.m1.1.1.3.2">0.67</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.1.m1.1c">\leq 0.67\%</annotation></semantics></math> difference), SVHN (<math id="S6.SS2.p1.2.m2.1" class="ltx_Math" alttext="\leq 0.67\%" display="inline"><semantics id="S6.SS2.p1.2.m2.1a"><mrow id="S6.SS2.p1.2.m2.1.1" xref="S6.SS2.p1.2.m2.1.1.cmml"><mi id="S6.SS2.p1.2.m2.1.1.2" xref="S6.SS2.p1.2.m2.1.1.2.cmml"></mi><mo id="S6.SS2.p1.2.m2.1.1.1" xref="S6.SS2.p1.2.m2.1.1.1.cmml">≤</mo><mrow id="S6.SS2.p1.2.m2.1.1.3" xref="S6.SS2.p1.2.m2.1.1.3.cmml"><mn id="S6.SS2.p1.2.m2.1.1.3.2" xref="S6.SS2.p1.2.m2.1.1.3.2.cmml">0.67</mn><mo id="S6.SS2.p1.2.m2.1.1.3.1" xref="S6.SS2.p1.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.2.m2.1b"><apply id="S6.SS2.p1.2.m2.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1"><leq id="S6.SS2.p1.2.m2.1.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1.1"></leq><csymbol cd="latexml" id="S6.SS2.p1.2.m2.1.1.2.cmml" xref="S6.SS2.p1.2.m2.1.1.2">absent</csymbol><apply id="S6.SS2.p1.2.m2.1.1.3.cmml" xref="S6.SS2.p1.2.m2.1.1.3"><csymbol cd="latexml" id="S6.SS2.p1.2.m2.1.1.3.1.cmml" xref="S6.SS2.p1.2.m2.1.1.3.1">percent</csymbol><cn type="float" id="S6.SS2.p1.2.m2.1.1.3.2.cmml" xref="S6.SS2.p1.2.m2.1.1.3.2">0.67</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.2.m2.1c">\leq 0.67\%</annotation></semantics></math> difference), CIFAR-10 (<math id="S6.SS2.p1.3.m3.1" class="ltx_Math" alttext="\leq 1.15\%" display="inline"><semantics id="S6.SS2.p1.3.m3.1a"><mrow id="S6.SS2.p1.3.m3.1.1" xref="S6.SS2.p1.3.m3.1.1.cmml"><mi id="S6.SS2.p1.3.m3.1.1.2" xref="S6.SS2.p1.3.m3.1.1.2.cmml"></mi><mo id="S6.SS2.p1.3.m3.1.1.1" xref="S6.SS2.p1.3.m3.1.1.1.cmml">≤</mo><mrow id="S6.SS2.p1.3.m3.1.1.3" xref="S6.SS2.p1.3.m3.1.1.3.cmml"><mn id="S6.SS2.p1.3.m3.1.1.3.2" xref="S6.SS2.p1.3.m3.1.1.3.2.cmml">1.15</mn><mo id="S6.SS2.p1.3.m3.1.1.3.1" xref="S6.SS2.p1.3.m3.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.3.m3.1b"><apply id="S6.SS2.p1.3.m3.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1"><leq id="S6.SS2.p1.3.m3.1.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1.1"></leq><csymbol cd="latexml" id="S6.SS2.p1.3.m3.1.1.2.cmml" xref="S6.SS2.p1.3.m3.1.1.2">absent</csymbol><apply id="S6.SS2.p1.3.m3.1.1.3.cmml" xref="S6.SS2.p1.3.m3.1.1.3"><csymbol cd="latexml" id="S6.SS2.p1.3.m3.1.1.3.1.cmml" xref="S6.SS2.p1.3.m3.1.1.3.1">percent</csymbol><cn type="float" id="S6.SS2.p1.3.m3.1.1.3.2.cmml" xref="S6.SS2.p1.3.m3.1.1.3.2">1.15</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.3.m3.1c">\leq 1.15\%</annotation></semantics></math> difference), Speech Command (<math id="S6.SS2.p1.4.m4.1" class="ltx_Math" alttext="\leq 1.32\%" display="inline"><semantics id="S6.SS2.p1.4.m4.1a"><mrow id="S6.SS2.p1.4.m4.1.1" xref="S6.SS2.p1.4.m4.1.1.cmml"><mi id="S6.SS2.p1.4.m4.1.1.2" xref="S6.SS2.p1.4.m4.1.1.2.cmml"></mi><mo id="S6.SS2.p1.4.m4.1.1.1" xref="S6.SS2.p1.4.m4.1.1.1.cmml">≤</mo><mrow id="S6.SS2.p1.4.m4.1.1.3" xref="S6.SS2.p1.4.m4.1.1.3.cmml"><mn id="S6.SS2.p1.4.m4.1.1.3.2" xref="S6.SS2.p1.4.m4.1.1.3.2.cmml">1.32</mn><mo id="S6.SS2.p1.4.m4.1.1.3.1" xref="S6.SS2.p1.4.m4.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.4.m4.1b"><apply id="S6.SS2.p1.4.m4.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1"><leq id="S6.SS2.p1.4.m4.1.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1.1"></leq><csymbol cd="latexml" id="S6.SS2.p1.4.m4.1.1.2.cmml" xref="S6.SS2.p1.4.m4.1.1.2">absent</csymbol><apply id="S6.SS2.p1.4.m4.1.1.3.cmml" xref="S6.SS2.p1.4.m4.1.1.3"><csymbol cd="latexml" id="S6.SS2.p1.4.m4.1.1.3.1.cmml" xref="S6.SS2.p1.4.m4.1.1.3.1">percent</csymbol><cn type="float" id="S6.SS2.p1.4.m4.1.1.3.2.cmml" xref="S6.SS2.p1.4.m4.1.1.3.2">1.32</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.4.m4.1c">\leq 1.32\%</annotation></semantics></math> difference) , and UTKFace (LUT-NN is even better). For the Speech Command task, LUT-NN can even improve accuracy by <math id="S6.SS2.p1.5.m5.1" class="ltx_Math" alttext="1.98\%" display="inline"><semantics id="S6.SS2.p1.5.m5.1a"><mrow id="S6.SS2.p1.5.m5.1.1" xref="S6.SS2.p1.5.m5.1.1.cmml"><mn id="S6.SS2.p1.5.m5.1.1.2" xref="S6.SS2.p1.5.m5.1.1.2.cmml">1.98</mn><mo id="S6.SS2.p1.5.m5.1.1.1" xref="S6.SS2.p1.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.5.m5.1b"><apply id="S6.SS2.p1.5.m5.1.1.cmml" xref="S6.SS2.p1.5.m5.1.1"><csymbol cd="latexml" id="S6.SS2.p1.5.m5.1.1.1.cmml" xref="S6.SS2.p1.5.m5.1.1.1">percent</csymbol><cn type="float" id="S6.SS2.p1.5.m5.1.1.2.cmml" xref="S6.SS2.p1.5.m5.1.1.2">1.98</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.5.m5.1c">1.98\%</annotation></semantics></math> for ResNet18.
Also for the UTKFace dataset, LUT-NN can even reduce MAE by at most 13.2%.
Expectedly, the accuracy difference for the more complex tasks, ImageNet and NLP, is a bit larger compared to the tasks above, with <math id="S6.SS2.p1.6.m6.1" class="ltx_Math" alttext="0.9\%" display="inline"><semantics id="S6.SS2.p1.6.m6.1a"><mrow id="S6.SS2.p1.6.m6.1.1" xref="S6.SS2.p1.6.m6.1.1.cmml"><mn id="S6.SS2.p1.6.m6.1.1.2" xref="S6.SS2.p1.6.m6.1.1.2.cmml">0.9</mn><mo id="S6.SS2.p1.6.m6.1.1.1" xref="S6.SS2.p1.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.6.m6.1b"><apply id="S6.SS2.p1.6.m6.1.1.cmml" xref="S6.SS2.p1.6.m6.1.1"><csymbol cd="latexml" id="S6.SS2.p1.6.m6.1.1.1.cmml" xref="S6.SS2.p1.6.m6.1.1.1">percent</csymbol><cn type="float" id="S6.SS2.p1.6.m6.1.1.2.cmml" xref="S6.SS2.p1.6.m6.1.1.2">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.6.m6.1c">0.9\%</annotation></semantics></math>~<math id="S6.SS2.p1.7.m7.1" class="ltx_Math" alttext="2.42\%" display="inline"><semantics id="S6.SS2.p1.7.m7.1a"><mrow id="S6.SS2.p1.7.m7.1.1" xref="S6.SS2.p1.7.m7.1.1.cmml"><mn id="S6.SS2.p1.7.m7.1.1.2" xref="S6.SS2.p1.7.m7.1.1.2.cmml">2.42</mn><mo id="S6.SS2.p1.7.m7.1.1.1" xref="S6.SS2.p1.7.m7.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.7.m7.1b"><apply id="S6.SS2.p1.7.m7.1.1.cmml" xref="S6.SS2.p1.7.m7.1.1"><csymbol cd="latexml" id="S6.SS2.p1.7.m7.1.1.1.cmml" xref="S6.SS2.p1.7.m7.1.1.1">percent</csymbol><cn type="float" id="S6.SS2.p1.7.m7.1.1.2.cmml" xref="S6.SS2.p1.7.m7.1.1.2">2.42</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.7.m7.1c">2.42\%</annotation></semantics></math> difference for ImageNet, and an averaged <math id="S6.SS2.p1.8.m8.1" class="ltx_Math" alttext="1.9\%" display="inline"><semantics id="S6.SS2.p1.8.m8.1a"><mrow id="S6.SS2.p1.8.m8.1.1" xref="S6.SS2.p1.8.m8.1.1.cmml"><mn id="S6.SS2.p1.8.m8.1.1.2" xref="S6.SS2.p1.8.m8.1.1.2.cmml">1.9</mn><mo id="S6.SS2.p1.8.m8.1.1.1" xref="S6.SS2.p1.8.m8.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.8.m8.1b"><apply id="S6.SS2.p1.8.m8.1.1.cmml" xref="S6.SS2.p1.8.m8.1.1"><csymbol cd="latexml" id="S6.SS2.p1.8.m8.1.1.1.cmml" xref="S6.SS2.p1.8.m8.1.1.1">percent</csymbol><cn type="float" id="S6.SS2.p1.8.m8.1.1.2.cmml" xref="S6.SS2.p1.8.m8.1.1.2">1.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.8.m8.1c">1.9\%</annotation></semantics></math> difference for NLP tasks.</p>
</div>
<div id="S6.SS2.p2" class="ltx_para ltx_noindent">
<p id="S6.SS2.p2.6" class="ltx_p"><span id="S6.SS2.p2.6.1" class="ltx_text ltx_font_bold">Operator speedup</span> Fig. <a href="#S6.F7" title="Figure 7 ‣ 6.2. Accuracy, latency, memory and power ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows the operator speedup selected from three models. The operator speedup for VGG11 ranges from <math id="S6.SS2.p2.1.m1.1" class="ltx_math_unparsed" alttext="4.28\times" display="inline"><semantics id="S6.SS2.p2.1.m1.1a"><mrow id="S6.SS2.p2.1.m1.1b"><mn id="S6.SS2.p2.1.m1.1.1">4.28</mn><mo lspace="0.222em" id="S6.SS2.p2.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p2.1.m1.1c">4.28\times</annotation></semantics></math> to <math id="S6.SS2.p2.2.m2.1" class="ltx_math_unparsed" alttext="5.43\times" display="inline"><semantics id="S6.SS2.p2.2.m2.1a"><mrow id="S6.SS2.p2.2.m2.1b"><mn id="S6.SS2.p2.2.m2.1.1">5.43</mn><mo lspace="0.222em" id="S6.SS2.p2.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p2.2.m2.1c">5.43\times</annotation></semantics></math> on ARM CPU, and from <math id="S6.SS2.p2.3.m3.1" class="ltx_math_unparsed" alttext="3.82\times" display="inline"><semantics id="S6.SS2.p2.3.m3.1a"><mrow id="S6.SS2.p2.3.m3.1b"><mn id="S6.SS2.p2.3.m3.1.1">3.82</mn><mo lspace="0.222em" id="S6.SS2.p2.3.m3.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p2.3.m3.1c">3.82\times</annotation></semantics></math> to <math id="S6.SS2.p2.4.m4.1" class="ltx_math_unparsed" alttext="3.86\times" display="inline"><semantics id="S6.SS2.p2.4.m4.1a"><mrow id="S6.SS2.p2.4.m4.1b"><mn id="S6.SS2.p2.4.m4.1.1">3.86</mn><mo lspace="0.222em" id="S6.SS2.p2.4.m4.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p2.4.m4.1c">3.86\times</annotation></semantics></math> on x86 CPU. The best speedup on BERT kernels are <math id="S6.SS2.p2.5.m5.1" class="ltx_math_unparsed" alttext="12.5\times" display="inline"><semantics id="S6.SS2.p2.5.m5.1a"><mrow id="S6.SS2.p2.5.m5.1b"><mn id="S6.SS2.p2.5.m5.1.1">12.5</mn><mo lspace="0.222em" id="S6.SS2.p2.5.m5.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p2.5.m5.1c">12.5\times</annotation></semantics></math> (ARM CPU) and <math id="S6.SS2.p2.6.m6.1" class="ltx_math_unparsed" alttext="10.3\times" display="inline"><semantics id="S6.SS2.p2.6.m6.1a"><mrow id="S6.SS2.p2.6.m6.1b"><mn id="S6.SS2.p2.6.m6.1.1">10.3</mn><mo lspace="0.222em" id="S6.SS2.p2.6.m6.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p2.6.m6.1c">10.3\times</annotation></semantics></math> (x86 CPU).</p>
</div>
<div id="S6.SS2.p3" class="ltx_para">
<p id="S6.SS2.p3.13" class="ltx_p">BERT operators achieve higher speedups due to larger <math id="S6.SS2.p3.1.m1.1" class="ltx_Math" alttext="M=768" display="inline"><semantics id="S6.SS2.p3.1.m1.1a"><mrow id="S6.SS2.p3.1.m1.1.1" xref="S6.SS2.p3.1.m1.1.1.cmml"><mi id="S6.SS2.p3.1.m1.1.1.2" xref="S6.SS2.p3.1.m1.1.1.2.cmml">M</mi><mo id="S6.SS2.p3.1.m1.1.1.1" xref="S6.SS2.p3.1.m1.1.1.1.cmml">=</mo><mn id="S6.SS2.p3.1.m1.1.1.3" xref="S6.SS2.p3.1.m1.1.1.3.cmml">768</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.1.m1.1b"><apply id="S6.SS2.p3.1.m1.1.1.cmml" xref="S6.SS2.p3.1.m1.1.1"><eq id="S6.SS2.p3.1.m1.1.1.1.cmml" xref="S6.SS2.p3.1.m1.1.1.1"></eq><ci id="S6.SS2.p3.1.m1.1.1.2.cmml" xref="S6.SS2.p3.1.m1.1.1.2">𝑀</ci><cn type="integer" id="S6.SS2.p3.1.m1.1.1.3.cmml" xref="S6.SS2.p3.1.m1.1.1.3">768</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.1.m1.1c">M=768</annotation></semantics></math> or <math id="S6.SS2.p3.2.m2.1" class="ltx_Math" alttext="3072" display="inline"><semantics id="S6.SS2.p3.2.m2.1a"><mn id="S6.SS2.p3.2.m2.1.1" xref="S6.SS2.p3.2.m2.1.1.cmml">3072</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.2.m2.1b"><cn type="integer" id="S6.SS2.p3.2.m2.1.1.cmml" xref="S6.SS2.p3.2.m2.1.1">3072</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.2.m2.1c">3072</annotation></semantics></math>, and longer sub-vector <math id="S6.SS2.p3.3.m3.1" class="ltx_Math" alttext="V=32" display="inline"><semantics id="S6.SS2.p3.3.m3.1a"><mrow id="S6.SS2.p3.3.m3.1.1" xref="S6.SS2.p3.3.m3.1.1.cmml"><mi id="S6.SS2.p3.3.m3.1.1.2" xref="S6.SS2.p3.3.m3.1.1.2.cmml">V</mi><mo id="S6.SS2.p3.3.m3.1.1.1" xref="S6.SS2.p3.3.m3.1.1.1.cmml">=</mo><mn id="S6.SS2.p3.3.m3.1.1.3" xref="S6.SS2.p3.3.m3.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.3.m3.1b"><apply id="S6.SS2.p3.3.m3.1.1.cmml" xref="S6.SS2.p3.3.m3.1.1"><eq id="S6.SS2.p3.3.m3.1.1.1.cmml" xref="S6.SS2.p3.3.m3.1.1.1"></eq><ci id="S6.SS2.p3.3.m3.1.1.2.cmml" xref="S6.SS2.p3.3.m3.1.1.2">𝑉</ci><cn type="integer" id="S6.SS2.p3.3.m3.1.1.3.cmml" xref="S6.SS2.p3.3.m3.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.3.m3.1c">V=32</annotation></semantics></math>. According to Table <a href="#S4.T1" title="Table 1 ‣ 4. cost reduction of LUT-NN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, the FLOPs of our proposed LUT-NN is <math id="S6.SS2.p3.4.m4.1" class="ltx_Math" alttext="ND(K+M/V)" display="inline"><semantics id="S6.SS2.p3.4.m4.1a"><mrow id="S6.SS2.p3.4.m4.1.1" xref="S6.SS2.p3.4.m4.1.1.cmml"><mi id="S6.SS2.p3.4.m4.1.1.3" xref="S6.SS2.p3.4.m4.1.1.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p3.4.m4.1.1.2" xref="S6.SS2.p3.4.m4.1.1.2.cmml">​</mo><mi id="S6.SS2.p3.4.m4.1.1.4" xref="S6.SS2.p3.4.m4.1.1.4.cmml">D</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p3.4.m4.1.1.2a" xref="S6.SS2.p3.4.m4.1.1.2.cmml">​</mo><mrow id="S6.SS2.p3.4.m4.1.1.1.1" xref="S6.SS2.p3.4.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="S6.SS2.p3.4.m4.1.1.1.1.2" xref="S6.SS2.p3.4.m4.1.1.1.1.1.cmml">(</mo><mrow id="S6.SS2.p3.4.m4.1.1.1.1.1" xref="S6.SS2.p3.4.m4.1.1.1.1.1.cmml"><mi id="S6.SS2.p3.4.m4.1.1.1.1.1.2" xref="S6.SS2.p3.4.m4.1.1.1.1.1.2.cmml">K</mi><mo id="S6.SS2.p3.4.m4.1.1.1.1.1.1" xref="S6.SS2.p3.4.m4.1.1.1.1.1.1.cmml">+</mo><mrow id="S6.SS2.p3.4.m4.1.1.1.1.1.3" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.cmml"><mi id="S6.SS2.p3.4.m4.1.1.1.1.1.3.2" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.2.cmml">M</mi><mo id="S6.SS2.p3.4.m4.1.1.1.1.1.3.1" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.1.cmml">/</mo><mi id="S6.SS2.p3.4.m4.1.1.1.1.1.3.3" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.3.cmml">V</mi></mrow></mrow><mo stretchy="false" id="S6.SS2.p3.4.m4.1.1.1.1.3" xref="S6.SS2.p3.4.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.4.m4.1b"><apply id="S6.SS2.p3.4.m4.1.1.cmml" xref="S6.SS2.p3.4.m4.1.1"><times id="S6.SS2.p3.4.m4.1.1.2.cmml" xref="S6.SS2.p3.4.m4.1.1.2"></times><ci id="S6.SS2.p3.4.m4.1.1.3.cmml" xref="S6.SS2.p3.4.m4.1.1.3">𝑁</ci><ci id="S6.SS2.p3.4.m4.1.1.4.cmml" xref="S6.SS2.p3.4.m4.1.1.4">𝐷</ci><apply id="S6.SS2.p3.4.m4.1.1.1.1.1.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1"><plus id="S6.SS2.p3.4.m4.1.1.1.1.1.1.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1.1.1"></plus><ci id="S6.SS2.p3.4.m4.1.1.1.1.1.2.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1.1.2">𝐾</ci><apply id="S6.SS2.p3.4.m4.1.1.1.1.1.3.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3"><divide id="S6.SS2.p3.4.m4.1.1.1.1.1.3.1.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.1"></divide><ci id="S6.SS2.p3.4.m4.1.1.1.1.1.3.2.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.2">𝑀</ci><ci id="S6.SS2.p3.4.m4.1.1.1.1.1.3.3.cmml" xref="S6.SS2.p3.4.m4.1.1.1.1.1.3.3">𝑉</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.4.m4.1c">ND(K+M/V)</annotation></semantics></math>, and the FLOPs of a normal MM is <math id="S6.SS2.p3.5.m5.1" class="ltx_Math" alttext="NDM" display="inline"><semantics id="S6.SS2.p3.5.m5.1a"><mrow id="S6.SS2.p3.5.m5.1.1" xref="S6.SS2.p3.5.m5.1.1.cmml"><mi id="S6.SS2.p3.5.m5.1.1.2" xref="S6.SS2.p3.5.m5.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p3.5.m5.1.1.1" xref="S6.SS2.p3.5.m5.1.1.1.cmml">​</mo><mi id="S6.SS2.p3.5.m5.1.1.3" xref="S6.SS2.p3.5.m5.1.1.3.cmml">D</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p3.5.m5.1.1.1a" xref="S6.SS2.p3.5.m5.1.1.1.cmml">​</mo><mi id="S6.SS2.p3.5.m5.1.1.4" xref="S6.SS2.p3.5.m5.1.1.4.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.5.m5.1b"><apply id="S6.SS2.p3.5.m5.1.1.cmml" xref="S6.SS2.p3.5.m5.1.1"><times id="S6.SS2.p3.5.m5.1.1.1.cmml" xref="S6.SS2.p3.5.m5.1.1.1"></times><ci id="S6.SS2.p3.5.m5.1.1.2.cmml" xref="S6.SS2.p3.5.m5.1.1.2">𝑁</ci><ci id="S6.SS2.p3.5.m5.1.1.3.cmml" xref="S6.SS2.p3.5.m5.1.1.3">𝐷</ci><ci id="S6.SS2.p3.5.m5.1.1.4.cmml" xref="S6.SS2.p3.5.m5.1.1.4">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.5.m5.1c">NDM</annotation></semantics></math>.
The FLOPs reduction by LUT-NN can be calculated by <math id="S6.SS2.p3.6.m6.1" class="ltx_Math" alttext="M/(K+M/V)" display="inline"><semantics id="S6.SS2.p3.6.m6.1a"><mrow id="S6.SS2.p3.6.m6.1.1" xref="S6.SS2.p3.6.m6.1.1.cmml"><mi id="S6.SS2.p3.6.m6.1.1.3" xref="S6.SS2.p3.6.m6.1.1.3.cmml">M</mi><mo id="S6.SS2.p3.6.m6.1.1.2" xref="S6.SS2.p3.6.m6.1.1.2.cmml">/</mo><mrow id="S6.SS2.p3.6.m6.1.1.1.1" xref="S6.SS2.p3.6.m6.1.1.1.1.1.cmml"><mo stretchy="false" id="S6.SS2.p3.6.m6.1.1.1.1.2" xref="S6.SS2.p3.6.m6.1.1.1.1.1.cmml">(</mo><mrow id="S6.SS2.p3.6.m6.1.1.1.1.1" xref="S6.SS2.p3.6.m6.1.1.1.1.1.cmml"><mi id="S6.SS2.p3.6.m6.1.1.1.1.1.2" xref="S6.SS2.p3.6.m6.1.1.1.1.1.2.cmml">K</mi><mo id="S6.SS2.p3.6.m6.1.1.1.1.1.1" xref="S6.SS2.p3.6.m6.1.1.1.1.1.1.cmml">+</mo><mrow id="S6.SS2.p3.6.m6.1.1.1.1.1.3" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.cmml"><mi id="S6.SS2.p3.6.m6.1.1.1.1.1.3.2" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.2.cmml">M</mi><mo id="S6.SS2.p3.6.m6.1.1.1.1.1.3.1" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.1.cmml">/</mo><mi id="S6.SS2.p3.6.m6.1.1.1.1.1.3.3" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.3.cmml">V</mi></mrow></mrow><mo stretchy="false" id="S6.SS2.p3.6.m6.1.1.1.1.3" xref="S6.SS2.p3.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.6.m6.1b"><apply id="S6.SS2.p3.6.m6.1.1.cmml" xref="S6.SS2.p3.6.m6.1.1"><divide id="S6.SS2.p3.6.m6.1.1.2.cmml" xref="S6.SS2.p3.6.m6.1.1.2"></divide><ci id="S6.SS2.p3.6.m6.1.1.3.cmml" xref="S6.SS2.p3.6.m6.1.1.3">𝑀</ci><apply id="S6.SS2.p3.6.m6.1.1.1.1.1.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1"><plus id="S6.SS2.p3.6.m6.1.1.1.1.1.1.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1.1.1"></plus><ci id="S6.SS2.p3.6.m6.1.1.1.1.1.2.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1.1.2">𝐾</ci><apply id="S6.SS2.p3.6.m6.1.1.1.1.1.3.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3"><divide id="S6.SS2.p3.6.m6.1.1.1.1.1.3.1.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.1"></divide><ci id="S6.SS2.p3.6.m6.1.1.1.1.1.3.2.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.2">𝑀</ci><ci id="S6.SS2.p3.6.m6.1.1.1.1.1.3.3.cmml" xref="S6.SS2.p3.6.m6.1.1.1.1.1.3.3">𝑉</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.6.m6.1c">M/(K+M/V)</annotation></semantics></math>. Therefore, the reduction is particularly large when <math id="S6.SS2.p3.7.m7.1" class="ltx_Math" alttext="M\gg K" display="inline"><semantics id="S6.SS2.p3.7.m7.1a"><mrow id="S6.SS2.p3.7.m7.1.1" xref="S6.SS2.p3.7.m7.1.1.cmml"><mi id="S6.SS2.p3.7.m7.1.1.2" xref="S6.SS2.p3.7.m7.1.1.2.cmml">M</mi><mo id="S6.SS2.p3.7.m7.1.1.1" xref="S6.SS2.p3.7.m7.1.1.1.cmml">≫</mo><mi id="S6.SS2.p3.7.m7.1.1.3" xref="S6.SS2.p3.7.m7.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.7.m7.1b"><apply id="S6.SS2.p3.7.m7.1.1.cmml" xref="S6.SS2.p3.7.m7.1.1"><csymbol cd="latexml" id="S6.SS2.p3.7.m7.1.1.1.cmml" xref="S6.SS2.p3.7.m7.1.1.1">much-greater-than</csymbol><ci id="S6.SS2.p3.7.m7.1.1.2.cmml" xref="S6.SS2.p3.7.m7.1.1.2">𝑀</ci><ci id="S6.SS2.p3.7.m7.1.1.3.cmml" xref="S6.SS2.p3.7.m7.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.7.m7.1c">M\gg K</annotation></semantics></math> and <math id="S6.SS2.p3.8.m8.1" class="ltx_Math" alttext="V\gg 1" display="inline"><semantics id="S6.SS2.p3.8.m8.1a"><mrow id="S6.SS2.p3.8.m8.1.1" xref="S6.SS2.p3.8.m8.1.1.cmml"><mi id="S6.SS2.p3.8.m8.1.1.2" xref="S6.SS2.p3.8.m8.1.1.2.cmml">V</mi><mo id="S6.SS2.p3.8.m8.1.1.1" xref="S6.SS2.p3.8.m8.1.1.1.cmml">≫</mo><mn id="S6.SS2.p3.8.m8.1.1.3" xref="S6.SS2.p3.8.m8.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.8.m8.1b"><apply id="S6.SS2.p3.8.m8.1.1.cmml" xref="S6.SS2.p3.8.m8.1.1"><csymbol cd="latexml" id="S6.SS2.p3.8.m8.1.1.1.cmml" xref="S6.SS2.p3.8.m8.1.1.1">much-greater-than</csymbol><ci id="S6.SS2.p3.8.m8.1.1.2.cmml" xref="S6.SS2.p3.8.m8.1.1.2">𝑉</ci><cn type="integer" id="S6.SS2.p3.8.m8.1.1.3.cmml" xref="S6.SS2.p3.8.m8.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.8.m8.1c">V\gg 1</annotation></semantics></math>. The kernel speedup gradually improves as the layer index increases for CNN models.
The reason is that the number of output channels increases from <math id="S6.SS2.p3.9.m9.1" class="ltx_Math" alttext="64" display="inline"><semantics id="S6.SS2.p3.9.m9.1a"><mn id="S6.SS2.p3.9.m9.1.1" xref="S6.SS2.p3.9.m9.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.9.m9.1b"><cn type="integer" id="S6.SS2.p3.9.m9.1.1.cmml" xref="S6.SS2.p3.9.m9.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.9.m9.1c">64</annotation></semantics></math> to <math id="S6.SS2.p3.10.m10.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S6.SS2.p3.10.m10.1a"><mn id="S6.SS2.p3.10.m10.1.1" xref="S6.SS2.p3.10.m10.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.10.m10.1b"><cn type="integer" id="S6.SS2.p3.10.m10.1.1.cmml" xref="S6.SS2.p3.10.m10.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.10.m10.1c">128</annotation></semantics></math>, <math id="S6.SS2.p3.11.m11.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S6.SS2.p3.11.m11.1a"><mn id="S6.SS2.p3.11.m11.1.1" xref="S6.SS2.p3.11.m11.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.11.m11.1b"><cn type="integer" id="S6.SS2.p3.11.m11.1.1.cmml" xref="S6.SS2.p3.11.m11.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.11.m11.1c">256</annotation></semantics></math>, and <math id="S6.SS2.p3.12.m12.1" class="ltx_Math" alttext="512" display="inline"><semantics id="S6.SS2.p3.12.m12.1a"><mn id="S6.SS2.p3.12.m12.1.1" xref="S6.SS2.p3.12.m12.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.12.m12.1b"><cn type="integer" id="S6.SS2.p3.12.m12.1.1.cmml" xref="S6.SS2.p3.12.m12.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.12.m12.1c">512</annotation></semantics></math> as the layer index increases.
The increased number of output channels <math id="S6.SS2.p3.13.m13.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S6.SS2.p3.13.m13.1a"><mi id="S6.SS2.p3.13.m13.1.1" xref="S6.SS2.p3.13.m13.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S6.SS2.p3.13.m13.1b"><ci id="S6.SS2.p3.13.m13.1.1.cmml" xref="S6.SS2.p3.13.m13.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p3.13.m13.1c">M</annotation></semantics></math> contributes to the speedup in these layers.</p>
</div>
<div id="S6.SS2.p4" class="ltx_para ltx_noindent">
<p id="S6.SS2.p4.9" class="ltx_p"><span id="S6.SS2.p4.9.1" class="ltx_text ltx_font_bold">Model speedup</span>
Fig. <a href="#S6.F8" title="Figure 8 ‣ 6.2. Accuracy, latency, memory and power ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> shows the end-to-end model latency.
LUT-NN achieves <math id="S6.SS2.p4.1.m1.1" class="ltx_Math" alttext="1.30" display="inline"><semantics id="S6.SS2.p4.1.m1.1a"><mn id="S6.SS2.p4.1.m1.1.1" xref="S6.SS2.p4.1.m1.1.1.cmml">1.30</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p4.1.m1.1b"><cn type="float" id="S6.SS2.p4.1.m1.1.1.cmml" xref="S6.SS2.p4.1.m1.1.1">1.30</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p4.1.m1.1c">1.30</annotation></semantics></math>~<math id="S6.SS2.p4.2.m2.1" class="ltx_math_unparsed" alttext="4.23\times" display="inline"><semantics id="S6.SS2.p4.2.m2.1a"><mrow id="S6.SS2.p4.2.m2.1b"><mn id="S6.SS2.p4.2.m2.1.1">4.23</mn><mo lspace="0.222em" id="S6.SS2.p4.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p4.2.m2.1c">4.23\times</annotation></semantics></math> speedup on ResNet18, SENet18 and VGG11 compared to TVM and ORT.
The best speedup is <math id="S6.SS2.p4.3.m3.1" class="ltx_math_unparsed" alttext="4.23\times" display="inline"><semantics id="S6.SS2.p4.3.m3.1a"><mrow id="S6.SS2.p4.3.m3.1b"><mn id="S6.SS2.p4.3.m3.1.1">4.23</mn><mo lspace="0.222em" id="S6.SS2.p4.3.m3.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p4.3.m3.1c">4.23\times</annotation></semantics></math> in VGG11 (for CIFAR10) on Pixel6 ARM CPU over ONNX Runtime.
The BERT model has higher throughput, and the speedups are <math id="S6.SS2.p4.4.m4.1" class="ltx_math_unparsed" alttext="5.6\times" display="inline"><semantics id="S6.SS2.p4.4.m4.1a"><mrow id="S6.SS2.p4.4.m4.1b"><mn id="S6.SS2.p4.4.m4.1.1">5.6</mn><mo lspace="0.222em" id="S6.SS2.p4.4.m4.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p4.4.m4.1c">5.6\times</annotation></semantics></math> and <math id="S6.SS2.p4.5.m5.1" class="ltx_math_unparsed" alttext="6.8\times" display="inline"><semantics id="S6.SS2.p4.5.m5.1a"><mrow id="S6.SS2.p4.5.m5.1b"><mn id="S6.SS2.p4.5.m5.1.1">6.8</mn><mo lspace="0.222em" id="S6.SS2.p4.5.m5.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p4.5.m5.1c">6.8\times</annotation></semantics></math> on the ARM CPU and the x86 CPU, respectively.
Compared to CNN models (<math id="S6.SS2.p4.6.m6.1" class="ltx_Math" alttext="M\leq 512" display="inline"><semantics id="S6.SS2.p4.6.m6.1a"><mrow id="S6.SS2.p4.6.m6.1.1" xref="S6.SS2.p4.6.m6.1.1.cmml"><mi id="S6.SS2.p4.6.m6.1.1.2" xref="S6.SS2.p4.6.m6.1.1.2.cmml">M</mi><mo id="S6.SS2.p4.6.m6.1.1.1" xref="S6.SS2.p4.6.m6.1.1.1.cmml">≤</mo><mn id="S6.SS2.p4.6.m6.1.1.3" xref="S6.SS2.p4.6.m6.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p4.6.m6.1b"><apply id="S6.SS2.p4.6.m6.1.1.cmml" xref="S6.SS2.p4.6.m6.1.1"><leq id="S6.SS2.p4.6.m6.1.1.1.cmml" xref="S6.SS2.p4.6.m6.1.1.1"></leq><ci id="S6.SS2.p4.6.m6.1.1.2.cmml" xref="S6.SS2.p4.6.m6.1.1.2">𝑀</ci><cn type="integer" id="S6.SS2.p4.6.m6.1.1.3.cmml" xref="S6.SS2.p4.6.m6.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p4.6.m6.1c">M\leq 512</annotation></semantics></math>), the BERT model has a larger input tensor (<math id="S6.SS2.p4.7.m7.2" class="ltx_Math" alttext="M=768,3072" display="inline"><semantics id="S6.SS2.p4.7.m7.2a"><mrow id="S6.SS2.p4.7.m7.2.3" xref="S6.SS2.p4.7.m7.2.3.cmml"><mi id="S6.SS2.p4.7.m7.2.3.2" xref="S6.SS2.p4.7.m7.2.3.2.cmml">M</mi><mo id="S6.SS2.p4.7.m7.2.3.1" xref="S6.SS2.p4.7.m7.2.3.1.cmml">=</mo><mrow id="S6.SS2.p4.7.m7.2.3.3.2" xref="S6.SS2.p4.7.m7.2.3.3.1.cmml"><mn id="S6.SS2.p4.7.m7.1.1" xref="S6.SS2.p4.7.m7.1.1.cmml">768</mn><mo id="S6.SS2.p4.7.m7.2.3.3.2.1" xref="S6.SS2.p4.7.m7.2.3.3.1.cmml">,</mo><mn id="S6.SS2.p4.7.m7.2.2" xref="S6.SS2.p4.7.m7.2.2.cmml">3072</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p4.7.m7.2b"><apply id="S6.SS2.p4.7.m7.2.3.cmml" xref="S6.SS2.p4.7.m7.2.3"><eq id="S6.SS2.p4.7.m7.2.3.1.cmml" xref="S6.SS2.p4.7.m7.2.3.1"></eq><ci id="S6.SS2.p4.7.m7.2.3.2.cmml" xref="S6.SS2.p4.7.m7.2.3.2">𝑀</ci><list id="S6.SS2.p4.7.m7.2.3.3.1.cmml" xref="S6.SS2.p4.7.m7.2.3.3.2"><cn type="integer" id="S6.SS2.p4.7.m7.1.1.cmml" xref="S6.SS2.p4.7.m7.1.1">768</cn><cn type="integer" id="S6.SS2.p4.7.m7.2.2.cmml" xref="S6.SS2.p4.7.m7.2.2">3072</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p4.7.m7.2c">M=768,3072</annotation></semantics></math>) to acquire better performance gains.
It also has a longer sub-vector length (<math id="S6.SS2.p4.8.m8.1" class="ltx_Math" alttext="V=32" display="inline"><semantics id="S6.SS2.p4.8.m8.1a"><mrow id="S6.SS2.p4.8.m8.1.1" xref="S6.SS2.p4.8.m8.1.1.cmml"><mi id="S6.SS2.p4.8.m8.1.1.2" xref="S6.SS2.p4.8.m8.1.1.2.cmml">V</mi><mo id="S6.SS2.p4.8.m8.1.1.1" xref="S6.SS2.p4.8.m8.1.1.1.cmml">=</mo><mn id="S6.SS2.p4.8.m8.1.1.3" xref="S6.SS2.p4.8.m8.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p4.8.m8.1b"><apply id="S6.SS2.p4.8.m8.1.1.cmml" xref="S6.SS2.p4.8.m8.1.1"><eq id="S6.SS2.p4.8.m8.1.1.1.cmml" xref="S6.SS2.p4.8.m8.1.1.1"></eq><ci id="S6.SS2.p4.8.m8.1.1.2.cmml" xref="S6.SS2.p4.8.m8.1.1.2">𝑉</ci><cn type="integer" id="S6.SS2.p4.8.m8.1.1.3.cmml" xref="S6.SS2.p4.8.m8.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p4.8.m8.1c">V=32</annotation></semantics></math>) compared to CNN models (<math id="S6.SS2.p4.9.m9.1" class="ltx_Math" alttext="V\leq 9" display="inline"><semantics id="S6.SS2.p4.9.m9.1a"><mrow id="S6.SS2.p4.9.m9.1.1" xref="S6.SS2.p4.9.m9.1.1.cmml"><mi id="S6.SS2.p4.9.m9.1.1.2" xref="S6.SS2.p4.9.m9.1.1.2.cmml">V</mi><mo id="S6.SS2.p4.9.m9.1.1.1" xref="S6.SS2.p4.9.m9.1.1.1.cmml">≤</mo><mn id="S6.SS2.p4.9.m9.1.1.3" xref="S6.SS2.p4.9.m9.1.1.3.cmml">9</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p4.9.m9.1b"><apply id="S6.SS2.p4.9.m9.1.1.cmml" xref="S6.SS2.p4.9.m9.1.1"><leq id="S6.SS2.p4.9.m9.1.1.1.cmml" xref="S6.SS2.p4.9.m9.1.1.1"></leq><ci id="S6.SS2.p4.9.m9.1.1.2.cmml" xref="S6.SS2.p4.9.m9.1.1.2">𝑉</ci><cn type="integer" id="S6.SS2.p4.9.m9.1.1.3.cmml" xref="S6.SS2.p4.9.m9.1.1.3">9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p4.9.m9.1c">V\leq 9</annotation></semantics></math>).</p>
</div>
<div id="S6.SS2.p5" class="ltx_para ltx_noindent">
<p id="S6.SS2.p5.2" class="ltx_p"><span id="S6.SS2.p5.2.1" class="ltx_text ltx_font_bold">Multi-thread speedup</span>
To evaluate the scaling capability of LUT-NN, Fig. <a href="#S6.F9" title="Figure 9 ‣ 6.2. Accuracy, latency, memory and power ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> shows the speedup with the thread increases, i.e., one to four
on x86 CPUs and one to two on the Pixel 6 ARM CPU. Note that we evaluate the scalability on the big cores (cores with the highest frequency) of ARM CPUs, to avoid the interference of asymmetric big and little cores. Pixel 6 has two big cores. Since Pixel 4 only has one big core, we skip it in this evaluation.
 The results show that LUT-NN achieves better scalability compared to the baselines.
It can reach 2.24~2.49<math id="S6.SS2.p5.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S6.SS2.p5.1.m1.1a"><mo id="S6.SS2.p5.1.m1.1.1" xref="S6.SS2.p5.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S6.SS2.p5.1.m1.1b"><times id="S6.SS2.p5.1.m1.1.1.cmml" xref="S6.SS2.p5.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p5.1.m1.1c">\times</annotation></semantics></math> speedup as the number of threads increases to 4.
Meanwhile, LUT-NN achieves a 2.17~2.82<math id="S6.SS2.p5.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S6.SS2.p5.2.m2.1a"><mo id="S6.SS2.p5.2.m2.1.1" xref="S6.SS2.p5.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S6.SS2.p5.2.m2.1b"><times id="S6.SS2.p5.2.m2.1.1.cmml" xref="S6.SS2.p5.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p5.2.m2.1c">\times</annotation></semantics></math> speedup over other baselines under the same number of threads.</p>
</div>
<div id="S6.SS2.p6" class="ltx_para ltx_noindent">
<p id="S6.SS2.p6.7" class="ltx_p"><span id="S6.SS2.p6.7.1" class="ltx_text ltx_font_bold">Model memory footprint</span>
Fig. <a href="#S6.F10" title="Figure 10 ‣ 6.2. Accuracy, latency, memory and power ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> compares the memory footprint of LUT-NN with baselines.
The results show that LUT-NN can save <math id="S6.SS2.p6.1.m1.1" class="ltx_Math" alttext="1.43" display="inline"><semantics id="S6.SS2.p6.1.m1.1a"><mn id="S6.SS2.p6.1.m1.1.1" xref="S6.SS2.p6.1.m1.1.1.cmml">1.43</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p6.1.m1.1b"><cn type="float" id="S6.SS2.p6.1.m1.1.1.cmml" xref="S6.SS2.p6.1.m1.1.1">1.43</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p6.1.m1.1c">1.43</annotation></semantics></math>~<math id="S6.SS2.p6.2.m2.1" class="ltx_math_unparsed" alttext="2.83\times" display="inline"><semantics id="S6.SS2.p6.2.m2.1a"><mrow id="S6.SS2.p6.2.m2.1b"><mn id="S6.SS2.p6.2.m2.1.1">2.83</mn><mo lspace="0.222em" id="S6.SS2.p6.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p6.2.m2.1c">2.83\times</annotation></semantics></math> memory for CNN models and <math id="S6.SS2.p6.3.m3.1" class="ltx_Math" alttext="4.76" display="inline"><semantics id="S6.SS2.p6.3.m3.1a"><mn id="S6.SS2.p6.3.m3.1.1" xref="S6.SS2.p6.3.m3.1.1.cmml">4.76</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p6.3.m3.1b"><cn type="float" id="S6.SS2.p6.3.m3.1.1.cmml" xref="S6.SS2.p6.3.m3.1.1">4.76</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p6.3.m3.1c">4.76</annotation></semantics></math>~<math id="S6.SS2.p6.4.m4.1" class="ltx_math_unparsed" alttext="6.49\times" display="inline"><semantics id="S6.SS2.p6.4.m4.1a"><mrow id="S6.SS2.p6.4.m4.1b"><mn id="S6.SS2.p6.4.m4.1.1">6.49</mn><mo lspace="0.222em" id="S6.SS2.p6.4.m4.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.p6.4.m4.1c">6.49\times</annotation></semantics></math> for BERT. For these CNN models, the sub-vector length is <math id="S6.SS2.p6.5.m5.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S6.SS2.p6.5.m5.1a"><mn id="S6.SS2.p6.5.m5.1.1" xref="S6.SS2.p6.5.m5.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p6.5.m5.1b"><cn type="integer" id="S6.SS2.p6.5.m5.1.1.cmml" xref="S6.SS2.p6.5.m5.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p6.5.m5.1c">9</annotation></semantics></math> or <math id="S6.SS2.p6.6.m6.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S6.SS2.p6.6.m6.1a"><mn id="S6.SS2.p6.6.m6.1.1" xref="S6.SS2.p6.6.m6.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p6.6.m6.1b"><cn type="integer" id="S6.SS2.p6.6.m6.1.1.cmml" xref="S6.SS2.p6.6.m6.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p6.6.m6.1c">4</annotation></semantics></math> for different operators, while the length is set to <math id="S6.SS2.p6.7.m7.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S6.SS2.p6.7.m7.1a"><mn id="S6.SS2.p6.7.m7.1.1" xref="S6.SS2.p6.7.m7.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p6.7.m7.1b"><cn type="integer" id="S6.SS2.p6.7.m7.1.1.cmml" xref="S6.SS2.p6.7.m7.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p6.7.m7.1c">16</annotation></semantics></math> for BERT.
Because longer sub-vectors result in a higher compression ratio on feature maps, BERT achieves lower memory costs than other models.</p>
</div>
<div id="S6.SS2.p7" class="ltx_para ltx_noindent">
<p id="S6.SS2.p7.2" class="ltx_p"><span id="S6.SS2.p7.2.1" class="ltx_text ltx_font_bold">Power</span>
We evaluate the power consumption of LUT-NN on Pixel 4.
The experiment results in Table <a href="#S6.T6" title="Table 6 ‣ 6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> demonstrate that LUT-NN reduces power consumption by <math id="S6.SS2.p7.1.m1.1" class="ltx_Math" alttext="15\%" display="inline"><semantics id="S6.SS2.p7.1.m1.1a"><mrow id="S6.SS2.p7.1.m1.1.1" xref="S6.SS2.p7.1.m1.1.1.cmml"><mn id="S6.SS2.p7.1.m1.1.1.2" xref="S6.SS2.p7.1.m1.1.1.2.cmml">15</mn><mo id="S6.SS2.p7.1.m1.1.1.1" xref="S6.SS2.p7.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p7.1.m1.1b"><apply id="S6.SS2.p7.1.m1.1.1.cmml" xref="S6.SS2.p7.1.m1.1.1"><csymbol cd="latexml" id="S6.SS2.p7.1.m1.1.1.1.cmml" xref="S6.SS2.p7.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S6.SS2.p7.1.m1.1.1.2.cmml" xref="S6.SS2.p7.1.m1.1.1.2">15</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p7.1.m1.1c">15\%</annotation></semantics></math>~<math id="S6.SS2.p7.2.m2.1" class="ltx_Math" alttext="41.7\%" display="inline"><semantics id="S6.SS2.p7.2.m2.1a"><mrow id="S6.SS2.p7.2.m2.1.1" xref="S6.SS2.p7.2.m2.1.1.cmml"><mn id="S6.SS2.p7.2.m2.1.1.2" xref="S6.SS2.p7.2.m2.1.1.2.cmml">41.7</mn><mo id="S6.SS2.p7.2.m2.1.1.1" xref="S6.SS2.p7.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p7.2.m2.1b"><apply id="S6.SS2.p7.2.m2.1.1.cmml" xref="S6.SS2.p7.2.m2.1.1"><csymbol cd="latexml" id="S6.SS2.p7.2.m2.1.1.1.cmml" xref="S6.SS2.p7.2.m2.1.1.1">percent</csymbol><cn type="float" id="S6.SS2.p7.2.m2.1.1.2.cmml" xref="S6.SS2.p7.2.m2.1.1.2">41.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p7.2.m2.1c">41.7\%</annotation></semantics></math> compared to TVM, which reveal that LUT-NN is an energy-efficient algorithm and system design.</p>
</div>
<figure id="S6.F7" class="ltx_figure"><img src="/html/2302.03213/assets/x9.png" id="S6.F7.g1" class="ltx_graphics ltx_centering ltx_img_square" width="267" height="294" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7. </span>

LUT-NN operator speedup over ORT and TVM on Pixel 6 and the x86 server CPU.
</figcaption>
</figure>
<figure id="S6.F8" class="ltx_figure"><img src="/html/2302.03213/assets/x10.png" id="S6.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="300" height="100" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8. </span>

End-to-end latency comparison between LUT-NN, ORT, and TVM (lower is better).
</figcaption>
</figure>
<figure id="S6.F9" class="ltx_figure"><img src="/html/2302.03213/assets/x11.png" id="S6.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="300" height="101" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9. </span>
Normalized multithread speedup (over TVM one thread) of LUT-NN, ORT, and TVM (higher is better).
</figcaption>
</figure>
<figure id="S6.F10" class="ltx_figure"><img src="/html/2302.03213/assets/x12.png" id="S6.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="276" height="138" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10. </span>

Model memory consumption of LUT-NN, ORT, and TVM on the x86 server CPU.
</figcaption>
</figure>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3. </span>Ablation Study</h3>

<div id="S6.SS3.p1" class="ltx_para">
<p id="S6.SS3.p1.1" class="ltx_p">We evaluate the effectiveness of the learned temperature and hyperparameters in the ablation study.
The hyperparameters include the number of centroids, the length of sub-vector, and the number of replaced linear computation operators.</p>
</div>
<figure id="S6.F11" class="ltx_figure"><img src="/html/2302.03213/assets/x13.png" id="S6.F11.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="131" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11. </span>

Learning curves of LUT-NN ResNet18 on CIFAR10 with temperature tuning methods.
“Annealing Temperature” (Orange) refers to manually annealing temperature from 1 to 1e-1.
Our learned temperature technique reaches 94.4% accuracy, higher than the annealing temperature (91.55%) and setting temperature as 1 (89.85%).
</figcaption>
</figure>
<div id="S6.SS3.p2" class="ltx_para ltx_noindent">
<p id="S6.SS3.p2.6" class="ltx_p"><span id="S6.SS3.p2.6.1" class="ltx_text ltx_font_bold">Learned temperature</span>
In Sec. <a href="#S3.SS2" title="3.2. Learned temperature ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>, we use gradient descent to learn the temperature.
To evaluate the effectiveness of our method, we compare three temperature tuning strategies together:
learned temperature, statically setting the temperature as <math id="S6.SS3.p2.1.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S6.SS3.p2.1.m1.1a"><mn id="S6.SS3.p2.1.m1.1.1" xref="S6.SS3.p2.1.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S6.SS3.p2.1.m1.1b"><cn type="integer" id="S6.SS3.p2.1.m1.1.1.cmml" xref="S6.SS3.p2.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p2.1.m1.1c">1</annotation></semantics></math>, and annealing temperature from <math id="S6.SS3.p2.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S6.SS3.p2.2.m2.1a"><mn id="S6.SS3.p2.2.m2.1.1" xref="S6.SS3.p2.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S6.SS3.p2.2.m2.1b"><cn type="integer" id="S6.SS3.p2.2.m2.1.1.cmml" xref="S6.SS3.p2.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p2.2.m2.1c">1</annotation></semantics></math> to <math id="S6.SS3.p2.3.m3.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="S6.SS3.p2.3.m3.1a"><mn id="S6.SS3.p2.3.m3.1.1" xref="S6.SS3.p2.3.m3.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="S6.SS3.p2.3.m3.1b"><cn type="float" id="S6.SS3.p2.3.m3.1.1.cmml" xref="S6.SS3.p2.3.m3.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p2.3.m3.1c">0.1</annotation></semantics></math>.
We compare the training curves in Fig. <a href="#S6.F11" title="Figure 11 ‣ 6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.
The figure suggests that our proposed learned temperature reaches the highest accuracy (<math id="S6.SS3.p2.4.m4.1" class="ltx_Math" alttext="94.4\%" display="inline"><semantics id="S6.SS3.p2.4.m4.1a"><mrow id="S6.SS3.p2.4.m4.1.1" xref="S6.SS3.p2.4.m4.1.1.cmml"><mn id="S6.SS3.p2.4.m4.1.1.2" xref="S6.SS3.p2.4.m4.1.1.2.cmml">94.4</mn><mo id="S6.SS3.p2.4.m4.1.1.1" xref="S6.SS3.p2.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p2.4.m4.1b"><apply id="S6.SS3.p2.4.m4.1.1.cmml" xref="S6.SS3.p2.4.m4.1.1"><csymbol cd="latexml" id="S6.SS3.p2.4.m4.1.1.1.cmml" xref="S6.SS3.p2.4.m4.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p2.4.m4.1.1.2.cmml" xref="S6.SS3.p2.4.m4.1.1.2">94.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p2.4.m4.1c">94.4\%</annotation></semantics></math>) and outperforms the statically setting (<math id="S6.SS3.p2.5.m5.1" class="ltx_Math" alttext="89.85\%" display="inline"><semantics id="S6.SS3.p2.5.m5.1a"><mrow id="S6.SS3.p2.5.m5.1.1" xref="S6.SS3.p2.5.m5.1.1.cmml"><mn id="S6.SS3.p2.5.m5.1.1.2" xref="S6.SS3.p2.5.m5.1.1.2.cmml">89.85</mn><mo id="S6.SS3.p2.5.m5.1.1.1" xref="S6.SS3.p2.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p2.5.m5.1b"><apply id="S6.SS3.p2.5.m5.1.1.cmml" xref="S6.SS3.p2.5.m5.1.1"><csymbol cd="latexml" id="S6.SS3.p2.5.m5.1.1.1.cmml" xref="S6.SS3.p2.5.m5.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p2.5.m5.1.1.2.cmml" xref="S6.SS3.p2.5.m5.1.1.2">89.85</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p2.5.m5.1c">89.85\%</annotation></semantics></math>) and the annealing temperature (<math id="S6.SS3.p2.6.m6.1" class="ltx_Math" alttext="91.55\%" display="inline"><semantics id="S6.SS3.p2.6.m6.1a"><mrow id="S6.SS3.p2.6.m6.1.1" xref="S6.SS3.p2.6.m6.1.1.cmml"><mn id="S6.SS3.p2.6.m6.1.1.2" xref="S6.SS3.p2.6.m6.1.1.2.cmml">91.55</mn><mo id="S6.SS3.p2.6.m6.1.1.1" xref="S6.SS3.p2.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p2.6.m6.1b"><apply id="S6.SS3.p2.6.m6.1.1.cmml" xref="S6.SS3.p2.6.m6.1.1"><csymbol cd="latexml" id="S6.SS3.p2.6.m6.1.1.1.cmml" xref="S6.SS3.p2.6.m6.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p2.6.m6.1.1.2.cmml" xref="S6.SS3.p2.6.m6.1.1.2">91.55</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p2.6.m6.1c">91.55\%</annotation></semantics></math>).
In addition, the learned temperature has a faster convergence speed.</p>
</div>
<div id="S6.SS3.p3" class="ltx_para ltx_noindent">
<p id="S6.SS3.p3.1" class="ltx_p"><span id="S6.SS3.p3.1.1" class="ltx_text ltx_font_bold">Impact of centroid number and sub-vector length</span>
The centroid number and sub-vector length do affect not only the inference throughput but also the model accuracy.
We present the ablation study for the impact of these two hyperparameters.
We follow the same experiment setting as Sec. <a href="#S6.SS1" title="6.1. Experiment methodologies and settings ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.1</span></a> on ResNet18.</p>
</div>
<div id="S6.SS3.p4" class="ltx_para">
<p id="S6.SS3.p4.1" class="ltx_p">Figs. <a href="#S6.F12" title="Figure 12 ‣ 6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> collect accuracy and FLOPs for the variant of centroid numbers and sub-vector lengths in ResNet18.
For this model, the sub-vector length significantly affects the model accuracy and worsens as the sub-vector length increases.
Each codebook has to handle higher dimensions as the sub-vector length grows, which cannot be accurately classified and harms the model accuracy.</p>
</div>
<div id="S6.SS3.p5" class="ltx_para">
<p id="S6.SS3.p5.2" class="ltx_p">The centroid number also affects accuracy and performance.
As the centroid number increases, each codebook can classify sub-vector in more fine-grained granularity, and it improves the accuracy of ResNet18.
To balance the model accuracy with performance, we set <math id="S6.SS3.p5.1.m1.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S6.SS3.p5.1.m1.1a"><mrow id="S6.SS3.p5.1.m1.1.1" xref="S6.SS3.p5.1.m1.1.1.cmml"><mi id="S6.SS3.p5.1.m1.1.1.2" xref="S6.SS3.p5.1.m1.1.1.2.cmml">K</mi><mo id="S6.SS3.p5.1.m1.1.1.1" xref="S6.SS3.p5.1.m1.1.1.1.cmml">=</mo><mn id="S6.SS3.p5.1.m1.1.1.3" xref="S6.SS3.p5.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p5.1.m1.1b"><apply id="S6.SS3.p5.1.m1.1.1.cmml" xref="S6.SS3.p5.1.m1.1.1"><eq id="S6.SS3.p5.1.m1.1.1.1.cmml" xref="S6.SS3.p5.1.m1.1.1.1"></eq><ci id="S6.SS3.p5.1.m1.1.1.2.cmml" xref="S6.SS3.p5.1.m1.1.1.2">𝐾</ci><cn type="integer" id="S6.SS3.p5.1.m1.1.1.3.cmml" xref="S6.SS3.p5.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p5.1.m1.1c">K=16</annotation></semantics></math> and <math id="S6.SS3.p5.2.m2.1" class="ltx_Math" alttext="V=9" display="inline"><semantics id="S6.SS3.p5.2.m2.1a"><mrow id="S6.SS3.p5.2.m2.1.1" xref="S6.SS3.p5.2.m2.1.1.cmml"><mi id="S6.SS3.p5.2.m2.1.1.2" xref="S6.SS3.p5.2.m2.1.1.2.cmml">V</mi><mo id="S6.SS3.p5.2.m2.1.1.1" xref="S6.SS3.p5.2.m2.1.1.1.cmml">=</mo><mn id="S6.SS3.p5.2.m2.1.1.3" xref="S6.SS3.p5.2.m2.1.1.3.cmml">9</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p5.2.m2.1b"><apply id="S6.SS3.p5.2.m2.1.1.cmml" xref="S6.SS3.p5.2.m2.1.1"><eq id="S6.SS3.p5.2.m2.1.1.1.cmml" xref="S6.SS3.p5.2.m2.1.1.1"></eq><ci id="S6.SS3.p5.2.m2.1.1.2.cmml" xref="S6.SS3.p5.2.m2.1.1.2">𝑉</ci><cn type="integer" id="S6.SS3.p5.2.m2.1.1.3.cmml" xref="S6.SS3.p5.2.m2.1.1.3">9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p5.2.m2.1c">V=9</annotation></semantics></math> for better accuracy and fewer GFLOPs than the vanilla ResNet18.</p>
</div>
<div id="S6.SS3.p6" class="ltx_para">
<p id="S6.SS3.p6.1" class="ltx_p">The optimal hyperparameter setting for DNNs is a long-standing research topic. Some neural network search techniques, such as weight sharing <cite class="ltx_cite ltx_citemacro_citep">(Cai et al<span class="ltx_text">.</span>, <a href="#bib.bib8" title="" class="ltx_ref">2019</a>)</cite> and search space search <cite class="ltx_cite ltx_citemacro_citep">(Radosavovic et al<span class="ltx_text">.</span>, <a href="#bib.bib49" title="" class="ltx_ref">2020</a>)</cite>, could apply to LUT-NN to reduce the searching cost.</p>
</div>
<figure id="S6.F12" class="ltx_figure"><img src="/html/2302.03213/assets/x14.png" id="S6.F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="115" height="57" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12. </span>

The scaling of centroid number and sub-vector length on ResNet18. x-axis is logarithmic.
</figcaption>
</figure>
<div id="S6.SS3.p7" class="ltx_para ltx_noindent">
<p id="S6.SS3.p7.1" class="ltx_p"><span id="S6.SS3.p7.1.1" class="ltx_text ltx_font_bold">The number of replaced operators</span>
As stated, to replace the front layers of BERT by table lookup can greatly impact accuracy, and we replace the last six layers in our default settings. Fig. <a href="#S6.F13" title="Figure 13 ‣ 6.3. Ablation Study ‣ 6. Evaluation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> shows the accuracy impact by replacing more layers.
We gradually replace the operators of more layers, from the last to the front layer, for BERT on the Semantic Textual Similarity Benchmark (STS-B) task.
We can see the accuracy drops obviously for the front three layers.</p>
</div>
<figure id="S6.F13" class="ltx_figure"><img src="/html/2302.03213/assets/x15.png" id="S6.F13.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="230" height="92" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13. </span>

The accuracy of LUT-NN based BERT with respect to the number of layers to replace. </figcaption>
</figure>
<div id="S6.SS3.p8" class="ltx_para ltx_noindent">
<p id="S6.SS3.p8.5" class="ltx_p"><span id="S6.SS3.p8.5.1" class="ltx_text ltx_font_bold">Speedup breakdown</span>
In Sec. <a href="#S5" title="5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we propose four inference optimization techniques, which include: ❶ memory optimization on centroid distance computations, ❷ intra-codebook parallelism, ❸ shuffle instructions table lookup, and ❹ mix-precision accumulation in Fig. <a href="#S5.F6" title="Figure 6 ‣ 5. Table lookup Inference Design ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
To better understand performance gains, we evaluate the execution of a convolution operator (<math id="S6.SS3.p8.1.m1.2" class="ltx_Math" alttext="C_{in}=C_{out}=64,k=3,s=1,H=W=56" display="inline"><semantics id="S6.SS3.p8.1.m1.2a"><mrow id="S6.SS3.p8.1.m1.2.2.2" xref="S6.SS3.p8.1.m1.2.2.3.cmml"><mrow id="S6.SS3.p8.1.m1.1.1.1.1" xref="S6.SS3.p8.1.m1.1.1.1.1.cmml"><msub id="S6.SS3.p8.1.m1.1.1.1.1.2" xref="S6.SS3.p8.1.m1.1.1.1.1.2.cmml"><mi id="S6.SS3.p8.1.m1.1.1.1.1.2.2" xref="S6.SS3.p8.1.m1.1.1.1.1.2.2.cmml">C</mi><mrow id="S6.SS3.p8.1.m1.1.1.1.1.2.3" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.cmml"><mi id="S6.SS3.p8.1.m1.1.1.1.1.2.3.2" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S6.SS3.p8.1.m1.1.1.1.1.2.3.1" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.1.cmml">​</mo><mi id="S6.SS3.p8.1.m1.1.1.1.1.2.3.3" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.3.cmml">n</mi></mrow></msub><mo id="S6.SS3.p8.1.m1.1.1.1.1.3" xref="S6.SS3.p8.1.m1.1.1.1.1.3.cmml">=</mo><msub id="S6.SS3.p8.1.m1.1.1.1.1.4" xref="S6.SS3.p8.1.m1.1.1.1.1.4.cmml"><mi id="S6.SS3.p8.1.m1.1.1.1.1.4.2" xref="S6.SS3.p8.1.m1.1.1.1.1.4.2.cmml">C</mi><mrow id="S6.SS3.p8.1.m1.1.1.1.1.4.3" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.cmml"><mi id="S6.SS3.p8.1.m1.1.1.1.1.4.3.2" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.SS3.p8.1.m1.1.1.1.1.4.3.1" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.1.cmml">​</mo><mi id="S6.SS3.p8.1.m1.1.1.1.1.4.3.3" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S6.SS3.p8.1.m1.1.1.1.1.4.3.1a" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.1.cmml">​</mo><mi id="S6.SS3.p8.1.m1.1.1.1.1.4.3.4" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.4.cmml">t</mi></mrow></msub><mo id="S6.SS3.p8.1.m1.1.1.1.1.5" xref="S6.SS3.p8.1.m1.1.1.1.1.5.cmml">=</mo><mn id="S6.SS3.p8.1.m1.1.1.1.1.6" xref="S6.SS3.p8.1.m1.1.1.1.1.6.cmml">64</mn></mrow><mo id="S6.SS3.p8.1.m1.2.2.2.3" xref="S6.SS3.p8.1.m1.2.2.3a.cmml">,</mo><mrow id="S6.SS3.p8.1.m1.2.2.2.2.2" xref="S6.SS3.p8.1.m1.2.2.2.2.3.cmml"><mrow id="S6.SS3.p8.1.m1.2.2.2.2.1.1" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.cmml"><mi id="S6.SS3.p8.1.m1.2.2.2.2.1.1.2" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.2.cmml">k</mi><mo id="S6.SS3.p8.1.m1.2.2.2.2.1.1.1" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.1.cmml">=</mo><mn id="S6.SS3.p8.1.m1.2.2.2.2.1.1.3" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.3.cmml">3</mn></mrow><mo id="S6.SS3.p8.1.m1.2.2.2.2.2.3" xref="S6.SS3.p8.1.m1.2.2.2.2.3a.cmml">,</mo><mrow id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.3.cmml"><mrow id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.cmml"><mi id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.2" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.2.cmml">s</mi><mo id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.1" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.1.cmml">=</mo><mn id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.3" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.3.cmml">1</mn></mrow><mo id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.3" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.3a.cmml">,</mo><mrow id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.cmml"><mi id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.2" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.2.cmml">H</mi><mo id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.3" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.3.cmml">=</mo><mi id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.4" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.4.cmml">W</mi><mo id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.5" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.5.cmml">=</mo><mn id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.6" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.6.cmml">56</mn></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p8.1.m1.2b"><apply id="S6.SS3.p8.1.m1.2.2.3.cmml" xref="S6.SS3.p8.1.m1.2.2.2"><csymbol cd="ambiguous" id="S6.SS3.p8.1.m1.2.2.3a.cmml" xref="S6.SS3.p8.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S6.SS3.p8.1.m1.1.1.1.1.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1"><and id="S6.SS3.p8.1.m1.1.1.1.1a.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1"></and><apply id="S6.SS3.p8.1.m1.1.1.1.1b.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1"><eq id="S6.SS3.p8.1.m1.1.1.1.1.3.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.3"></eq><apply id="S6.SS3.p8.1.m1.1.1.1.1.2.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S6.SS3.p8.1.m1.1.1.1.1.2.1.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2">subscript</csymbol><ci id="S6.SS3.p8.1.m1.1.1.1.1.2.2.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2.2">𝐶</ci><apply id="S6.SS3.p8.1.m1.1.1.1.1.2.3.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3"><times id="S6.SS3.p8.1.m1.1.1.1.1.2.3.1.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.1"></times><ci id="S6.SS3.p8.1.m1.1.1.1.1.2.3.2.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.2">𝑖</ci><ci id="S6.SS3.p8.1.m1.1.1.1.1.2.3.3.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.2.3.3">𝑛</ci></apply></apply><apply id="S6.SS3.p8.1.m1.1.1.1.1.4.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S6.SS3.p8.1.m1.1.1.1.1.4.1.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4">subscript</csymbol><ci id="S6.SS3.p8.1.m1.1.1.1.1.4.2.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4.2">𝐶</ci><apply id="S6.SS3.p8.1.m1.1.1.1.1.4.3.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3"><times id="S6.SS3.p8.1.m1.1.1.1.1.4.3.1.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.1"></times><ci id="S6.SS3.p8.1.m1.1.1.1.1.4.3.2.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.2">𝑜</ci><ci id="S6.SS3.p8.1.m1.1.1.1.1.4.3.3.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.3">𝑢</ci><ci id="S6.SS3.p8.1.m1.1.1.1.1.4.3.4.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.4.3.4">𝑡</ci></apply></apply></apply><apply id="S6.SS3.p8.1.m1.1.1.1.1c.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1"><eq id="S6.SS3.p8.1.m1.1.1.1.1.5.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.5"></eq><share href="#S6.SS3.p8.1.m1.1.1.1.1.4.cmml" id="S6.SS3.p8.1.m1.1.1.1.1d.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1"></share><cn type="integer" id="S6.SS3.p8.1.m1.1.1.1.1.6.cmml" xref="S6.SS3.p8.1.m1.1.1.1.1.6">64</cn></apply></apply><apply id="S6.SS3.p8.1.m1.2.2.2.2.3.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S6.SS3.p8.1.m1.2.2.2.2.3a.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S6.SS3.p8.1.m1.2.2.2.2.1.1.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1"><eq id="S6.SS3.p8.1.m1.2.2.2.2.1.1.1.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.1"></eq><ci id="S6.SS3.p8.1.m1.2.2.2.2.1.1.2.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.2">𝑘</ci><cn type="integer" id="S6.SS3.p8.1.m1.2.2.2.2.1.1.3.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.1.1.3">3</cn></apply><apply id="S6.SS3.p8.1.m1.2.2.2.2.2.2.3.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S6.SS3.p8.1.m1.2.2.2.2.2.2.3a.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1"><eq id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.1.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.1"></eq><ci id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.2.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.2">𝑠</ci><cn type="integer" id="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.3.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.1.1.3">1</cn></apply><apply id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2"><and id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2a.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2"></and><apply id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2b.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2"><eq id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.3.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.3"></eq><ci id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.2.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.2">𝐻</ci><ci id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.4.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.4">𝑊</ci></apply><apply id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2c.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2"><eq id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.5.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.5"></eq><share href="#S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.4.cmml" id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2d.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2"></share><cn type="integer" id="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.6.cmml" xref="S6.SS3.p8.1.m1.2.2.2.2.2.2.2.2.6">56</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p8.1.m1.2c">C_{in}=C_{out}=64,k=3,s=1,H=W=56</annotation></semantics></math>, the second layer of ResNet18) on Pixel 6 when different techniques are enabled.
The results show that ❸ (<math id="S6.SS3.p8.2.m2.1" class="ltx_Math" alttext="44.6\%" display="inline"><semantics id="S6.SS3.p8.2.m2.1a"><mrow id="S6.SS3.p8.2.m2.1.1" xref="S6.SS3.p8.2.m2.1.1.cmml"><mn id="S6.SS3.p8.2.m2.1.1.2" xref="S6.SS3.p8.2.m2.1.1.2.cmml">44.6</mn><mo id="S6.SS3.p8.2.m2.1.1.1" xref="S6.SS3.p8.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p8.2.m2.1b"><apply id="S6.SS3.p8.2.m2.1.1.cmml" xref="S6.SS3.p8.2.m2.1.1"><csymbol cd="latexml" id="S6.SS3.p8.2.m2.1.1.1.cmml" xref="S6.SS3.p8.2.m2.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p8.2.m2.1.1.2.cmml" xref="S6.SS3.p8.2.m2.1.1.2">44.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p8.2.m2.1c">44.6\%</annotation></semantics></math>) saves the most execution time, followed by methods ❶ (<math id="S6.SS3.p8.3.m3.1" class="ltx_Math" alttext="18.5\%" display="inline"><semantics id="S6.SS3.p8.3.m3.1a"><mrow id="S6.SS3.p8.3.m3.1.1" xref="S6.SS3.p8.3.m3.1.1.cmml"><mn id="S6.SS3.p8.3.m3.1.1.2" xref="S6.SS3.p8.3.m3.1.1.2.cmml">18.5</mn><mo id="S6.SS3.p8.3.m3.1.1.1" xref="S6.SS3.p8.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p8.3.m3.1b"><apply id="S6.SS3.p8.3.m3.1.1.cmml" xref="S6.SS3.p8.3.m3.1.1"><csymbol cd="latexml" id="S6.SS3.p8.3.m3.1.1.1.cmml" xref="S6.SS3.p8.3.m3.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p8.3.m3.1.1.2.cmml" xref="S6.SS3.p8.3.m3.1.1.2">18.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p8.3.m3.1c">18.5\%</annotation></semantics></math>) and ❷ (<math id="S6.SS3.p8.4.m4.1" class="ltx_Math" alttext="16.4\%" display="inline"><semantics id="S6.SS3.p8.4.m4.1a"><mrow id="S6.SS3.p8.4.m4.1.1" xref="S6.SS3.p8.4.m4.1.1.cmml"><mn id="S6.SS3.p8.4.m4.1.1.2" xref="S6.SS3.p8.4.m4.1.1.2.cmml">16.4</mn><mo id="S6.SS3.p8.4.m4.1.1.1" xref="S6.SS3.p8.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p8.4.m4.1b"><apply id="S6.SS3.p8.4.m4.1.1.cmml" xref="S6.SS3.p8.4.m4.1.1"><csymbol cd="latexml" id="S6.SS3.p8.4.m4.1.1.1.cmml" xref="S6.SS3.p8.4.m4.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p8.4.m4.1.1.2.cmml" xref="S6.SS3.p8.4.m4.1.1.2">16.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p8.4.m4.1c">16.4\%</annotation></semantics></math>). Finally, method ❹ delivers a minor improvement (<math id="S6.SS3.p8.5.m5.1" class="ltx_Math" alttext="4.1\%" display="inline"><semantics id="S6.SS3.p8.5.m5.1a"><mrow id="S6.SS3.p8.5.m5.1.1" xref="S6.SS3.p8.5.m5.1.1.cmml"><mn id="S6.SS3.p8.5.m5.1.1.2" xref="S6.SS3.p8.5.m5.1.1.2.cmml">4.1</mn><mo id="S6.SS3.p8.5.m5.1.1.1" xref="S6.SS3.p8.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS3.p8.5.m5.1b"><apply id="S6.SS3.p8.5.m5.1.1.cmml" xref="S6.SS3.p8.5.m5.1.1"><csymbol cd="latexml" id="S6.SS3.p8.5.m5.1.1.1.cmml" xref="S6.SS3.p8.5.m5.1.1.1">percent</csymbol><cn type="float" id="S6.SS3.p8.5.m5.1.1.2.cmml" xref="S6.SS3.p8.5.m5.1.1.2">4.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.p8.5.m5.1c">4.1\%</annotation></semantics></math>).
The results show that our inference design and optimization can take advantage of ARM/x86 ISA’s special instructions, which outperform conventional operators.</p>
</div>
<div id="S6.SS3.p9" class="ltx_para ltx_noindent">
<p id="S6.SS3.p9.1" class="ltx_p"><span id="S6.SS3.p9.1.1" class="ltx_text ltx_font_bold">Scalar quantization level for lookup table</span>
As explained in Sec. <a href="#S3.SS3" title="3.3. Scalar quantized lookup table ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>, LUT-NN leverages scalar quantization to reduce the table size. In this subsection, we evaluate the accuracy impact of different quantization levels by using ResNet18 for the CIFAR10 dataset as an example.
The accuracy of LUT-NN is 94.44% with FP32 lookup table, 94.40% with INT8 lookup table, and 94.27% with INT4 lookup table. The results show that our quantization-aware training can achieve similar accuracy by using lower bits compared to FP32. We select INT8 in our current implementation as its the tradeoff of accuracy and speedup. INT4, on the other hand, is not supported by the hardware SIMD units directly. The exploration of lower-bit lookup table could be a potential future work.</p>
</div>
<figure id="S6.T6" class="ltx_table">
<table id="S6.T6.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T6.1.1.1" class="ltx_tr">
<th id="S6.T6.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t">Model</th>
<th id="S6.T6.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">LUT-NN v.s. TVM Avg. power (W)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T6.1.2.1" class="ltx_tr">
<td id="S6.T6.1.2.1.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">BERT</td>
<td id="S6.T6.1.2.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.6/3.7</td>
</tr>
<tr id="S6.T6.1.3.2" class="ltx_tr">
<td id="S6.T6.1.3.2.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">ResNet18</td>
<td id="S6.T6.1.3.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.6/3.0</td>
</tr>
<tr id="S6.T6.1.4.3" class="ltx_tr">
<td id="S6.T6.1.4.3.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">ResNet18 (CIFAR)</td>
<td id="S6.T6.1.4.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.6/3.3</td>
</tr>
<tr id="S6.T6.1.5.4" class="ltx_tr">
<td id="S6.T6.1.5.4.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">SENET18</td>
<td id="S6.T6.1.5.4.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.6/2.9</td>
</tr>
<tr id="S6.T6.1.6.5" class="ltx_tr">
<td id="S6.T6.1.6.5.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">SENET18 (CIFAR)</td>
<td id="S6.T6.1.6.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.8/3.2</td>
</tr>
<tr id="S6.T6.1.7.6" class="ltx_tr">
<td id="S6.T6.1.7.6.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">VGG11</td>
<td id="S6.T6.1.7.6.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">2.3/2.9</td>
</tr>
<tr id="S6.T6.1.8.7" class="ltx_tr">
<td id="S6.T6.1.8.7.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t">VGG11 (CIFAR)</td>
<td id="S6.T6.1.8.7.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">2.7/3.3</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6. </span>Power (deducted SoC idle power) on Pixel 4.</figcaption>
</figure>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7. </span>Related Work</h2>

<div id="S7.p1" class="ltx_para ltx_noindent">
<p id="S7.p1.1" class="ltx_p"><span id="S7.p1.1.1" class="ltx_text ltx_font_bold">Approximated matrix multiplication</span> Traditional approximated matrix multiplication techniques focus on minimizing the difference between the ground truth and the approximated output.
For example, Magen et al. <cite class="ltx_cite ltx_citemacro_citep">(Magen and Zouzias, <a href="#bib.bib43" title="" class="ltx_ref">2011</a>)</cite> use random projection to map two matrices to a lower-dimensional subspace, and then perform accurate multiplication on the mapped matrices.
<span id="S7.p1.1.2" class="ltx_text ltx_font_smallcaps">Maddness</span> <cite class="ltx_cite ltx_citemacro_citep">(Blalock and Guttag, <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite> uses product quantization to learn the most typical features (also known as centroids) in the input distribution. <span id="S7.p1.1.3" class="ltx_text ltx_font_smallcaps">Maddness</span> then stores the products of centroids and the weight matrix in advance in a lookup table, so that the product of any input matrix and weight matrix can be approximated by the result stored in the table.
To conclude, traditional methods will suffer from severe accuracy loss when used in neural network inference if the approximation is too harsh.
Our method can avoid this defect by integrating itself into the end-to-end training process.</p>
</div>
<div id="S7.p2" class="ltx_para">
<p id="S7.p2.1" class="ltx_p">Also, previously, some attempts have been made to approximate the weights of NN with low-rank tensor expansion <cite class="ltx_cite ltx_citemacro_citep">(Jaderberg et al<span class="ltx_text">.</span>, <a href="#bib.bib31" title="" class="ltx_ref">2014</a>; Denil et al<span class="ltx_text">.</span>, <a href="#bib.bib14" title="" class="ltx_ref">2013</a>; Denton et al<span class="ltx_text">.</span>, <a href="#bib.bib15" title="" class="ltx_ref">2014</a>)</cite>.
Specifically, LCNN <cite class="ltx_cite ltx_citemacro_citep">(Bagherinezhad et al<span class="ltx_text">.</span>, <a href="#bib.bib3" title="" class="ltx_ref">2017</a>)</cite> is also a kind of low-rank approximation.
With LCNN, a dictionary is trained and the NN weights can be represented by linear combinations of the dictionary items.
Afterward, LCNN transforms a vanilla convolution into lookup operations in the dictionary.
Our work is different from LCNN in that our work learns centroids for each layer’s input feature map but not the weight.</p>
</div>
<div id="S7.p3" class="ltx_para ltx_noindent">
<p id="S7.p3.1" class="ltx_p"><span id="S7.p3.1.1" class="ltx_text ltx_font_bold">Product quantization for DNN</span>
Product Quantization <cite class="ltx_cite ltx_citemacro_citep">(Matsui et al<span class="ltx_text">.</span>, <a href="#bib.bib44" title="" class="ltx_ref">2018</a>)</cite> is a popular and successful method for large-scale approximated nearest neighbor search <cite class="ltx_cite ltx_citemacro_citep">(Jegou et al<span class="ltx_text">.</span>, <a href="#bib.bib34" title="" class="ltx_ref">2010</a>)</cite>.
In recent years, researchers have integrated product quantization into neural networks for various purposes.
For example, <cite class="ltx_cite ltx_citemacro_citep">(Yu et al<span class="ltx_text">.</span>, <a href="#bib.bib59" title="" class="ltx_ref">2018</a>; Yue et al<span class="ltx_text">.</span>, <a href="#bib.bib60" title="" class="ltx_ref">2016</a>; Jain et al<span class="ltx_text">.</span>, <a href="#bib.bib32" title="" class="ltx_ref">2017</a>; Klein and Wolf, <a href="#bib.bib37" title="" class="ltx_ref">2017</a>)</cite> incorporated product quantization as a layer in a convolutional neural network to obtain a compact and discriminative image representation.
<cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib9" title="" class="ltx_ref">2020a</a>)</cite> utilized Product Quantization to compress the embedding layer in NLP models.
<cite class="ltx_cite ltx_citemacro_citep">(Stock et al<span class="ltx_text">.</span>, <a href="#bib.bib52" title="" class="ltx_ref">2020</a>; Gong et al<span class="ltx_text">.</span>, <a href="#bib.bib18" title="" class="ltx_ref">2014</a>; Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib11" title="" class="ltx_ref">2020b</a>)</cite> compressed the weight matrix for neural networks with Product Quantization.
However, our approach differs from the above methods that our purpose is to achieve end-to-end neural network inference acceleration.</p>
</div>
<div id="S7.p4" class="ltx_para ltx_noindent">
<p id="S7.p4.1" class="ltx_p"><span id="S7.p4.1.1" class="ltx_text ltx_font_bold">Scalar quantization</span>
Scalar quantization methods aim to reduce scalar bits in neural networks, e.g., from float representation to INT8 representation.
They can both compress the neural network bits and accelerate the inference.
Jacob et al. <cite class="ltx_cite ltx_citemacro_citep">(Jacob et al<span class="ltx_text">.</span>, <a href="#bib.bib30" title="" class="ltx_ref">2018</a>)</cite> proposed an INT8 quantization method with both a training scheme and an efficient kernel implementation.
Other researchers <cite class="ltx_cite ltx_citemacro_citep">(Courbariaux et al<span class="ltx_text">.</span>, <a href="#bib.bib12" title="" class="ltx_ref">2015</a>; Zhou et al<span class="ltx_text">.</span>, <a href="#bib.bib63" title="" class="ltx_ref">2016b</a>; Alemdar et al<span class="ltx_text">.</span>, <a href="#bib.bib2" title="" class="ltx_ref">2017</a>; Guo et al<span class="ltx_text">.</span>, <a href="#bib.bib22" title="" class="ltx_ref">2022</a>; Banner et al<span class="ltx_text">.</span>, <a href="#bib.bib4" title="" class="ltx_ref">2019</a>; McKinstry et al<span class="ltx_text">.</span>, <a href="#bib.bib46" title="" class="ltx_ref">2018</a>)</cite> further proved that 1/2/4-bit quantization is sufficient for image classification.
More recently, Wang et al. <cite class="ltx_cite ltx_citemacro_citep">(Wang et al<span class="ltx_text">.</span>, <a href="#bib.bib55" title="" class="ltx_ref">2022</a>)</cite> proposes a new approach to quantization using trainable lookup tables which can fit the distributions in different layers and have small additional computational costs.
Although we leverage scalar quantization to compress the size of the lookup table (Sec. <a href="#S3.SS3" title="3.3. Scalar quantized lookup table ‣ 3. Differentiable Centroid Learning for DNN ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>), the main idea of our work is different from these quantization methods.
Our work focuses on learning the most typical features of each NN layer and computing the results of these features in advance for future input tensors, in order to both compress NN weights and accelerate NN inference.</p>
</div>
<div id="S7.p5" class="ltx_para ltx_noindent">
<p id="S7.p5.1" class="ltx_p"><span id="S7.p5.1.1" class="ltx_text ltx_font_bold">Inference result reuse</span>
There are a range of works that exploit the duplicated inputs, to reuse their inference results and
skip further model inference.
Cachier <cite class="ltx_cite ltx_citemacro_citep">(Drolia et al<span class="ltx_text">.</span>, <a href="#bib.bib17" title="" class="ltx_ref">2017</a>)</cite> reuses results among nearby users. Potluck <cite class="ltx_cite ltx_citemacro_citep">(Guo and Hu, <a href="#bib.bib24" title="" class="ltx_ref">2018</a>)</cite> reuses results across applications. Euphrate <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al<span class="ltx_text">.</span>, <a href="#bib.bib64" title="" class="ltx_ref">2018</a>)</cite> reuses results of temporal pixel motion for continuous vision. FoggyCache <cite class="ltx_cite ltx_citemacro_citep">(Guo et al<span class="ltx_text">.</span>, <a href="#bib.bib23" title="" class="ltx_ref">2018</a>)</cite> reuses results across devices. DeepCache reuses results of similar regions in recent frames of consecutive videos. MCDNN <cite class="ltx_cite ltx_citemacro_citep">(Han et al<span class="ltx_text">.</span>, <a href="#bib.bib26" title="" class="ltx_ref">2016</a>)</cite> reuses results across applications. Semantic Memory <cite class="ltx_cite ltx_citemacro_citep">(Li et al<span class="ltx_text">.</span>, <a href="#bib.bib39" title="" class="ltx_ref">2021</a>)</cite> reuses results from some layers for continuous vision.
These works rely on specific application inputs.
Our work, by comparison, optimizes the model itself to reduce size and FLOPS.</p>
</div>
<div id="S7.p6" class="ltx_para ltx_noindent">
<p id="S7.p6.1" class="ltx_p"><span id="S7.p6.1.1" class="ltx_text ltx_font_bold">Lookup table for DNN</span>
Different from our work that learns and stores the centroid results in the table, there are works that store other results in the tables. LogicNets <cite class="ltx_cite ltx_citemacro_citep">(Umuroglu et al<span class="ltx_text">.</span>, <a href="#bib.bib53" title="" class="ltx_ref">2020</a>)</cite> maps artificial neurons to truth tables in FPGA. It also compresses the model by available pruning and scalar quantization techniques. Ramanathan et al. <cite class="ltx_cite ltx_citemacro_citep">(Ramanathan et al<span class="ltx_text">.</span>, <a href="#bib.bib50" title="" class="ltx_ref">2020</a>)</cite> store the results of bitline computing in Processing in Cache hardware in the table, to accelerate DNN. These works do not target neural network optimization or training techniques like LUT-NN.</p>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8. </span>Future opportunity discussion</h2>

<div id="S8.p1" class="ltx_para ltx_noindent">
<p id="S8.p1.1" class="ltx_p"><span id="S8.p1.1.1" class="ltx_text ltx_font_bold">Hardware implication </span> Potentially, LUT-NN can greatly reduce FLOPs, e.g., <math id="S8.p1.1.m1.1" class="ltx_math_unparsed" alttext="16\times" display="inline"><semantics id="S8.p1.1.m1.1a"><mrow id="S8.p1.1.m1.1b"><mn id="S8.p1.1.m1.1.1">16</mn><mo lspace="0.222em" id="S8.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S8.p1.1.m1.1c">16\times</annotation></semantics></math> for BERT. However, current hardware restrains LUT-NN from showing its full potential. The reason is that current hardware is optimized for linear computation, but not table lookup. Compared to direct Multiply-Add support for MM in hardware,
LUT-NN has to run <code id="S8.p1.1.2" class="ltx_verbatim ltx_font_typewriter">argmin</code> partially sequentially to return the index of the nearest centroid, and then lookup in the table followed by aggregation. Besides, the SIMD width limits the options of the number of centroids. These are obviously inefficient.</p>
</div>
<div id="S8.p2" class="ltx_para">
<p id="S8.p2.1" class="ltx_p">Even on this unfriendly hardware, LUT-NN has shown the benefits of table lookup on all dimensions, particularly for latency, power, and memory reduction. An accelerator or function unit supporting the first-class parallel table-lookup pipeline could even more significantly reduce the inference cost. LUT-NN opens new opportunity for hardware design.</p>
</div>
<div id="S8.p3" class="ltx_para ltx_noindent">
<p id="S8.p3.1" class="ltx_p"><span id="S8.p3.1.1" class="ltx_text ltx_font_bold">Replace more operator types </span>
Currently LUT-NN replaces linear computation operators by table lookup, including MM, convolution, and fully connected. The scaled dot-product attention (<math id="S8.p3.1.m1.1" class="ltx_Math" alttext="&lt;2\%" display="inline"><semantics id="S8.p3.1.m1.1a"><mrow id="S8.p3.1.m1.1.1" xref="S8.p3.1.m1.1.1.cmml"><mi id="S8.p3.1.m1.1.1.2" xref="S8.p3.1.m1.1.1.2.cmml"></mi><mo id="S8.p3.1.m1.1.1.1" xref="S8.p3.1.m1.1.1.1.cmml">&lt;</mo><mrow id="S8.p3.1.m1.1.1.3" xref="S8.p3.1.m1.1.1.3.cmml"><mn id="S8.p3.1.m1.1.1.3.2" xref="S8.p3.1.m1.1.1.3.2.cmml">2</mn><mo id="S8.p3.1.m1.1.1.3.1" xref="S8.p3.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S8.p3.1.m1.1b"><apply id="S8.p3.1.m1.1.1.cmml" xref="S8.p3.1.m1.1.1"><lt id="S8.p3.1.m1.1.1.1.cmml" xref="S8.p3.1.m1.1.1.1"></lt><csymbol cd="latexml" id="S8.p3.1.m1.1.1.2.cmml" xref="S8.p3.1.m1.1.1.2">absent</csymbol><apply id="S8.p3.1.m1.1.1.3.cmml" xref="S8.p3.1.m1.1.1.3"><csymbol cd="latexml" id="S8.p3.1.m1.1.1.3.1.cmml" xref="S8.p3.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S8.p3.1.m1.1.1.3.2.cmml" xref="S8.p3.1.m1.1.1.3.2">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S8.p3.1.m1.1c">&lt;2\%</annotation></semantics></math> of total latency) in attention layers have no weights, and cannot precompute the results with centroids to save in tables, so we do not replace it by lookup table in this work. A possible way to replace it by lookup table is to precompute and save the production results of centroids. In addition, how to replace the non-linear operators, such as the activation functions, by LUT-NN can be future work.</p>
</div>
<div id="S8.p4" class="ltx_para ltx_noindent">
<p id="S8.p4.2" class="ltx_p"><span id="S8.p4.2.1" class="ltx_text ltx_font_bold">Learning for hashing </span>
As introduced in Sec. <a href="#S2.SS1" title="2.1. Product Quantization: Background ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>, hashing can be used to avoid the Euclidean distance calculation of <math id="S8.p4.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S8.p4.1.m1.1a"><mi id="S8.p4.1.m1.1.1" xref="S8.p4.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S8.p4.1.m1.1b"><ci id="S8.p4.1.m1.1.1.cmml" xref="S8.p4.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S8.p4.1.m1.1c">k</annotation></semantics></math>-means encoding, but at the cost of higher quantization error (refer to Fig. <a href="#S2.F3.sf2" title="In Figure 3 ‣ 2.3. Motivation: Poor results of PQ for DNN ‣ 2. Background and Motivation ‣ LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3b</span></a>). We evaluate the potential of hashing for DNN inference. Since hashing is not differentiable, LUT-NN uses hashing after the centroids are learned. Our current results show that to achieve similar model accuracy as Euclidean distance for encoding, we have to use a 12-level decision tree for hashing. Compared to distance calculation, it can further reduce FLOPs by 30% to <math id="S8.p4.2.m2.1" class="ltx_math_unparsed" alttext="3\times" display="inline"><semantics id="S8.p4.2.m2.1a"><mrow id="S8.p4.2.m2.1b"><mn id="S8.p4.2.m2.1.1">3</mn><mo lspace="0.222em" id="S8.p4.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S8.p4.2.m2.1c">3\times</annotation></semantics></math>. It could be possible to also integrate hashing into the backpropagation to learn the hashing functions. This could reduce the depth of the tree and further reduce the encoding cost.</p>
</div>
<div id="S8.p5" class="ltx_para">
<p id="S8.p5.1" class="ltx_p">On current commodity hardware without direct hashing support, hashing costs even higher than Euclidean distance calculation, since the tree traversing is sequential without SIMD support. Therefore, we still use Euclidean distance for encoding in LUT-NN now. Future hardware can also integrate hashing units for further speedup.</p>
</div>
<div id="S8.p6" class="ltx_para ltx_noindent">
<p id="S8.p6.1" class="ltx_p"><span id="S8.p6.1.1" class="ltx_text ltx_font_bold">LUT-NN on GPUs and FPGAs</span>
NVIDIA GPUs provide the efficient shuffle instruction, i.e., “prmt”, achieving Tera-OPs throughput. Similar as the LUT-NN implementation on the CPUs, which heavily depends on the shuffle instructions, this GPU shuffle instruction can be leveraged for parallelly looking up the approximated computation results in the tables.
FPGAs contain a vast array of lookup tables implemented through reconfigurable memory, which LUT-NN can utilize for the storage and retrieval of codebooks. Additionally, FPGAs are equipped with Block RAM, a primary memory resource that enables efficient data storage. It can be employed for LUT-NN’s lookup tables.</p>
</div>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9. </span>Conclusion</h2>

<div id="S9.p1" class="ltx_para">
<p id="S9.p1.1" class="ltx_p">This paper takes the first step towards DNN inference by table lookup. A new paradigm potentially brings significant benefits to the DNN inference ecosystem, simplifying the inference software and hardware design and decoupling with the DNN algorithm updates. By the centroid learning technique for DNN, LUT-NN achieves comparable accuracy for complex tasks with much less resource cost.
However, LUT-NN still has space to be improved, such as the learning technique to improve accuracy and hardware support for table lookup, which calls for future efforts from the community.</p>
</div>
</section>
<section id="S10" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">10. </span>Acknowledgement</h2>

<div id="S10.p1" class="ltx_para">
<p id="S10.p1.1" class="ltx_p">The authors thank the anonymous shepherd and reviewers for their valuable comments. The work of Yunxin Liu was supported by the National Key R&amp;D Program of China (No. 2022YFF0604501) and Xiaomi Foundation. The work of Deng Cai was supported by the National Nature Science Foundation of China (Grant Nos: 62273302, 62036009, 61936006).</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(1)</span>
<span class="ltx_bibblock">        




</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Alemdar et al<span id="bib.bib2.2.2.1" class="ltx_text">.</span> (2017)</span>
<span class="ltx_bibblock">
Hande Alemdar, Vincent Leroy, Adrien Prost-Boucle, and Frédéric Pétrot. 2017.

</span>
<span class="ltx_bibblock">Ternary neural networks for resource-efficient AI applications. In <em id="bib.bib2.3.1" class="ltx_emph ltx_font_italic">2017 international joint conference on neural networks (IJCNN)</em>. IEEE, 2547–2554.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bagherinezhad et al<span id="bib.bib3.2.2.1" class="ltx_text">.</span> (2017)</span>
<span class="ltx_bibblock">
Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. 2017.

</span>
<span class="ltx_bibblock">Lcnn: Lookup-based convolutional neural network. In <em id="bib.bib3.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 7120–7129.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Banner et al<span id="bib.bib4.2.2.1" class="ltx_text">.</span> (2019)</span>
<span class="ltx_bibblock">
Ron Banner, Yury Nahshan, and Daniel Soudry. 2019.

</span>
<span class="ltx_bibblock">Post training 4-bit quantization of convolutional networks for rapid-deployment.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.3.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em> 32 (2019).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blalock et al<span id="bib.bib5.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. 2020.

</span>
<span class="ltx_bibblock">What is the state of neural network pruning?

</span>
<span class="ltx_bibblock"><em id="bib.bib5.3.1" class="ltx_emph ltx_font_italic">Proceedings of machine learning and systems</em> 2 (2020), 129–146.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blalock and Guttag (2021)</span>
<span class="ltx_bibblock">
Davis Blalock and John Guttag. 2021.

</span>
<span class="ltx_bibblock">Multiplying Matrices Without Multiplying. In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th International Conference on Machine Learning</em> <em id="bib.bib6.2.2" class="ltx_emph ltx_font_italic">(Proceedings of Machine Learning Research, Vol. 139)</em>, Marina Meila and Tong Zhang (Eds.). PMLR, 992–1004.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://proceedings.mlr.press/v139/blalock21a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v139/blalock21a.html</a>

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blalock and Guttag (2017)</span>
<span class="ltx_bibblock">
Davis W Blalock and John V Guttag. 2017.

</span>
<span class="ltx_bibblock">Bolt: Accelerated data mining with fast vector compression. In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining</em>. 727–735.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cai et al<span id="bib.bib8.2.2.1" class="ltx_text">.</span> (2019)</span>
<span class="ltx_bibblock">
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2019.

</span>
<span class="ltx_bibblock">Once-for-all: Train one network and specialize it for efficient deployment.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.3.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1908.09791</em> (2019).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al<span id="bib.bib9.2.2.1" class="ltx_text">.</span> (2020a)</span>
<span class="ltx_bibblock">
Ting Chen, Lala Li, and Yizhou Sun. 2020a.

</span>
<span class="ltx_bibblock">Differentiable product quantization for end-to-end embedding compression. In <em id="bib.bib9.3.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>. PMLR, 1617–1626.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al<span id="bib.bib10.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al<span id="bib.bib10.3.1" class="ltx_text">.</span> 2018.

</span>
<span class="ltx_bibblock">TVM: An automated end-to-end optimizing compiler for deep learning. In <em id="bib.bib10.4.1" class="ltx_emph ltx_font_italic">13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)</em>. 578–594.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al<span id="bib.bib11.2.2.1" class="ltx_text">.</span> (2020b)</span>
<span class="ltx_bibblock">
Weihan Chen, Peisong Wang, and Jian Cheng. 2020b.

</span>
<span class="ltx_bibblock">Towards Convolutional Neural Networks Compression via Global&amp;Progressive Product Quantization.. In <em id="bib.bib11.3.1" class="ltx_emph ltx_font_italic">BMVC</em>.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Courbariaux et al<span id="bib.bib12.2.2.1" class="ltx_text">.</span> (2015)</span>
<span class="ltx_bibblock">
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015.

</span>
<span class="ltx_bibblock">Binaryconnect: Training deep neural networks with binary weights during propagations.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.3.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em> 28 (2015).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al<span id="bib.bib13.2.2.1" class="ltx_text">.</span> (2009)</span>
<span class="ltx_bibblock">
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.

</span>
<span class="ltx_bibblock">ImageNet: A large-scale hierarchical image database. In <em id="bib.bib13.3.1" class="ltx_emph ltx_font_italic">2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA</em>. IEEE Computer Society, 248–255.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/CVPR.2009.5206848" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/CVPR.2009.5206848</a>

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Denil et al<span id="bib.bib14.2.2.1" class="ltx_text">.</span> (2013)</span>
<span class="ltx_bibblock">
Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas. 2013.

</span>
<span class="ltx_bibblock">Predicting parameters in deep learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.3.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em> 26 (2013).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Denton et al<span id="bib.bib15.2.2.1" class="ltx_text">.</span> (2014)</span>
<span class="ltx_bibblock">
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014.

</span>
<span class="ltx_bibblock">Exploiting linear structure within convolutional networks for efficient evaluation.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.3.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em> 27 (2014).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al<span id="bib.bib16.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.

</span>
<span class="ltx_bibblock">Bert: Pre-training of deep bidirectional transformers for language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.3.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1810.04805</em> (2018).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Drolia et al<span id="bib.bib17.2.2.1" class="ltx_text">.</span> (2017)</span>
<span class="ltx_bibblock">
Utsav Drolia, Katherine Guo, Jiaqi Tan, Rajeev Gandhi, and Priya Narasimhan. 2017.

</span>
<span class="ltx_bibblock">Cachier: Edge-caching for recognition applications. In <em id="bib.bib17.3.1" class="ltx_emph ltx_font_italic">2017 IEEE 37th international conference on distributed computing systems (ICDCS)</em>. IEEE, 276–286.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gong et al<span id="bib.bib18.2.2.1" class="ltx_text">.</span> (2014)</span>
<span class="ltx_bibblock">
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014.

</span>
<span class="ltx_bibblock">Compressing deep convolutional networks using vector quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.3.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1412.6115</em> (2014).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Google (2019)</span>
<span class="ltx_bibblock">
Google. 2019.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">TensorFlow: An end-to-end open source machine learning platform</em>.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://www.tensorflow.org/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.tensorflow.org/</a>

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gou et al<span id="bib.bib20.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.

</span>
<span class="ltx_bibblock">Knowledge distillation: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.3.1" class="ltx_emph ltx_font_italic">International Journal of Computer Vision</em> 129 (2021), 1789–1819.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gray and Neuhoff (1998)</span>
<span class="ltx_bibblock">
R.M. Gray and D.L. Neuhoff. 1998.

</span>
<span class="ltx_bibblock">Quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Information Theory</em> 44, 6 (1998), 2325–2383.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/18.720541" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/18.720541</a>

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al<span id="bib.bib22.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Cong Guo, Chen Zhang, Jingwen Leng, Zihan Liu, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. 2022.

</span>
<span class="ltx_bibblock">ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization. In <em id="bib.bib22.3.1" class="ltx_emph ltx_font_italic">55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, October 1-5, 2022</em>. IEEE, 1414–1433.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/MICRO56248.2022.00095" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/MICRO56248.2022.00095</a>

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al<span id="bib.bib23.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Peizhen Guo, Bo Hu, Rui Li, and Wenjun Hu. 2018.

</span>
<span class="ltx_bibblock">Foggycache: Cross-device approximate computation reuse. In <em id="bib.bib23.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 24th annual international conference on mobile computing and networking</em>. 19–34.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo and Hu (2018)</span>
<span class="ltx_bibblock">
Peizhen Guo and Wenjun Hu. 2018.

</span>
<span class="ltx_bibblock">Potluck: Cross-application approximate deduplication for computation-intensive mobile applications. In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems</em>. 271–284.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al<span id="bib.bib25.2.2.1" class="ltx_text">.</span> (2015)</span>
<span class="ltx_bibblock">
Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. 2015.

</span>
<span class="ltx_bibblock">Quantization based Fast Inner Product Search.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.3.1" class="ltx_emph ltx_font_italic">CoRR</em> abs/1509.01469 (2015).

</span>
<span class="ltx_bibblock">arXiv:1509.01469

<a target="_blank" href="http://arxiv.org/abs/1509.01469" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1509.01469</a>

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al<span id="bib.bib26.2.2.1" class="ltx_text">.</span> (2016)</span>
<span class="ltx_bibblock">
Seungyeop Han, Haichen Shen, Matthai Philipose, Sharad Agarwal, Alec Wolman, and Arvind Krishnamurthy. 2016.

</span>
<span class="ltx_bibblock">Mcdnn: An approximation-based execution framework for deep stream processing under resource constraints. In <em id="bib.bib26.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services</em>. 123–136.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al<span id="bib.bib27.2.2.1" class="ltx_text">.</span> (2013)</span>
<span class="ltx_bibblock">
Kaiming He, Fang Wen, and Jian Sun. 2013.

</span>
<span class="ltx_bibblock">K-Means Hashing: An Affinity-Preserving Quantization Method for Learning Binary Compact Codes. In <em id="bib.bib27.3.1" class="ltx_emph ltx_font_italic">2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23-28, 2013</em>. IEEE Computer Society, 2938–2945.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/CVPR.2013.378" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/CVPR.2013.378</a>

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al<span id="bib.bib28.2.2.1" class="ltx_text">.</span> (2016)</span>
<span class="ltx_bibblock">
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.

</span>
<span class="ltx_bibblock">Deep residual learning for image recognition. In <em id="bib.bib28.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 770–778.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al<span id="bib.bib29.2.2.1" class="ltx_text">.</span> (2015)</span>
<span class="ltx_bibblock">
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.

</span>
<span class="ltx_bibblock">Distilling the Knowledge in a Neural Network.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.48550/ARXIV.1503.02531" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/ARXIV.1503.02531</a>

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jacob et al<span id="bib.bib30.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018.

</span>
<span class="ltx_bibblock">Quantization and training of neural networks for efficient integer-arithmetic-only inference. In <em id="bib.bib30.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 2704–2713.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jaderberg et al<span id="bib.bib31.2.2.1" class="ltx_text">.</span> (2014)</span>
<span class="ltx_bibblock">
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. 2014.

</span>
<span class="ltx_bibblock">Speeding up convolutional neural networks with low rank expansions.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.3.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1405.3866</em> (2014).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jain et al<span id="bib.bib32.2.2.1" class="ltx_text">.</span> (2017)</span>
<span class="ltx_bibblock">
Himalaya Jain, Joaquin Zepeda, Patrick Pérez, and Rémi Gribonval. 2017.

</span>
<span class="ltx_bibblock">Subic: A supervised, structured binary code for image search. In <em id="bib.bib32.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer vision</em>. 833–842.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jang et al<span id="bib.bib33.2.2.1" class="ltx_text">.</span> (2016)</span>
<span class="ltx_bibblock">
Eric Jang, Shixiang Gu, and Ben Poole. 2016.

</span>
<span class="ltx_bibblock">Categorical Reparameterization with Gumbel-Softmax.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.48550/ARXIV.1611.01144" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/ARXIV.1611.01144</a>

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jegou et al<span id="bib.bib34.2.2.1" class="ltx_text">.</span> (2010)</span>
<span class="ltx_bibblock">
Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010.

</span>
<span class="ltx_bibblock">Product quantization for nearest neighbor search.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.3.1" class="ltx_emph ltx_font_italic">IEEE transactions on pattern analysis and machine intelligence</em> 33, 1 (2010), 117–128.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiao et al<span id="bib.bib35.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Yang Jiao, Liang Han, and Xin Long. 2020.

</span>
<span class="ltx_bibblock">Hanguang 800 NPU - The Ultimate AI Inference Solution for Data Centers. In <em id="bib.bib35.3.1" class="ltx_emph ltx_font_italic">IEEE Hot Chips 32 Symposium, HCS 2020, Palo Alto, CA, USA, August 16-18, 2020</em>. IEEE, 1–29.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/HCS49909.2020.9220619" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/HCS49909.2020.9220619</a>

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jouppi et al<span id="bib.bib36.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Norman P. Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B. Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, Thomas Norrie, Nishant Patil, Sushma Prasad, Cliff Young, Zongwei Zhou, and David Patterson. 2021.

</span>
<span class="ltx_bibblock">Ten Lessons from Three Generations Shaped Google’s TPUv4i <em id="bib.bib36.3.1" class="ltx_emph ltx_font_italic">(ISCA ’21)</em>. IEEE Press, 1–14.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/ISCA52012.2021.00010" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/ISCA52012.2021.00010</a>

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Klein and Wolf (2017)</span>
<span class="ltx_bibblock">
Benjamin Klein and Lior Wolf. 2017.

</span>
<span class="ltx_bibblock">In defense of product quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1711.08589</em> 2, 3 (2017), 4.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krizhevsky (2009)</span>
<span class="ltx_bibblock">
Alex Krizhevsky. 2009.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Learning multiple layers of features from tiny images</em>.

</span>
<span class="ltx_bibblock">Technical Report.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al<span id="bib.bib39.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Yun Li, Chen Zhang, Shihao Han, Li Lyna Zhang, Baoqun Yin, Yunxin Liu, and Mengwei Xu. 2021.

</span>
<span class="ltx_bibblock">Boosting mobile CNN inference through semantic memory. In <em id="bib.bib39.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 29th ACM International Conference on Multimedia</em>. 2362–2371.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liang et al<span id="bib.bib40.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Rendong Liang, Ting Cao, Jicheng Wen, Manni Wang, Yang Wang, Jianhua Zou, and Yunxin Liu. 2022.

</span>
<span class="ltx_bibblock">Romou: Rapidly generate high-performance tensor kernels for mobile gpus. In <em id="bib.bib40.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 28th Annual International Conference on Mobile Computing And Networking</em>. 487–500.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lloyd (1982)</span>
<span class="ltx_bibblock">
Stuart P. Lloyd. 1982.

</span>
<span class="ltx_bibblock">Least squares quantization in PCM.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">IEEE Trans. Inf. Theory</em> 28, 2 (1982), 129–136.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/TIT.1982.1056489" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/TIT.1982.1056489</a>

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">MACE (2020)</span>
<span class="ltx_bibblock">
MACE. 2020.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://github.com/XiaoMi/mace." title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/XiaoMi/mace.</a>

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Magen and Zouzias (2011)</span>
<span class="ltx_bibblock">
Avner Magen and Anastasios Zouzias. 2011.

</span>
<span class="ltx_bibblock">Low rank matrix-valued Chernoff bounds and approximate matrix multiplication. In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms</em>. SIAM, 1422–1436.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Matsui et al<span id="bib.bib44.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin’ichi Satoh. 2018.

</span>
<span class="ltx_bibblock">A survey of product quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.3.1" class="ltx_emph ltx_font_italic">ITE Transactions on Media Technology and Applications</em> 6, 1 (2018), 2–10.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McCarter and Dronen (2022)</span>
<span class="ltx_bibblock">
Calvin McCarter and Nicholas Dronen. 2022.

</span>
<span class="ltx_bibblock">Look-ups are not (yet) all you need for deep learning inference.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2207.05808</em> (2022).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McKinstry et al<span id="bib.bib46.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani, John V Arthur, Izzet B Yildiz, and Dharmendra S Modha. 2018.

</span>
<span class="ltx_bibblock">Discovering low-precision networks close to full-precision networks for efficient embedded inference.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.3.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1809.04191</em> (2018).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Microsoft (2019)</span>
<span class="ltx_bibblock">
Microsoft. 2019.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">ONNX Runtime</em>.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://github.com/microsoft/onnxruntime" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/microsoft/onnxruntime</a>

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Netzer et al<span id="bib.bib48.2.2.1" class="ltx_text">.</span> (2011)</span>
<span class="ltx_bibblock">
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. 2011.

</span>
<span class="ltx_bibblock">Reading digits in natural images with unsupervised feature learning. In <em id="bib.bib48.3.1" class="ltx_emph ltx_font_italic">NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011</em>.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radosavovic et al<span id="bib.bib49.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. 2020.

</span>
<span class="ltx_bibblock">Designing network design spaces. In <em id="bib.bib49.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>. 10428–10436.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ramanathan et al<span id="bib.bib50.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Akshay Krishna Ramanathan, Gurpreet S. Kalsi, Srivatsa Srinivasa, Tarun Makesh Chandran, Kamlesh R. Pillai, Om Ji Omer, Vijaykrishnan Narayanan, and Sreenivas Subramoney. 2020.

</span>
<span class="ltx_bibblock">Look-Up Table based Energy Efficient Processing in Cache Support for Neural Network Acceleration. In <em id="bib.bib50.3.1" class="ltx_emph ltx_font_italic">53rd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2020, Athens, Greece, October 17-21, 2020</em>. IEEE, 88–101.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/MICRO50266.2020.00020" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/MICRO50266.2020.00020</a>

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stallkamp et al<span id="bib.bib51.2.2.1" class="ltx_text">.</span> (2011)</span>
<span class="ltx_bibblock">
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. 2011.

</span>
<span class="ltx_bibblock">The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In <em id="bib.bib51.3.1" class="ltx_emph ltx_font_italic">The 2011 International Joint Conference on Neural Networks</em>. 1453–1460.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/IJCNN.2011.6033395" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/IJCNN.2011.6033395</a>

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stock et al<span id="bib.bib52.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. 2020.

</span>
<span class="ltx_bibblock">And the Bit Goes Down: Revisiting the Quantization of Neural Networks. In <em id="bib.bib52.3.1" class="ltx_emph ltx_font_italic">ICLR 2020-Eighth International Conference on Learning Representations</em>. 1–11.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Umuroglu et al<span id="bib.bib53.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Yaman Umuroglu, Yash Akhauri, Nicholas James Fraser, and Michaela Blott. 2020.

</span>
<span class="ltx_bibblock">LogicNets: Co-designed neural networks and circuits for extreme-throughput applications. In <em id="bib.bib53.3.1" class="ltx_emph ltx_font_italic">2020 30th International Conference on Field-Programmable Logic and Applications (FPL)</em>. IEEE, 291–297.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al<span id="bib.bib54.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018.

</span>
<span class="ltx_bibblock">GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.3.1" class="ltx_emph ltx_font_italic">CoRR</em> abs/1804.07461 (2018).

</span>
<span class="ltx_bibblock">arXiv:1804.07461

<a target="_blank" href="http://arxiv.org/abs/1804.07461" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1804.07461</a>

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al<span id="bib.bib55.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Longguang Wang, Xiaoyu Dong, Yingqian Wang, Li Liu, Wei An, and Yulan Guo. 2022.

</span>
<span class="ltx_bibblock">Learnable lookup table for neural network quantization. In <em id="bib.bib55.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>. 12423–12433.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al<span id="bib.bib56.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Manni Wang, Shaohua Ding, Ting Cao, Yunxin Liu, and Fengyuan Xu. 2021.

</span>
<span class="ltx_bibblock">Asymo: scalable and efficient deep-learning inference on asymmetric mobile cpus. In <em id="bib.bib56.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 27th Annual International Conference on Mobile Computing and Networking</em>. 215–228.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Warden (2018)</span>
<span class="ltx_bibblock">
Pete Warden. 2018.

</span>
<span class="ltx_bibblock">Speech commands: A dataset for limited-vocabulary speech recognition.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1804.03209</em> (2018).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wechsler et al<span id="bib.bib58.2.2.1" class="ltx_text">.</span> (2019)</span>
<span class="ltx_bibblock">
Ofri Wechsler, Michael Behar, and Bharat Daga. 2019.

</span>
<span class="ltx_bibblock">Spring Hill (NNP-I 1000) Intel’s Data Center Inference Chip. In <em id="bib.bib58.3.1" class="ltx_emph ltx_font_italic">2019 IEEE Hot Chips 31 Symposium (HCS)</em>. 1–12.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/HOTCHIPS.2019.8875671" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/HOTCHIPS.2019.8875671</a>

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al<span id="bib.bib59.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Tan Yu, Junsong Yuan, Chen Fang, and Hailin Jin. 2018.

</span>
<span class="ltx_bibblock">Product quantization network for fast image retrieval. In <em id="bib.bib59.3.1" class="ltx_emph ltx_font_italic">Proceedings of the European Conference on Computer Vision (ECCV)</em>. 186–201.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yue et al<span id="bib.bib60.2.2.1" class="ltx_text">.</span> (2016)</span>
<span class="ltx_bibblock">
Cao Yue, M Long, J Wang, Zhu Han, and Q Wen. 2016.

</span>
<span class="ltx_bibblock">Deep quantization network for efficient image retrieval. In <em id="bib.bib60.3.1" class="ltx_emph ltx_font_italic">Proc. 13th AAAI Conf. Artif. Intell.</em> 3457–3463.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al<span id="bib.bib61.2.2.1" class="ltx_text">.</span> (2017)</span>
<span class="ltx_bibblock">
Zhifei Zhang, Yang Song, and Hairong Qi. 2017.

</span>
<span class="ltx_bibblock">Age progression/regression by conditional adversarial autoencoder. In <em id="bib.bib61.3.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 5810–5818.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al<span id="bib.bib62.2.2.1" class="ltx_text">.</span> (2016a)</span>
<span class="ltx_bibblock">
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. 2016a.

</span>
<span class="ltx_bibblock">DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients.

</span>
<span class="ltx_bibblock"><em id="bib.bib62.3.1" class="ltx_emph ltx_font_italic">CoRR</em> abs/1606.06160 (2016).

</span>
<span class="ltx_bibblock">arXiv:1606.06160

<a target="_blank" href="http://arxiv.org/abs/1606.06160" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1606.06160</a>

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al<span id="bib.bib63.2.2.1" class="ltx_text">.</span> (2016b)</span>
<span class="ltx_bibblock">
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. 2016b.

</span>
<span class="ltx_bibblock">Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.3.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1606.06160</em> (2016).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al<span id="bib.bib64.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Yuhao Zhu, Anand Samajdar, Matthew Mattina, and Paul Whatmough. 2018.

</span>
<span class="ltx_bibblock">Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision. In <em id="bib.bib64.3.1" class="ltx_emph ltx_font_italic">2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)</em>. IEEE, 547–560.

</span>
<span class="ltx_bibblock">
</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2302.03212" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2302.03213" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2302.03213">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2302.03213" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2302.03214" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Mar  1 03:03:23 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2302.03213
On-device Deep Neural Network (DNN)  inference consumes significant computing resources and development efforts. To alleviate that, we propose LUT-NN, the first system to empower inference by table lookup, to reduce inference cost. LUT-NN learns the typical features for each operator, named centroid, and precompute the results for these centroids to save in lookup tables. During inference, the results of the closest centroids with the inputs can be read directly from the table, as the approximated outputs without computations. LUT-NN integrates two major novel techniques: (1)  differentiable centroid learning through backpropagation, which adapts three levels of approximation to minimize the accuracy impact by centroids; (2)  table lookup inference execution, which comprehensively considers different levels of parallelism, memory access reduction, and dedicated hardware units for optimal performance. LUT-NN is evaluated on multiple real tasks, covering image and speech recognition, and nature language processing. Compared to related work, LUT-NN improves accuracy by 66% to 92%, achieving similar level with the original models. LUT-NN reduces the cost at all dimensions, including FLOPs () , model size () , latency () , memory () , and power () .
abstract
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
Abstract
2302.03213
1.IntroductionDNNs are widely used in mobile applications, offering users unparalleled intelligent services. However, DNNs are computation hungry workloads, mainly composed of linear computation operators, heavily stressing the limited hardware resources on mobile devices.As a result, huge efforts have been taken for efficient and affordable DNN inference on mobile devices, such as model compression, sophisticated computing operator optimization, tensor compilers for operator generation, and customized DNN accelerators. These methods require redesigning or re-implementing model structures, computation operators, or accelerators repeatedly for diverse deployment scenarios.Different from these directions, this paper explores a new possibility to replace computation operators of DNNs, for reduced inference cost, as well as the tedious efforts of operator development. To achieve this, we first rethink the essence of inference. Each layer of a DNN model is to output another level of features given the input features. For example, the front layers of an image recognition model output low-level features (e.g., edges and lines) , and subsequent layers output high-level features (e.g., faces and objects) . The fact is that the features of different images for each layer have semantic similarity. A vivid example is that though cats and horses are different input images for a model, their ear features result in similar output for the layer that extracts it. Same for language tasks, similar words of different sentences could have similar embeddings and output results.Based on this similarity, the question is that whether the typical features can be learned for each computation operator of DNNs, so that the output of these typical features can represent the output of diverse features. If so, by precomputing and saving the output of typical features, the output of future inputs can be read directly without computation.Towards answering this question, we propose LUT-NN, a novel system to empower DNN inference by table lookup. The system learns the typical features, namedcentroid, for each linear computation operator, and precompute the results for these centroids to save in lookup tables. During inference, the results of the closest centroids with the input features can be read directly from the table as the approximated output of this operator. The required computation cost for inference is significantly reduced. Fig.1shows an overview of LUT-NN using real data samples of a model. LUT-NN integrates two major techniques: (1)  differentiable centroid learning through backpropagation, and (2)  table-lookup inference execution.Maddnessinitiates the centroid learning for matrix multiplication (MM) , leveraging the technique of Product Quantization (PQ) . PQ is an effective vector quantization algorithm widely used for dataset compression. It compresses the dataset by clustering the vectors in the dataset first, and then learning the centroids to represent vectors in each cluster. By using PQ,Maddnesslearns the centroids from the training dataset for a single-operator image classifier, and precomputes the MM results for the centroids, as the approximation for future input matrices.However, as we will show in the paper, directly applyingMaddnessto the computation operators of a DNN model results in poor accuracy (80% accuracy drops for CIFAR-10) . A similar conclusion is also from McCarteret al..
We first analyze the issue and expose the main reason for the poor result isthe different optimization goal of centroid learning for PQ and DNNs.The goal of PQ is to learn the centroids that can minimize the total error/distance between centroids and vectors, while the goal of DNNs is to minimize the final loss function. The two goals have no direct relationship. Without considering the loss function, errors introduced by centroids inMaddnessare accumulated layer by layer, resulting in poor accuracy, making the resulting DNN model unreliable and undeployable in real-world applications.Figure 1.LUT-NN transforms model linear-computation layers to table lookup for inference.Therefore, we identify the key factor for DNN centroid learning is topass the model loss to each operator through backpropagation, and iteratively adjust the centroids by the gradients to minimize the model loss. However, the challenge is that currently it is not possible to learn centroids through backpropogation, since encoding a vector to the closest centroid e.g.,argminor hashing inMaddness, is not differentiable, and thus the gradients cannot be calculated.To solve this challenge, we propose the noveldifferentiable centroid learningtechnique for DNN. Through backpropagation, it adapts three different levels of approximation introduced by centroids to model inference by three methods, and minimize the accuracy loss. (1)  To enable gradient calculation and iteratively adjust centroids, LUT-NN usessoft-PQmethod. It uses the continuous approximation ofargmax, i.e.,softmax, for backward pass. The forward pass still usesargminfor model loss calculation, sinceargminwill be used for inference. Soft-PQ can adapt the approximation introduced by centroids to reduce accuracy loss. (2)  Since the use ofsoftmaxintroduces approximation toargmin, LUT-NN useslearned temperaturemethod, to learn the hyperparametertemperatureofsoftmaxfor each operator, to tradeoff accuracy loss and learning convergence. (3)  Since the lookup tables for precomputed results are the main cost for LUT-NN, scalar quantization is used to reduce the table size. To adapt the approximation introduced by this scalar quantization, LUT-NN usesquantization-aware training, which uses quantized tables in the forward pass and real-value tables in the backward pass, to reduce accuracy loss. By the techniques above, the centroids can be learned, which empowers table lookup to replace the diverse DNN linear computations while maintain similar-level of model accuracy.LUT-NN achieves a high FLOPs reduction (up to 16) . To run LUT-NN on resource-limited devices, we design and implement thetable lookup inference executionon commodity ARM and x86 CPUs. The challenge is that table lookup is a new inference paradigm, and no direct support from current systems and accelerators. Our design comprehensively considers different levels of hardware parallelism, memory access reduction, and dedicated hardware units, to fully utilize hardware resources. Specifically, we align the size of codebook and lookup table to the width of SIMD instructions for data-level parallelism, interleave the instructions ofargminreduction for instruction-level parallelism, reside the codebook in an inner loop for cache locality, and leverage the shuffle instruction for fast table access.We evaluate LUT-NN on a range of tasks, including image recognition tasks: CIFAR, GTSRB, SVHN, and ImageNet; speech recognition task: Google Speech Command; nature language processing (NLP)  tasks: GLUEand a regression task: UTKFaceage prediction, and cover both CNN and BERT models. For all these tasks, compared to directly applyingMaddness, LUT-NN can improve the accuracy by 66% to 92%. The accuracy difference between LUT-NN and the original DNN models ranges from +1.98% (Speech Command)  to -2.42% (ImageNet) .For inference efficiency, LUT-NN can outperform baselines in all dimensions.
The selected baselines include both the state-of-the-art hand-written inference system ONNX Runtime, and also the well-tuned TVMgenerated kernels as the performance upper-bound of linear computations. The detailed results include: FLOPs reduces; model (or disk)  size reduces; end-to-end inference latency reduces; memory usage reduces; and peak power reduces.
The real speedup is less than the FLOPs reduction due to the unfriendly support for table lookup in hardware. A customized table-lookup unit or accelerator could unleash the full potential of LUT-NN.LUT-NN takes the first trial towards this new table-lookup inference paradigm, which potentially much simplifies the inference system and hardware design and improves efficiency. To sum up, the key contributions of this paper are as follows. The code is open sourced athttps://github.com/lutnn.•LUT-NN is the first to empower DNN inference by table lookup to reduce inference cost.•It can achieve similar level of model accuracy by the novel differentiable centroid learning technique for DNNs through backpropagation.•It designs the new table lookup operation to enable this new inference paradigm on commodity CPUs.•We implement the learning and inference pipelines. Superior results are shown on all evaluated dimensions.
introduction
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
1.Introduction
2302.03213
2.Background and MotivationEach data sample in the training set can be assumed as a vector. This section will first introduce the concept of vector quantization and its efficient solution, PQ, and then show the poor results of directly applying PQ to DNN inference.2.1.Product Quantization: BackgroundQuantization has been well studied in information theory. It reduces the cardinality of a dataset by usingcentroidsto represent the data samples. The set of centroids is called acodebook. For vector quantization, the dataset is composed of-dimension vectors. The vector quantizer can encode each vector to a centroid in the codebook.To reduce the cost of vector quantization, PQ is proposed. The essence of it is to decompose the high-dimensional vector space into the Cartesian product of sub-vector spaces and then quantize these sub-vector spaces separately. As shown in Fig.2(a) , it splits the-dimension vector intodistinct-dimension sub-vectors () . The sub-vectors are quantized separately usingcodebooks. The quantization result of the vector is the concatenation of thecentroids.Centroid learningWe now formulize the PQ process. To quantize a-dimension vector, PQ needs to learn the centroids from a training datasetcomposed of vectors with the same distribution as. PQ first decomposes the vectors in the dataset intodistinct-dimension sub-vectors, notated as(marked in different colors in Fig.2) . To make optimal quantization, the centroid learning process is to find thecentroids(i.e., thecodebook)  forby-means, which can minimize the distance sum of each sub-vectorand its nearest centroid, as shown in Eq.1.(1) Sub-vector encodingWith the learned centroids, for an input vector, PQ can encode it as the concatenation of the nearest centroids for each sub-vector. The encoding function for a sub-vector is shown in Eq.2. By this vector decomposition method, the centroids can representdifferent vectors by onlymemory cost.(2) Hashing for acceleration with bigger errorLearning centroids is a NP-hard problem. Vanilla PQ uses-means to learn centroids and encode sub-vectors.-means satisfies Lloyd optimal conditionsand can get a local optimal quantization error. However, the-means encoding costs high to compute the Euclidean distance for each sub-vector with each centroid, as shown in Eq.2.To reduce the encoding cost, some works propose hashing methods to encode sub-vectors, butat the cost of higher quantization error. Hashing hashes a sub-vector to one of thebuckets. For example,maddnessselects a 4-level balanced binary regression tree of the hashing function family, with each leaf as a hash bucket. A sub-vector is encoded by traversing the tree from the root, and moving to left or right child if the value of certain indices is above or below a threshold.This paper will evaluate both distance-based encoding and hashing-based encoding methods.Figure 2.Product Quantization for AMM. Each color marks a sub-vector dataset.2.2.PQ for AMMPQ can be used for approximated matrix multiplication (AMM) . The essence is to approximate the matrix multiplication by the centroids’ multiplication.To formulize it, for matrix multiplication,andare the rows ofandrespectively. The centroid codebooks foris. For a layer of DNN,can be the input feature maps, andcan be the weights (note that convolution can be computed as matrix multiplication too) . Sinceis constant, the multiplication of all the centroids andcan be precomputed to construct a lookup table, as shown in Fig.2(b) . The table construction function foris shown in Eq.3.(3) The matrix multiplication can then be approximated by looking up and aggregating the results of the nearest centroids in the precompute table, formulated in Eq.4. Here considers thefunction withrepresentation for, i.e., the nearest centroid is marked as 1 and others as 0,.(4) 2.3.Motivation: Poor results of PQ for DNNSince DNN models are composed of MM, a direct thought is that can we replace MM in a DNN model by the PQ-based AMM. However, results show that directly applying it to a DNN model leads to poor accuracy.Fig.3shows the accuracy of using PQ-based AMM for ResNet-20 on CIFAR-10, as well as the Mean Square Error (MSE)  of the replaced model and the original model. We replace the MM from the last to the first layer by PQ-based AMM. Fig.3auses vanilla PQ with-means for encoding, while Fig.3busesmaddnesswith hashing for encoding.Results show the accuracy keeps dropping while more layers are replaced by AMM, because the error of the AMMs is accumulated. As expected, vanilla PQ shows better results thanmaddness, since-means introduces smaller quantization error than hashing.maddnesscan maintain accuracy when only the last layer is replaced.
This is consistent with themaddnesspaper, which only replaces the last layer, i.e., fully-connect, by AMM. However, if we replace the last two layers, the accuracy sharply drops by 30%, and finally ends in 10% accuracy. Vanilla PQ drops by 30% when the last six layers are replaced, and also ends in 10% accuracy.(a) Vanilla PQ-based AMM(b) maddnessAMMFigure 3.Model accuracy keeps decreasing because MSE keeps increasing, while more MMs (from the last to the first)  are replaced by PQ-based AMM, for ResNet20 on CIFAR-10.maddnessleads to more accuracy loss because using hashing for encoding.Reason for poor accuracyWe expose the key reason for the poor accuracy of PQ-based AMM is thatthe optimization goal of PQ and DNN learning is different. As shown in Eq.1, the goal of PQ is to minimize quantization error, i.e., learn the centroids to minimize the distance of each sub-vector and its nearest centroid.
On the other hand, the learning goal of DNN is to minimize the final loss function, through backpropagation to iteratively adjust the model parameters of each layer. Without considering the loss function, when more layers use AMM, the approximation error gets accumulated, as shown in Fig.3. To get better accuracy, it is necessary to learn the centroids by the DNN training process.However, the challenge is that PQ centroid learning and encoding functions shown in Eqs.1and2are not differentiable, and cannot use backpropagation to calculate gradient.
This paper thus proposes the soft-PQ technique, which empowers the centroid learning for a DNN by backpropagation and gradient descent.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
2.Background and Motivation
2302.03213
3.Differentiable Centroid Learning for DNNAs exposed above, the key factor to empower table lookup for DNNs is to learn the centroids of each layer through backpropagation to minimize the model loss. However, the main challenge is that theargminfunction in vanilla PQ, according to Eq.2, is not differentiable. We therefore propose the differentiable centroid learning for DNN. The learning integrates three methods to adapt three levels of approximation.Firstly, we propose soft-PQ method, which leverages the continuous and differentiable approximation toargmaxfunction, i.e.,softmax, for backpropagation. We approximate thecentroid result as the weighted sum of all centroid results (Sec.3.1) .
Prior work has usedsoftmaxfor differentiable PQ, but only for input embedding compression. Our work is the first to enable soft-PQ for end-to-end DNNs.Secondly, the use ofsoftmaxintroduces another level of approximation to usingargmin, which could also lead to reduced model accuracy. We propose a learnable temperature method to address this challenge to adjust the approximation error ofsoftmaxwithargminfor each layer.
The temperature of each layer’ssoftmaxcan be learned through backpropagation (Sec.3.2) .Thirdly, to reduce memory cost of the lookup tables, LUT-NN applies scalar quantization on the lookup tables. However, this also introduces a level of approximation. Similarly, we propose quantization-aware training to adapt this approximation during centroid learning (Sec.3.3) .Figure 4.Soft PQ learns centroids. Forward pass usesas the encoding function, and backward uses. Each color marks a sub-vector dataset using one codebook.is thecentroid for sub-vector.3.1.Backpropagation through soft-PQAs discussed in Sec.2.1, vanilla PQ employs-means to learn centroids from the dataset, and the encoding functionutilizesargminto encode a sub-vector as the nearest centroid, represented by aonehotvectoras shown in Fig.4. The sub-vector AMM result can be read directly from the lookup table by.However, to apply PQ to the entire DNN model and minimize the model loss, the centroids for each layer must be learned from backpropagation and gradient descent. We utilize the smoothargmaxfunction, i.e.,softmax, as the encoding function for backpropagation, as shown in Eq.5. Here,represents thetemperaturehyperparameter, which will be further discussed in Sec.3.2.(5) For a codebook withcentroids, thesoftmaxfunction takes a vector ofdistance results between the sub-vectorand each centroidas input.
It normalizes the input to a probability distribution that adds up to 1. According to the definition ofsoftmax, each probability is proportional to the exponent of the distance, i.e.,.
The closer the centroid is to the sub-vector, the higher the probability will be.
The encoding for a sub-vector is transformed from a deterministiconehotvector into a probability vector.
For example, as illustrated in Fig.4, the output of thesoftmaxencoding function is, where 0.64 is the probability of the nearest centroid. The sub-vector AMM result is then obtained as the dot product of the probability vector and the lookup table entries.Soft-PQ centroid learning:Usingsoftmax, the centroid learning process of our soft-PQ for the entire model is illustrated in Fig.4.
During the forward pass, the onehotargminfunction is utilized to calculate the model output and loss, as model inference will also useargminfor simplicity.
The backward pass utilizessoftmaxas the encoding function to calculate gradients, adjust centroids via gradient descent, and rebuild lookup tables with the updated centroids for the next training iteration.
Based on Eq.4, the sub-vector AMM in soft-PQ is formulated as Eq.6.(6) Here,represents thestop gradientoperator. It serves as an identity function during the forward pass to enable the use ofencoding inargmin. During the backward pass, it drops gradients inside it to enableto generate gradients viasoftmax.The initial value is critical for learning convergence and accuracy. Therefore, we use the centroids learned by-means from vanilla PQ to initialize the centroids and lookup tables.3.2.Learned temperatureThetemperaturehyperparameterofsoftmaxcontrols the approximation error ofsoftmaxtoargmax. Shown in Fig.5a, as, when,, i.e., the output probability distribution approaches uniform distribution. When,, i.e., the probability of the largestapproaching 1 and the others is 0.Therefore, there is a tradeoff between small and large temperature. For small temperature,softmaxis close toonehotargmax, but the training is difficult since the variance of gradients is large. For larger temperature, the approximation error is increased, but the variance of gradients is smaller.Works before normally setas a fixed value (mostly 1) , or anneal it from a large number to a small one during training, but never analyze how to set it reasonably. This is because currently for a DNN model,softmaxis only used by the output layer to produce class probability, or used by the input layer to produce symbol embedding. The approximation error barely impacts model accuracy. However, our soft-PQ employssoftmaxin every layer. The accumulated error can decrease accuracy without propersettings.We thus propose to learn the temperature for each layer, also during backpropagation while centroid learning. Fig.5bshows the learnedfor each layer for ResNet18. The value for each layer is different, and thus not practical to tune by hand.
According to the CIFAR10 accuracy experiments,
training with the learned temperature technique only spendsiterations of training with temperature setting to 1 to achieve 85% accuracy.
Detailed results will show in Sec.6.(a) (b) Figure 5.(a)  The output probability distribution ofat different temperature, for four centroids as an example.approaches uniform distribution, andapproaches. (b)  The learnedfor each layer of ResNet18.3.3.Scalar quantized lookup tableLookup tables are the main disk and memory cost. We reduce the table size by scalar quantization (e.g., FP32 to INT8) . We leverage the classic range-based linear quantization. The formula is, in whichis the real value,is the scaling factor,is the quantized value, andis the zero point.
We use symmetric quantization, sois forced to be 0, and the quantized range is. The scaling factoris calculated as the max absolute value in the table divided by half of the range, i.e.,.Quantized lookup tables introduce another level of approximation. Similar totemperature, we thus quantize the tables during centroid learning, to minimize the loss function. Inspired by Jacobet al., the backpropagation uses lookup tables in real values, so that they can be adjusted in small
amounts. The forward pass uses quantized lookup tables as in the inference to calculate the loss. Results show that by this learning method, the quantized lookup table has little impact on the model accuracy.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
3.Differentiable Centroid Learning for DNN
2302.03213
4.cost reduction of LUT-NNBefore introducing the inference design of LUT-NN, this section analyzes the theoretical FLOPs and model size reduction by LUT-NN.
The two primary factors of LUT-NN are the number of centroids and the length of sub-vectors, representing a tradeoff between cost and accuracy.
We conduct the analysis with these two factors.According to the output formula Eq.4of a PQ-based AMM in LUT-NN, the main cost is from the encoding function, which calculates the Euclidean distance of the sub-vector with each centroid. After that, the cost is from table lookup with the encoding result (i.e., index of the closest centroid) , and the result aggregation of sub-vectors. For file size, the major cost is from the lookup tables, which saves the dot product result of each centroid and the according sub-vectors in the weight matrix. The size of codebooks is relatively small, since the sub-vectors on the same column share one codebook.Therefore, we analyze the FLOPs of encoding, table lookup and aggregation, as well as the size of lookup tables as the cost of a LUT-NN AMM, to compare it with normal MM in Table1. Since convolution can be transformed to MM by im2col, its cost also follows these formulas. For a convolution,is the number of output channels,is the number of input channelsfilter size2, andis heightwidth.The number of centroidsand the sub-vector lengthare two hyperparameters of LUT-NN. They are tradeoffs between accuracy and cost (refer to Sec.6.3) .
The more centroidsand shorter sub-vectormay lead to higher accuracy, but will increase the cost of LUT-NN. Table2shows the calculated GFLOPs and model size by the formulas in Table1for different models, using typicalsettings in LUT-NN. These typical settings can achieve comparable accuracy with the original model, and also align with the SIMD width for high performance. Similar to other hyperparameters in DNN training,andcan be set by grid search, evolutionary search, or other popular searching methods.It is clear that LUT-NN can achieve both computation and model size saving. The FLOPs saving is becauseis normally smaller than. For example, the number of output channels i.e.,, for ResNet50 is normally 128, 256, or 512, so the FLOPs can be reduced bywhen. For ResNet20, the number of output channel is 16, 32, and 64, so the FLOPs is reduced bywhen.: input matrix: weight matrix: the length of a sub-vector: the number of centroids in a codebook forOursMMFLOPsSizeTable 1.FLOPs and disk size of a LUT-NN AMM compared to MM (FP32) .ModelGFLOPsoriginal(8, 9) (16, 9) ResNet18 (CIFAR10) 0.5550.0980.132SENet18 (CIFAR10) 0.5560.0980.131VGG11 (CIFAR10) 0.6060.0850.102ResNet181.8140.4120.515SENet181.8140.4120.515VGG117.6091.1321.334original(16, 32) (16, 16) BERT2.7590.1690.254ModelDisk Size (MB) original(8, 9) (16, 9) ResNet18 (CIFAR10) 42.5912.5523.13SENet18 (CIFAR10) 42.9210.9421.53VGG11 (CIFAR10) 35.188.4516.88ResNet1844.5512.5723.16SENet1844.8812.9123.49VGG1137.1210.3918.82original(16, 32) (16, 16) BERT162.2623.0543.30Table 2.Theoretical GFLOPs and model size of typical (,) , calculated by formulas in Table1.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
4.cost reduction of LUT-NN
2302.03213
5.Table lookup Inference DesignIn this section, we present the design and optimization of the inference system that supports LUT-NN.
Fig.6depicts the LUT-NN model inference architecture, comprising the Closest Centroid Search stage and the Table Read and Accumulation stage.
In the Closest Centroid Search stage, LUT-NN first computes the distance between input tensors and centroids and searches the nearest centroids for each input tensor.
In the Table Read and Accumulation stage, LUT-NN reads the pre-computed results from the lookup table and accumulate results.
LUT-NN inference design fully takes advantage of the features of current hardware architectures to optimize the design and implementation of Table Lookup.
These optimizations improve inference by utilizing memory hierarchy, utilizing shuffle instructions, and utilizing mixed-precision accumulation instructions.Figure 6.Table Lookup Inference Design5.1.Closest centroid searchThe centroid search takes the most computation costs in LUT-NN, and efficient centroid search is critical for performance efficiency compared to conventional computational methods.
TheClosest Centroid Searchstage first calculates the distance between input tensors and centroids.
We can represent this calculation as matrix multiplications of input tensors and codebook matrices.
Then, it searches for the nearest centroids for each input sub-vector.However, the design of centroid search in LUT-NN presents challenges that involve leveraging the features of existing hardware architectures.
Firstly, LUT-NN distance computation is irregular-shaped (tall-and-skinny)  and difficult to optimize using BLAS libraries.
For example, the XNNPACK library achieves onlyGFLOP/s on Pixel 6 for the second layer of LUT-NN based ResNet18, which accounts for onlyof peak performance.
The height of the input tensor ()  is usually much larger than the length of the sub-vector ()  and the number of centroids ()  on each codebook.
Therefore, the operation intensity of the distance calculation can be approximated by.
Since the length of the sub-vector and the number of centroids are small in LUT-NN, the operation intensityis also small.
The distance computation becomes a memory-intensive matrix multiplication.Therefore, we focus primarily on optimizing memory access for centroid distance computations (❶ in Fig.6)  to leverage memory hierarchy in current hardware.
To optimize memory access for centroid distance computations (❶ in Fig.6) , we reduce memory access overhead by keeping frequently accessed data in registers and caches as much as possible.
So we design acentroid-stationary computationscheme to reside centroid matrices in registers and reorder centroid matrix loads in the inner loop to keep them in the cache as long as possible.
The centroid-stationary computation keepscentroids in the cache for each codebook, which only requires reading these centroids once from DRAM.
Consequently, it only requires reading aninput tensor from DRAM once, reducing memory bandwidth costs and improving performance.Second, after computing centroid distances, the Closest Centroid Search stage must identify the nearest centroid for each input sub-vector and generate the centroid index (❷ in Fig.6) .
It can be represented by anargminfunction, which finds an index with the shortest distance.
However, searching for the nearest centroid for each input sub-vector is a data-dependent operation.
To find the closest one, we must compare each distance sequentially, which is RAW (Read After Write)  dependent and hard to be parallelized on CPUs.We proposeintra-codebook parallelismto optimize the Closest Centroid Search stage.
Intra-codebook parallelism searches the nearest centroid for the input sub-vector on a codebook in parallel.
We slice a codebook into multiple sub-codebooks and compare each distance between sub-vector and centroids in each sub-codebook.
Then, we interleave each sub-codebook’s execution, and avoid data dependency in closest centroid search.
We merge the compared distances by reduction and find the index corresponding to the closest centroid.
It leverages instruction-level parallelism to improve hardware utilization and performance.5.2.Table read and accumulationLUT-NN obtains the indices of the nearest centroids for the input sub-vectors after closest centroid search, and leveragesTable Read and Accumulationstage to compute the final results.
This stage first reads the pre-computed results from the corresponding lookup table through indices (❸ in Fig.6)  and completes the computation by accumulation operation (❹ in Fig.6) .
For example, convolution operators directly read out filter’s outputs from lookup tables and accumulate each input channel’s result for output channels.However, Table Read and Accumulation stage introduces additional overhead in inference.
We enhance inference efficiency by skillfully utilizing widely available and supported instructions in commercial hardware.
First, table read is difficult to parallelize and introduces additional indirect memory accesses, which exaggerate the memory overhead on lookup tables.
Since we quantize lookup table into INT8 in Sec.3.3, we leverage widely supported SIMDshuffleinstructions (x86:pshfband ARM:tbl)  to achieve parallel and efficient table read.
We demonstrate the implementation of table read using shuffle instructions in Fig.6.
The shuffle instructionpermutes each byte of a vector based on an index vector and stores the shuffled bytes in the result vector register in each clock cycle.
On 128-bit wide SIMD, a vectorized table read instruction handles 16 sub-vector lookups ()  onresults ()  simultaneously, greatly simplifying table read and reducing overheads.Second, the accumulation still has comparable computation costs to the entire process.
For example, when a codebook handlesindex lookups on alookup table, it coststable reads ()  andaccumulation adds.
Therefore, the vectorized lookup table leaves accumulation operations as the performance bottleneck of table reads.
The number of parallel processing units within a SIMD instruction is called SIMD lanes.
We typically set the number of centroids to 16 (K=16)  to maximize the utilization of all SIMD instruction lanes.
Since a higher number of SIMD lanes leads to higher throughput on the same width of SIMD instruction (e.g., the vectorized INT16 add instruction has the half throughput of INT8 on a 128-bit SIMD) ,
we maximize accumulation throughput by mixed precision accumulation. It first accumulates results in INT16 to utilize more SIMD lanes and then gathers INT16 to INT32 to avoid overflow.5.3.Utilization of memory hierarchyAs LUT-NN is composed of many memory accesses, we explain more details for its memory hierachy usage.The frequent-accessed codebook (for centroids)  is kept in the inner loop during distance calculation, so that the hardware cache can leaverage the locality to keep the codebook in cache.
The size of a codebook for one layer is normally small, in KB level (calculated by sub-vector lengthnumber of centroidsnumber of codebooksnumber of bytes) . For example, the largest codebook in our tested models is 288 KB. It can be held in L2 cache on many mobile devices.The lookup table (for precomputed results)  is larger, in MB level. It can be held in L3 cache. Even if it is larger than L3 cache, the memory access overhead could still be hidden in our implementation. LUT-NN uses Shuffle instructions to parallelize table lookup, turning the pattern of reading index numbers and lookup tables from random to sequential. The memory access is thus predictable and easy to prefetch. Furthermore, we interleave computation and memory access instructions within the kernel, which can overlap computation and memory access time.DatasetModelCentroidLearning RateTemperatureLearning RateWeightDecayBatchSize# of EpochsOptimizerLRSchedulerCIFAR10ResNet18SENet18VGG110256200AdamCosineAnnealingGRSRBSpeech CommandsUTKFace150SVHN200ImageNet150GLUEBERT323AdamWConstantTable 3.Soft-PQ training settings for centroid and temperature learning.
We choose the best fine-tuning learning rate among {,,,} based on the accuracy of the validation set for BERT, following.ModelResNet18SENet18VGG11DatasetLUT-NNMADDNESSbaselineLUT-NNMADDNESSbaselineLUT-NNMADDNESSbaselineCIFAR1094.4010.0195.2694.4810.6595.4793.8922.8795.04GTSRB98.734.5398.8098.365.6898.8498.555.7099.22SpeechCommands93.701.4991.7293.041.4994.3693.381.4993.11SVHN96.0020.6896.6796.2220.1296.6096.2329.9796.62UTKFace4.9110.515.574.7411.025.465.6924.575.85ImageNet67.380.1069.7668.210.1770.6368.040.1668.33Table 4.LUT-NN achieves comparable accuracy
with the original models, significantly outperformingMaddness. UTKFace is the lower the better. Others are the higher the better.Dataset TaskSingleSentenceSimilarity andParaphraseNatural LanguageInferenceSST-2QQPQNLIRTEAverageTraining Dataset Size67k364k105k2.5kTest Dataset Size1.8k391k5.4k3kBERT base (%) 93.571.290.566.480.4LUT-NN (%) 92.469.687.464.778.5Table 5.LUT-NN achieves comparable accuracy to BERT-base on NLP tasks selected from GLUE.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
5.Table lookup Inference Design
2302.03213
6.Evaluation6.1.Experiment methodologies and settingsDataset and ModelsOur experiment tests LUT-NN’s effectiveness on image recognition, speech recognition, and NLP tasks.
For image recognition and speech recognition, we evaluate VGG, ResNet, and SENet family models on different datasets, including CIFAR-10, GTSRB (German Traffic Sign Recognition Benchmark) , Google Speech Command, SVHN (Street View House Numbers) , UTKFace, and the large-scale complex ImageNetdataset.
The CIFAR-10 dataset consists ofimages inclasses, withfor training andfor validation.
The GTSRB dataset hasclasses of traffic signs, withtraining images andtest images.
Google Speech Command is an audio dataset containingrecordings ofwords, designed for keyword spotting on edge devices.
SVHN is a dataset includingreal-world images of house numbers forclasses.
UTKFace dataset consists offace images labeled by age, gender, and ethnicity. We use the age prediction task to test the regression ability of LUT-NN. We will report Mean Average Error (MAE)  for this task.
ImageNet dataset hasimages for training andfor validation.On the large ImageNet dataset, ResNet18 and SENet18 models use 7x7 convolution for the first layer and 3x3 max-pooling, while other datasets use 3x3 convolution for the first layer and no max-pooling.
The final three dense layers of the original VGG models are replaced by an average pooling layer and a dense layer to reduce model size for mobile deployment, following the general practice for VGG deployment. Except for ImageNet, the first max pooling layer is removed for VGG. In evaluation results, the models for ImageNet use the model names, and the models for other datasets are marked as ”Model (CIFAR10) ” to differentiate.To evaluate LUT-NN for NLP tasks, we used BERT models and the tasks from GLUE benchmark.Soft-PQ training settingTable3lists the detailed training hyperparameter settings.
Except for the initial learning rate, we followed standard practices for all other training procedures.
As the learned temperature requires a larger learning rate to converge quickly to a sub-optimal solution, we use different learning rates for the centroid and temperature learning.
Prior to soft-PQ training, we initialize centroids using-means clustering.
Specifically, we apply the original model inference to a randomly sampled sub-dataset (consisting of 1024 training samples)  and collect each operator’s inputs.
We then utilize the-means algorithm to cluster each operator’s inputs and obtain the initial centroids.LUT-NN system settingsIn order to balance accuracy and cost, we can specify various settings for LUT-NN, including the number of centroids in a codebook () , the length of the sub-vector ()  (as shown in Table1) , and the operators to be replaced by table lookup.
The default setting forandis to align with both the feature size and the length of SIMD instructions.
Specifically, we setforconvolution,forconvolution, We will also evaluate other settings in Sec.6.3ablation study.For operators to replace, we replace all convolution operators for CNN models by table lookup except the first one. This is because as explained by Zhouet al., the layer interacted with the model input is very sensitive to accuracy. We also observe this phenomenon. Replacing the first layer by table-lookup can lead to obvious accuracy drop. This issue is even more serious for BERT. Replacing the first two layers results in 80% accuracy loss (refer to Fig.13) .
Therefore, we do not replace these layers by table lookup. For BERT, other than explicitly stated, we replace the fully connected operators of the last 6 layers. The results with different replaced layers for BERT will be shown in Section6.3ablation study.Evaluation platformsLUT-NN is implemented on ARM Neon and Intel SIMD ISA (Instruction Set Architecture) , and can run in a single or multi threads. We evaluate LUT-NN on two mobile devices Google Pixel 4 and 6, which are equipped with Cortex-A76 (GHz)  and Cortex-X1 (GHz) , respectively.
In addition, we evaluate LUT-NN on a desktop CPU (Intel Core i7-4790)  running atGHz and a server CPU (Xeon Silver 4210)  running atGHz.Comparison baselinesThe accuracy baselines are (1)  original models and (2) Maddness, respectively. The inference performance baselines are ONNX Runtime (ORT)  v1.12.1and TVM v0.9.0, as the SOTA hand-written performance, and the auto-tuned performance upper bound for linear computation opeartors. We tune the TVM baselines using AutoScheduler and AutoTVM initerations.6.2.Accuracy, latency, memory and powerAccuracyTable4and5summarize the accuracy comparison. Remarkably, to compare with directly applyingmaddness, LUT-NN can improve accuracy by 66% to 92%, achieving similar-level of accuracy with the original models. LUT-NN behaves particularly well on GTSRB (difference) , SVHN (difference) , CIFAR-10 (difference) , Speech Command (difference)  , and UTKFace (LUT-NN is even better) . For the Speech Command task, LUT-NN can even improve accuracy byfor ResNet18.
Also for the UTKFace dataset, LUT-NN can even reduce MAE by at most 13.2%.
Expectedly, the accuracy difference for the more complex tasks, ImageNet and NLP, is a bit larger compared to the tasks above, with~difference for ImageNet, and an averageddifference for NLP tasks.Operator speedupFig.7shows the operator speedup selected from three models. The operator speedup for VGG11 ranges fromtoon ARM CPU, and fromtoon x86 CPU. The best speedup on BERT kernels are(ARM CPU)  and(x86 CPU) .BERT operators achieve higher speedups due to largeror, and longer sub-vector. According to Table1, the FLOPs of our proposed LUT-NN is, and the FLOPs of a normal MM is.
The FLOPs reduction by LUT-NN can be calculated by. Therefore, the reduction is particularly large whenand. The kernel speedup gradually improves as the layer index increases for CNN models.
The reason is that the number of output channels increases fromto,, andas the layer index increases.
The increased number of output channelscontributes to the speedup in these layers.Model speedupFig.8shows the end-to-end model latency.
LUT-NN achieves~speedup on ResNet18, SENet18 and VGG11 compared to TVM and ORT.
The best speedup isin VGG11 (for CIFAR10)  on Pixel6 ARM CPU over ONNX Runtime.
The BERT model has higher throughput, and the speedups areandon the ARM CPU and the x86 CPU, respectively.
Compared to CNN models () , the BERT model has a larger input tensor ()  to acquire better performance gains.
It also has a longer sub-vector length ()  compared to CNN models () .Multi-thread speedupTo evaluate the scaling capability of LUT-NN, Fig.9shows the speedup with the thread increases, i.e., one to four
on x86 CPUs and one to two on the Pixel 6 ARM CPU. Note that we evaluate the scalability on the big cores (cores with the highest frequency)  of ARM CPUs, to avoid the interference of asymmetric big and little cores. Pixel 6 has two big cores. Since Pixel 4 only has one big core, we skip it in this evaluation.
 The results show that LUT-NN achieves better scalability compared to the baselines.
It can reach 2.24~2.49speedup as the number of threads increases to 4.
Meanwhile, LUT-NN achieves a 2.17~2.82speedup over other baselines under the same number of threads.Model memory footprintFig.10compares the memory footprint of LUT-NN with baselines.
The results show that LUT-NN can save~memory for CNN models and~for BERT. For these CNN models, the sub-vector length isorfor different operators, while the length is set tofor BERT.
Because longer sub-vectors result in a higher compression ratio on feature maps, BERT achieves lower memory costs than other models.PowerWe evaluate the power consumption of LUT-NN on Pixel 4.
The experiment results in Table6demonstrate that LUT-NN reduces power consumption by~compared to TVM, which reveal that LUT-NN is an energy-efficient algorithm and system design.Figure 7.LUT-NN operator speedup over ORT and TVM on Pixel 6 and the x86 server CPU.Figure 8.End-to-end latency comparison between LUT-NN, ORT, and TVM (lower is better) .Figure 9.Normalized multithread speedup (over TVM one thread)  of LUT-NN, ORT, and TVM (higher is better) .Figure 10.Model memory consumption of LUT-NN, ORT, and TVM on the x86 server CPU.6.3.Ablation StudyWe evaluate the effectiveness of the learned temperature and hyperparameters in the ablation study.
The hyperparameters include the number of centroids, the length of sub-vector, and the number of replaced linear computation operators.Figure 11.Learning curves of LUT-NN ResNet18 on CIFAR10 with temperature tuning methods.
“Annealing Temperature” (Orange)  refers to manually annealing temperature from 1 to 1e-1.
Our learned temperature technique reaches 94.4% accuracy, higher than the annealing temperature (91.55%)  and setting temperature as 1 (89.85%) .Learned temperatureIn Sec.3.2, we use gradient descent to learn the temperature.
To evaluate the effectiveness of our method, we compare three temperature tuning strategies together:
learned temperature, statically setting the temperature as, and annealing temperature fromto.
We compare the training curves in Fig.11.
The figure suggests that our proposed learned temperature reaches the highest accuracy ()  and outperforms the statically setting ()  and the annealing temperature () .
In addition, the learned temperature has a faster convergence speed.Impact of centroid number and sub-vector lengthThe centroid number and sub-vector length do affect not only the inference throughput but also the model accuracy.
We present the ablation study for the impact of these two hyperparameters.
We follow the same experiment setting as Sec.6.1on ResNet18.Figs.12collect accuracy and FLOPs for the variant of centroid numbers and sub-vector lengths in ResNet18.
For this model, the sub-vector length significantly affects the model accuracy and worsens as the sub-vector length increases.
Each codebook has to handle higher dimensions as the sub-vector length grows, which cannot be accurately classified and harms the model accuracy.The centroid number also affects accuracy and performance.
As the centroid number increases, each codebook can classify sub-vector in more fine-grained granularity, and it improves the accuracy of ResNet18.
To balance the model accuracy with performance, we setandfor better accuracy and fewer GFLOPs than the vanilla ResNet18.The optimal hyperparameter setting for DNNs is a long-standing research topic. Some neural network search techniques, such as weight sharingand search space search, could apply to LUT-NN to reduce the searching cost.Figure 12.The scaling of centroid number and sub-vector length on ResNet18. x-axis is logarithmic.The number of replaced operatorsAs stated, to replace the front layers of BERT by table lookup can greatly impact accuracy, and we replace the last six layers in our default settings. Fig.13shows the accuracy impact by replacing more layers.
We gradually replace the operators of more layers, from the last to the front layer, for BERT on the Semantic Textual Similarity Benchmark (STS-B)  task.
We can see the accuracy drops obviously for the front three layers.Figure 13.The accuracy of LUT-NN based BERT with respect to the number of layers to replace.Speedup breakdownIn Sec.5, we propose four inference optimization techniques, which include: ❶ memory optimization on centroid distance computations, ❷ intra-codebook parallelism, ❸ shuffle instructions table lookup, and ❹ mix-precision accumulation in Fig.6.
To better understand performance gains, we evaluate the execution of a convolution operator (, the second layer of ResNet18)  on Pixel 6 when different techniques are enabled.
The results show that ❸ ()  saves the most execution time, followed by methods ❶ ()  and ❷ () . Finally, method ❹ delivers a minor improvement () .
The results show that our inference design and optimization can take advantage of ARM/x86 ISA’s special instructions, which outperform conventional operators.Scalar quantization level for lookup tableAs explained in Sec.3.3, LUT-NN leverages scalar quantization to reduce the table size. In this subsection, we evaluate the accuracy impact of different quantization levels by using ResNet18 for the CIFAR10 dataset as an example.
The accuracy of LUT-NN is 94.44% with FP32 lookup table, 94.40% with INT8 lookup table, and 94.27% with INT4 lookup table. The results show that our quantization-aware training can achieve similar accuracy by using lower bits compared to FP32. We select INT8 in our current implementation as its the tradeoff of accuracy and speedup. INT4, on the other hand, is not supported by the hardware SIMD units directly. The exploration of lower-bit lookup table could be a potential future work.ModelLUT-NN v.s. TVM Avg. power (W) BERT2.6/3.7ResNet182.6/3.0ResNet18 (CIFAR) 2.6/3.3SENET182.6/2.9SENET18 (CIFAR) 2.8/3.2VGG112.3/2.9VGG11 (CIFAR) 2.7/3.3Table 6.Power (deducted SoC idle power)  on Pixel 4.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
6.Evaluation
2302.03213
7.Related WorkApproximated matrix multiplicationTraditional approximated matrix multiplication techniques focus on minimizing the difference between the ground truth and the approximated output.
For example, Magen et al.use random projection to map two matrices to a lower-dimensional subspace, and then perform accurate multiplication on the mapped matrices.Maddnessuses product quantization to learn the most typical features (also known as centroids)  in the input distribution.Maddnessthen stores the products of centroids and the weight matrix in advance in a lookup table, so that the product of any input matrix and weight matrix can be approximated by the result stored in the table.
To conclude, traditional methods will suffer from severe accuracy loss when used in neural network inference if the approximation is too harsh.
Our method can avoid this defect by integrating itself into the end-to-end training process.Also, previously, some attempts have been made to approximate the weights of NN with low-rank tensor expansion.
Specifically, LCNNis also a kind of low-rank approximation.
With LCNN, a dictionary is trained and the NN weights can be represented by linear combinations of the dictionary items.
Afterward, LCNN transforms a vanilla convolution into lookup operations in the dictionary.
Our work is different from LCNN in that our work learns centroids for each layer’s input feature map but not the weight.Product quantization for DNNProduct Quantizationis a popular and successful method for large-scale approximated nearest neighbor search.
In recent years, researchers have integrated product quantization into neural networks for various purposes.
For example,incorporated product quantization as a layer in a convolutional neural network to obtain a compact and discriminative image representation.utilized Product Quantization to compress the embedding layer in NLP models.compressed the weight matrix for neural networks with Product Quantization.
However, our approach differs from the above methods that our purpose is to achieve end-to-end neural network inference acceleration.Scalar quantizationScalar quantization methods aim to reduce scalar bits in neural networks, e.g., from float representation to INT8 representation.
They can both compress the neural network bits and accelerate the inference.
Jacob et al.proposed an INT8 quantization method with both a training scheme and an efficient kernel implementation.
Other researchersfurther proved that 1/2/4-bit quantization is sufficient for image classification.
More recently, Wang et al.proposes a new approach to quantization using trainable lookup tables which can fit the distributions in different layers and have small additional computational costs.
Although we leverage scalar quantization to compress the size of the lookup table (Sec.3.3) , the main idea of our work is different from these quantization methods.
Our work focuses on learning the most typical features of each NN layer and computing the results of these features in advance for future input tensors, in order to both compress NN weights and accelerate NN inference.Inference result reuseThere are a range of works that exploit the duplicated inputs, to reuse their inference results and
skip further model inference.
Cachierreuses results among nearby users. Potluckreuses results across applications. Euphratereuses results of temporal pixel motion for continuous vision. FoggyCachereuses results across devices. DeepCache reuses results of similar regions in recent frames of consecutive videos. MCDNNreuses results across applications. Semantic Memoryreuses results from some layers for continuous vision.
These works rely on specific application inputs.
Our work, by comparison, optimizes the model itself to reduce size and FLOPS.Lookup table for DNNDifferent from our work that learns and stores the centroid results in the table, there are works that store other results in the tables. LogicNetsmaps artificial neurons to truth tables in FPGA. It also compresses the model by available pruning and scalar quantization techniques. Ramanathan et al.store the results of bitline computing in Processing in Cache hardware in the table, to accelerate DNN. These works do not target neural network optimization or training techniques like LUT-NN.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
7.Related Work
2302.03213
8.Future opportunity discussionHardware implicationPotentially, LUT-NN can greatly reduce FLOPs, e.g.,for BERT. However, current hardware restrains LUT-NN from showing its full potential. The reason is that current hardware is optimized for linear computation, but not table lookup. Compared to direct Multiply-Add support for MM in hardware,
LUT-NN has to runargminpartially sequentially to return the index of the nearest centroid, and then lookup in the table followed by aggregation. Besides, the SIMD width limits the options of the number of centroids. These are obviously inefficient.Even on this unfriendly hardware, LUT-NN has shown the benefits of table lookup on all dimensions, particularly for latency, power, and memory reduction. An accelerator or function unit supporting the first-class parallel table-lookup pipeline could even more significantly reduce the inference cost. LUT-NN opens new opportunity for hardware design.Replace more operator typesCurrently LUT-NN replaces linear computation operators by table lookup, including MM, convolution, and fully connected. The scaled dot-product attention (of total latency)  in attention layers have no weights, and cannot precompute the results with centroids to save in tables, so we do not replace it by lookup table in this work. A possible way to replace it by lookup table is to precompute and save the production results of centroids. In addition, how to replace the non-linear operators, such as the activation functions, by LUT-NN can be future work.Learning for hashingAs introduced in Sec.2.1, hashing can be used to avoid the Euclidean distance calculation of-means encoding, but at the cost of higher quantization error (refer to Fig.3b) . We evaluate the potential of hashing for DNN inference. Since hashing is not differentiable, LUT-NN uses hashing after the centroids are learned. Our current results show that to achieve similar model accuracy as Euclidean distance for encoding, we have to use a 12-level decision tree for hashing. Compared to distance calculation, it can further reduce FLOPs by 30% to. It could be possible to also integrate hashing into the backpropagation to learn the hashing functions. This could reduce the depth of the tree and further reduce the encoding cost.On current commodity hardware without direct hashing support, hashing costs even higher than Euclidean distance calculation, since the tree traversing is sequential without SIMD support. Therefore, we still use Euclidean distance for encoding in LUT-NN now. Future hardware can also integrate hashing units for further speedup.LUT-NN on GPUs and FPGAsNVIDIA GPUs provide the efficient shuffle instruction, i.e., “prmt”, achieving Tera-OPs throughput. Similar as the LUT-NN implementation on the CPUs, which heavily depends on the shuffle instructions, this GPU shuffle instruction can be leveraged for parallelly looking up the approximated computation results in the tables.
FPGAs contain a vast array of lookup tables implemented through reconfigurable memory, which LUT-NN can utilize for the storage and retrieval of codebooks. Additionally, FPGAs are equipped with Block RAM, a primary memory resource that enables efficient data storage. It can be employed for LUT-NN’s lookup tables.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
8.Future opportunity discussion
2302.03213
9.ConclusionThis paper takes the first step towards DNN inference by table lookup. A new paradigm potentially brings significant benefits to the DNN inference ecosystem, simplifying the inference software and hardware design and decoupling with the DNN algorithm updates. By the centroid learning technique for DNN, LUT-NN achieves comparable accuracy for complex tasks with much less resource cost.
However, LUT-NN still has space to be improved, such as the learning technique to improve accuracy and hardware support for table lookup, which calls for future efforts from the community.
body
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
9.Conclusion
2302.03213
10.AcknowledgementThe authors thank the anonymous shepherd and reviewers for their valuable comments. The work of Yunxin Liu was supported by the National Key R&D Program of China (No. 2022YFF0604501)  and Xiaomi Foundation. The work of Deng Cai was supported by the National Nature Science Foundation of China (Grant Nos: 62273302, 62036009, 61936006) .
conclusion
LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup
10.Acknowledgement