<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.04093] Scaling and evaluating sparse autoencoders</title><meta property="og:description" content="Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Since language models learn many c…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Scaling and evaluating sparse autoencoders">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Scaling and evaluating sparse autoencoders">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.04093">

<!--Generated on Fri Jul  5 19:17:37 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Scaling and evaluating sparse autoencoders</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Leo Gao 
<br class="ltx_break">&amp;Tom Dupré la Tour 
<br class="ltx_break">&amp;Henk Tillman<sup id="id3.3.id1" class="ltx_sup">†</sup> 
<br class="ltx_break">Gabriel Goh 
<br class="ltx_break">&amp;Rajan Troll 
<br class="ltx_break">&amp;Alec Radford 
<br class="ltx_break">Ilya Sutskever 
<br class="ltx_break">&amp;Jan Leike 
<br class="ltx_break">&amp;Jeffrey Wu<sup id="id4.4.id2" class="ltx_sup">†</sup> 
<br class="ltx_break">OpenAI
</span><span class="ltx_author_notes">Primary Contributor. Correspondence to lg@openai.com. Core Research Contributor. This project was conducted by the Superalignment Interpretability team. Author contributions statement in <a href="#A9" title="Appendix I Contributions ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix I</span></a>.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id5.id1" class="ltx_p">Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Since language models learn many concepts, autoencoders need to be very large to recover all relevant features. However, studying the properties of autoencoder scaling is difficult due to the need to balance reconstruction and sparsity objectives and the presence of dead latents. We propose using k-sparse autoencoders <cite class="ltx_cite ltx_citemacro_citep">(Makhzani and Frey, <a href="#bib.bib36" title="" class="ltx_ref">2013</a>)</cite> to directly control sparsity, simplifying tuning and improving the reconstruction-sparsity frontier. Additionally, we find modifications that result in few dead latents, even at the largest scales we tried. Using these techniques, we find clean scaling laws with respect to autoencoder size and sparsity. We also introduce several new metrics for evaluating feature quality based on the recovery of hypothesized features, the explainability of activation patterns, and the sparsity of downstream effects. These metrics all generally improve with autoencoder size. To demonstrate the scalability of our approach, we train a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens. We release <a target="_blank" href="https://github.com/openai/sparse_autoencoder" title="" class="ltx_ref ltx_href">code and autoencoders for open-source models</a>, as well as a <a target="_blank" href="https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html" title="" class="ltx_ref ltx_href">visualizer</a>.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Our open source code can be found at <a target="_blank" href="https://github.com/openai/sparse_autoencoder" title="" class="ltx_ref ltx_href">https://github.com/openai/sparse_autoencoder</a> and our visualizer is hosted at <a target="_blank" href="https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html" title="" class="ltx_ref ltx_href">https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html</a></span></span></span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Sparse autoencoders (SAEs) have shown great promise for finding features <cite class="ltx_cite ltx_citemacro_citep">(Cunningham et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>; Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>; Templeton et al., <a href="#bib.bib64" title="" class="ltx_ref">2024</a>; Goh, <a href="#bib.bib16" title="" class="ltx_ref">2016</a>)</cite> and circuits <cite class="ltx_cite ltx_citemacro_citep">(Marks et al., <a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite> in language models. Unfortunately, they are difficult to train due to their extreme sparsity, so prior work has primarily focused on training relatively small sparse autoencoders on small language models.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">We develop a state-of-the-art methodology to reliably train extremely wide and sparse autoencoders with very few dead latents on the activations of any language model. We systematically study the scaling laws with respect to sparsity, autoencoder size, and language model size. To demonstrate that our methodology can scale reliably, we train a 16 million latent autoencoder on GPT-4 <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> residual stream activations.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Because improving reconstruction and sparsity is not the ultimate objective of sparse autoencoders, we also explore better methods for quantifying autoencoder quality. We study quantities corresponding to: whether certain hypothesized features were recovered, whether downstream effects are sparse, and whether features can be explained with both high precision and recall.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Our contributions:</p>
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">In <a href="#S2" title="2 Methods ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 2</span></a>, we describe a state-of-the-art recipe for training sparse autoencoders.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">In <a href="#S3" title="3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3</span></a>, we demonstrate clean scaling laws and scale to large numbers of latents.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">In <a href="#S4" title="4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4</span></a>, we introduce metrics of latent quality and find larger sparse autoencoders are generally better according to these metrics.</p>
</div>
</li>
</ol>
<p id="S1.p4.2" class="ltx_p">We also release <a target="_blank" href="https://github.com/openai/sparse_autoencoder" title="" class="ltx_ref ltx_href">code</a>, a full suite of GPT-2 small autoencoders,
and <a target="_blank" href="https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html" title="" class="ltx_ref ltx_href">a feature visualizer</a> for GPT-2 small autoencoders and the 16 million latent GPT-4 autoencoder.</p>
</div>
<figure id="S1.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="162" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/x2.png" id="S1.F1.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="221" height="159" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.6.3.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">
Scaling laws for TopK autoencoders trained on GPT-4 activations.
(Left) Optimal loss for a fixed compute budget.
(Right) Joint scaling law of loss at convergence with fixed number of total latents <math id="S1.F1.3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S1.F1.3.1.m1.1b"><mi id="S1.F1.3.1.m1.1.1" xref="S1.F1.3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S1.F1.3.1.m1.1c"><ci id="S1.F1.3.1.m1.1.1.cmml" xref="S1.F1.3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.3.1.m1.1d">n</annotation></semantics></math> and fixed sparsity (number of active latents) <math id="S1.F1.4.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.F1.4.2.m2.1b"><mi id="S1.F1.4.2.m2.1.1" xref="S1.F1.4.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.F1.4.2.m2.1c"><ci id="S1.F1.4.2.m2.1.1.cmml" xref="S1.F1.4.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.4.2.m2.1d">k</annotation></semantics></math>. Details in <a href="#S3" title="3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3</span></a>.
</span></figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Methods</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Setup</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.3" class="ltx_p"><span id="S2.SS1.p1.3.1" class="ltx_text ltx_font_bold">Inputs:</span> We train autoencoders on the residual streams of both GPT-2 small <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib51" title="" class="ltx_ref">2019</a>)</cite> and models from
a series of increasing sized models sharing GPT-4 architecture and training setup,
including GPT-4 itself <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>All presented results either have qualitatively similar results on both GPT-2 small and GPT-4 models, or were only ran on one model class.</span></span></span>.
We choose a layer near the end of the network, which should contain many features without being specialized for next-token predictions (see <a href="#A6.SS1" title="F.1 Impact of different locations ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section F.1</span></a> for more discussion). Specifically, we use a layer <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="\frac{5}{6}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mfrac id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mn id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">5</mn><mn id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml">6</mn></mfrac><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><divide id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"></divide><cn type="integer" id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">5</cn><cn type="integer" id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">\frac{5}{6}</annotation></semantics></math> of the way into the network for GPT-4 series models, and we use layer 8 (<math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="\frac{3}{4}" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mfrac id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml"><mn id="S2.SS1.p1.2.m2.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.cmml">3</mn><mn id="S2.SS1.p1.2.m2.1.1.3" xref="S2.SS1.p1.2.m2.1.1.3.cmml">4</mn></mfrac><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"><divide id="S2.SS1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"></divide><cn type="integer" id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.2">3</cn><cn type="integer" id="S2.SS1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">\frac{3}{4}</annotation></semantics></math> of the way) for GPT-2 small.
We use a context length of 64 tokens for all experiments.
We subtract the mean over the <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="d_{\text{model}}" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><msub id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml"><mi id="S2.SS1.p1.3.m3.1.1.2" xref="S2.SS1.p1.3.m3.1.1.2.cmml">d</mi><mtext id="S2.SS1.p1.3.m3.1.1.3" xref="S2.SS1.p1.3.m3.1.1.3a.cmml">model</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><apply id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1.2">𝑑</ci><ci id="S2.SS1.p1.3.m3.1.1.3a.cmml" xref="S2.SS1.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S2.SS1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3">model</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">d_{\text{model}}</annotation></semantics></math> dimension and normalize to all inputs to unit norm, prior to passing to the autoencoder (or computing reconstruction errors).</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p"><span id="S2.SS1.p2.1.1" class="ltx_text ltx_font_bold">Evaluation:</span> After training, we evaluate autoencoders on sparsity L<sub id="S2.SS1.p2.1.2" class="ltx_sub">0</sub>, and reconstruction mean-squared error (MSE). We report a normalized version of all MSE numbers, where we divide by a baseline reconstruction error of always predicting the mean activations.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.1" class="ltx_p"><span id="S2.SS1.p3.1.1" class="ltx_text ltx_font_bold">Hyperparameters:</span> To simplify analysis, we do not consider learning rate warmup or decay unless otherwise noted. We sweep learning rates at small scales and extrapolate the trend of optimal learning rates for large scale. See <a href="#A1" title="Appendix A Optimization ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix A</span></a> for other optimization details.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Baseline: ReLU autoencoders</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.2" class="ltx_p">For an input vector <math id="S2.SS2.p1.1.m1.1" class="ltx_Math" alttext="x\in\mathbb{R}^{d}" display="inline"><semantics id="S2.SS2.p1.1.m1.1a"><mrow id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml"><mi id="S2.SS2.p1.1.m1.1.1.2" xref="S2.SS2.p1.1.m1.1.1.2.cmml">x</mi><mo id="S2.SS2.p1.1.m1.1.1.1" xref="S2.SS2.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS2.p1.1.m1.1.1.3" xref="S2.SS2.p1.1.m1.1.1.3.cmml"><mi id="S2.SS2.p1.1.m1.1.1.3.2" xref="S2.SS2.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS2.p1.1.m1.1.1.3.3" xref="S2.SS2.p1.1.m1.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.1b"><apply id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1"><in id="S2.SS2.p1.1.m1.1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1.1"></in><ci id="S2.SS2.p1.1.m1.1.1.2.cmml" xref="S2.SS2.p1.1.m1.1.1.2">𝑥</ci><apply id="S2.SS2.p1.1.m1.1.1.3.cmml" xref="S2.SS2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p1.1.m1.1.1.3.1.cmml" xref="S2.SS2.p1.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p1.1.m1.1.1.3.2.cmml" xref="S2.SS2.p1.1.m1.1.1.3.2">ℝ</ci><ci id="S2.SS2.p1.1.m1.1.1.3.3.cmml" xref="S2.SS2.p1.1.m1.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.1c">x\in\mathbb{R}^{d}</annotation></semantics></math> from the residual stream, and <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><mi id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><ci id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">n</annotation></semantics></math> latent dimensions, we use baseline ReLU autoencoders from <cite class="ltx_cite ltx_citemacro_citep">(Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>. The encoder and decoder are defined by:</p>
<table id="S2.E1" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S2.E1X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle z" display="inline"><semantics id="S2.E1X.2.1.1.m1.1a"><mi id="S2.E1X.2.1.1.m1.1.1" xref="S2.E1X.2.1.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S2.E1X.2.1.1.m1.1b"><ci id="S2.E1X.2.1.1.m1.1.1.cmml" xref="S2.E1X.2.1.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E1X.2.1.1.m1.1c">\displaystyle z</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1X.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle=\text{ReLU}(W_{\text{enc}}(x-b_{\text{pre}})+b_{\text{enc}})" display="inline"><semantics id="S2.E1X.3.2.2.m1.1a"><mrow id="S2.E1X.3.2.2.m1.1.1" xref="S2.E1X.3.2.2.m1.1.1.cmml"><mi id="S2.E1X.3.2.2.m1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.3.cmml"></mi><mo id="S2.E1X.3.2.2.m1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.2.cmml">=</mo><mrow id="S2.E1X.3.2.2.m1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.cmml"><mtext id="S2.E1X.3.2.2.m1.1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.1.3a.cmml">ReLU</mtext><mo lspace="0em" rspace="0em" id="S2.E1X.3.2.2.m1.1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.1.2.cmml">​</mo><mrow id="S2.E1X.3.2.2.m1.1.1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E1X.3.2.2.m1.1.1.1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1X.3.2.2.m1.1.1.1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.cmml"><mrow id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.cmml"><msub id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.2.cmml">W</mi><mtext id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.3a.cmml">enc</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mo id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">b</mi><mtext id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml">pre</mtext></msub></mrow><mo stretchy="false" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.2.cmml">+</mo><msub id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.2" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.2.cmml">b</mi><mtext id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.3a.cmml">enc</mtext></msub></mrow><mo stretchy="false" id="S2.E1X.3.2.2.m1.1.1.1.1.1.3" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1X.3.2.2.m1.1b"><apply id="S2.E1X.3.2.2.m1.1.1.cmml" xref="S2.E1X.3.2.2.m1.1.1"><eq id="S2.E1X.3.2.2.m1.1.1.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.2"></eq><csymbol cd="latexml" id="S2.E1X.3.2.2.m1.1.1.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.3">absent</csymbol><apply id="S2.E1X.3.2.2.m1.1.1.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1"><times id="S2.E1X.3.2.2.m1.1.1.1.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.2"></times><ci id="S2.E1X.3.2.2.m1.1.1.1.3a.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.3"><mtext id="S2.E1X.3.2.2.m1.1.1.1.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.3">ReLU</mtext></ci><apply id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1"><plus id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.2"></plus><apply id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1"><times id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.2"></times><apply id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.2">𝑊</ci><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.3a.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.3.3">enc</mtext></ci></apply><apply id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1"><minus id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.1"></minus><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.2">𝑏</ci><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.3.3">pre</mtext></ci></apply></apply></apply><apply id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.2">𝑏</ci><ci id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.3a.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1X.3.2.2.m1.1.1.1.1.1.1.3.3">enc</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1X.3.2.2.m1.1c">\displaystyle=\text{ReLU}(W_{\text{enc}}(x-b_{\text{pre}})+b_{\text{enc}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(1)</span></td>
</tr>
<tr id="S2.E1Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1Xa.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\hat{x}" display="inline"><semantics id="S2.E1Xa.2.1.1.m1.1a"><mover accent="true" id="S2.E1Xa.2.1.1.m1.1.1" xref="S2.E1Xa.2.1.1.m1.1.1.cmml"><mi id="S2.E1Xa.2.1.1.m1.1.1.2" xref="S2.E1Xa.2.1.1.m1.1.1.2.cmml">x</mi><mo id="S2.E1Xa.2.1.1.m1.1.1.1" xref="S2.E1Xa.2.1.1.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S2.E1Xa.2.1.1.m1.1b"><apply id="S2.E1Xa.2.1.1.m1.1.1.cmml" xref="S2.E1Xa.2.1.1.m1.1.1"><ci id="S2.E1Xa.2.1.1.m1.1.1.1.cmml" xref="S2.E1Xa.2.1.1.m1.1.1.1">^</ci><ci id="S2.E1Xa.2.1.1.m1.1.1.2.cmml" xref="S2.E1Xa.2.1.1.m1.1.1.2">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1Xa.2.1.1.m1.1c">\displaystyle\hat{x}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1Xa.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle=W_{\text{dec}}z+b_{\text{pre}}" display="inline"><semantics id="S2.E1Xa.3.2.2.m1.1a"><mrow id="S2.E1Xa.3.2.2.m1.1.1" xref="S2.E1Xa.3.2.2.m1.1.1.cmml"><mi id="S2.E1Xa.3.2.2.m1.1.1.2" xref="S2.E1Xa.3.2.2.m1.1.1.2.cmml"></mi><mo id="S2.E1Xa.3.2.2.m1.1.1.1" xref="S2.E1Xa.3.2.2.m1.1.1.1.cmml">=</mo><mrow id="S2.E1Xa.3.2.2.m1.1.1.3" xref="S2.E1Xa.3.2.2.m1.1.1.3.cmml"><mrow id="S2.E1Xa.3.2.2.m1.1.1.3.2" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.cmml"><msub id="S2.E1Xa.3.2.2.m1.1.1.3.2.2" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2.cmml"><mi id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.2" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2.2.cmml">W</mi><mtext id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.3" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2.3a.cmml">dec</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E1Xa.3.2.2.m1.1.1.3.2.1" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.1.cmml">​</mo><mi id="S2.E1Xa.3.2.2.m1.1.1.3.2.3" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.3.cmml">z</mi></mrow><mo id="S2.E1Xa.3.2.2.m1.1.1.3.1" xref="S2.E1Xa.3.2.2.m1.1.1.3.1.cmml">+</mo><msub id="S2.E1Xa.3.2.2.m1.1.1.3.3" xref="S2.E1Xa.3.2.2.m1.1.1.3.3.cmml"><mi id="S2.E1Xa.3.2.2.m1.1.1.3.3.2" xref="S2.E1Xa.3.2.2.m1.1.1.3.3.2.cmml">b</mi><mtext id="S2.E1Xa.3.2.2.m1.1.1.3.3.3" xref="S2.E1Xa.3.2.2.m1.1.1.3.3.3a.cmml">pre</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1Xa.3.2.2.m1.1b"><apply id="S2.E1Xa.3.2.2.m1.1.1.cmml" xref="S2.E1Xa.3.2.2.m1.1.1"><eq id="S2.E1Xa.3.2.2.m1.1.1.1.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.1"></eq><csymbol cd="latexml" id="S2.E1Xa.3.2.2.m1.1.1.2.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.2">absent</csymbol><apply id="S2.E1Xa.3.2.2.m1.1.1.3.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3"><plus id="S2.E1Xa.3.2.2.m1.1.1.3.1.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.1"></plus><apply id="S2.E1Xa.3.2.2.m1.1.1.3.2.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2"><times id="S2.E1Xa.3.2.2.m1.1.1.3.2.1.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.1"></times><apply id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.1.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2">subscript</csymbol><ci id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.2.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2.2">𝑊</ci><ci id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.3a.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2.3"><mtext mathsize="70%" id="S2.E1Xa.3.2.2.m1.1.1.3.2.2.3.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.2.3">dec</mtext></ci></apply><ci id="S2.E1Xa.3.2.2.m1.1.1.3.2.3.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.2.3">𝑧</ci></apply><apply id="S2.E1Xa.3.2.2.m1.1.1.3.3.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E1Xa.3.2.2.m1.1.1.3.3.1.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.3">subscript</csymbol><ci id="S2.E1Xa.3.2.2.m1.1.1.3.3.2.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.3.2">𝑏</ci><ci id="S2.E1Xa.3.2.2.m1.1.1.3.3.3a.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.3.3"><mtext mathsize="70%" id="S2.E1Xa.3.2.2.m1.1.1.3.3.3.cmml" xref="S2.E1Xa.3.2.2.m1.1.1.3.3.3">pre</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1Xa.3.2.2.m1.1c">\displaystyle=W_{\text{dec}}z+b_{\text{pre}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S2.SS2.p1.12" class="ltx_p">with <math id="S2.SS2.p1.3.m1.1" class="ltx_Math" alttext="W_{\text{enc}}\in\mathbb{R}^{n\times d}" display="inline"><semantics id="S2.SS2.p1.3.m1.1a"><mrow id="S2.SS2.p1.3.m1.1.1" xref="S2.SS2.p1.3.m1.1.1.cmml"><msub id="S2.SS2.p1.3.m1.1.1.2" xref="S2.SS2.p1.3.m1.1.1.2.cmml"><mi id="S2.SS2.p1.3.m1.1.1.2.2" xref="S2.SS2.p1.3.m1.1.1.2.2.cmml">W</mi><mtext id="S2.SS2.p1.3.m1.1.1.2.3" xref="S2.SS2.p1.3.m1.1.1.2.3a.cmml">enc</mtext></msub><mo id="S2.SS2.p1.3.m1.1.1.1" xref="S2.SS2.p1.3.m1.1.1.1.cmml">∈</mo><msup id="S2.SS2.p1.3.m1.1.1.3" xref="S2.SS2.p1.3.m1.1.1.3.cmml"><mi id="S2.SS2.p1.3.m1.1.1.3.2" xref="S2.SS2.p1.3.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p1.3.m1.1.1.3.3" xref="S2.SS2.p1.3.m1.1.1.3.3.cmml"><mi id="S2.SS2.p1.3.m1.1.1.3.3.2" xref="S2.SS2.p1.3.m1.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p1.3.m1.1.1.3.3.1" xref="S2.SS2.p1.3.m1.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p1.3.m1.1.1.3.3.3" xref="S2.SS2.p1.3.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m1.1b"><apply id="S2.SS2.p1.3.m1.1.1.cmml" xref="S2.SS2.p1.3.m1.1.1"><in id="S2.SS2.p1.3.m1.1.1.1.cmml" xref="S2.SS2.p1.3.m1.1.1.1"></in><apply id="S2.SS2.p1.3.m1.1.1.2.cmml" xref="S2.SS2.p1.3.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.3.m1.1.1.2.1.cmml" xref="S2.SS2.p1.3.m1.1.1.2">subscript</csymbol><ci id="S2.SS2.p1.3.m1.1.1.2.2.cmml" xref="S2.SS2.p1.3.m1.1.1.2.2">𝑊</ci><ci id="S2.SS2.p1.3.m1.1.1.2.3a.cmml" xref="S2.SS2.p1.3.m1.1.1.2.3"><mtext mathsize="70%" id="S2.SS2.p1.3.m1.1.1.2.3.cmml" xref="S2.SS2.p1.3.m1.1.1.2.3">enc</mtext></ci></apply><apply id="S2.SS2.p1.3.m1.1.1.3.cmml" xref="S2.SS2.p1.3.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p1.3.m1.1.1.3.1.cmml" xref="S2.SS2.p1.3.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p1.3.m1.1.1.3.2.cmml" xref="S2.SS2.p1.3.m1.1.1.3.2">ℝ</ci><apply id="S2.SS2.p1.3.m1.1.1.3.3.cmml" xref="S2.SS2.p1.3.m1.1.1.3.3"><times id="S2.SS2.p1.3.m1.1.1.3.3.1.cmml" xref="S2.SS2.p1.3.m1.1.1.3.3.1"></times><ci id="S2.SS2.p1.3.m1.1.1.3.3.2.cmml" xref="S2.SS2.p1.3.m1.1.1.3.3.2">𝑛</ci><ci id="S2.SS2.p1.3.m1.1.1.3.3.3.cmml" xref="S2.SS2.p1.3.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m1.1c">W_{\text{enc}}\in\mathbb{R}^{n\times d}</annotation></semantics></math>, <math id="S2.SS2.p1.4.m2.1" class="ltx_Math" alttext="b_{\text{enc}}\in\mathbb{R}^{n}" display="inline"><semantics id="S2.SS2.p1.4.m2.1a"><mrow id="S2.SS2.p1.4.m2.1.1" xref="S2.SS2.p1.4.m2.1.1.cmml"><msub id="S2.SS2.p1.4.m2.1.1.2" xref="S2.SS2.p1.4.m2.1.1.2.cmml"><mi id="S2.SS2.p1.4.m2.1.1.2.2" xref="S2.SS2.p1.4.m2.1.1.2.2.cmml">b</mi><mtext id="S2.SS2.p1.4.m2.1.1.2.3" xref="S2.SS2.p1.4.m2.1.1.2.3a.cmml">enc</mtext></msub><mo id="S2.SS2.p1.4.m2.1.1.1" xref="S2.SS2.p1.4.m2.1.1.1.cmml">∈</mo><msup id="S2.SS2.p1.4.m2.1.1.3" xref="S2.SS2.p1.4.m2.1.1.3.cmml"><mi id="S2.SS2.p1.4.m2.1.1.3.2" xref="S2.SS2.p1.4.m2.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS2.p1.4.m2.1.1.3.3" xref="S2.SS2.p1.4.m2.1.1.3.3.cmml">n</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m2.1b"><apply id="S2.SS2.p1.4.m2.1.1.cmml" xref="S2.SS2.p1.4.m2.1.1"><in id="S2.SS2.p1.4.m2.1.1.1.cmml" xref="S2.SS2.p1.4.m2.1.1.1"></in><apply id="S2.SS2.p1.4.m2.1.1.2.cmml" xref="S2.SS2.p1.4.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.4.m2.1.1.2.1.cmml" xref="S2.SS2.p1.4.m2.1.1.2">subscript</csymbol><ci id="S2.SS2.p1.4.m2.1.1.2.2.cmml" xref="S2.SS2.p1.4.m2.1.1.2.2">𝑏</ci><ci id="S2.SS2.p1.4.m2.1.1.2.3a.cmml" xref="S2.SS2.p1.4.m2.1.1.2.3"><mtext mathsize="70%" id="S2.SS2.p1.4.m2.1.1.2.3.cmml" xref="S2.SS2.p1.4.m2.1.1.2.3">enc</mtext></ci></apply><apply id="S2.SS2.p1.4.m2.1.1.3.cmml" xref="S2.SS2.p1.4.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p1.4.m2.1.1.3.1.cmml" xref="S2.SS2.p1.4.m2.1.1.3">superscript</csymbol><ci id="S2.SS2.p1.4.m2.1.1.3.2.cmml" xref="S2.SS2.p1.4.m2.1.1.3.2">ℝ</ci><ci id="S2.SS2.p1.4.m2.1.1.3.3.cmml" xref="S2.SS2.p1.4.m2.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m2.1c">b_{\text{enc}}\in\mathbb{R}^{n}</annotation></semantics></math>, <math id="S2.SS2.p1.5.m3.1" class="ltx_Math" alttext="W_{\text{dec}}\in\mathbb{R}^{d\times n}" display="inline"><semantics id="S2.SS2.p1.5.m3.1a"><mrow id="S2.SS2.p1.5.m3.1.1" xref="S2.SS2.p1.5.m3.1.1.cmml"><msub id="S2.SS2.p1.5.m3.1.1.2" xref="S2.SS2.p1.5.m3.1.1.2.cmml"><mi id="S2.SS2.p1.5.m3.1.1.2.2" xref="S2.SS2.p1.5.m3.1.1.2.2.cmml">W</mi><mtext id="S2.SS2.p1.5.m3.1.1.2.3" xref="S2.SS2.p1.5.m3.1.1.2.3a.cmml">dec</mtext></msub><mo id="S2.SS2.p1.5.m3.1.1.1" xref="S2.SS2.p1.5.m3.1.1.1.cmml">∈</mo><msup id="S2.SS2.p1.5.m3.1.1.3" xref="S2.SS2.p1.5.m3.1.1.3.cmml"><mi id="S2.SS2.p1.5.m3.1.1.3.2" xref="S2.SS2.p1.5.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p1.5.m3.1.1.3.3" xref="S2.SS2.p1.5.m3.1.1.3.3.cmml"><mi id="S2.SS2.p1.5.m3.1.1.3.3.2" xref="S2.SS2.p1.5.m3.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p1.5.m3.1.1.3.3.1" xref="S2.SS2.p1.5.m3.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p1.5.m3.1.1.3.3.3" xref="S2.SS2.p1.5.m3.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m3.1b"><apply id="S2.SS2.p1.5.m3.1.1.cmml" xref="S2.SS2.p1.5.m3.1.1"><in id="S2.SS2.p1.5.m3.1.1.1.cmml" xref="S2.SS2.p1.5.m3.1.1.1"></in><apply id="S2.SS2.p1.5.m3.1.1.2.cmml" xref="S2.SS2.p1.5.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m3.1.1.2.1.cmml" xref="S2.SS2.p1.5.m3.1.1.2">subscript</csymbol><ci id="S2.SS2.p1.5.m3.1.1.2.2.cmml" xref="S2.SS2.p1.5.m3.1.1.2.2">𝑊</ci><ci id="S2.SS2.p1.5.m3.1.1.2.3a.cmml" xref="S2.SS2.p1.5.m3.1.1.2.3"><mtext mathsize="70%" id="S2.SS2.p1.5.m3.1.1.2.3.cmml" xref="S2.SS2.p1.5.m3.1.1.2.3">dec</mtext></ci></apply><apply id="S2.SS2.p1.5.m3.1.1.3.cmml" xref="S2.SS2.p1.5.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m3.1.1.3.1.cmml" xref="S2.SS2.p1.5.m3.1.1.3">superscript</csymbol><ci id="S2.SS2.p1.5.m3.1.1.3.2.cmml" xref="S2.SS2.p1.5.m3.1.1.3.2">ℝ</ci><apply id="S2.SS2.p1.5.m3.1.1.3.3.cmml" xref="S2.SS2.p1.5.m3.1.1.3.3"><times id="S2.SS2.p1.5.m3.1.1.3.3.1.cmml" xref="S2.SS2.p1.5.m3.1.1.3.3.1"></times><ci id="S2.SS2.p1.5.m3.1.1.3.3.2.cmml" xref="S2.SS2.p1.5.m3.1.1.3.3.2">𝑑</ci><ci id="S2.SS2.p1.5.m3.1.1.3.3.3.cmml" xref="S2.SS2.p1.5.m3.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m3.1c">W_{\text{dec}}\in\mathbb{R}^{d\times n}</annotation></semantics></math>, and <math id="S2.SS2.p1.6.m4.1" class="ltx_Math" alttext="b_{\text{pre}}\in\mathbb{R}^{d}" display="inline"><semantics id="S2.SS2.p1.6.m4.1a"><mrow id="S2.SS2.p1.6.m4.1.1" xref="S2.SS2.p1.6.m4.1.1.cmml"><msub id="S2.SS2.p1.6.m4.1.1.2" xref="S2.SS2.p1.6.m4.1.1.2.cmml"><mi id="S2.SS2.p1.6.m4.1.1.2.2" xref="S2.SS2.p1.6.m4.1.1.2.2.cmml">b</mi><mtext id="S2.SS2.p1.6.m4.1.1.2.3" xref="S2.SS2.p1.6.m4.1.1.2.3a.cmml">pre</mtext></msub><mo id="S2.SS2.p1.6.m4.1.1.1" xref="S2.SS2.p1.6.m4.1.1.1.cmml">∈</mo><msup id="S2.SS2.p1.6.m4.1.1.3" xref="S2.SS2.p1.6.m4.1.1.3.cmml"><mi id="S2.SS2.p1.6.m4.1.1.3.2" xref="S2.SS2.p1.6.m4.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS2.p1.6.m4.1.1.3.3" xref="S2.SS2.p1.6.m4.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m4.1b"><apply id="S2.SS2.p1.6.m4.1.1.cmml" xref="S2.SS2.p1.6.m4.1.1"><in id="S2.SS2.p1.6.m4.1.1.1.cmml" xref="S2.SS2.p1.6.m4.1.1.1"></in><apply id="S2.SS2.p1.6.m4.1.1.2.cmml" xref="S2.SS2.p1.6.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m4.1.1.2.1.cmml" xref="S2.SS2.p1.6.m4.1.1.2">subscript</csymbol><ci id="S2.SS2.p1.6.m4.1.1.2.2.cmml" xref="S2.SS2.p1.6.m4.1.1.2.2">𝑏</ci><ci id="S2.SS2.p1.6.m4.1.1.2.3a.cmml" xref="S2.SS2.p1.6.m4.1.1.2.3"><mtext mathsize="70%" id="S2.SS2.p1.6.m4.1.1.2.3.cmml" xref="S2.SS2.p1.6.m4.1.1.2.3">pre</mtext></ci></apply><apply id="S2.SS2.p1.6.m4.1.1.3.cmml" xref="S2.SS2.p1.6.m4.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m4.1.1.3.1.cmml" xref="S2.SS2.p1.6.m4.1.1.3">superscript</csymbol><ci id="S2.SS2.p1.6.m4.1.1.3.2.cmml" xref="S2.SS2.p1.6.m4.1.1.3.2">ℝ</ci><ci id="S2.SS2.p1.6.m4.1.1.3.3.cmml" xref="S2.SS2.p1.6.m4.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m4.1c">b_{\text{pre}}\in\mathbb{R}^{d}</annotation></semantics></math>. The training loss is defined by <math id="S2.SS2.p1.7.m5.2" class="ltx_Math" alttext="\mathcal{L}=||x-\hat{x}||^{2}_{2}+\lambda||z||_{1}" display="inline"><semantics id="S2.SS2.p1.7.m5.2a"><mrow id="S2.SS2.p1.7.m5.2.2" xref="S2.SS2.p1.7.m5.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p1.7.m5.2.2.3" xref="S2.SS2.p1.7.m5.2.2.3.cmml">ℒ</mi><mo id="S2.SS2.p1.7.m5.2.2.2" xref="S2.SS2.p1.7.m5.2.2.2.cmml">=</mo><mrow id="S2.SS2.p1.7.m5.2.2.1" xref="S2.SS2.p1.7.m5.2.2.1.cmml"><msubsup id="S2.SS2.p1.7.m5.2.2.1.1" xref="S2.SS2.p1.7.m5.2.2.1.1.cmml"><mrow id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.2" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.2.1.cmml">‖</mo><mrow id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.2" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.2.cmml">x</mi><mo id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.1" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.cmml"><mi id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.2" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.2.cmml">x</mi><mo id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.1" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.1.cmml">^</mo></mover></mrow><mo stretchy="false" id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.3" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.SS2.p1.7.m5.2.2.1.1.3" xref="S2.SS2.p1.7.m5.2.2.1.1.3.cmml">2</mn><mn id="S2.SS2.p1.7.m5.2.2.1.1.1.3" xref="S2.SS2.p1.7.m5.2.2.1.1.1.3.cmml">2</mn></msubsup><mo id="S2.SS2.p1.7.m5.2.2.1.2" xref="S2.SS2.p1.7.m5.2.2.1.2.cmml">+</mo><mrow id="S2.SS2.p1.7.m5.2.2.1.3" xref="S2.SS2.p1.7.m5.2.2.1.3.cmml"><mi id="S2.SS2.p1.7.m5.2.2.1.3.2" xref="S2.SS2.p1.7.m5.2.2.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.7.m5.2.2.1.3.1" xref="S2.SS2.p1.7.m5.2.2.1.3.1.cmml">​</mo><msub id="S2.SS2.p1.7.m5.2.2.1.3.3" xref="S2.SS2.p1.7.m5.2.2.1.3.3.cmml"><mrow id="S2.SS2.p1.7.m5.2.2.1.3.3.2.2" xref="S2.SS2.p1.7.m5.2.2.1.3.3.2.1.cmml"><mo stretchy="false" id="S2.SS2.p1.7.m5.2.2.1.3.3.2.2.1" xref="S2.SS2.p1.7.m5.2.2.1.3.3.2.1.1.cmml">‖</mo><mi id="S2.SS2.p1.7.m5.1.1" xref="S2.SS2.p1.7.m5.1.1.cmml">z</mi><mo stretchy="false" id="S2.SS2.p1.7.m5.2.2.1.3.3.2.2.2" xref="S2.SS2.p1.7.m5.2.2.1.3.3.2.1.1.cmml">‖</mo></mrow><mn id="S2.SS2.p1.7.m5.2.2.1.3.3.3" xref="S2.SS2.p1.7.m5.2.2.1.3.3.3.cmml">1</mn></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.7.m5.2b"><apply id="S2.SS2.p1.7.m5.2.2.cmml" xref="S2.SS2.p1.7.m5.2.2"><eq id="S2.SS2.p1.7.m5.2.2.2.cmml" xref="S2.SS2.p1.7.m5.2.2.2"></eq><ci id="S2.SS2.p1.7.m5.2.2.3.cmml" xref="S2.SS2.p1.7.m5.2.2.3">ℒ</ci><apply id="S2.SS2.p1.7.m5.2.2.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1"><plus id="S2.SS2.p1.7.m5.2.2.1.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.2"></plus><apply id="S2.SS2.p1.7.m5.2.2.1.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.7.m5.2.2.1.1.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1">subscript</csymbol><apply id="S2.SS2.p1.7.m5.2.2.1.1.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.7.m5.2.2.1.1.1.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1">superscript</csymbol><apply id="S2.SS2.p1.7.m5.2.2.1.1.1.1.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS2.p1.7.m5.2.2.1.1.1.1.2.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.2">norm</csymbol><apply id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1"><minus id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.1"></minus><ci id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3"><ci id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.1">^</ci><ci id="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.1.1.1.3.2">𝑥</ci></apply></apply></apply><cn type="integer" id="S2.SS2.p1.7.m5.2.2.1.1.1.3.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.1.3">2</cn></apply><cn type="integer" id="S2.SS2.p1.7.m5.2.2.1.1.3.cmml" xref="S2.SS2.p1.7.m5.2.2.1.1.3">2</cn></apply><apply id="S2.SS2.p1.7.m5.2.2.1.3.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3"><times id="S2.SS2.p1.7.m5.2.2.1.3.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.1"></times><ci id="S2.SS2.p1.7.m5.2.2.1.3.2.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.2">𝜆</ci><apply id="S2.SS2.p1.7.m5.2.2.1.3.3.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.3"><csymbol cd="ambiguous" id="S2.SS2.p1.7.m5.2.2.1.3.3.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.3">subscript</csymbol><apply id="S2.SS2.p1.7.m5.2.2.1.3.3.2.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.3.2.2"><csymbol cd="latexml" id="S2.SS2.p1.7.m5.2.2.1.3.3.2.1.1.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.3.2.2.1">norm</csymbol><ci id="S2.SS2.p1.7.m5.1.1.cmml" xref="S2.SS2.p1.7.m5.1.1">𝑧</ci></apply><cn type="integer" id="S2.SS2.p1.7.m5.2.2.1.3.3.3.cmml" xref="S2.SS2.p1.7.m5.2.2.1.3.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.7.m5.2c">\mathcal{L}=||x-\hat{x}||^{2}_{2}+\lambda||z||_{1}</annotation></semantics></math>, where <math id="S2.SS2.p1.8.m6.1" class="ltx_Math" alttext="||x-\hat{x}||^{2}_{2}" display="inline"><semantics id="S2.SS2.p1.8.m6.1a"><msubsup id="S2.SS2.p1.8.m6.1.1" xref="S2.SS2.p1.8.m6.1.1.cmml"><mrow id="S2.SS2.p1.8.m6.1.1.1.1.1" xref="S2.SS2.p1.8.m6.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p1.8.m6.1.1.1.1.1.2" xref="S2.SS2.p1.8.m6.1.1.1.1.2.1.cmml">‖</mo><mrow id="S2.SS2.p1.8.m6.1.1.1.1.1.1" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p1.8.m6.1.1.1.1.1.1.2" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.2.cmml">x</mi><mo id="S2.SS2.p1.8.m6.1.1.1.1.1.1.1" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S2.SS2.p1.8.m6.1.1.1.1.1.1.3" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.cmml"><mi id="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.2" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.2.cmml">x</mi><mo id="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.1" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.1.cmml">^</mo></mover></mrow><mo stretchy="false" id="S2.SS2.p1.8.m6.1.1.1.1.1.3" xref="S2.SS2.p1.8.m6.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.SS2.p1.8.m6.1.1.3" xref="S2.SS2.p1.8.m6.1.1.3.cmml">2</mn><mn id="S2.SS2.p1.8.m6.1.1.1.3" xref="S2.SS2.p1.8.m6.1.1.1.3.cmml">2</mn></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.8.m6.1b"><apply id="S2.SS2.p1.8.m6.1.1.cmml" xref="S2.SS2.p1.8.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.8.m6.1.1.2.cmml" xref="S2.SS2.p1.8.m6.1.1">subscript</csymbol><apply id="S2.SS2.p1.8.m6.1.1.1.cmml" xref="S2.SS2.p1.8.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.8.m6.1.1.1.2.cmml" xref="S2.SS2.p1.8.m6.1.1">superscript</csymbol><apply id="S2.SS2.p1.8.m6.1.1.1.1.2.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS2.p1.8.m6.1.1.1.1.2.1.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.2">norm</csymbol><apply id="S2.SS2.p1.8.m6.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1"><minus id="S2.SS2.p1.8.m6.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.1"></minus><ci id="S2.SS2.p1.8.m6.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.3"><ci id="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.1.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.1">^</ci><ci id="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.2.cmml" xref="S2.SS2.p1.8.m6.1.1.1.1.1.1.3.2">𝑥</ci></apply></apply></apply><cn type="integer" id="S2.SS2.p1.8.m6.1.1.1.3.cmml" xref="S2.SS2.p1.8.m6.1.1.1.3">2</cn></apply><cn type="integer" id="S2.SS2.p1.8.m6.1.1.3.cmml" xref="S2.SS2.p1.8.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.8.m6.1c">||x-\hat{x}||^{2}_{2}</annotation></semantics></math> is the reconstruction MSE, <math id="S2.SS2.p1.9.m7.1" class="ltx_Math" alttext="||z||_{1}" display="inline"><semantics id="S2.SS2.p1.9.m7.1a"><msub id="S2.SS2.p1.9.m7.1.2" xref="S2.SS2.p1.9.m7.1.2.cmml"><mrow id="S2.SS2.p1.9.m7.1.2.2.2" xref="S2.SS2.p1.9.m7.1.2.2.1.cmml"><mo stretchy="false" id="S2.SS2.p1.9.m7.1.2.2.2.1" xref="S2.SS2.p1.9.m7.1.2.2.1.1.cmml">‖</mo><mi id="S2.SS2.p1.9.m7.1.1" xref="S2.SS2.p1.9.m7.1.1.cmml">z</mi><mo stretchy="false" id="S2.SS2.p1.9.m7.1.2.2.2.2" xref="S2.SS2.p1.9.m7.1.2.2.1.1.cmml">‖</mo></mrow><mn id="S2.SS2.p1.9.m7.1.2.3" xref="S2.SS2.p1.9.m7.1.2.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.9.m7.1b"><apply id="S2.SS2.p1.9.m7.1.2.cmml" xref="S2.SS2.p1.9.m7.1.2"><csymbol cd="ambiguous" id="S2.SS2.p1.9.m7.1.2.1.cmml" xref="S2.SS2.p1.9.m7.1.2">subscript</csymbol><apply id="S2.SS2.p1.9.m7.1.2.2.1.cmml" xref="S2.SS2.p1.9.m7.1.2.2.2"><csymbol cd="latexml" id="S2.SS2.p1.9.m7.1.2.2.1.1.cmml" xref="S2.SS2.p1.9.m7.1.2.2.2.1">norm</csymbol><ci id="S2.SS2.p1.9.m7.1.1.cmml" xref="S2.SS2.p1.9.m7.1.1">𝑧</ci></apply><cn type="integer" id="S2.SS2.p1.9.m7.1.2.3.cmml" xref="S2.SS2.p1.9.m7.1.2.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.9.m7.1c">||z||_{1}</annotation></semantics></math> is an L<sub id="S2.SS2.p1.12.1" class="ltx_sub">1</sub> penalty promoting sparsity in latent activations <math id="S2.SS2.p1.11.m9.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S2.SS2.p1.11.m9.1a"><mi id="S2.SS2.p1.11.m9.1.1" xref="S2.SS2.p1.11.m9.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.11.m9.1b"><ci id="S2.SS2.p1.11.m9.1.1.cmml" xref="S2.SS2.p1.11.m9.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.11.m9.1c">z</annotation></semantics></math>, and <math id="S2.SS2.p1.12.m10.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S2.SS2.p1.12.m10.1a"><mi id="S2.SS2.p1.12.m10.1.1" xref="S2.SS2.p1.12.m10.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.12.m10.1b"><ci id="S2.SS2.p1.12.m10.1.1.cmml" xref="S2.SS2.p1.12.m10.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.12.m10.1c">\lambda</annotation></semantics></math> is a hyperparameter that needs to be tuned.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>TopK activation function</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.2" class="ltx_p">We use a <math id="S2.SS3.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS3.p1.1.m1.1a"><mi id="S2.SS3.p1.1.m1.1.1" xref="S2.SS3.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.1.m1.1b"><ci id="S2.SS3.p1.1.m1.1.1.cmml" xref="S2.SS3.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.1.m1.1c">k</annotation></semantics></math>-sparse autoencoder <cite class="ltx_cite ltx_citemacro_citep">(Makhzani and Frey, <a href="#bib.bib36" title="" class="ltx_ref">2013</a>)</cite>, which directly controls the number of active latents by using an activation function (TopK) that only keeps the <math id="S2.SS3.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS3.p1.2.m2.1a"><mi id="S2.SS3.p1.2.m2.1.1" xref="S2.SS3.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.2.m2.1b"><ci id="S2.SS3.p1.2.m2.1.1.cmml" xref="S2.SS3.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.2.m2.1c">k</annotation></semantics></math> largest latents, zeroing the rest. The encoder is thus defined as:</p>
<table id="S2.E2" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S2.E2X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle z" display="inline"><semantics id="S2.E2X.2.1.1.m1.1a"><mi id="S2.E2X.2.1.1.m1.1.1" xref="S2.E2X.2.1.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S2.E2X.2.1.1.m1.1b"><ci id="S2.E2X.2.1.1.m1.1.1.cmml" xref="S2.E2X.2.1.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E2X.2.1.1.m1.1c">\displaystyle z</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E2X.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle=\text{TopK}(W_{\text{enc}}(x-b_{\text{pre}}))" display="inline"><semantics id="S2.E2X.3.2.2.m1.1a"><mrow id="S2.E2X.3.2.2.m1.1.1" xref="S2.E2X.3.2.2.m1.1.1.cmml"><mi id="S2.E2X.3.2.2.m1.1.1.3" xref="S2.E2X.3.2.2.m1.1.1.3.cmml"></mi><mo id="S2.E2X.3.2.2.m1.1.1.2" xref="S2.E2X.3.2.2.m1.1.1.2.cmml">=</mo><mrow id="S2.E2X.3.2.2.m1.1.1.1" xref="S2.E2X.3.2.2.m1.1.1.1.cmml"><mtext id="S2.E2X.3.2.2.m1.1.1.1.3" xref="S2.E2X.3.2.2.m1.1.1.1.3a.cmml">TopK</mtext><mo lspace="0em" rspace="0em" id="S2.E2X.3.2.2.m1.1.1.1.2" xref="S2.E2X.3.2.2.m1.1.1.1.2.cmml">​</mo><mrow id="S2.E2X.3.2.2.m1.1.1.1.1.1" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2X.3.2.2.m1.1.1.1.1.1.2" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E2X.3.2.2.m1.1.1.1.1.1.1" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.cmml"><msub id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.2" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.2.cmml">W</mi><mtext id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.3" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.3a.cmml">enc</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.2" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.2" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.2" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mo id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.2" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">b</mi><mtext id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.3" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.3a.cmml">pre</mtext></msub></mrow><mo stretchy="false" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.3" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.E2X.3.2.2.m1.1.1.1.1.1.3" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2X.3.2.2.m1.1b"><apply id="S2.E2X.3.2.2.m1.1.1.cmml" xref="S2.E2X.3.2.2.m1.1.1"><eq id="S2.E2X.3.2.2.m1.1.1.2.cmml" xref="S2.E2X.3.2.2.m1.1.1.2"></eq><csymbol cd="latexml" id="S2.E2X.3.2.2.m1.1.1.3.cmml" xref="S2.E2X.3.2.2.m1.1.1.3">absent</csymbol><apply id="S2.E2X.3.2.2.m1.1.1.1.cmml" xref="S2.E2X.3.2.2.m1.1.1.1"><times id="S2.E2X.3.2.2.m1.1.1.1.2.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.2"></times><ci id="S2.E2X.3.2.2.m1.1.1.1.3a.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.3"><mtext id="S2.E2X.3.2.2.m1.1.1.1.3.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.3">TopK</mtext></ci><apply id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1"><times id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.2"></times><apply id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.2">𝑊</ci><ci id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.3a.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.3.3">enc</mtext></ci></apply><apply id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1"><minus id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.1"></minus><ci id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.2">𝑏</ci><ci id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E2X.3.2.2.m1.1.1.1.1.1.1.1.1.1.3.3">pre</mtext></ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2X.3.2.2.m1.1c">\displaystyle=\text{TopK}(W_{\text{enc}}(x-b_{\text{pre}}))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(2)</span></td>
</tr>
</tbody>
</table>
<p id="S2.SS3.p1.3" class="ltx_p">and the decoder is unchanged. The training loss is simply <math id="S2.SS3.p1.3.m1.1" class="ltx_Math" alttext="\mathcal{L}=||x-\hat{x}||^{2}_{2}" display="inline"><semantics id="S2.SS3.p1.3.m1.1a"><mrow id="S2.SS3.p1.3.m1.1.1" xref="S2.SS3.p1.3.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p1.3.m1.1.1.3" xref="S2.SS3.p1.3.m1.1.1.3.cmml">ℒ</mi><mo id="S2.SS3.p1.3.m1.1.1.2" xref="S2.SS3.p1.3.m1.1.1.2.cmml">=</mo><msubsup id="S2.SS3.p1.3.m1.1.1.1" xref="S2.SS3.p1.3.m1.1.1.1.cmml"><mrow id="S2.SS3.p1.3.m1.1.1.1.1.1.1" xref="S2.SS3.p1.3.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS3.p1.3.m1.1.1.1.1.1.1.2" xref="S2.SS3.p1.3.m1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.cmml"><mi id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.2" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.2.cmml">x</mi><mo id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.1" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.2" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mo id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.1" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.1.cmml">^</mo></mover></mrow><mo stretchy="false" id="S2.SS3.p1.3.m1.1.1.1.1.1.1.3" xref="S2.SS3.p1.3.m1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S2.SS3.p1.3.m1.1.1.1.3" xref="S2.SS3.p1.3.m1.1.1.1.3.cmml">2</mn><mn id="S2.SS3.p1.3.m1.1.1.1.1.3" xref="S2.SS3.p1.3.m1.1.1.1.1.3.cmml">2</mn></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.3.m1.1b"><apply id="S2.SS3.p1.3.m1.1.1.cmml" xref="S2.SS3.p1.3.m1.1.1"><eq id="S2.SS3.p1.3.m1.1.1.2.cmml" xref="S2.SS3.p1.3.m1.1.1.2"></eq><ci id="S2.SS3.p1.3.m1.1.1.3.cmml" xref="S2.SS3.p1.3.m1.1.1.3">ℒ</ci><apply id="S2.SS3.p1.3.m1.1.1.1.cmml" xref="S2.SS3.p1.3.m1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.3.m1.1.1.1.2.cmml" xref="S2.SS3.p1.3.m1.1.1.1">subscript</csymbol><apply id="S2.SS3.p1.3.m1.1.1.1.1.cmml" xref="S2.SS3.p1.3.m1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.3.m1.1.1.1.1.2.cmml" xref="S2.SS3.p1.3.m1.1.1.1">superscript</csymbol><apply id="S2.SS3.p1.3.m1.1.1.1.1.1.2.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="S2.SS3.p1.3.m1.1.1.1.1.1.2.1.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.2">norm</csymbol><apply id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1"><minus id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.1"></minus><ci id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3"><ci id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.1">^</ci><ci id="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.1.1.1.3.2">𝑥</ci></apply></apply></apply><cn type="integer" id="S2.SS3.p1.3.m1.1.1.1.1.3.cmml" xref="S2.SS3.p1.3.m1.1.1.1.1.3">2</cn></apply><cn type="integer" id="S2.SS3.p1.3.m1.1.1.1.3.cmml" xref="S2.SS3.p1.3.m1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.3.m1.1c">\mathcal{L}=||x-\hat{x}||^{2}_{2}</annotation></semantics></math>.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">Using <math id="S2.SS3.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS3.p2.1.m1.1a"><mi id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><ci id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">k</annotation></semantics></math>-sparse autoencoders has a number of benefits:</p>
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.3" class="ltx_p">It removes the need for the L<sub id="S2.I1.i1.p1.3.1" class="ltx_sub">1</sub> penalty. L<sub id="S2.I1.i1.p1.3.2" class="ltx_sub">1</sub> is an imperfect approximation of L<sub id="S2.I1.i1.p1.3.3" class="ltx_sub">0</sub>, and it introduces a bias of shrinking all positive activations toward zero
(<a href="#S5.SS1" title="5.1 TopK prevents activation shrinkage ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5.1</span></a>).</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.3" class="ltx_p">It enables setting the L<sub id="S2.I1.i2.p1.3.1" class="ltx_sub">0</sub> directly, as opposed to tuning an L<sub id="S2.I1.i2.p1.3.2" class="ltx_sub">1</sub> coefficient <math id="S2.I1.i2.p1.3.m3.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S2.I1.i2.p1.3.m3.1a"><mi id="S2.I1.i2.p1.3.m3.1.1" xref="S2.I1.i2.p1.3.m3.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.3.m3.1b"><ci id="S2.I1.i2.p1.3.m3.1.1.cmml" xref="S2.I1.i2.p1.3.m3.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.3.m3.1c">\lambda</annotation></semantics></math>, enabling simpler model comparison and rapid iteration. It can also be used in combination with arbitrary activation functions.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>In our code, we also apply a ReLU to guarantee activations to be non-negative. However, the training curves are indistinguishable, as the <math id="footnote3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="footnote3.m1.1b"><mi id="footnote3.m1.1.1" xref="footnote3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="footnote3.m1.1c"><ci id="footnote3.m1.1.1.cmml" xref="footnote3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m1.1d">k</annotation></semantics></math> largest activations are almost always positive for reasonable choices of <math id="footnote3.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="footnote3.m2.1b"><mi id="footnote3.m2.1.1" xref="footnote3.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="footnote3.m2.1c"><ci id="footnote3.m2.1.1.cmml" xref="footnote3.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m2.1d">k</annotation></semantics></math>.</span></span></span></p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p">It empirically outperforms baseline ReLU autoencoders on the sparsity-reconstruction frontier (<a href="#S2.F2" title="Figure 2 ‣ 2.3 TopK activation function ‣ 2 Methods ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 2</span></a>a), and this gap increases with scale (<a href="#S2.F2" title="Figure 2 ‣ 2.3 TopK activation function ‣ 2 Methods ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 2</span></a>b).</p>
</div>
</li>
<li id="S2.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i4.p1" class="ltx_para">
<p id="S2.I1.i4.p1.1" class="ltx_p">It increases monosemanticity of random activating examples by effectively clamping small activations to zero
(<a href="#S4.SS3" title="4.3 Finding simple explanations for features ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.3</span></a>).</p>
</div>
</li>
</ul>
</div>
<figure id="S2.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F2.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.04093/assets/x3.png" id="S2.F2.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="326" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.sf1.4.2.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S2.F2.sf1.2.1" class="ltx_text" style="font-size:90%;">At a fixed number of latents (<math id="S2.F2.sf1.2.1.m1.1" class="ltx_Math" alttext="n=32768" display="inline"><semantics id="S2.F2.sf1.2.1.m1.1b"><mrow id="S2.F2.sf1.2.1.m1.1.1" xref="S2.F2.sf1.2.1.m1.1.1.cmml"><mi id="S2.F2.sf1.2.1.m1.1.1.2" xref="S2.F2.sf1.2.1.m1.1.1.2.cmml">n</mi><mo id="S2.F2.sf1.2.1.m1.1.1.1" xref="S2.F2.sf1.2.1.m1.1.1.1.cmml">=</mo><mn id="S2.F2.sf1.2.1.m1.1.1.3" xref="S2.F2.sf1.2.1.m1.1.1.3.cmml">32768</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.sf1.2.1.m1.1c"><apply id="S2.F2.sf1.2.1.m1.1.1.cmml" xref="S2.F2.sf1.2.1.m1.1.1"><eq id="S2.F2.sf1.2.1.m1.1.1.1.cmml" xref="S2.F2.sf1.2.1.m1.1.1.1"></eq><ci id="S2.F2.sf1.2.1.m1.1.1.2.cmml" xref="S2.F2.sf1.2.1.m1.1.1.2">𝑛</ci><cn type="integer" id="S2.F2.sf1.2.1.m1.1.1.3.cmml" xref="S2.F2.sf1.2.1.m1.1.1.3">32768</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.sf1.2.1.m1.1d">n=32768</annotation></semantics></math>), TopK has a better reconstruction-sparsity trade off than ReLU and ProLU, and is comparable to Gated.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F2.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.04093/assets/x4.png" id="S2.F2.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="325" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.sf2.4.2.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S2.F2.sf2.2.1" class="ltx_text" style="font-size:90%;">At a fixed sparsity level (<math id="S2.F2.sf2.2.1.m1.1" class="ltx_Math" alttext="L_{0}=128" display="inline"><semantics id="S2.F2.sf2.2.1.m1.1b"><mrow id="S2.F2.sf2.2.1.m1.1.1" xref="S2.F2.sf2.2.1.m1.1.1.cmml"><msub id="S2.F2.sf2.2.1.m1.1.1.2" xref="S2.F2.sf2.2.1.m1.1.1.2.cmml"><mi id="S2.F2.sf2.2.1.m1.1.1.2.2" xref="S2.F2.sf2.2.1.m1.1.1.2.2.cmml">L</mi><mn id="S2.F2.sf2.2.1.m1.1.1.2.3" xref="S2.F2.sf2.2.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="S2.F2.sf2.2.1.m1.1.1.1" xref="S2.F2.sf2.2.1.m1.1.1.1.cmml">=</mo><mn id="S2.F2.sf2.2.1.m1.1.1.3" xref="S2.F2.sf2.2.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.sf2.2.1.m1.1c"><apply id="S2.F2.sf2.2.1.m1.1.1.cmml" xref="S2.F2.sf2.2.1.m1.1.1"><eq id="S2.F2.sf2.2.1.m1.1.1.1.cmml" xref="S2.F2.sf2.2.1.m1.1.1.1"></eq><apply id="S2.F2.sf2.2.1.m1.1.1.2.cmml" xref="S2.F2.sf2.2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.F2.sf2.2.1.m1.1.1.2.1.cmml" xref="S2.F2.sf2.2.1.m1.1.1.2">subscript</csymbol><ci id="S2.F2.sf2.2.1.m1.1.1.2.2.cmml" xref="S2.F2.sf2.2.1.m1.1.1.2.2">𝐿</ci><cn type="integer" id="S2.F2.sf2.2.1.m1.1.1.2.3.cmml" xref="S2.F2.sf2.2.1.m1.1.1.2.3">0</cn></apply><cn type="integer" id="S2.F2.sf2.2.1.m1.1.1.3.cmml" xref="S2.F2.sf2.2.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.sf2.2.1.m1.1d">L_{0}=128</annotation></semantics></math>), scaling laws are steeper for TopK than ReLU.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note"><span id="footnote5.1.1.1" class="ltx_text" style="font-size:111%;">5</span></span>Non-TopK cannot set a precise L<sub id="footnote5.5" class="ltx_sub"><span id="footnote5.5.1" class="ltx_text" style="font-size:111%;">0</span></sub>, so we interpolated using a piecewise linear function in log-log space.</span></span></span>
<br class="ltx_break"></span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.2.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.3.2" class="ltx_text" style="font-size:90%;">Comparison between TopK and other activation functions.</span></figcaption>
</figure>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Preventing dead latents</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.1" class="ltx_p">Dead latents pose another significant difficulty in autoencoder training. In larger autoencoders, an increasingly large proportion of latents stop activating entirely at some point in training. For example, <cite class="ltx_cite ltx_citemacro_citet">Templeton et al. (<a href="#bib.bib64" title="" class="ltx_ref">2024</a>)</cite> train a 34 million latent autoencoder with only 12 million alive latents, and in our ablations we find up to 90% dead latents<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>We follow <cite class="ltx_cite ltx_citemacro_citet">Templeton et al. (<a href="#bib.bib64" title="" class="ltx_ref">2024</a>)</cite>, and consider a latent dead if it has not activated in 10 million tokens</span></span></span> when no mitigations are applied (<a href="#A3.F15" title="Figure 15 ‣ C.1 Dead latent prevention ‣ Appendix C Training ablations ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 15</span></a>). This results in substantially worse MSE and makes training computationally wasteful.
We find two important ingredients for preventing dead latents: we initialize the encoder to the transpose of the decoder, and we use an auxiliary loss that models reconstruction error using the top-<math id="S2.SS4.p1.1.m1.1" class="ltx_Math" alttext="k_{\text{aux}}" display="inline"><semantics id="S2.SS4.p1.1.m1.1a"><msub id="S2.SS4.p1.1.m1.1.1" xref="S2.SS4.p1.1.m1.1.1.cmml"><mi id="S2.SS4.p1.1.m1.1.1.2" xref="S2.SS4.p1.1.m1.1.1.2.cmml">k</mi><mtext id="S2.SS4.p1.1.m1.1.1.3" xref="S2.SS4.p1.1.m1.1.1.3a.cmml">aux</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.1.m1.1b"><apply id="S2.SS4.p1.1.m1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.1.m1.1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS4.p1.1.m1.1.1.2.cmml" xref="S2.SS4.p1.1.m1.1.1.2">𝑘</ci><ci id="S2.SS4.p1.1.m1.1.1.3a.cmml" xref="S2.SS4.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S2.SS4.p1.1.m1.1.1.3.cmml" xref="S2.SS4.p1.1.m1.1.1.3">aux</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.1.m1.1c">k_{\text{aux}}</annotation></semantics></math> dead latents (see <a href="#A1.SS2" title="A.2 Auxiliary loss ‣ Appendix A Optimization ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section A.2</span></a> for more details). Using these techniques, even in our largest (16 million latent) autoencoder only 7% of latents are dead.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Scaling laws</h2>

<figure id="S3.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S3.F4.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:208.1pt;"><img src="/html/2406.04093/assets/x5.png" id="S3.F4.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="319" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F4.1.1.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S3.F4.1.2.2" class="ltx_text" style="font-size:90%;">Varying the learning rate jointly with the number of latents. Number of tokens to convergence shown above each point.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S3.F4.4" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:208.1pt;"><img src="/html/2406.04093/assets/x6.png" id="S3.F4.2.g1" class="ltx_graphics ltx_img_landscape" width="460" height="328" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F4.4.3.2.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S3.F4.4.2.1" class="ltx_text" style="font-size:90%;">Larger subject models in the GPT-4 family require more latents to get to the same MSE (<math id="S3.F4.4.2.1.m1.1" class="ltx_Math" alttext="k=32" display="inline"><semantics id="S3.F4.4.2.1.m1.1b"><mrow id="S3.F4.4.2.1.m1.1.1" xref="S3.F4.4.2.1.m1.1.1.cmml"><mi id="S3.F4.4.2.1.m1.1.1.2" xref="S3.F4.4.2.1.m1.1.1.2.cmml">k</mi><mo id="S3.F4.4.2.1.m1.1.1.1" xref="S3.F4.4.2.1.m1.1.1.1.cmml">=</mo><mn id="S3.F4.4.2.1.m1.1.1.3" xref="S3.F4.4.2.1.m1.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F4.4.2.1.m1.1c"><apply id="S3.F4.4.2.1.m1.1.1.cmml" xref="S3.F4.4.2.1.m1.1.1"><eq id="S3.F4.4.2.1.m1.1.1.1.cmml" xref="S3.F4.4.2.1.m1.1.1.1"></eq><ci id="S3.F4.4.2.1.m1.1.1.2.cmml" xref="S3.F4.4.2.1.m1.1.1.2">𝑘</ci><cn type="integer" id="S3.F4.4.2.1.m1.1.1.3.cmml" xref="S3.F4.4.2.1.m1.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.4.2.1.m1.1d">k=32</annotation></semantics></math>).</span></figcaption>
</figure>
</div>
</div>
</figure>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Number of latents</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">Due to the broad capabilities of frontier models such as GPT-4, we hypothesize that faithfully representing model state will require large numbers of sparse features. We consider two primary approaches to choose autoencoder size and token budget:</p>
</div>
<section id="S3.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.1 </span>Training to compute-MSE frontier (<math id="S3.SS1.SSS1.1.m1.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="S3.SS1.SSS1.1.m1.1b"><mrow id="S3.SS1.SSS1.1.m1.1.2" xref="S3.SS1.SSS1.1.m1.1.2.cmml"><mi id="S3.SS1.SSS1.1.m1.1.2.2" xref="S3.SS1.SSS1.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS1.1.m1.1.2.1" xref="S3.SS1.SSS1.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS1.1.m1.1.2.3.2" xref="S3.SS1.SSS1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS1.1.m1.1.2.3.2.1" xref="S3.SS1.SSS1.1.m1.1.2.cmml">(</mo><mi id="S3.SS1.SSS1.1.m1.1.1" xref="S3.SS1.SSS1.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="S3.SS1.SSS1.1.m1.1.2.3.2.2" xref="S3.SS1.SSS1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.1.m1.1c"><apply id="S3.SS1.SSS1.1.m1.1.2.cmml" xref="S3.SS1.SSS1.1.m1.1.2"><times id="S3.SS1.SSS1.1.m1.1.2.1.cmml" xref="S3.SS1.SSS1.1.m1.1.2.1"></times><ci id="S3.SS1.SSS1.1.m1.1.2.2.cmml" xref="S3.SS1.SSS1.1.m1.1.2.2">𝐿</ci><ci id="S3.SS1.SSS1.1.m1.1.1.cmml" xref="S3.SS1.SSS1.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.1.m1.1d">L(C)</annotation></semantics></math>)</h4>

<div id="S3.SS1.SSS1.p1" class="ltx_para">
<p id="S3.SS1.SSS1.p1.1" class="ltx_p">Firstly, following <cite class="ltx_cite ltx_citemacro_citet">Lindsey et al. (<a href="#bib.bib33" title="" class="ltx_ref">2024</a>)</cite>, we train autoencoders to the optimal MSE given the available compute, disregarding convergence. This method was introduced for pre-training language models <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib25" title="" class="ltx_ref">2020</a>; Hoffmann et al., <a href="#bib.bib22" title="" class="ltx_ref">2022</a>)</cite>. We find that MSE follows a power law <math id="S3.SS1.SSS1.p1.1.m1.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="S3.SS1.SSS1.p1.1.m1.1a"><mrow id="S3.SS1.SSS1.p1.1.m1.1.2" xref="S3.SS1.SSS1.p1.1.m1.1.2.cmml"><mi id="S3.SS1.SSS1.p1.1.m1.1.2.2" xref="S3.SS1.SSS1.p1.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS1.p1.1.m1.1.2.1" xref="S3.SS1.SSS1.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS1.p1.1.m1.1.2.3.2" xref="S3.SS1.SSS1.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS1.p1.1.m1.1.2.3.2.1" xref="S3.SS1.SSS1.p1.1.m1.1.2.cmml">(</mo><mi id="S3.SS1.SSS1.p1.1.m1.1.1" xref="S3.SS1.SSS1.p1.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="S3.SS1.SSS1.p1.1.m1.1.2.3.2.2" xref="S3.SS1.SSS1.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p1.1.m1.1b"><apply id="S3.SS1.SSS1.p1.1.m1.1.2.cmml" xref="S3.SS1.SSS1.p1.1.m1.1.2"><times id="S3.SS1.SSS1.p1.1.m1.1.2.1.cmml" xref="S3.SS1.SSS1.p1.1.m1.1.2.1"></times><ci id="S3.SS1.SSS1.p1.1.m1.1.2.2.cmml" xref="S3.SS1.SSS1.p1.1.m1.1.2.2">𝐿</ci><ci id="S3.SS1.SSS1.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS1.p1.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p1.1.m1.1c">L(C)</annotation></semantics></math> of compute, though the smallest models are off trend
(<a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a>).</p>
</div>
<div id="S3.SS1.SSS1.p2" class="ltx_para">
<p id="S3.SS1.SSS1.p2.2" class="ltx_p">However, latents are the important artifact of training (not reconstruction predictions), whereas for language models we typically care only about token predictions. Comparing MSE across different <math id="S3.SS1.SSS1.p2.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS1.SSS1.p2.1.m1.1a"><mi id="S3.SS1.SSS1.p2.1.m1.1.1" xref="S3.SS1.SSS1.p2.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p2.1.m1.1b"><ci id="S3.SS1.SSS1.p2.1.m1.1.1.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p2.1.m1.1c">n</annotation></semantics></math> is thus not a fair comparison — the latents have a looser information bottleneck with larger <math id="S3.SS1.SSS1.p2.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS1.SSS1.p2.2.m2.1a"><mi id="S3.SS1.SSS1.p2.2.m2.1.1" xref="S3.SS1.SSS1.p2.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p2.2.m2.1b"><ci id="S3.SS1.SSS1.p2.2.m2.1.1.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p2.2.m2.1c">n</annotation></semantics></math>, so lower MSE is more easily achieved. Thus, this approach is arguably unprincipled for autoencoder training.</p>
</div>
</section>
<section id="S3.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.2 </span>Training to convergence (<math id="S3.SS1.SSS2.1.m1.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="S3.SS1.SSS2.1.m1.1b"><mrow id="S3.SS1.SSS2.1.m1.1.2" xref="S3.SS1.SSS2.1.m1.1.2.cmml"><mi id="S3.SS1.SSS2.1.m1.1.2.2" xref="S3.SS1.SSS2.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.1.m1.1.2.1" xref="S3.SS1.SSS2.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS2.1.m1.1.2.3.2" xref="S3.SS1.SSS2.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS2.1.m1.1.2.3.2.1" xref="S3.SS1.SSS2.1.m1.1.2.cmml">(</mo><mi id="S3.SS1.SSS2.1.m1.1.1" xref="S3.SS1.SSS2.1.m1.1.1.cmml">N</mi><mo stretchy="false" id="S3.SS1.SSS2.1.m1.1.2.3.2.2" xref="S3.SS1.SSS2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.1.m1.1c"><apply id="S3.SS1.SSS2.1.m1.1.2.cmml" xref="S3.SS1.SSS2.1.m1.1.2"><times id="S3.SS1.SSS2.1.m1.1.2.1.cmml" xref="S3.SS1.SSS2.1.m1.1.2.1"></times><ci id="S3.SS1.SSS2.1.m1.1.2.2.cmml" xref="S3.SS1.SSS2.1.m1.1.2.2">𝐿</ci><ci id="S3.SS1.SSS2.1.m1.1.1.cmml" xref="S3.SS1.SSS2.1.m1.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.1.m1.1d">L(N)</annotation></semantics></math>)</h4>

<div id="S3.SS1.SSS2.p1" class="ltx_para">
<p id="S3.SS1.SSS2.p1.3" class="ltx_p">We also look at training autoencoders to convergence (within some <math id="S3.SS1.SSS2.p1.1.m1.1" class="ltx_Math" alttext="\epsilon" display="inline"><semantics id="S3.SS1.SSS2.p1.1.m1.1a"><mi id="S3.SS1.SSS2.p1.1.m1.1.1" xref="S3.SS1.SSS2.p1.1.m1.1.1.cmml">ϵ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p1.1.m1.1b"><ci id="S3.SS1.SSS2.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS2.p1.1.m1.1.1">italic-ϵ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p1.1.m1.1c">\epsilon</annotation></semantics></math>). This gives a bound on the best possible reconstruction achievable by our training method if we disregard compute efficiency. In practice, we would ideally train to some intermediate token budget between <math id="S3.SS1.SSS2.p1.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="S3.SS1.SSS2.p1.2.m2.1a"><mrow id="S3.SS1.SSS2.p1.2.m2.1.2" xref="S3.SS1.SSS2.p1.2.m2.1.2.cmml"><mi id="S3.SS1.SSS2.p1.2.m2.1.2.2" xref="S3.SS1.SSS2.p1.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.p1.2.m2.1.2.1" xref="S3.SS1.SSS2.p1.2.m2.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS2.p1.2.m2.1.2.3.2" xref="S3.SS1.SSS2.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS2.p1.2.m2.1.2.3.2.1" xref="S3.SS1.SSS2.p1.2.m2.1.2.cmml">(</mo><mi id="S3.SS1.SSS2.p1.2.m2.1.1" xref="S3.SS1.SSS2.p1.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="S3.SS1.SSS2.p1.2.m2.1.2.3.2.2" xref="S3.SS1.SSS2.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p1.2.m2.1b"><apply id="S3.SS1.SSS2.p1.2.m2.1.2.cmml" xref="S3.SS1.SSS2.p1.2.m2.1.2"><times id="S3.SS1.SSS2.p1.2.m2.1.2.1.cmml" xref="S3.SS1.SSS2.p1.2.m2.1.2.1"></times><ci id="S3.SS1.SSS2.p1.2.m2.1.2.2.cmml" xref="S3.SS1.SSS2.p1.2.m2.1.2.2">𝐿</ci><ci id="S3.SS1.SSS2.p1.2.m2.1.1.cmml" xref="S3.SS1.SSS2.p1.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p1.2.m2.1c">L(N)</annotation></semantics></math> and <math id="S3.SS1.SSS2.p1.3.m3.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="S3.SS1.SSS2.p1.3.m3.1a"><mrow id="S3.SS1.SSS2.p1.3.m3.1.2" xref="S3.SS1.SSS2.p1.3.m3.1.2.cmml"><mi id="S3.SS1.SSS2.p1.3.m3.1.2.2" xref="S3.SS1.SSS2.p1.3.m3.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.p1.3.m3.1.2.1" xref="S3.SS1.SSS2.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS2.p1.3.m3.1.2.3.2" xref="S3.SS1.SSS2.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS2.p1.3.m3.1.2.3.2.1" xref="S3.SS1.SSS2.p1.3.m3.1.2.cmml">(</mo><mi id="S3.SS1.SSS2.p1.3.m3.1.1" xref="S3.SS1.SSS2.p1.3.m3.1.1.cmml">C</mi><mo stretchy="false" id="S3.SS1.SSS2.p1.3.m3.1.2.3.2.2" xref="S3.SS1.SSS2.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p1.3.m3.1b"><apply id="S3.SS1.SSS2.p1.3.m3.1.2.cmml" xref="S3.SS1.SSS2.p1.3.m3.1.2"><times id="S3.SS1.SSS2.p1.3.m3.1.2.1.cmml" xref="S3.SS1.SSS2.p1.3.m3.1.2.1"></times><ci id="S3.SS1.SSS2.p1.3.m3.1.2.2.cmml" xref="S3.SS1.SSS2.p1.3.m3.1.2.2">𝐿</ci><ci id="S3.SS1.SSS2.p1.3.m3.1.1.cmml" xref="S3.SS1.SSS2.p1.3.m3.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p1.3.m3.1c">L(C)</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.SSS2.p2" class="ltx_para">
<p id="S3.SS1.SSS2.p2.3" class="ltx_p">We find that the largest learning rate that converges scales with <math id="S3.SS1.SSS2.p2.1.m1.1" class="ltx_Math" alttext="1/\sqrt{n}" display="inline"><semantics id="S3.SS1.SSS2.p2.1.m1.1a"><mrow id="S3.SS1.SSS2.p2.1.m1.1.1" xref="S3.SS1.SSS2.p2.1.m1.1.1.cmml"><mn id="S3.SS1.SSS2.p2.1.m1.1.1.2" xref="S3.SS1.SSS2.p2.1.m1.1.1.2.cmml">1</mn><mo id="S3.SS1.SSS2.p2.1.m1.1.1.1" xref="S3.SS1.SSS2.p2.1.m1.1.1.1.cmml">/</mo><msqrt id="S3.SS1.SSS2.p2.1.m1.1.1.3" xref="S3.SS1.SSS2.p2.1.m1.1.1.3.cmml"><mi id="S3.SS1.SSS2.p2.1.m1.1.1.3.2" xref="S3.SS1.SSS2.p2.1.m1.1.1.3.2.cmml">n</mi></msqrt></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p2.1.m1.1b"><apply id="S3.SS1.SSS2.p2.1.m1.1.1.cmml" xref="S3.SS1.SSS2.p2.1.m1.1.1"><divide id="S3.SS1.SSS2.p2.1.m1.1.1.1.cmml" xref="S3.SS1.SSS2.p2.1.m1.1.1.1"></divide><cn type="integer" id="S3.SS1.SSS2.p2.1.m1.1.1.2.cmml" xref="S3.SS1.SSS2.p2.1.m1.1.1.2">1</cn><apply id="S3.SS1.SSS2.p2.1.m1.1.1.3.cmml" xref="S3.SS1.SSS2.p2.1.m1.1.1.3"><root id="S3.SS1.SSS2.p2.1.m1.1.1.3a.cmml" xref="S3.SS1.SSS2.p2.1.m1.1.1.3"></root><ci id="S3.SS1.SSS2.p2.1.m1.1.1.3.2.cmml" xref="S3.SS1.SSS2.p2.1.m1.1.1.3.2">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p2.1.m1.1c">1/\sqrt{n}</annotation></semantics></math> (<a href="#S3.F4" title="Figure 4 ‣ 3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a>). We also find that the optimal learning rate for <math id="S3.SS1.SSS2.p2.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="S3.SS1.SSS2.p2.2.m2.1a"><mrow id="S3.SS1.SSS2.p2.2.m2.1.2" xref="S3.SS1.SSS2.p2.2.m2.1.2.cmml"><mi id="S3.SS1.SSS2.p2.2.m2.1.2.2" xref="S3.SS1.SSS2.p2.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.p2.2.m2.1.2.1" xref="S3.SS1.SSS2.p2.2.m2.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS2.p2.2.m2.1.2.3.2" xref="S3.SS1.SSS2.p2.2.m2.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS2.p2.2.m2.1.2.3.2.1" xref="S3.SS1.SSS2.p2.2.m2.1.2.cmml">(</mo><mi id="S3.SS1.SSS2.p2.2.m2.1.1" xref="S3.SS1.SSS2.p2.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="S3.SS1.SSS2.p2.2.m2.1.2.3.2.2" xref="S3.SS1.SSS2.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p2.2.m2.1b"><apply id="S3.SS1.SSS2.p2.2.m2.1.2.cmml" xref="S3.SS1.SSS2.p2.2.m2.1.2"><times id="S3.SS1.SSS2.p2.2.m2.1.2.1.cmml" xref="S3.SS1.SSS2.p2.2.m2.1.2.1"></times><ci id="S3.SS1.SSS2.p2.2.m2.1.2.2.cmml" xref="S3.SS1.SSS2.p2.2.m2.1.2.2">𝐿</ci><ci id="S3.SS1.SSS2.p2.2.m2.1.1.cmml" xref="S3.SS1.SSS2.p2.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p2.2.m2.1c">L(N)</annotation></semantics></math> is about four times smaller than the optimal learning rate for <math id="S3.SS1.SSS2.p2.3.m3.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="S3.SS1.SSS2.p2.3.m3.1a"><mrow id="S3.SS1.SSS2.p2.3.m3.1.2" xref="S3.SS1.SSS2.p2.3.m3.1.2.cmml"><mi id="S3.SS1.SSS2.p2.3.m3.1.2.2" xref="S3.SS1.SSS2.p2.3.m3.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.p2.3.m3.1.2.1" xref="S3.SS1.SSS2.p2.3.m3.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS2.p2.3.m3.1.2.3.2" xref="S3.SS1.SSS2.p2.3.m3.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS2.p2.3.m3.1.2.3.2.1" xref="S3.SS1.SSS2.p2.3.m3.1.2.cmml">(</mo><mi id="S3.SS1.SSS2.p2.3.m3.1.1" xref="S3.SS1.SSS2.p2.3.m3.1.1.cmml">C</mi><mo stretchy="false" id="S3.SS1.SSS2.p2.3.m3.1.2.3.2.2" xref="S3.SS1.SSS2.p2.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p2.3.m3.1b"><apply id="S3.SS1.SSS2.p2.3.m3.1.2.cmml" xref="S3.SS1.SSS2.p2.3.m3.1.2"><times id="S3.SS1.SSS2.p2.3.m3.1.2.1.cmml" xref="S3.SS1.SSS2.p2.3.m3.1.2.1"></times><ci id="S3.SS1.SSS2.p2.3.m3.1.2.2.cmml" xref="S3.SS1.SSS2.p2.3.m3.1.2.2">𝐿</ci><ci id="S3.SS1.SSS2.p2.3.m3.1.1.cmml" xref="S3.SS1.SSS2.p2.3.m3.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p2.3.m3.1c">L(C)</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.SSS2.p3" class="ltx_para">
<p id="S3.SS1.SSS2.p3.2" class="ltx_p">We find that the number of tokens to convergence increases as approximately <math id="S3.SS1.SSS2.p3.1.m1.1" class="ltx_Math" alttext="\Theta(n^{0.6})" display="inline"><semantics id="S3.SS1.SSS2.p3.1.m1.1a"><mrow id="S3.SS1.SSS2.p3.1.m1.1.1" xref="S3.SS1.SSS2.p3.1.m1.1.1.cmml"><mi mathvariant="normal" id="S3.SS1.SSS2.p3.1.m1.1.1.3" xref="S3.SS1.SSS2.p3.1.m1.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.p3.1.m1.1.1.2" xref="S3.SS1.SSS2.p3.1.m1.1.1.2.cmml">​</mo><mrow id="S3.SS1.SSS2.p3.1.m1.1.1.1.1" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.2" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.cmml"><mi id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.2" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.2.cmml">n</mi><mn id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.3" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.3.cmml">0.6</mn></msup><mo stretchy="false" id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.3" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p3.1.m1.1b"><apply id="S3.SS1.SSS2.p3.1.m1.1.1.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1"><times id="S3.SS1.SSS2.p3.1.m1.1.1.2.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1.2"></times><ci id="S3.SS1.SSS2.p3.1.m1.1.1.3.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1.3">Θ</ci><apply id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.2.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.2">𝑛</ci><cn type="float" id="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.3.cmml" xref="S3.SS1.SSS2.p3.1.m1.1.1.1.1.1.3">0.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p3.1.m1.1c">\Theta(n^{0.6})</annotation></semantics></math> for GPT-2 small and <math id="S3.SS1.SSS2.p3.2.m2.1" class="ltx_Math" alttext="\Theta(n^{0.65})" display="inline"><semantics id="S3.SS1.SSS2.p3.2.m2.1a"><mrow id="S3.SS1.SSS2.p3.2.m2.1.1" xref="S3.SS1.SSS2.p3.2.m2.1.1.cmml"><mi mathvariant="normal" id="S3.SS1.SSS2.p3.2.m2.1.1.3" xref="S3.SS1.SSS2.p3.2.m2.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS2.p3.2.m2.1.1.2" xref="S3.SS1.SSS2.p3.2.m2.1.1.2.cmml">​</mo><mrow id="S3.SS1.SSS2.p3.2.m2.1.1.1.1" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.2" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.cmml"><mi id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.2" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.2.cmml">n</mi><mn id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.3" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.3.cmml">0.65</mn></msup><mo stretchy="false" id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.3" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS2.p3.2.m2.1b"><apply id="S3.SS1.SSS2.p3.2.m2.1.1.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1"><times id="S3.SS1.SSS2.p3.2.m2.1.1.2.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1.2"></times><ci id="S3.SS1.SSS2.p3.2.m2.1.1.3.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1.3">Θ</ci><apply id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1">superscript</csymbol><ci id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.2.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.2">𝑛</ci><cn type="float" id="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.3.cmml" xref="S3.SS1.SSS2.p3.2.m2.1.1.1.1.1.3">0.65</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS2.p3.2.m2.1c">\Theta(n^{0.65})</annotation></semantics></math> for GPT-4 (<a href="#A1.F11" title="Figure 11 ‣ Appendix A Optimization ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 11</span></a>). This must break at some point – if token budget continues to increase sublinearly, the number of tokens each latent receives gradient signal on would approach zero.<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>One slight complication is that in the infinite width limit, TopK autoencoders with our initialization scheme are actually optimal at init using our init scheme (<a href="#A1.SS1" title="A.1 Initialization ‣ Appendix A Optimization ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section A.1</span></a>), so this allows for an exponent very slightly less than <math id="footnote7.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="footnote7.m1.1b"><mn id="footnote7.m1.1.1" xref="footnote7.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="footnote7.m1.1c"><cn type="integer" id="footnote7.m1.1.1.cmml" xref="footnote7.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m1.1d">1</annotation></semantics></math>; however, this happens very slowly with <math id="footnote7.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="footnote7.m2.1b"><mi id="footnote7.m2.1.1" xref="footnote7.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="footnote7.m2.1c"><ci id="footnote7.m2.1.1.cmml" xref="footnote7.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m2.1d">n</annotation></semantics></math> so is unlikely to be a major factor at realistic scales.</span></span></span></p>
</div>
</section>
<section id="S3.SS1.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.3 </span>Irreducible loss</h4>

<div id="S3.SS1.SSS3.p1" class="ltx_para">
<p id="S3.SS1.SSS3.p1.4" class="ltx_p">Scaling laws sometimes include an irreducible loss term <math id="S3.SS1.SSS3.p1.1.m1.1" class="ltx_Math" alttext="e" display="inline"><semantics id="S3.SS1.SSS3.p1.1.m1.1a"><mi id="S3.SS1.SSS3.p1.1.m1.1.1" xref="S3.SS1.SSS3.p1.1.m1.1.1.cmml">e</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS3.p1.1.m1.1b"><ci id="S3.SS1.SSS3.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS3.p1.1.m1.1.1">𝑒</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS3.p1.1.m1.1c">e</annotation></semantics></math>, such that <math id="S3.SS1.SSS3.p1.2.m2.1" class="ltx_Math" alttext="y=\alpha x^{\beta}+e" display="inline"><semantics id="S3.SS1.SSS3.p1.2.m2.1a"><mrow id="S3.SS1.SSS3.p1.2.m2.1.1" xref="S3.SS1.SSS3.p1.2.m2.1.1.cmml"><mi id="S3.SS1.SSS3.p1.2.m2.1.1.2" xref="S3.SS1.SSS3.p1.2.m2.1.1.2.cmml">y</mi><mo id="S3.SS1.SSS3.p1.2.m2.1.1.1" xref="S3.SS1.SSS3.p1.2.m2.1.1.1.cmml">=</mo><mrow id="S3.SS1.SSS3.p1.2.m2.1.1.3" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.cmml"><mrow id="S3.SS1.SSS3.p1.2.m2.1.1.3.2" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.cmml"><mi id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.2" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.1" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.1.cmml">​</mo><msup id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.cmml"><mi id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.2" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.2.cmml">x</mi><mi id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.3" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.3.cmml">β</mi></msup></mrow><mo id="S3.SS1.SSS3.p1.2.m2.1.1.3.1" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.1.cmml">+</mo><mi id="S3.SS1.SSS3.p1.2.m2.1.1.3.3" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.3.cmml">e</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS3.p1.2.m2.1b"><apply id="S3.SS1.SSS3.p1.2.m2.1.1.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1"><eq id="S3.SS1.SSS3.p1.2.m2.1.1.1.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.1"></eq><ci id="S3.SS1.SSS3.p1.2.m2.1.1.2.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.2">𝑦</ci><apply id="S3.SS1.SSS3.p1.2.m2.1.1.3.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3"><plus id="S3.SS1.SSS3.p1.2.m2.1.1.3.1.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.1"></plus><apply id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2"><times id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.1.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.1"></times><ci id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.2.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.2">𝛼</ci><apply id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.1.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3">superscript</csymbol><ci id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.2.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.2">𝑥</ci><ci id="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.3.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.2.3.3">𝛽</ci></apply></apply><ci id="S3.SS1.SSS3.p1.2.m2.1.1.3.3.cmml" xref="S3.SS1.SSS3.p1.2.m2.1.1.3.3">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS3.p1.2.m2.1c">y=\alpha x^{\beta}+e</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Henighan et al., <a href="#bib.bib20" title="" class="ltx_ref">2020</a>)</cite>. We find that including an irreducible loss term substantially improves the quality of our fits for both <math id="S3.SS1.SSS3.p1.3.m3.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="S3.SS1.SSS3.p1.3.m3.1a"><mrow id="S3.SS1.SSS3.p1.3.m3.1.2" xref="S3.SS1.SSS3.p1.3.m3.1.2.cmml"><mi id="S3.SS1.SSS3.p1.3.m3.1.2.2" xref="S3.SS1.SSS3.p1.3.m3.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS3.p1.3.m3.1.2.1" xref="S3.SS1.SSS3.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS3.p1.3.m3.1.2.3.2" xref="S3.SS1.SSS3.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS3.p1.3.m3.1.2.3.2.1" xref="S3.SS1.SSS3.p1.3.m3.1.2.cmml">(</mo><mi id="S3.SS1.SSS3.p1.3.m3.1.1" xref="S3.SS1.SSS3.p1.3.m3.1.1.cmml">C</mi><mo stretchy="false" id="S3.SS1.SSS3.p1.3.m3.1.2.3.2.2" xref="S3.SS1.SSS3.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS3.p1.3.m3.1b"><apply id="S3.SS1.SSS3.p1.3.m3.1.2.cmml" xref="S3.SS1.SSS3.p1.3.m3.1.2"><times id="S3.SS1.SSS3.p1.3.m3.1.2.1.cmml" xref="S3.SS1.SSS3.p1.3.m3.1.2.1"></times><ci id="S3.SS1.SSS3.p1.3.m3.1.2.2.cmml" xref="S3.SS1.SSS3.p1.3.m3.1.2.2">𝐿</ci><ci id="S3.SS1.SSS3.p1.3.m3.1.1.cmml" xref="S3.SS1.SSS3.p1.3.m3.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS3.p1.3.m3.1c">L(C)</annotation></semantics></math> and <math id="S3.SS1.SSS3.p1.4.m4.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="S3.SS1.SSS3.p1.4.m4.1a"><mrow id="S3.SS1.SSS3.p1.4.m4.1.2" xref="S3.SS1.SSS3.p1.4.m4.1.2.cmml"><mi id="S3.SS1.SSS3.p1.4.m4.1.2.2" xref="S3.SS1.SSS3.p1.4.m4.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS3.p1.4.m4.1.2.1" xref="S3.SS1.SSS3.p1.4.m4.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS3.p1.4.m4.1.2.3.2" xref="S3.SS1.SSS3.p1.4.m4.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS3.p1.4.m4.1.2.3.2.1" xref="S3.SS1.SSS3.p1.4.m4.1.2.cmml">(</mo><mi id="S3.SS1.SSS3.p1.4.m4.1.1" xref="S3.SS1.SSS3.p1.4.m4.1.1.cmml">N</mi><mo stretchy="false" id="S3.SS1.SSS3.p1.4.m4.1.2.3.2.2" xref="S3.SS1.SSS3.p1.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS3.p1.4.m4.1b"><apply id="S3.SS1.SSS3.p1.4.m4.1.2.cmml" xref="S3.SS1.SSS3.p1.4.m4.1.2"><times id="S3.SS1.SSS3.p1.4.m4.1.2.1.cmml" xref="S3.SS1.SSS3.p1.4.m4.1.2.1"></times><ci id="S3.SS1.SSS3.p1.4.m4.1.2.2.cmml" xref="S3.SS1.SSS3.p1.4.m4.1.2.2">𝐿</ci><ci id="S3.SS1.SSS3.p1.4.m4.1.1.cmml" xref="S3.SS1.SSS3.p1.4.m4.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS3.p1.4.m4.1c">L(N)</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.SSS3.p2" class="ltx_para">
<p id="S3.SS1.SSS3.p2.1" class="ltx_p">It was initially not clear to us that there should be a nonzero irreducible loss.
One possibility is that there are other kinds of structures in the activations. In the extreme case, unstructured noise in the activations is substantially harder to model and would have an exponent close to zero (<a href="#A7" title="Appendix G Irreducible loss term ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix G</span></a>). Existence of some unstructured noise would explain a bend in the power law.</p>
</div>
</section>
<section id="S3.SS1.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.4 </span>Jointly fitting sparsity (<math id="S3.SS1.SSS4.1.m1.2" class="ltx_Math" alttext="L(N,K)" display="inline"><semantics id="S3.SS1.SSS4.1.m1.2b"><mrow id="S3.SS1.SSS4.1.m1.2.3" xref="S3.SS1.SSS4.1.m1.2.3.cmml"><mi id="S3.SS1.SSS4.1.m1.2.3.2" xref="S3.SS1.SSS4.1.m1.2.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS4.1.m1.2.3.1" xref="S3.SS1.SSS4.1.m1.2.3.1.cmml">​</mo><mrow id="S3.SS1.SSS4.1.m1.2.3.3.2" xref="S3.SS1.SSS4.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS1.SSS4.1.m1.2.3.3.2.1" xref="S3.SS1.SSS4.1.m1.2.3.3.1.cmml">(</mo><mi id="S3.SS1.SSS4.1.m1.1.1" xref="S3.SS1.SSS4.1.m1.1.1.cmml">N</mi><mo id="S3.SS1.SSS4.1.m1.2.3.3.2.2" xref="S3.SS1.SSS4.1.m1.2.3.3.1.cmml">,</mo><mi id="S3.SS1.SSS4.1.m1.2.2" xref="S3.SS1.SSS4.1.m1.2.2.cmml">K</mi><mo stretchy="false" id="S3.SS1.SSS4.1.m1.2.3.3.2.3" xref="S3.SS1.SSS4.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.1.m1.2c"><apply id="S3.SS1.SSS4.1.m1.2.3.cmml" xref="S3.SS1.SSS4.1.m1.2.3"><times id="S3.SS1.SSS4.1.m1.2.3.1.cmml" xref="S3.SS1.SSS4.1.m1.2.3.1"></times><ci id="S3.SS1.SSS4.1.m1.2.3.2.cmml" xref="S3.SS1.SSS4.1.m1.2.3.2">𝐿</ci><interval closure="open" id="S3.SS1.SSS4.1.m1.2.3.3.1.cmml" xref="S3.SS1.SSS4.1.m1.2.3.3.2"><ci id="S3.SS1.SSS4.1.m1.1.1.cmml" xref="S3.SS1.SSS4.1.m1.1.1">𝑁</ci><ci id="S3.SS1.SSS4.1.m1.2.2.cmml" xref="S3.SS1.SSS4.1.m1.2.2">𝐾</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.1.m1.2d">L(N,K)</annotation></semantics></math>)</h4>

<div id="S3.SS1.SSS4.p1" class="ltx_para">
<p id="S3.SS1.SSS4.p1.5" class="ltx_p">We find that MSE follows a joint scaling law along the number of latents <math id="S3.SS1.SSS4.p1.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS1.SSS4.p1.1.m1.1a"><mi id="S3.SS1.SSS4.p1.1.m1.1.1" xref="S3.SS1.SSS4.p1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.1.m1.1b"><ci id="S3.SS1.SSS4.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS4.p1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.1.m1.1c">n</annotation></semantics></math> and the sparsity level <math id="S3.SS1.SSS4.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.SSS4.p1.2.m2.1a"><mi id="S3.SS1.SSS4.p1.2.m2.1.1" xref="S3.SS1.SSS4.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.2.m2.1b"><ci id="S3.SS1.SSS4.p1.2.m2.1.1.cmml" xref="S3.SS1.SSS4.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.2.m2.1c">k</annotation></semantics></math> (<a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a>b). Because reconstruction becomes trivial as <math id="S3.SS1.SSS4.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.SSS4.p1.3.m3.1a"><mi id="S3.SS1.SSS4.p1.3.m3.1.1" xref="S3.SS1.SSS4.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.3.m3.1b"><ci id="S3.SS1.SSS4.p1.3.m3.1.1.cmml" xref="S3.SS1.SSS4.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.3.m3.1c">k</annotation></semantics></math> approaches <math id="S3.SS1.SSS4.p1.4.m4.1" class="ltx_Math" alttext="d_{model}" display="inline"><semantics id="S3.SS1.SSS4.p1.4.m4.1a"><msub id="S3.SS1.SSS4.p1.4.m4.1.1" xref="S3.SS1.SSS4.p1.4.m4.1.1.cmml"><mi id="S3.SS1.SSS4.p1.4.m4.1.1.2" xref="S3.SS1.SSS4.p1.4.m4.1.1.2.cmml">d</mi><mrow id="S3.SS1.SSS4.p1.4.m4.1.1.3" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.cmml"><mi id="S3.SS1.SSS4.p1.4.m4.1.1.3.2" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS4.p1.4.m4.1.1.3.1" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S3.SS1.SSS4.p1.4.m4.1.1.3.3" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS4.p1.4.m4.1.1.3.1a" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S3.SS1.SSS4.p1.4.m4.1.1.3.4" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS4.p1.4.m4.1.1.3.1b" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S3.SS1.SSS4.p1.4.m4.1.1.3.5" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS4.p1.4.m4.1.1.3.1c" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S3.SS1.SSS4.p1.4.m4.1.1.3.6" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.6.cmml">l</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.4.m4.1b"><apply id="S3.SS1.SSS4.p1.4.m4.1.1.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS4.p1.4.m4.1.1.1.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS1.SSS4.p1.4.m4.1.1.2.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.2">𝑑</ci><apply id="S3.SS1.SSS4.p1.4.m4.1.1.3.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3"><times id="S3.SS1.SSS4.p1.4.m4.1.1.3.1.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.1"></times><ci id="S3.SS1.SSS4.p1.4.m4.1.1.3.2.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.2">𝑚</ci><ci id="S3.SS1.SSS4.p1.4.m4.1.1.3.3.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.3">𝑜</ci><ci id="S3.SS1.SSS4.p1.4.m4.1.1.3.4.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.4">𝑑</ci><ci id="S3.SS1.SSS4.p1.4.m4.1.1.3.5.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.5">𝑒</ci><ci id="S3.SS1.SSS4.p1.4.m4.1.1.3.6.cmml" xref="S3.SS1.SSS4.p1.4.m4.1.1.3.6">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.4.m4.1c">d_{model}</annotation></semantics></math>,
this scaling law only holds for the small <math id="S3.SS1.SSS4.p1.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.SSS4.p1.5.m5.1a"><mi id="S3.SS1.SSS4.p1.5.m5.1.1" xref="S3.SS1.SSS4.p1.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.5.m5.1b"><ci id="S3.SS1.SSS4.p1.5.m5.1.1.cmml" xref="S3.SS1.SSS4.p1.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.5.m5.1c">k</annotation></semantics></math> regime.
Our joint scaling law fit on GPT-4 autoencoders is:</p>
<table id="S3.E3" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E3X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3X.2.1.1.m1.16" class="ltx_Math" alttext="\displaystyle L(n,k)=\exp(\alpha+\beta_{k}\log(k)+\beta_{n}\log(n)+\gamma\log(k)\log(n))+\exp(\zeta+\eta\log(k))" display="inline"><semantics id="S3.E3X.2.1.1.m1.16a"><mrow id="S3.E3X.2.1.1.m1.16.16" xref="S3.E3X.2.1.1.m1.16.16.cmml"><mrow id="S3.E3X.2.1.1.m1.16.16.4" xref="S3.E3X.2.1.1.m1.16.16.4.cmml"><mi id="S3.E3X.2.1.1.m1.16.16.4.2" xref="S3.E3X.2.1.1.m1.16.16.4.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.E3X.2.1.1.m1.16.16.4.1" xref="S3.E3X.2.1.1.m1.16.16.4.1.cmml">​</mo><mrow id="S3.E3X.2.1.1.m1.16.16.4.3.2" xref="S3.E3X.2.1.1.m1.16.16.4.3.1.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.16.16.4.3.2.1" xref="S3.E3X.2.1.1.m1.16.16.4.3.1.cmml">(</mo><mi id="S3.E3X.2.1.1.m1.1.1" xref="S3.E3X.2.1.1.m1.1.1.cmml">n</mi><mo id="S3.E3X.2.1.1.m1.16.16.4.3.2.2" xref="S3.E3X.2.1.1.m1.16.16.4.3.1.cmml">,</mo><mi id="S3.E3X.2.1.1.m1.2.2" xref="S3.E3X.2.1.1.m1.2.2.cmml">k</mi><mo stretchy="false" id="S3.E3X.2.1.1.m1.16.16.4.3.2.3" xref="S3.E3X.2.1.1.m1.16.16.4.3.1.cmml">)</mo></mrow></mrow><mo id="S3.E3X.2.1.1.m1.16.16.3" xref="S3.E3X.2.1.1.m1.16.16.3.cmml">=</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2" xref="S3.E3X.2.1.1.m1.16.16.2.cmml"><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.2.cmml"><mi id="S3.E3X.2.1.1.m1.11.11" xref="S3.E3X.2.1.1.m1.11.11.cmml">exp</mi><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1a" xref="S3.E3X.2.1.1.m1.15.15.1.1.2.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.2.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.2.cmml">(</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.cmml"><mi id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.2.cmml">α</mi><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.cmml"><msub id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.cmml"><mi id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.2.cmml">β</mi><mi id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.3" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.3.cmml">k</mi></msub><mo lspace="0.167em" rspace="0em" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.1.cmml">​</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.1.cmml"><mi id="S3.E3X.2.1.1.m1.3.3" xref="S3.E3X.2.1.1.m1.3.3.cmml">log</mi><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.2a" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.1.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.2.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.1.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.2.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.1.cmml">(</mo><mi id="S3.E3X.2.1.1.m1.4.4" xref="S3.E3X.2.1.1.m1.4.4.cmml">k</mi><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.2.1.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1a" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.cmml"><msub id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.cmml"><mi id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.2.cmml">β</mi><mi id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.3" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.3.cmml">n</mi></msub><mo lspace="0.167em" rspace="0em" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.1.cmml">​</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.1.cmml"><mi id="S3.E3X.2.1.1.m1.5.5" xref="S3.E3X.2.1.1.m1.5.5.cmml">log</mi><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.2a" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.1.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.2.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.1.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.2.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.1.cmml">(</mo><mi id="S3.E3X.2.1.1.m1.6.6" xref="S3.E3X.2.1.1.m1.6.6.cmml">n</mi><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.2.1.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1b" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.cmml"><mi id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.2.cmml">γ</mi><mo lspace="0.167em" rspace="0em" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.1.cmml">​</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.1.cmml"><mi id="S3.E3X.2.1.1.m1.7.7" xref="S3.E3X.2.1.1.m1.7.7.cmml">log</mi><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.2a" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.1.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.2.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.1.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.2.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.1.cmml">(</mo><mi id="S3.E3X.2.1.1.m1.8.8" xref="S3.E3X.2.1.1.m1.8.8.cmml">k</mi><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.2.1.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.1.cmml">)</mo></mrow></mrow><mo lspace="0.167em" rspace="0em" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.1a" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.1.cmml">​</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.1.cmml"><mi id="S3.E3X.2.1.1.m1.9.9" xref="S3.E3X.2.1.1.m1.9.9.cmml">log</mi><mo id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.2a" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.1.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.2.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.1.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.2.1.1" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.1.cmml">(</mo><mi id="S3.E3X.2.1.1.m1.10.10" xref="S3.E3X.2.1.1.m1.10.10.cmml">n</mi><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.2.1.2" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo stretchy="false" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.3" xref="S3.E3X.2.1.1.m1.15.15.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.E3X.2.1.1.m1.16.16.2.3" xref="S3.E3X.2.1.1.m1.16.16.2.3.cmml">+</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2.2.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.2.cmml"><mi id="S3.E3X.2.1.1.m1.14.14" xref="S3.E3X.2.1.1.m1.14.14.cmml">exp</mi><mo id="S3.E3X.2.1.1.m1.16.16.2.2.1a" xref="S3.E3X.2.1.1.m1.16.16.2.2.2.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2.2.1.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.2.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.2" xref="S3.E3X.2.1.1.m1.16.16.2.2.2.cmml">(</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.cmml"><mi id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.2" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.2.cmml">ζ</mi><mo id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.1.cmml">+</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.cmml"><mi id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.2" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.2.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.1.cmml">​</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.2" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.1.cmml"><mi id="S3.E3X.2.1.1.m1.12.12" xref="S3.E3X.2.1.1.m1.12.12.cmml">log</mi><mo id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.2a" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.1.cmml">⁡</mo><mrow id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.2.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.1.cmml"><mo stretchy="false" id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.2.1.1" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.1.cmml">(</mo><mi id="S3.E3X.2.1.1.m1.13.13" xref="S3.E3X.2.1.1.m1.13.13.cmml">k</mi><mo stretchy="false" id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.2.1.2" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo stretchy="false" id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.3" xref="S3.E3X.2.1.1.m1.16.16.2.2.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3X.2.1.1.m1.16b"><apply id="S3.E3X.2.1.1.m1.16.16.cmml" xref="S3.E3X.2.1.1.m1.16.16"><eq id="S3.E3X.2.1.1.m1.16.16.3.cmml" xref="S3.E3X.2.1.1.m1.16.16.3"></eq><apply id="S3.E3X.2.1.1.m1.16.16.4.cmml" xref="S3.E3X.2.1.1.m1.16.16.4"><times id="S3.E3X.2.1.1.m1.16.16.4.1.cmml" xref="S3.E3X.2.1.1.m1.16.16.4.1"></times><ci id="S3.E3X.2.1.1.m1.16.16.4.2.cmml" xref="S3.E3X.2.1.1.m1.16.16.4.2">𝐿</ci><interval closure="open" id="S3.E3X.2.1.1.m1.16.16.4.3.1.cmml" xref="S3.E3X.2.1.1.m1.16.16.4.3.2"><ci id="S3.E3X.2.1.1.m1.1.1.cmml" xref="S3.E3X.2.1.1.m1.1.1">𝑛</ci><ci id="S3.E3X.2.1.1.m1.2.2.cmml" xref="S3.E3X.2.1.1.m1.2.2">𝑘</ci></interval></apply><apply id="S3.E3X.2.1.1.m1.16.16.2.cmml" xref="S3.E3X.2.1.1.m1.16.16.2"><plus id="S3.E3X.2.1.1.m1.16.16.2.3.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.3"></plus><apply id="S3.E3X.2.1.1.m1.15.15.1.1.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1"><exp id="S3.E3X.2.1.1.m1.11.11.cmml" xref="S3.E3X.2.1.1.m1.11.11"></exp><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1"><plus id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.1"></plus><ci id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.2">𝛼</ci><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3"><times id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.1"></times><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.2">𝛽</ci><ci id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.3.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.2.3">𝑘</ci></apply><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.3.3.2"><log id="S3.E3X.2.1.1.m1.3.3.cmml" xref="S3.E3X.2.1.1.m1.3.3"></log><ci id="S3.E3X.2.1.1.m1.4.4.cmml" xref="S3.E3X.2.1.1.m1.4.4">𝑘</ci></apply></apply><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4"><times id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.1"></times><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2"><csymbol cd="ambiguous" id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2">subscript</csymbol><ci id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.2">𝛽</ci><ci id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.3.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.2.3">𝑛</ci></apply><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.4.3.2"><log id="S3.E3X.2.1.1.m1.5.5.cmml" xref="S3.E3X.2.1.1.m1.5.5"></log><ci id="S3.E3X.2.1.1.m1.6.6.cmml" xref="S3.E3X.2.1.1.m1.6.6">𝑛</ci></apply></apply><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5"><times id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.1"></times><ci id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.2.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.2">𝛾</ci><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.3.2"><log id="S3.E3X.2.1.1.m1.7.7.cmml" xref="S3.E3X.2.1.1.m1.7.7"></log><ci id="S3.E3X.2.1.1.m1.8.8.cmml" xref="S3.E3X.2.1.1.m1.8.8">𝑘</ci></apply><apply id="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.1.cmml" xref="S3.E3X.2.1.1.m1.15.15.1.1.1.1.1.5.4.2"><log id="S3.E3X.2.1.1.m1.9.9.cmml" xref="S3.E3X.2.1.1.m1.9.9"></log><ci id="S3.E3X.2.1.1.m1.10.10.cmml" xref="S3.E3X.2.1.1.m1.10.10">𝑛</ci></apply></apply></apply></apply><apply id="S3.E3X.2.1.1.m1.16.16.2.2.2.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1"><exp id="S3.E3X.2.1.1.m1.14.14.cmml" xref="S3.E3X.2.1.1.m1.14.14"></exp><apply id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1"><plus id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.1.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.1"></plus><ci id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.2.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.2">𝜁</ci><apply id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3"><times id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.1.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.1"></times><ci id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.2.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.2">𝜂</ci><apply id="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.1.cmml" xref="S3.E3X.2.1.1.m1.16.16.2.2.1.1.1.3.3.2"><log id="S3.E3X.2.1.1.m1.12.12.cmml" xref="S3.E3X.2.1.1.m1.12.12"></log><ci id="S3.E3X.2.1.1.m1.13.13.cmml" xref="S3.E3X.2.1.1.m1.13.13">𝑘</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3X.2.1.1.m1.16c">\displaystyle L(n,k)=\exp(\alpha+\beta_{k}\log(k)+\beta_{n}\log(n)+\gamma\log(k)\log(n))+\exp(\zeta+\eta\log(k))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(3)</span></td>
</tr>
</tbody>
</table>
<p id="S3.SS1.SSS4.p1.16" class="ltx_p">with <math id="S3.SS1.SSS4.p1.6.m1.1" class="ltx_Math" alttext="\alpha=-0.50" display="inline"><semantics id="S3.SS1.SSS4.p1.6.m1.1a"><mrow id="S3.SS1.SSS4.p1.6.m1.1.1" xref="S3.SS1.SSS4.p1.6.m1.1.1.cmml"><mi id="S3.SS1.SSS4.p1.6.m1.1.1.2" xref="S3.SS1.SSS4.p1.6.m1.1.1.2.cmml">α</mi><mo id="S3.SS1.SSS4.p1.6.m1.1.1.1" xref="S3.SS1.SSS4.p1.6.m1.1.1.1.cmml">=</mo><mrow id="S3.SS1.SSS4.p1.6.m1.1.1.3" xref="S3.SS1.SSS4.p1.6.m1.1.1.3.cmml"><mo id="S3.SS1.SSS4.p1.6.m1.1.1.3a" xref="S3.SS1.SSS4.p1.6.m1.1.1.3.cmml">−</mo><mn id="S3.SS1.SSS4.p1.6.m1.1.1.3.2" xref="S3.SS1.SSS4.p1.6.m1.1.1.3.2.cmml">0.50</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.6.m1.1b"><apply id="S3.SS1.SSS4.p1.6.m1.1.1.cmml" xref="S3.SS1.SSS4.p1.6.m1.1.1"><eq id="S3.SS1.SSS4.p1.6.m1.1.1.1.cmml" xref="S3.SS1.SSS4.p1.6.m1.1.1.1"></eq><ci id="S3.SS1.SSS4.p1.6.m1.1.1.2.cmml" xref="S3.SS1.SSS4.p1.6.m1.1.1.2">𝛼</ci><apply id="S3.SS1.SSS4.p1.6.m1.1.1.3.cmml" xref="S3.SS1.SSS4.p1.6.m1.1.1.3"><minus id="S3.SS1.SSS4.p1.6.m1.1.1.3.1.cmml" xref="S3.SS1.SSS4.p1.6.m1.1.1.3"></minus><cn type="float" id="S3.SS1.SSS4.p1.6.m1.1.1.3.2.cmml" xref="S3.SS1.SSS4.p1.6.m1.1.1.3.2">0.50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.6.m1.1c">\alpha=-0.50</annotation></semantics></math>, <math id="S3.SS1.SSS4.p1.7.m2.1" class="ltx_Math" alttext="\beta_{k}=0.26" display="inline"><semantics id="S3.SS1.SSS4.p1.7.m2.1a"><mrow id="S3.SS1.SSS4.p1.7.m2.1.1" xref="S3.SS1.SSS4.p1.7.m2.1.1.cmml"><msub id="S3.SS1.SSS4.p1.7.m2.1.1.2" xref="S3.SS1.SSS4.p1.7.m2.1.1.2.cmml"><mi id="S3.SS1.SSS4.p1.7.m2.1.1.2.2" xref="S3.SS1.SSS4.p1.7.m2.1.1.2.2.cmml">β</mi><mi id="S3.SS1.SSS4.p1.7.m2.1.1.2.3" xref="S3.SS1.SSS4.p1.7.m2.1.1.2.3.cmml">k</mi></msub><mo id="S3.SS1.SSS4.p1.7.m2.1.1.1" xref="S3.SS1.SSS4.p1.7.m2.1.1.1.cmml">=</mo><mn id="S3.SS1.SSS4.p1.7.m2.1.1.3" xref="S3.SS1.SSS4.p1.7.m2.1.1.3.cmml">0.26</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.7.m2.1b"><apply id="S3.SS1.SSS4.p1.7.m2.1.1.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1"><eq id="S3.SS1.SSS4.p1.7.m2.1.1.1.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1.1"></eq><apply id="S3.SS1.SSS4.p1.7.m2.1.1.2.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS4.p1.7.m2.1.1.2.1.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1.2">subscript</csymbol><ci id="S3.SS1.SSS4.p1.7.m2.1.1.2.2.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1.2.2">𝛽</ci><ci id="S3.SS1.SSS4.p1.7.m2.1.1.2.3.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1.2.3">𝑘</ci></apply><cn type="float" id="S3.SS1.SSS4.p1.7.m2.1.1.3.cmml" xref="S3.SS1.SSS4.p1.7.m2.1.1.3">0.26</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.7.m2.1c">\beta_{k}=0.26</annotation></semantics></math>, <math id="S3.SS1.SSS4.p1.8.m3.1" class="ltx_Math" alttext="\beta_{n}=-0.017" display="inline"><semantics id="S3.SS1.SSS4.p1.8.m3.1a"><mrow id="S3.SS1.SSS4.p1.8.m3.1.1" xref="S3.SS1.SSS4.p1.8.m3.1.1.cmml"><msub id="S3.SS1.SSS4.p1.8.m3.1.1.2" xref="S3.SS1.SSS4.p1.8.m3.1.1.2.cmml"><mi id="S3.SS1.SSS4.p1.8.m3.1.1.2.2" xref="S3.SS1.SSS4.p1.8.m3.1.1.2.2.cmml">β</mi><mi id="S3.SS1.SSS4.p1.8.m3.1.1.2.3" xref="S3.SS1.SSS4.p1.8.m3.1.1.2.3.cmml">n</mi></msub><mo id="S3.SS1.SSS4.p1.8.m3.1.1.1" xref="S3.SS1.SSS4.p1.8.m3.1.1.1.cmml">=</mo><mrow id="S3.SS1.SSS4.p1.8.m3.1.1.3" xref="S3.SS1.SSS4.p1.8.m3.1.1.3.cmml"><mo id="S3.SS1.SSS4.p1.8.m3.1.1.3a" xref="S3.SS1.SSS4.p1.8.m3.1.1.3.cmml">−</mo><mn id="S3.SS1.SSS4.p1.8.m3.1.1.3.2" xref="S3.SS1.SSS4.p1.8.m3.1.1.3.2.cmml">0.017</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.8.m3.1b"><apply id="S3.SS1.SSS4.p1.8.m3.1.1.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1"><eq id="S3.SS1.SSS4.p1.8.m3.1.1.1.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.1"></eq><apply id="S3.SS1.SSS4.p1.8.m3.1.1.2.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS4.p1.8.m3.1.1.2.1.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.2">subscript</csymbol><ci id="S3.SS1.SSS4.p1.8.m3.1.1.2.2.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.2.2">𝛽</ci><ci id="S3.SS1.SSS4.p1.8.m3.1.1.2.3.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.2.3">𝑛</ci></apply><apply id="S3.SS1.SSS4.p1.8.m3.1.1.3.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.3"><minus id="S3.SS1.SSS4.p1.8.m3.1.1.3.1.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.3"></minus><cn type="float" id="S3.SS1.SSS4.p1.8.m3.1.1.3.2.cmml" xref="S3.SS1.SSS4.p1.8.m3.1.1.3.2">0.017</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.8.m3.1c">\beta_{n}=-0.017</annotation></semantics></math>, <math id="S3.SS1.SSS4.p1.9.m4.1" class="ltx_Math" alttext="\gamma=-0.042" display="inline"><semantics id="S3.SS1.SSS4.p1.9.m4.1a"><mrow id="S3.SS1.SSS4.p1.9.m4.1.1" xref="S3.SS1.SSS4.p1.9.m4.1.1.cmml"><mi id="S3.SS1.SSS4.p1.9.m4.1.1.2" xref="S3.SS1.SSS4.p1.9.m4.1.1.2.cmml">γ</mi><mo id="S3.SS1.SSS4.p1.9.m4.1.1.1" xref="S3.SS1.SSS4.p1.9.m4.1.1.1.cmml">=</mo><mrow id="S3.SS1.SSS4.p1.9.m4.1.1.3" xref="S3.SS1.SSS4.p1.9.m4.1.1.3.cmml"><mo id="S3.SS1.SSS4.p1.9.m4.1.1.3a" xref="S3.SS1.SSS4.p1.9.m4.1.1.3.cmml">−</mo><mn id="S3.SS1.SSS4.p1.9.m4.1.1.3.2" xref="S3.SS1.SSS4.p1.9.m4.1.1.3.2.cmml">0.042</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.9.m4.1b"><apply id="S3.SS1.SSS4.p1.9.m4.1.1.cmml" xref="S3.SS1.SSS4.p1.9.m4.1.1"><eq id="S3.SS1.SSS4.p1.9.m4.1.1.1.cmml" xref="S3.SS1.SSS4.p1.9.m4.1.1.1"></eq><ci id="S3.SS1.SSS4.p1.9.m4.1.1.2.cmml" xref="S3.SS1.SSS4.p1.9.m4.1.1.2">𝛾</ci><apply id="S3.SS1.SSS4.p1.9.m4.1.1.3.cmml" xref="S3.SS1.SSS4.p1.9.m4.1.1.3"><minus id="S3.SS1.SSS4.p1.9.m4.1.1.3.1.cmml" xref="S3.SS1.SSS4.p1.9.m4.1.1.3"></minus><cn type="float" id="S3.SS1.SSS4.p1.9.m4.1.1.3.2.cmml" xref="S3.SS1.SSS4.p1.9.m4.1.1.3.2">0.042</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.9.m4.1c">\gamma=-0.042</annotation></semantics></math>, <math id="S3.SS1.SSS4.p1.10.m5.1" class="ltx_Math" alttext="\zeta=-1.32" display="inline"><semantics id="S3.SS1.SSS4.p1.10.m5.1a"><mrow id="S3.SS1.SSS4.p1.10.m5.1.1" xref="S3.SS1.SSS4.p1.10.m5.1.1.cmml"><mi id="S3.SS1.SSS4.p1.10.m5.1.1.2" xref="S3.SS1.SSS4.p1.10.m5.1.1.2.cmml">ζ</mi><mo id="S3.SS1.SSS4.p1.10.m5.1.1.1" xref="S3.SS1.SSS4.p1.10.m5.1.1.1.cmml">=</mo><mrow id="S3.SS1.SSS4.p1.10.m5.1.1.3" xref="S3.SS1.SSS4.p1.10.m5.1.1.3.cmml"><mo id="S3.SS1.SSS4.p1.10.m5.1.1.3a" xref="S3.SS1.SSS4.p1.10.m5.1.1.3.cmml">−</mo><mn id="S3.SS1.SSS4.p1.10.m5.1.1.3.2" xref="S3.SS1.SSS4.p1.10.m5.1.1.3.2.cmml">1.32</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.10.m5.1b"><apply id="S3.SS1.SSS4.p1.10.m5.1.1.cmml" xref="S3.SS1.SSS4.p1.10.m5.1.1"><eq id="S3.SS1.SSS4.p1.10.m5.1.1.1.cmml" xref="S3.SS1.SSS4.p1.10.m5.1.1.1"></eq><ci id="S3.SS1.SSS4.p1.10.m5.1.1.2.cmml" xref="S3.SS1.SSS4.p1.10.m5.1.1.2">𝜁</ci><apply id="S3.SS1.SSS4.p1.10.m5.1.1.3.cmml" xref="S3.SS1.SSS4.p1.10.m5.1.1.3"><minus id="S3.SS1.SSS4.p1.10.m5.1.1.3.1.cmml" xref="S3.SS1.SSS4.p1.10.m5.1.1.3"></minus><cn type="float" id="S3.SS1.SSS4.p1.10.m5.1.1.3.2.cmml" xref="S3.SS1.SSS4.p1.10.m5.1.1.3.2">1.32</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.10.m5.1c">\zeta=-1.32</annotation></semantics></math>, and <math id="S3.SS1.SSS4.p1.11.m6.1" class="ltx_Math" alttext="\eta=-0.085" display="inline"><semantics id="S3.SS1.SSS4.p1.11.m6.1a"><mrow id="S3.SS1.SSS4.p1.11.m6.1.1" xref="S3.SS1.SSS4.p1.11.m6.1.1.cmml"><mi id="S3.SS1.SSS4.p1.11.m6.1.1.2" xref="S3.SS1.SSS4.p1.11.m6.1.1.2.cmml">η</mi><mo id="S3.SS1.SSS4.p1.11.m6.1.1.1" xref="S3.SS1.SSS4.p1.11.m6.1.1.1.cmml">=</mo><mrow id="S3.SS1.SSS4.p1.11.m6.1.1.3" xref="S3.SS1.SSS4.p1.11.m6.1.1.3.cmml"><mo id="S3.SS1.SSS4.p1.11.m6.1.1.3a" xref="S3.SS1.SSS4.p1.11.m6.1.1.3.cmml">−</mo><mn id="S3.SS1.SSS4.p1.11.m6.1.1.3.2" xref="S3.SS1.SSS4.p1.11.m6.1.1.3.2.cmml">0.085</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.11.m6.1b"><apply id="S3.SS1.SSS4.p1.11.m6.1.1.cmml" xref="S3.SS1.SSS4.p1.11.m6.1.1"><eq id="S3.SS1.SSS4.p1.11.m6.1.1.1.cmml" xref="S3.SS1.SSS4.p1.11.m6.1.1.1"></eq><ci id="S3.SS1.SSS4.p1.11.m6.1.1.2.cmml" xref="S3.SS1.SSS4.p1.11.m6.1.1.2">𝜂</ci><apply id="S3.SS1.SSS4.p1.11.m6.1.1.3.cmml" xref="S3.SS1.SSS4.p1.11.m6.1.1.3"><minus id="S3.SS1.SSS4.p1.11.m6.1.1.3.1.cmml" xref="S3.SS1.SSS4.p1.11.m6.1.1.3"></minus><cn type="float" id="S3.SS1.SSS4.p1.11.m6.1.1.3.2.cmml" xref="S3.SS1.SSS4.p1.11.m6.1.1.3.2">0.085</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.11.m6.1c">\eta=-0.085</annotation></semantics></math>.
We can see that <math id="S3.SS1.SSS4.p1.12.m7.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S3.SS1.SSS4.p1.12.m7.1a"><mi id="S3.SS1.SSS4.p1.12.m7.1.1" xref="S3.SS1.SSS4.p1.12.m7.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.12.m7.1b"><ci id="S3.SS1.SSS4.p1.12.m7.1.1.cmml" xref="S3.SS1.SSS4.p1.12.m7.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.12.m7.1c">\gamma</annotation></semantics></math> is negative,
which means that the scaling law <math id="S3.SS1.SSS4.p1.13.m8.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="S3.SS1.SSS4.p1.13.m8.1a"><mrow id="S3.SS1.SSS4.p1.13.m8.1.2" xref="S3.SS1.SSS4.p1.13.m8.1.2.cmml"><mi id="S3.SS1.SSS4.p1.13.m8.1.2.2" xref="S3.SS1.SSS4.p1.13.m8.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSS4.p1.13.m8.1.2.1" xref="S3.SS1.SSS4.p1.13.m8.1.2.1.cmml">​</mo><mrow id="S3.SS1.SSS4.p1.13.m8.1.2.3.2" xref="S3.SS1.SSS4.p1.13.m8.1.2.cmml"><mo stretchy="false" id="S3.SS1.SSS4.p1.13.m8.1.2.3.2.1" xref="S3.SS1.SSS4.p1.13.m8.1.2.cmml">(</mo><mi id="S3.SS1.SSS4.p1.13.m8.1.1" xref="S3.SS1.SSS4.p1.13.m8.1.1.cmml">N</mi><mo stretchy="false" id="S3.SS1.SSS4.p1.13.m8.1.2.3.2.2" xref="S3.SS1.SSS4.p1.13.m8.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.13.m8.1b"><apply id="S3.SS1.SSS4.p1.13.m8.1.2.cmml" xref="S3.SS1.SSS4.p1.13.m8.1.2"><times id="S3.SS1.SSS4.p1.13.m8.1.2.1.cmml" xref="S3.SS1.SSS4.p1.13.m8.1.2.1"></times><ci id="S3.SS1.SSS4.p1.13.m8.1.2.2.cmml" xref="S3.SS1.SSS4.p1.13.m8.1.2.2">𝐿</ci><ci id="S3.SS1.SSS4.p1.13.m8.1.1.cmml" xref="S3.SS1.SSS4.p1.13.m8.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.13.m8.1c">L(N)</annotation></semantics></math> gets steeper as <math id="S3.SS1.SSS4.p1.14.m9.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.SSS4.p1.14.m9.1a"><mi id="S3.SS1.SSS4.p1.14.m9.1.1" xref="S3.SS1.SSS4.p1.14.m9.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.14.m9.1b"><ci id="S3.SS1.SSS4.p1.14.m9.1.1.cmml" xref="S3.SS1.SSS4.p1.14.m9.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.14.m9.1c">k</annotation></semantics></math> increases. <math id="S3.SS1.SSS4.p1.15.m10.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S3.SS1.SSS4.p1.15.m10.1a"><mi id="S3.SS1.SSS4.p1.15.m10.1.1" xref="S3.SS1.SSS4.p1.15.m10.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.15.m10.1b"><ci id="S3.SS1.SSS4.p1.15.m10.1.1.cmml" xref="S3.SS1.SSS4.p1.15.m10.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.15.m10.1c">\eta</annotation></semantics></math> is negative too, which means that the irreducible loss decreases with <math id="S3.SS1.SSS4.p1.16.m11.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.SSS4.p1.16.m11.1a"><mi id="S3.SS1.SSS4.p1.16.m11.1.1" xref="S3.SS1.SSS4.p1.16.m11.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS4.p1.16.m11.1b"><ci id="S3.SS1.SSS4.p1.16.m11.1.1.cmml" xref="S3.SS1.SSS4.p1.16.m11.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS4.p1.16.m11.1c">k</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Subject model size <math id="S3.SS2.1.m1.1" class="ltx_Math" alttext="L_{s}(N)" display="inline"><semantics id="S3.SS2.1.m1.1b"><mrow id="S3.SS2.1.m1.1.2" xref="S3.SS2.1.m1.1.2.cmml"><msub id="S3.SS2.1.m1.1.2.2" xref="S3.SS2.1.m1.1.2.2.cmml"><mi id="S3.SS2.1.m1.1.2.2.2" xref="S3.SS2.1.m1.1.2.2.2.cmml">L</mi><mi id="S3.SS2.1.m1.1.2.2.3" xref="S3.SS2.1.m1.1.2.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.1.m1.1.2.1" xref="S3.SS2.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS2.1.m1.1.2.3.2" xref="S3.SS2.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS2.1.m1.1.2.3.2.1" xref="S3.SS2.1.m1.1.2.cmml">(</mo><mi id="S3.SS2.1.m1.1.1" xref="S3.SS2.1.m1.1.1.cmml">N</mi><mo stretchy="false" id="S3.SS2.1.m1.1.2.3.2.2" xref="S3.SS2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.1.m1.1c"><apply id="S3.SS2.1.m1.1.2.cmml" xref="S3.SS2.1.m1.1.2"><times id="S3.SS2.1.m1.1.2.1.cmml" xref="S3.SS2.1.m1.1.2.1"></times><apply id="S3.SS2.1.m1.1.2.2.cmml" xref="S3.SS2.1.m1.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.1.m1.1.2.2.1.cmml" xref="S3.SS2.1.m1.1.2.2">subscript</csymbol><ci id="S3.SS2.1.m1.1.2.2.2.cmml" xref="S3.SS2.1.m1.1.2.2.2">𝐿</ci><ci id="S3.SS2.1.m1.1.2.2.3.cmml" xref="S3.SS2.1.m1.1.2.2.3">𝑠</ci></apply><ci id="S3.SS2.1.m1.1.1.cmml" xref="S3.SS2.1.m1.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.1.m1.1d">L_{s}(N)</annotation></semantics></math>
</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Since language models are likely to keep growing in size, we would also like to understand how sparse autoencoders scale as the subject models get larger.
We find that if we hold <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">k</annotation></semantics></math> constant, larger subject models require larger autoencoders to achieve the same MSE, and the exponent is worse (<a href="#S3.F4" title="Figure 4 ‣ 3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a>).</p>
</div>
<figure id="S3.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S3.F5.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.04093/assets/x7.png" id="S3.F5.sf1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="347" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F5.sf1.4.2.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S3.F5.sf1.2.1" class="ltx_text" style="font-size:90%;">For a fixed number of latents (<math id="S3.F5.sf1.2.1.m1.1" class="ltx_Math" alttext="n=2^{17}=131072" display="inline"><semantics id="S3.F5.sf1.2.1.m1.1b"><mrow id="S3.F5.sf1.2.1.m1.1.1" xref="S3.F5.sf1.2.1.m1.1.1.cmml"><mi id="S3.F5.sf1.2.1.m1.1.1.2" xref="S3.F5.sf1.2.1.m1.1.1.2.cmml">n</mi><mo id="S3.F5.sf1.2.1.m1.1.1.3" xref="S3.F5.sf1.2.1.m1.1.1.3.cmml">=</mo><msup id="S3.F5.sf1.2.1.m1.1.1.4" xref="S3.F5.sf1.2.1.m1.1.1.4.cmml"><mn id="S3.F5.sf1.2.1.m1.1.1.4.2" xref="S3.F5.sf1.2.1.m1.1.1.4.2.cmml">2</mn><mn id="S3.F5.sf1.2.1.m1.1.1.4.3" xref="S3.F5.sf1.2.1.m1.1.1.4.3.cmml">17</mn></msup><mo id="S3.F5.sf1.2.1.m1.1.1.5" xref="S3.F5.sf1.2.1.m1.1.1.5.cmml">=</mo><mn id="S3.F5.sf1.2.1.m1.1.1.6" xref="S3.F5.sf1.2.1.m1.1.1.6.cmml">131072</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F5.sf1.2.1.m1.1c"><apply id="S3.F5.sf1.2.1.m1.1.1.cmml" xref="S3.F5.sf1.2.1.m1.1.1"><and id="S3.F5.sf1.2.1.m1.1.1a.cmml" xref="S3.F5.sf1.2.1.m1.1.1"></and><apply id="S3.F5.sf1.2.1.m1.1.1b.cmml" xref="S3.F5.sf1.2.1.m1.1.1"><eq id="S3.F5.sf1.2.1.m1.1.1.3.cmml" xref="S3.F5.sf1.2.1.m1.1.1.3"></eq><ci id="S3.F5.sf1.2.1.m1.1.1.2.cmml" xref="S3.F5.sf1.2.1.m1.1.1.2">𝑛</ci><apply id="S3.F5.sf1.2.1.m1.1.1.4.cmml" xref="S3.F5.sf1.2.1.m1.1.1.4"><csymbol cd="ambiguous" id="S3.F5.sf1.2.1.m1.1.1.4.1.cmml" xref="S3.F5.sf1.2.1.m1.1.1.4">superscript</csymbol><cn type="integer" id="S3.F5.sf1.2.1.m1.1.1.4.2.cmml" xref="S3.F5.sf1.2.1.m1.1.1.4.2">2</cn><cn type="integer" id="S3.F5.sf1.2.1.m1.1.1.4.3.cmml" xref="S3.F5.sf1.2.1.m1.1.1.4.3">17</cn></apply></apply><apply id="S3.F5.sf1.2.1.m1.1.1c.cmml" xref="S3.F5.sf1.2.1.m1.1.1"><eq id="S3.F5.sf1.2.1.m1.1.1.5.cmml" xref="S3.F5.sf1.2.1.m1.1.1.5"></eq><share href="#S3.F5.sf1.2.1.m1.1.1.4.cmml" id="S3.F5.sf1.2.1.m1.1.1d.cmml" xref="S3.F5.sf1.2.1.m1.1.1"></share><cn type="integer" id="S3.F5.sf1.2.1.m1.1.1.6.cmml" xref="S3.F5.sf1.2.1.m1.1.1.6">131072</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.sf1.2.1.m1.1d">n=2^{17}=131072</annotation></semantics></math>), the downstream-loss/sparsity trade-off is better for TopK autoencoders than for other activation functions.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S3.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.04093/assets/x8.png" id="S3.F5.sf2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F5.sf2.4.2.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S3.F5.sf2.2.1" class="ltx_text" style="font-size:90%;">For a fixed sparsity level (<math id="S3.F5.sf2.2.1.m1.1" class="ltx_Math" alttext="L_{0}=128" display="inline"><semantics id="S3.F5.sf2.2.1.m1.1b"><mrow id="S3.F5.sf2.2.1.m1.1.1" xref="S3.F5.sf2.2.1.m1.1.1.cmml"><msub id="S3.F5.sf2.2.1.m1.1.1.2" xref="S3.F5.sf2.2.1.m1.1.1.2.cmml"><mi id="S3.F5.sf2.2.1.m1.1.1.2.2" xref="S3.F5.sf2.2.1.m1.1.1.2.2.cmml">L</mi><mn id="S3.F5.sf2.2.1.m1.1.1.2.3" xref="S3.F5.sf2.2.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="S3.F5.sf2.2.1.m1.1.1.1" xref="S3.F5.sf2.2.1.m1.1.1.1.cmml">=</mo><mn id="S3.F5.sf2.2.1.m1.1.1.3" xref="S3.F5.sf2.2.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F5.sf2.2.1.m1.1c"><apply id="S3.F5.sf2.2.1.m1.1.1.cmml" xref="S3.F5.sf2.2.1.m1.1.1"><eq id="S3.F5.sf2.2.1.m1.1.1.1.cmml" xref="S3.F5.sf2.2.1.m1.1.1.1"></eq><apply id="S3.F5.sf2.2.1.m1.1.1.2.cmml" xref="S3.F5.sf2.2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.F5.sf2.2.1.m1.1.1.2.1.cmml" xref="S3.F5.sf2.2.1.m1.1.1.2">subscript</csymbol><ci id="S3.F5.sf2.2.1.m1.1.1.2.2.cmml" xref="S3.F5.sf2.2.1.m1.1.1.2.2">𝐿</ci><cn type="integer" id="S3.F5.sf2.2.1.m1.1.1.2.3.cmml" xref="S3.F5.sf2.2.1.m1.1.1.2.3">0</cn></apply><cn type="integer" id="S3.F5.sf2.2.1.m1.1.1.3.cmml" xref="S3.F5.sf2.2.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.sf2.2.1.m1.1d">L_{0}=128</annotation></semantics></math>), a given MSE level leads to a lower downstream-loss for TopK autoencoders than for other activation functions.</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S3.F5.3.2" class="ltx_text" style="font-size:90%;">Comparison between TopK and other activation functions on downstream loss. Comparisons done for GPT-2 small, see <a href="#A2.F13" title="Figure 13 ‣ B.2 Baseline hyperparameters ‣ Appendix B Other training details ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 13</span></a> for GPT-4.</span></figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Evaluation</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">We demonstrated in <a href="#S3" title="3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3</span></a> that our larger autoencoders scale well in terms of MSE and sparsity (see also a comparison of activation functions in <a href="#S5.SS2" title="5.2 Comparison with other activation functions ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5.2</span></a>). However, the end goal of autoencoders is not to improve the sparsity-reconstruction frontier (which degenerates in the limit<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>Improving the reconstruction-sparsity frontier is not always strictly better. An infinitely wide maximally sparse (<math id="footnote8.m1.1" class="ltx_Math" alttext="k=1" display="inline"><semantics id="footnote8.m1.1b"><mrow id="footnote8.m1.1.1" xref="footnote8.m1.1.1.cmml"><mi id="footnote8.m1.1.1.2" xref="footnote8.m1.1.1.2.cmml">k</mi><mo id="footnote8.m1.1.1.1" xref="footnote8.m1.1.1.1.cmml">=</mo><mn id="footnote8.m1.1.1.3" xref="footnote8.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote8.m1.1c"><apply id="footnote8.m1.1.1.cmml" xref="footnote8.m1.1.1"><eq id="footnote8.m1.1.1.1.cmml" xref="footnote8.m1.1.1.1"></eq><ci id="footnote8.m1.1.1.2.cmml" xref="footnote8.m1.1.1.2">𝑘</ci><cn type="integer" id="footnote8.m1.1.1.3.cmml" xref="footnote8.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote8.m1.1d">k=1</annotation></semantics></math>) autoencoder can perfectly reconstruct by assigning latents densely in <math id="footnote8.m2.1" class="ltx_Math" alttext="\mathbb{R}^{d}" display="inline"><semantics id="footnote8.m2.1b"><msup id="footnote8.m2.1.1" xref="footnote8.m2.1.1.cmml"><mi id="footnote8.m2.1.1.2" xref="footnote8.m2.1.1.2.cmml">ℝ</mi><mi id="footnote8.m2.1.1.3" xref="footnote8.m2.1.1.3.cmml">d</mi></msup><annotation-xml encoding="MathML-Content" id="footnote8.m2.1c"><apply id="footnote8.m2.1.1.cmml" xref="footnote8.m2.1.1"><csymbol cd="ambiguous" id="footnote8.m2.1.1.1.cmml" xref="footnote8.m2.1.1">superscript</csymbol><ci id="footnote8.m2.1.1.2.cmml" xref="footnote8.m2.1.1.2">ℝ</ci><ci id="footnote8.m2.1.1.3.cmml" xref="footnote8.m2.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote8.m2.1d">\mathbb{R}^{d}</annotation></semantics></math>, while being completely structure-less and uninteresting.</span></span></span>), but rather to find features useful for applications, such as mechanistic interpretability.
Therefore, we measure autoencoder quality with the following metrics:</p>
</div>
<div id="S4.p2" class="ltx_para">
<ol id="S4.I1" class="ltx_enumerate">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S4.I1.i1.p1" class="ltx_para">
<p id="S4.I1.i1.p1.1" class="ltx_p"><span id="S4.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Downstream loss</span>: How good is the language model loss if the residual stream latent is replaced with the autoencoder reconstruction of that latent? (<a href="#S4.SS1" title="4.1 Downstream loss ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.1</span></a>)</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S4.I1.i2.p1" class="ltx_para">
<p id="S4.I1.i2.p1.1" class="ltx_p"><span id="S4.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Probe loss</span>: Do autoencoders recover features that we believe they might have? (<a href="#S4.SS2" title="4.2 Recovering known features with 1d probes ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.2</span></a>)</p>
</div>
</li>
<li id="S4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S4.I1.i3.p1" class="ltx_para">
<p id="S4.I1.i3.p1.1" class="ltx_p"><span id="S4.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Explainability</span>: Are there simple explanations that are both necessary and sufficient for the activation of the autoencoder latent? (<a href="#S4.SS3" title="4.3 Finding simple explanations for features ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.3</span></a>)</p>
</div>
</li>
<li id="S4.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S4.I1.i4.p1" class="ltx_para">
<p id="S4.I1.i4.p1.1" class="ltx_p"><span id="S4.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Ablation sparsity</span>: Does ablating individual latents have a sparse effect on downstream logits? (<a href="#S4.SS5" title="4.5 Sparsity of ablation effects ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.5</span></a>)</p>
</div>
</li>
</ol>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.4" class="ltx_p">These metrics provide evidence that autoencoders generally get better when the number of total latents increases.
The impact of the number of active latents L<sub id="S4.p3.4.1" class="ltx_sub">0</sub> is more complicated. Increasing L<sub id="S4.p3.4.2" class="ltx_sub">0</sub> makes explanations based on token patterns worse, but makes probe loss and ablation sparsity better. All of these trends also break when L<sub id="S4.p3.4.3" class="ltx_sub">0</sub> gets close to <math id="S4.p3.4.m4.1" class="ltx_Math" alttext="d_{\text{model}}" display="inline"><semantics id="S4.p3.4.m4.1a"><msub id="S4.p3.4.m4.1.1" xref="S4.p3.4.m4.1.1.cmml"><mi id="S4.p3.4.m4.1.1.2" xref="S4.p3.4.m4.1.1.2.cmml">d</mi><mtext id="S4.p3.4.m4.1.1.3" xref="S4.p3.4.m4.1.1.3a.cmml">model</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p3.4.m4.1b"><apply id="S4.p3.4.m4.1.1.cmml" xref="S4.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S4.p3.4.m4.1.1.1.cmml" xref="S4.p3.4.m4.1.1">subscript</csymbol><ci id="S4.p3.4.m4.1.1.2.cmml" xref="S4.p3.4.m4.1.1.2">𝑑</ci><ci id="S4.p3.4.m4.1.1.3a.cmml" xref="S4.p3.4.m4.1.1.3"><mtext mathsize="70%" id="S4.p3.4.m4.1.1.3.cmml" xref="S4.p3.4.m4.1.1.3">model</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.4.m4.1c">d_{\text{model}}</annotation></semantics></math>, a regime in which latents also become quite dense (see <a href="#A5.SS5" title="E.5 Solutions with dense latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.5</span></a> for detailed discussion).</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Downstream loss</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.3" class="ltx_p">An autoencoder with non-zero reconstruction error may not succeed at modeling the features most relevant for behavior <cite class="ltx_cite ltx_citemacro_citep">(Braun et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite>.
To measure whether we model features relevant to language modeling, we follow prior work <cite class="ltx_cite ltx_citemacro_citep">(Bills et al., <a href="#bib.bib3" title="" class="ltx_ref">2023</a>; Cunningham et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>; Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>; Braun et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite> and consider downstream Kullback-Leibler (KL) divergence and cross-entropy loss.<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>Because a perfect reconstruction would lead to a non-zero cross-entropy loss, we actually consider the difference to the perfect-autoencoder cross-entropy (“delta cross-entropy”).</span></span></span> In both cases, we test an autoencoder by replacing the residual stream by the reconstructed value during the forward pass, and seeing how it affects downstream predictions. We find that <math id="S4.SS1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS1.p1.1.m1.1a"><mi id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><ci id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">k</annotation></semantics></math>-sparse autoencoders improve more on downstream loss than on MSE over prior methods (<a href="#S3.F5" title="Figure 5 ‣ 3.2 Subject model size 𝐿_𝑠⁢(𝑁) ‣ 3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 5</span></a>a).
We also find that MSE has a clean power law relationship with both KL divergence, and difference of cross entropy loss (<a href="#S3.F5" title="Figure 5 ‣ 3.2 Subject model size 𝐿_𝑠⁢(𝑁) ‣ 3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 5</span></a>b), when keeping sparsity L<sub id="S4.SS1.p1.3.1" class="ltx_sub">0</sub> fixed and only varying autoencoder size. Note that while this trend is clean for our trained autoencoders, we can observe instances where it breaks such as when modulating <math id="S4.SS1.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS1.p1.3.m3.1a"><mi id="S4.SS1.p1.3.m3.1.1" xref="S4.SS1.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.3.m3.1b"><ci id="S4.SS1.p1.3.m3.1.1.cmml" xref="S4.SS1.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">k</annotation></semantics></math> at test time (see <a href="#S5.SS3" title="5.3 Progressive recovery ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5.3</span></a>).</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">One additional issue is that raw loss numbers alone are difficult to interpret—we would like to know how good it is in an absolute sense. Prior work <cite class="ltx_cite ltx_citemacro_citep">(Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>; Rajamanoharan et al., <a href="#bib.bib52" title="" class="ltx_ref">2024</a>)</cite> use the loss of ablating activations to zero as a baseline and report the fraction of loss recovered from that baseline. However, because ablating the residual stream to zero causes very high downstream loss, this means that even very poorly explaining the behavior can result in high scores.<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>For completeness, the zero-ablation fidelity metric of our 16M autoencoder is 98.2%.</span></span></span></p>
</div>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p">Instead, we believe a more natural metric is to consider the relative amount of pretraining compute needed to train a language model of comparable downstream loss. For example, when our 16 million latent autoencoder is substituted into GPT-4, we get a language modeling loss corresponding to 10% of the pretraining compute of GPT-4.</p>
</div>
<figure id="S4.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F6.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_2m_probe_grid.png" id="S4.F6.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="392" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S4.F6.sf1.3.2" class="ltx_text" style="font-size:90%;">Probe loss</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F6.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_2m_ablatevote_sparsity_l1l2.png" id="S4.F6.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="392" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S4.F6.sf2.3.2" class="ltx_text" style="font-size:90%;">Logit diff sparsity</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.13.6.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S4.F6.10.5" class="ltx_text" style="font-size:90%;">The probe loss and logit diff metrics as a function of number of total latents <math id="S4.F6.6.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.F6.6.1.m1.1b"><mi id="S4.F6.6.1.m1.1.1" xref="S4.F6.6.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.F6.6.1.m1.1c"><ci id="S4.F6.6.1.m1.1.1.cmml" xref="S4.F6.6.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F6.6.1.m1.1d">n</annotation></semantics></math> and active latents <math id="S4.F6.7.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.F6.7.2.m2.1b"><mi id="S4.F6.7.2.m2.1.1" xref="S4.F6.7.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.F6.7.2.m2.1c"><ci id="S4.F6.7.2.m2.1.1.cmml" xref="S4.F6.7.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F6.7.2.m2.1d">k</annotation></semantics></math>, for GPT-2 small autoencoders. More total latents (higher <math id="S4.F6.8.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.F6.8.3.m3.1b"><mi id="S4.F6.8.3.m3.1.1" xref="S4.F6.8.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.F6.8.3.m3.1c"><ci id="S4.F6.8.3.m3.1.1.cmml" xref="S4.F6.8.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F6.8.3.m3.1d">n</annotation></semantics></math>) generally improves all metrics (yellow = better). Both metrics are worse at L<sub id="S4.F6.10.5.1" class="ltx_sub">0</sub> <math id="S4.F6.10.5.m5.1" class="ltx_Math" alttext="=512" display="inline"><semantics id="S4.F6.10.5.m5.1b"><mrow id="S4.F6.10.5.m5.1.1" xref="S4.F6.10.5.m5.1.1.cmml"><mi id="S4.F6.10.5.m5.1.1.2" xref="S4.F6.10.5.m5.1.1.2.cmml"></mi><mo id="S4.F6.10.5.m5.1.1.1" xref="S4.F6.10.5.m5.1.1.1.cmml">=</mo><mn id="S4.F6.10.5.m5.1.1.3" xref="S4.F6.10.5.m5.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F6.10.5.m5.1c"><apply id="S4.F6.10.5.m5.1.1.cmml" xref="S4.F6.10.5.m5.1.1"><eq id="S4.F6.10.5.m5.1.1.1.cmml" xref="S4.F6.10.5.m5.1.1.1"></eq><csymbol cd="latexml" id="S4.F6.10.5.m5.1.1.2.cmml" xref="S4.F6.10.5.m5.1.1.2">absent</csymbol><cn type="integer" id="S4.F6.10.5.m5.1.1.3.cmml" xref="S4.F6.10.5.m5.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F6.10.5.m5.1d">=512</annotation></semantics></math>, a regime in which solutions are dense (see <a href="#A5.SS5" title="E.5 Solutions with dense latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.5</span></a>).
</span></figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Recovering known features with 1d probes</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">If we expect that a specific feature (e.g sentiment, language identification) should be discovered by a high quality autoencoder, then one metric of autoencoder quality is to check whether these features are present.
Based on this intuition, we curated a set of 61 binary classification datasets (details in <a href="#A8.T1" title="Table 1 ‣ Appendix H Further probe based evaluations results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 1</span></a>). For each task, we train a 1d logistic probe on each latent using the Newton-Raphson method to predict the task, and record the best cross entropy loss (across latents).<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>This is similar to the approach of <cite class="ltx_cite ltx_citemacro_citet">Gurnee et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>, but always using <math id="footnote11.m1.1" class="ltx_Math" alttext="k=1" display="inline"><semantics id="footnote11.m1.1b"><mrow id="footnote11.m1.1.1" xref="footnote11.m1.1.1.cmml"><mi id="footnote11.m1.1.1.2" xref="footnote11.m1.1.1.2.cmml">k</mi><mo id="footnote11.m1.1.1.1" xref="footnote11.m1.1.1.1.cmml">=</mo><mn id="footnote11.m1.1.1.3" xref="footnote11.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote11.m1.1c"><apply id="footnote11.m1.1.1.cmml" xref="footnote11.m1.1.1"><eq id="footnote11.m1.1.1.1.cmml" xref="footnote11.m1.1.1.1"></eq><ci id="footnote11.m1.1.1.2.cmml" xref="footnote11.m1.1.1.2">𝑘</ci><cn type="integer" id="footnote11.m1.1.1.3.cmml" xref="footnote11.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote11.m1.1d">k=1</annotation></semantics></math>, and regressing on autoencoder latents instead of neurons.</span></span></span> That is:</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<table id="S4.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E4.m1.5" class="ltx_Math" alttext="\min_{i,w,b}\mathbb{E}\left[y\log\sigma\left(wz_{i}+b\right)+\left(1-y\right)\log\left(1-\sigma\left(wz_{i}+b\right)\right)\right]" display="block"><semantics id="S4.E4.m1.5a"><mrow id="S4.E4.m1.5.5" xref="S4.E4.m1.5.5.cmml"><mrow id="S4.E4.m1.5.5.3" xref="S4.E4.m1.5.5.3.cmml"><munder id="S4.E4.m1.5.5.3.1" xref="S4.E4.m1.5.5.3.1.cmml"><mi id="S4.E4.m1.5.5.3.1.2" xref="S4.E4.m1.5.5.3.1.2.cmml">min</mi><mrow id="S4.E4.m1.3.3.3.5" xref="S4.E4.m1.3.3.3.4.cmml"><mi id="S4.E4.m1.1.1.1.1" xref="S4.E4.m1.1.1.1.1.cmml">i</mi><mo id="S4.E4.m1.3.3.3.5.1" xref="S4.E4.m1.3.3.3.4.cmml">,</mo><mi id="S4.E4.m1.2.2.2.2" xref="S4.E4.m1.2.2.2.2.cmml">w</mi><mo id="S4.E4.m1.3.3.3.5.2" xref="S4.E4.m1.3.3.3.4.cmml">,</mo><mi id="S4.E4.m1.3.3.3.3" xref="S4.E4.m1.3.3.3.3.cmml">b</mi></mrow></munder><mo lspace="0.167em" id="S4.E4.m1.5.5.3a" xref="S4.E4.m1.5.5.3.cmml">⁡</mo><mi id="S4.E4.m1.5.5.3.2" xref="S4.E4.m1.5.5.3.2.cmml">𝔼</mi></mrow><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.2" xref="S4.E4.m1.5.5.2.cmml">​</mo><mrow id="S4.E4.m1.5.5.1.1" xref="S4.E4.m1.5.5.1.2.cmml"><mo id="S4.E4.m1.5.5.1.1.2" xref="S4.E4.m1.5.5.1.2.1.cmml">[</mo><mrow id="S4.E4.m1.5.5.1.1.1" xref="S4.E4.m1.5.5.1.1.1.cmml"><mrow id="S4.E4.m1.5.5.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.cmml"><mi id="S4.E4.m1.5.5.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.1.3.cmml">y</mi><mo lspace="0.167em" rspace="0em" id="S4.E4.m1.5.5.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="S4.E4.m1.5.5.1.1.1.1.4" xref="S4.E4.m1.5.5.1.1.1.1.4.cmml"><mi id="S4.E4.m1.5.5.1.1.1.1.4.1" xref="S4.E4.m1.5.5.1.1.1.1.4.1.cmml">log</mi><mo lspace="0.167em" id="S4.E4.m1.5.5.1.1.1.1.4a" xref="S4.E4.m1.5.5.1.1.1.1.4.cmml">⁡</mo><mi id="S4.E4.m1.5.5.1.1.1.1.4.2" xref="S4.E4.m1.5.5.1.1.1.1.4.2.cmml">σ</mi></mrow><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.1.1.1.1.2a" xref="S4.E4.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="S4.E4.m1.5.5.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.cmml"><mo id="S4.E4.m1.5.5.1.1.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E4.m1.5.5.1.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.cmml"><mrow id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.2" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.2.cmml">w</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.1" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.cmml"><mi id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.2" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.2.cmml">z</mi><mi id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.3" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.3.cmml">i</mi></msub></mrow><mo id="S4.E4.m1.5.5.1.1.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="S4.E4.m1.5.5.1.1.1.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.3.cmml">b</mi></mrow><mo id="S4.E4.m1.5.5.1.1.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E4.m1.5.5.1.1.1.4" xref="S4.E4.m1.5.5.1.1.1.4.cmml">+</mo><mrow id="S4.E4.m1.5.5.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.3.cmml"><mrow id="S4.E4.m1.5.5.1.1.1.2.1.1" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.cmml"><mo id="S4.E4.m1.5.5.1.1.1.2.1.1.2" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.cmml">(</mo><mrow id="S4.E4.m1.5.5.1.1.1.2.1.1.1" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.cmml"><mn id="S4.E4.m1.5.5.1.1.1.2.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.2.cmml">1</mn><mo id="S4.E4.m1.5.5.1.1.1.2.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.1.cmml">−</mo><mi id="S4.E4.m1.5.5.1.1.1.2.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.3.cmml">y</mi></mrow><mo id="S4.E4.m1.5.5.1.1.1.2.1.1.3" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S4.E4.m1.5.5.1.1.1.3.3" xref="S4.E4.m1.5.5.1.1.1.3.3.cmml">​</mo><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1" xref="S4.E4.m1.5.5.1.1.1.3.2.2.cmml"><mi id="S4.E4.m1.4.4" xref="S4.E4.m1.4.4.cmml">log</mi><mo id="S4.E4.m1.5.5.1.1.1.3.2.1a" xref="S4.E4.m1.5.5.1.1.1.3.2.2.cmml">⁡</mo><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1.1" xref="S4.E4.m1.5.5.1.1.1.3.2.2.cmml"><mo id="S4.E4.m1.5.5.1.1.1.3.2.1.1.2" xref="S4.E4.m1.5.5.1.1.1.3.2.2.cmml">(</mo><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.cmml"><mn id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.3.cmml">1</mn><mo id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.2.cmml">−</mo><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.cmml"><mi id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.2.cmml">​</mo><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.cmml"><mo id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.cmml"><mrow id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.2" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.2.cmml">w</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.1" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.cmml"><mi id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.2" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.2.cmml">z</mi><mi id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.3" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.3.cmml">i</mi></msub></mrow><mo id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.3.cmml">b</mi></mrow><mo id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S4.E4.m1.5.5.1.1.1.3.2.1.1.3" xref="S4.E4.m1.5.5.1.1.1.3.2.2.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S4.E4.m1.5.5.1.1.3" xref="S4.E4.m1.5.5.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E4.m1.5b"><apply id="S4.E4.m1.5.5.cmml" xref="S4.E4.m1.5.5"><times id="S4.E4.m1.5.5.2.cmml" xref="S4.E4.m1.5.5.2"></times><apply id="S4.E4.m1.5.5.3.cmml" xref="S4.E4.m1.5.5.3"><apply id="S4.E4.m1.5.5.3.1.cmml" xref="S4.E4.m1.5.5.3.1"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.3.1.1.cmml" xref="S4.E4.m1.5.5.3.1">subscript</csymbol><min id="S4.E4.m1.5.5.3.1.2.cmml" xref="S4.E4.m1.5.5.3.1.2"></min><list id="S4.E4.m1.3.3.3.4.cmml" xref="S4.E4.m1.3.3.3.5"><ci id="S4.E4.m1.1.1.1.1.cmml" xref="S4.E4.m1.1.1.1.1">𝑖</ci><ci id="S4.E4.m1.2.2.2.2.cmml" xref="S4.E4.m1.2.2.2.2">𝑤</ci><ci id="S4.E4.m1.3.3.3.3.cmml" xref="S4.E4.m1.3.3.3.3">𝑏</ci></list></apply><ci id="S4.E4.m1.5.5.3.2.cmml" xref="S4.E4.m1.5.5.3.2">𝔼</ci></apply><apply id="S4.E4.m1.5.5.1.2.cmml" xref="S4.E4.m1.5.5.1.1"><csymbol cd="latexml" id="S4.E4.m1.5.5.1.2.1.cmml" xref="S4.E4.m1.5.5.1.1.2">delimited-[]</csymbol><apply id="S4.E4.m1.5.5.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1"><plus id="S4.E4.m1.5.5.1.1.1.4.cmml" xref="S4.E4.m1.5.5.1.1.1.4"></plus><apply id="S4.E4.m1.5.5.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1"><times id="S4.E4.m1.5.5.1.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.2"></times><ci id="S4.E4.m1.5.5.1.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.3">𝑦</ci><apply id="S4.E4.m1.5.5.1.1.1.1.4.cmml" xref="S4.E4.m1.5.5.1.1.1.1.4"><log id="S4.E4.m1.5.5.1.1.1.1.4.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.4.1"></log><ci id="S4.E4.m1.5.5.1.1.1.1.4.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.4.2">𝜎</ci></apply><apply id="S4.E4.m1.5.5.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1"><plus id="S4.E4.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.1"></plus><apply id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2"><times id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.1"></times><ci id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.2">𝑤</ci><apply id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.2">𝑧</ci><ci id="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.2.3.3">𝑖</ci></apply></apply><ci id="S4.E4.m1.5.5.1.1.1.1.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1.1.3">𝑏</ci></apply></apply><apply id="S4.E4.m1.5.5.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3"><times id="S4.E4.m1.5.5.1.1.1.3.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3.3"></times><apply id="S4.E4.m1.5.5.1.1.1.2.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.2.1.1"><minus id="S4.E4.m1.5.5.1.1.1.2.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.1"></minus><cn type="integer" id="S4.E4.m1.5.5.1.1.1.2.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.2">1</cn><ci id="S4.E4.m1.5.5.1.1.1.2.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.2.1.1.1.3">𝑦</ci></apply><apply id="S4.E4.m1.5.5.1.1.1.3.2.2.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1"><log id="S4.E4.m1.4.4.cmml" xref="S4.E4.m1.4.4"></log><apply id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1"><minus id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.2"></minus><cn type="integer" id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.3">1</cn><apply id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1"><times id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.2"></times><ci id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.3">𝜎</ci><apply id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1"><plus id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.1"></plus><apply id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2"><times id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.1"></times><ci id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.2">𝑤</ci><apply id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.1.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.2.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.2">𝑧</ci><ci id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.2.3.3">𝑖</ci></apply></apply><ci id="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.3.2.1.1.1.1.1.1.1.3">𝑏</ci></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E4.m1.5c">\min_{i,w,b}\mathbb{E}\left[y\log\sigma\left(wz_{i}+b\right)+\left(1-y\right)\log\left(1-\sigma\left(wz_{i}+b\right)\right)\right]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.3" class="ltx_p">where <math id="S4.SS2.p3.1.m1.1" class="ltx_Math" alttext="z_{i}" display="inline"><semantics id="S4.SS2.p3.1.m1.1a"><msub id="S4.SS2.p3.1.m1.1.1" xref="S4.SS2.p3.1.m1.1.1.cmml"><mi id="S4.SS2.p3.1.m1.1.1.2" xref="S4.SS2.p3.1.m1.1.1.2.cmml">z</mi><mi id="S4.SS2.p3.1.m1.1.1.3" xref="S4.SS2.p3.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.1.m1.1b"><apply id="S4.SS2.p3.1.m1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p3.1.m1.1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.p3.1.m1.1.1.2.cmml" xref="S4.SS2.p3.1.m1.1.1.2">𝑧</ci><ci id="S4.SS2.p3.1.m1.1.1.3.cmml" xref="S4.SS2.p3.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.1.m1.1c">z_{i}</annotation></semantics></math> is the <math id="S4.SS2.p3.2.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S4.SS2.p3.2.m2.1a"><mi id="S4.SS2.p3.2.m2.1.1" xref="S4.SS2.p3.2.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.2.m2.1b"><ci id="S4.SS2.p3.2.m2.1.1.cmml" xref="S4.SS2.p3.2.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.2.m2.1c">i</annotation></semantics></math>th pre-activation autoencoder latent, and <math id="S4.SS2.p3.3.m3.1" class="ltx_Math" alttext="y" display="inline"><semantics id="S4.SS2.p3.3.m3.1a"><mi id="S4.SS2.p3.3.m3.1.1" xref="S4.SS2.p3.3.m3.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.3.m3.1b"><ci id="S4.SS2.p3.3.m3.1.1.cmml" xref="S4.SS2.p3.3.m3.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.3.m3.1c">y</annotation></semantics></math> is a binary label.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.1" class="ltx_p">Results on GPT-2 small are shown in <a href="#S4.F6.sf1" title="6(a) ‣ Figure 6 ‣ 4.1 Downstream loss ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">6(a)</span></a>. We find that probe score increases and then decreases as <math id="S4.SS2.p4.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS2.p4.1.m1.1a"><mi id="S4.SS2.p4.1.m1.1.1" xref="S4.SS2.p4.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.1.m1.1b"><ci id="S4.SS2.p4.1.m1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.1.m1.1c">k</annotation></semantics></math> increases. We find that TopK generally achieves better probe scores than ReLU (<a href="#A5.F23" title="Figure 23 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 23</span></a>), and both are substantially better
than when using directly residual stream channels. See <a href="#A8.F32" title="Figure 32 ‣ Appendix H Further probe based evaluations results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 32</span></a> for results on several GPT-4 autoencoders: we observe that this metric improves throughout training, despite there being no supervised training signal; and we find that it beats a baseline using channels of the residual stream. See <a href="#A8.F33" title="Figure 33 ‣ Appendix H Further probe based evaluations results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 33</span></a> for scores broken down by component.</p>
</div>
<div id="S4.SS2.p5" class="ltx_para">
<p id="S4.SS2.p5.1" class="ltx_p">This metric has the advantage that it is computationally cheap. However, it also has a major limitation, which is that it leans on strong assumptions about what kinds of features are natural.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Finding simple explanations for features</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">Anecdotally, our autoencoders find many features that have quickly recognizable patterns that suggest explanations when viewing random activations (<a href="#A5.SS1" title="E.1 Subjective latent quality ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.1</span></a>). However, this can create an “illusion” of interpretability <cite class="ltx_cite ltx_citemacro_citep">(Bolukbasi et al., <a href="#bib.bib6" title="" class="ltx_ref">2021</a>)</cite>, where explanations are overly broad, and thus have good recall but poor precision.
For example, <cite class="ltx_cite ltx_citemacro_citet">Bills et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite> propose an automated interpretability score which disproportionately depends on recall. They find a feature activating at the end of the phrase “don’t stop” or “can’t stop”, but an explanation activating on all instances of “stop” achieves a high interpretability score. As we scale autoencoders and the features get sparser and more specific, this kind of failure becomes more severe.</p>
</div>
<figure id="S4.F7" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div class="ltx_pagination ltx_figure_panel ltx_role_start_2_columns"></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S4.F7.2" class="ltx_p ltx_align_left ltx_figure_panel"><span id="S4.F7.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">(a) A feature with precision = 0.97, recall = 0.56<span id="S4.F7.2.1.1" class="ltx_text ltx_font_medium"> 
<br class="ltx_break"></span>(b) A feature with precision = 0.06, recall = 0.05<span id="S4.F7.2.1.2" class="ltx_text ltx_font_medium"></span></span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div class="ltx_pagination ltx_figure_panel ltx_role_end_2_columns"></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S4.F7.3" class="ltx_p ltx_figure_panel"><span class="ltx_rule" style="width:100%;height:1px;background:black;display:inline-block;"> </span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div class="ltx_pagination ltx_figure_panel ltx_role_start_2_columns"></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S4.F7.4" class="ltx_p ltx_align_left ltx_figure_panel"><span id="S4.F7.4.1" class="ltx_text" style="font-size:70%;">Recall Eval Sequences:
<br class="ltx_break">

<span id="S4.F7.4.1.1" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.2" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.3" class="ltx_text" style="background-color:#FFFFFF;">Therefore</span><span id="S4.F7.4.1.4" class="ltx_text" style="background-color:#FFFFFF;"> she</span><span id="S4.F7.4.1.5" class="ltx_text" style="background-color:#FFFFFF;">’s</span><span id="S4.F7.4.1.6" class="ltx_text" style="background-color:#FFFFFF;"> not</span><span id="S4.F7.4.1.7" class="ltx_text" style="background-color:#FFFFFF;"> going</span><span id="S4.F7.4.1.8" class="ltx_text" style="background-color:#FFFFFF;"> to</span><span id="S4.F7.4.1.9" class="ltx_text" style="background-color:#FFFFFF;"> play</span><span id="S4.F7.4.1.10" class="ltx_text" style="background-color:#FFFFFF;"> pr</span>
<span id="S4.F7.4.1.11" class="ltx_tabular">
<span id="S4.F7.4.1.11.1" class="ltx_tr">
<span id="S4.F7.4.1.11.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.11.1.1.1" class="ltx_text"><span id="S4.F7.4.1.11.1.1.1.1" class="ltx_text" style="background-color:#93E066;">anks</span></span></span></span>
</span><span id="S4.F7.4.1.12" class="ltx_text" style="background-color:#FFFFFF;"> but</span><span id="S4.F7.4.1.13" class="ltx_text" style="background-color:#FFFFFF;"> to</span><span id="S4.F7.4.1.14" class="ltx_text" style="background-color:#FFFFFF;"> give</span><span id="S4.F7.4.1.15" class="ltx_text" style="background-color:#FFFFFF;"> a</span><span id="S4.F7.4.1.16" class="ltx_text" style="background-color:#FFFFFF;"> sweet</span><span id="S4.F7.4.1.17" class="ltx_text" style="background-color:#FFFFFF;"> time</span><span id="S4.F7.4.1.18" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.4.1.19" class="ltx_text" style="background-color:#FFFFFF;">a</span><span id="S4.F7.4.1.20" class="ltx_text" style="background-color:#FFFFFF;"> rend</span><span id="S4.F7.4.1.21" class="ltx_text" style="background-color:#FFFFFF;">ezvous</span>
<br class="ltx_break">
<span id="S4.F7.4.1.22" class="ltx_text" style="background-color:#FFFFFF;"> choice</span><span id="S4.F7.4.1.23" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.24" class="ltx_text" style="background-color:#FFFFFF;"> [</span><span id="S4.F7.4.1.25" class="ltx_text" style="background-color:#FFFFFF;">Warning</span><span id="S4.F7.4.1.26" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.4.1.27" class="ltx_text" style="background-color:#FFFFFF;"> R</span><span id="S4.F7.4.1.28" class="ltx_text" style="background-color:#FFFFFF;">ipe</span><span id="S4.F7.4.1.29" class="ltx_text" style="background-color:#FFFFFF;"> with</span><span id="S4.F7.4.1.30" class="ltx_text" style="background-color:#FFFFFF;"> Wine</span><span id="S4.F7.4.1.31" class="ltx_text" style="background-color:#FFFFFF;"> P</span><span id="S4.F7.4.1.32" class="ltx_text" style="background-color:#D2DB66;">uns</span><span id="S4.F7.4.1.33" class="ltx_text" style="background-color:#FFFFFF;">]</span><span id="S4.F7.4.1.34" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.35" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.36" class="ltx_text" style="background-color:#FFFFFF;">V</span><span id="S4.F7.4.1.37" class="ltx_text" style="background-color:#FFFFFF;">ital</span><span id="S4.F7.4.1.38" class="ltx_text" style="background-color:#FFFFFF;"> L</span><span id="S4.F7.4.1.39" class="ltx_text" style="background-color:#FFFFFF;">acer</span><span id="S4.F7.4.1.40" class="ltx_text" style="background-color:#FFFFFF;">da</span><span id="S4.F7.4.1.41" class="ltx_text" style="background-color:#FFFFFF;"> (</span><span id="S4.F7.4.1.42" class="ltx_text" style="background-color:#FFFFFF;">September</span>
<br class="ltx_break">
<span id="S4.F7.4.1.43" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.44" class="ltx_text" style="background-color:#FFFFFF;"> [</span><span id="S4.F7.4.1.45" class="ltx_text" style="background-color:#FFFFFF;">Warning</span><span id="S4.F7.4.1.46" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.4.1.47" class="ltx_text" style="background-color:#FFFFFF;"> Develop</span><span id="S4.F7.4.1.48" class="ltx_text" style="background-color:#FFFFFF;">ed</span><span id="S4.F7.4.1.49" class="ltx_text" style="background-color:#FFFFFF;"> with</span><span id="S4.F7.4.1.50" class="ltx_text" style="background-color:#FFFFFF;"> Evolution</span><span id="S4.F7.4.1.51" class="ltx_text" style="background-color:#FFFFFF;">ary</span><span id="S4.F7.4.1.52" class="ltx_text" style="background-color:#FFFFFF;"> P</span><span id="S4.F7.4.1.53" class="ltx_text" style="background-color:#CFD866;">uns</span><span id="S4.F7.4.1.54" class="ltx_text" style="background-color:#FFFFFF;">]</span><span id="S4.F7.4.1.55" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.56" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.57" class="ltx_text" style="background-color:#FFFFFF;">Mail</span><span id="S4.F7.4.1.58" class="ltx_text" style="background-color:#FFFFFF;">bags</span><span id="S4.F7.4.1.59" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.60" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.61" class="ltx_text" style="background-color:#FFFFFF;">T</span><span id="S4.F7.4.1.62" class="ltx_text" style="background-color:#FFFFFF;">rial</span><span id="S4.F7.4.1.63" class="ltx_text" style="background-color:#FFFFFF;"> &amp;</span>
<br class="ltx_break">
<span id="S4.F7.4.1.64" class="ltx_text" style="background-color:#FFFFFF;">al</span><span id="S4.F7.4.1.65" class="ltx_text" style="background-color:#FFFFFF;">6</span><span id="S4.F7.4.1.66" class="ltx_text" style="background-color:#FFFFFF;">606</span><span id="S4.F7.4.1.67" class="ltx_text" style="background-color:#FFFFFF;"> Luke</span><span id="S4.F7.4.1.68" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.69" class="ltx_text" style="background-color:#FFFFFF;">A</span><span id="S4.F7.4.1.70" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.71" class="ltx_text" style="background-color:#FFFFFF;">M</span><span id="S4.F7.4.1.72" class="ltx_text" style="background-color:#FFFFFF;"> me</span><span id="S4.F7.4.1.73" class="ltx_text" style="background-color:#FFFFFF;">p</span><span id="S4.F7.4.1.74" class="ltx_text" style="background-color:#DDE466;">wn</span><span id="S4.F7.4.1.75" class="ltx_text" style="background-color:#FFFFFF;">12</span><span id="S4.F7.4.1.76" class="ltx_text" style="background-color:#FFFFFF;"> Thorn</span><span id="S4.F7.4.1.77" class="ltx_text" style="background-color:#FFFFFF;">y</span><span id="S4.F7.4.1.78" class="ltx_text" style="background-color:#FFFFFF;"> [</span><span id="S4.F7.4.1.79" class="ltx_text" style="background-color:#FFFFFF;">B</span><span id="S4.F7.4.1.80" class="ltx_text" style="background-color:#FFFFFF;">]</span><span id="S4.F7.4.1.81" class="ltx_text" style="background-color:#FFFFFF;">Available</span><span id="S4.F7.4.1.82" class="ltx_text" style="background-color:#FFFFFF;"> voice</span><span id="S4.F7.4.1.83" class="ltx_text" style="background-color:#FFFFFF;"> actors</span><span id="S4.F7.4.1.84" class="ltx_text" style="background-color:#FFFFFF;">,</span>
<br class="ltx_break">
<span id="S4.F7.4.1.85" class="ltx_text" style="background-color:#FFFFFF;"> his</span><span id="S4.F7.4.1.86" class="ltx_text" style="background-color:#FFFFFF;"> picture</span><span id="S4.F7.4.1.87" class="ltx_text" style="background-color:#FFFFFF;"> of</span><span id="S4.F7.4.1.88" class="ltx_text" style="background-color:#FFFFFF;"> President</span><span id="S4.F7.4.1.89" class="ltx_text" style="background-color:#FFFFFF;"> Putin</span><span id="S4.F7.4.1.90" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.91" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.92" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.93" class="ltx_text" style="background-color:#FFFFFF;">President</span><span id="S4.F7.4.1.94" class="ltx_text" style="background-color:#FFFFFF;"> Pr</span>
<span id="S4.F7.4.1.95" class="ltx_tabular">
<span id="S4.F7.4.1.95.1" class="ltx_tr">
<span id="S4.F7.4.1.95.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.95.1.1.1" class="ltx_text"><span id="S4.F7.4.1.95.1.1.1.1" class="ltx_text" style="background-color:#A7E066;">ank</span></span></span></span>
</span><span id="S4.F7.4.1.96" class="ltx_text" style="background-color:#FFFFFF;">ster</span><span id="S4.F7.4.1.97" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.98" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.99" class="ltx_text" style="background-color:#FFFFFF;">D</span><span id="S4.F7.4.1.100" class="ltx_text" style="background-color:#FFFFFF;">mit</span><span id="S4.F7.4.1.101" class="ltx_text" style="background-color:#FFFFFF;">ry</span><span id="S4.F7.4.1.102" class="ltx_text" style="background-color:#FFFFFF;"> Rog</span><span id="S4.F7.4.1.103" class="ltx_text" style="background-color:#FFFFFF;">oz</span><span id="S4.F7.4.1.104" class="ltx_text" style="background-color:#FFFFFF;">in</span><span id="S4.F7.4.1.105" class="ltx_text" style="background-color:#FFFFFF;">,</span>
<br class="ltx_break">
Precision Eval Sequences:
<br class="ltx_break">

<span id="S4.F7.4.1.106" class="ltx_text" style="background-color:#FFFFFF;"> away</span><span id="S4.F7.4.1.107" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.4.1.108" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.109" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.110" class="ltx_text" style="background-color:#FFFFFF;">It</span><span id="S4.F7.4.1.111" class="ltx_text" style="background-color:#FFFFFF;">ching</span><span id="S4.F7.4.1.112" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.113" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.114" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.115" class="ltx_text" style="background-color:#FFFFFF;">P</span>
<span id="S4.F7.4.1.116" class="ltx_tabular">
<span id="S4.F7.4.1.116.1" class="ltx_tr">
<span id="S4.F7.4.1.116.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.116.1.1.1" class="ltx_text"><span id="S4.F7.4.1.116.1.1.1.1" class="ltx_text" style="background-color:#CCD766;">rick</span></span></span></span>
</span><span id="S4.F7.4.1.117" class="ltx_text" style="background-color:#FFFFFF;">ling</span><span id="S4.F7.4.1.118" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.119" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.120" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.121" class="ltx_text" style="background-color:#FFFFFF;">These</span><span id="S4.F7.4.1.122" class="ltx_text" style="background-color:#FFFFFF;"> are</span><span id="S4.F7.4.1.123" class="ltx_text" style="background-color:#FFFFFF;"> not</span><span id="S4.F7.4.1.124" class="ltx_text" style="background-color:#FFFFFF;"> all</span><span id="S4.F7.4.1.125" class="ltx_text" style="background-color:#FFFFFF;"> of</span><span id="S4.F7.4.1.126" class="ltx_text" style="background-color:#FFFFFF;"> the</span>
<br class="ltx_break">
<span id="S4.F7.4.1.127" class="ltx_text" style="background-color:#FFFFFF;">%</span><span id="S4.F7.4.1.128" class="ltx_text" style="background-color:#FFFFFF;"> 70</span><span id="S4.F7.4.1.129" class="ltx_text" style="background-color:#FFFFFF;">%</span><span id="S4.F7.4.1.130" class="ltx_text" style="background-color:#FFFFFF;"> 99</span><span id="S4.F7.4.1.131" class="ltx_text" style="background-color:#FFFFFF;">%</span><span id="S4.F7.4.1.132" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.133" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.4.1.134" class="ltx_text" style="background-color:#FFFFFF;">71</span><span id="S4.F7.4.1.135" class="ltx_text" style="background-color:#FFFFFF;"> No</span><span id="S4.F7.4.1.136" class="ltx_text" style="background-color:#FFFFFF;">P</span>
<span id="S4.F7.4.1.137" class="ltx_tabular">
<span id="S4.F7.4.1.137.1" class="ltx_tr">
<span id="S4.F7.4.1.137.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.137.1.1.1" class="ltx_text"><span id="S4.F7.4.1.137.1.1.1.1" class="ltx_text" style="background-color:#C9E066;">wn</span></span></span></span>
</span><span id="S4.F7.4.1.138" class="ltx_text" style="background-color:#FFFFFF;">1</span><span id="S4.F7.4.1.139" class="ltx_text" style="background-color:#FFFFFF;">nt</span><span id="S4.F7.4.1.140" class="ltx_text" style="background-color:#FFFFFF;">ended</span><span id="S4.F7.4.1.141" class="ltx_text" style="background-color:#FFFFFF;"> i</span><span id="S4.F7.4.1.142" class="ltx_text" style="background-color:#FFFFFF;">7</span><span id="S4.F7.4.1.143" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.4.1.144" class="ltx_text" style="background-color:#FFFFFF;">950</span><span id="S4.F7.4.1.145" class="ltx_text" style="background-color:#FFFFFF;"> 3</span><span id="S4.F7.4.1.146" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.4.1.147" class="ltx_text" style="background-color:#FFFFFF;">4</span>
<br class="ltx_break">
<span id="S4.F7.4.1.148" class="ltx_text" style="background-color:#FFFFFF;">ott</span><span id="S4.F7.4.1.149" class="ltx_text" style="background-color:#FFFFFF;">one</span><span id="S4.F7.4.1.150" class="ltx_text" style="background-color:#FFFFFF;">e</span><span id="S4.F7.4.1.151" class="ltx_text" style="background-color:#FFFFFF;"> Wh</span><span id="S4.F7.4.1.152" class="ltx_text" style="background-color:#FFFFFF;">ims</span><span id="S4.F7.4.1.153" class="ltx_text" style="background-color:#FFFFFF;">ic</span><span id="S4.F7.4.1.154" class="ltx_text" style="background-color:#FFFFFF;">ott</span><span id="S4.F7.4.1.155" class="ltx_text" style="background-color:#FFFFFF;"> M</span><span id="S4.F7.4.1.156" class="ltx_text" style="background-color:#FFFFFF;"> 0</span><span id="S4.F7.4.1.157" class="ltx_text" style="background-color:#FFFFFF;"> Pr</span>
<span id="S4.F7.4.1.158" class="ltx_tabular">
<span id="S4.F7.4.1.158.1" class="ltx_tr">
<span id="S4.F7.4.1.158.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.158.1.1.1" class="ltx_text"><span id="S4.F7.4.1.158.1.1.1.1" class="ltx_text" style="background-color:#A3E066;">ank</span></span></span></span>
</span><span id="S4.F7.4.1.159" class="ltx_text" style="background-color:#FFFFFF;">ster</span><span id="S4.F7.4.1.160" class="ltx_text" style="background-color:#FFFFFF;"> /</span><span id="S4.F7.4.1.161" class="ltx_text" style="background-color:#FFFFFF;"> Inf</span><span id="S4.F7.4.1.162" class="ltx_text" style="background-color:#FFFFFF;">iltr</span><span id="S4.F7.4.1.163" class="ltx_text" style="background-color:#FFFFFF;">ator</span><span id="S4.F7.4.1.164" class="ltx_text" style="background-color:#FFFFFF;"> Pr</span>
<span id="S4.F7.4.1.165" class="ltx_tabular">
<span id="S4.F7.4.1.165.1" class="ltx_tr">
<span id="S4.F7.4.1.165.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.165.1.1.1" class="ltx_text"><span id="S4.F7.4.1.165.1.1.1.1" class="ltx_text" style="background-color:#B6E066;">ank</span></span></span></span>
</span><span id="S4.F7.4.1.166" class="ltx_text" style="background-color:#FFFFFF;">ster</span><span id="S4.F7.4.1.167" class="ltx_text" style="background-color:#FFFFFF;"> /</span><span id="S4.F7.4.1.168" class="ltx_text" style="background-color:#FFFFFF;"> Inf</span>
<br class="ltx_break">
<span id="S4.F7.4.1.169" class="ltx_text" style="background-color:#FFFFFF;"> of</span><span id="S4.F7.4.1.170" class="ltx_text" style="background-color:#FFFFFF;"> Chimera</span><span id="S4.F7.4.1.171" class="ltx_text" style="background-color:#FFFFFF;"> players</span><span id="S4.F7.4.1.172" class="ltx_text" style="background-color:#FFFFFF;"> using</span><span id="S4.F7.4.1.173" class="ltx_text" style="background-color:#FFFFFF;"> one</span><span id="S4.F7.4.1.174" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.4.1.175" class="ltx_text" style="background-color:#FFFFFF;">dimensional</span><span id="S4.F7.4.1.176" class="ltx_text" style="background-color:#FFFFFF;"> teams</span><span id="S4.F7.4.1.177" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.4.1.178" class="ltx_text" style="background-color:#FFFFFF;"> Pr</span>
<span id="S4.F7.4.1.179" class="ltx_tabular">
<span id="S4.F7.4.1.179.1" class="ltx_tr">
<span id="S4.F7.4.1.179.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.179.1.1.1" class="ltx_text"><span id="S4.F7.4.1.179.1.1.1.1" class="ltx_text" style="background-color:#99E066;">ank</span></span></span></span>
</span>
<br class="ltx_break">
<span id="S4.F7.4.1.180" class="ltx_text" style="background-color:#FFFFFF;"> of</span><span id="S4.F7.4.1.181" class="ltx_text" style="background-color:#FFFFFF;"> Chimera</span><span id="S4.F7.4.1.182" class="ltx_text" style="background-color:#FFFFFF;"> players</span><span id="S4.F7.4.1.183" class="ltx_text" style="background-color:#FFFFFF;"> using</span><span id="S4.F7.4.1.184" class="ltx_text" style="background-color:#FFFFFF;"> one</span><span id="S4.F7.4.1.185" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.4.1.186" class="ltx_text" style="background-color:#FFFFFF;">dimensional</span><span id="S4.F7.4.1.187" class="ltx_text" style="background-color:#FFFFFF;"> teams</span><span id="S4.F7.4.1.188" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.4.1.189" class="ltx_text" style="background-color:#FFFFFF;"> Pr</span>
<span id="S4.F7.4.1.190" class="ltx_tabular">
<span id="S4.F7.4.1.190.1" class="ltx_tr">
<span id="S4.F7.4.1.190.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.4.1.190.1.1.1" class="ltx_text"><span id="S4.F7.4.1.190.1.1.1.1" class="ltx_text" style="background-color:#A5E066;">ank</span></span></span></span>
</span><span id="S4.F7.4.1.191" class="ltx_text" style="background-color:#FFFFFF;">ster</span><span id="S4.F7.4.1.192" class="ltx_text" style="background-color:#FFFFFF;"> Sp</span><span id="S4.F7.4.1.193" class="ltx_text" style="background-color:#FFFFFF;">ore</span><span id="S4.F7.4.1.194" class="ltx_text" style="background-color:#FFFFFF;">S</span><span id="S4.F7.4.1.195" class="ltx_text" style="background-color:#FFFFFF;">eed</span><span id="S4.F7.4.1.196" class="ltx_text" style="background-color:#FFFFFF;">,</span><span id="S4.F7.4.1.197" class="ltx_text" style="background-color:#FFFFFF;"> Shell</span><span id="S4.F7.4.1.198" class="ltx_text" style="background-color:#FFFFFF;"> Smash</span><span id="S4.F7.4.1.199" class="ltx_text" style="background-color:#FFFFFF;">,</span><span id="S4.F7.4.1.200" class="ltx_text" style="background-color:#FFFFFF;"> Poison</span>
<br class="ltx_break"></span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S4.F7.5" class="ltx_p ltx_align_left ltx_figure_panel"><span id="S4.F7.5.1" class="ltx_text" style="font-size:70%;">Recall Eval Sequences:
<br class="ltx_break">

<span id="S4.F7.5.1.1" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.2" class="ltx_text" style="background-color:#FFFFFF;">M</span><span id="S4.F7.5.1.3" class="ltx_text" style="background-color:#FFFFFF;">eal</span><span id="S4.F7.5.1.4" class="ltx_text" style="background-color:#FFFFFF;"> 1</span><span id="S4.F7.5.1.5" class="ltx_text" style="background-color:#FFFFFF;"> 06</span><span id="S4.F7.5.1.6" class="ltx_text" style="background-color:#FFFFFF;">h</span><span id="S4.F7.5.1.7" class="ltx_text" style="background-color:#EFF266;">30</span><span id="S4.F7.5.1.8" class="ltx_text" style="background-color:#FFFFFF;"> O</span><span id="S4.F7.5.1.9" class="ltx_text" style="background-color:#FFFFFF;">ats</span><span id="S4.F7.5.1.10" class="ltx_text" style="background-color:#FFFFFF;">,</span><span id="S4.F7.5.1.11" class="ltx_text" style="background-color:#FFFFFF;"> double</span><span id="S4.F7.5.1.12" class="ltx_text" style="background-color:#FFFFFF;"> serving</span><span id="S4.F7.5.1.13" class="ltx_text" style="background-color:#FFFFFF;"> of</span><span id="S4.F7.5.1.14" class="ltx_text" style="background-color:#FFFFFF;"> Whe</span><span id="S4.F7.5.1.15" class="ltx_text" style="background-color:#FFFFFF;">y</span><span id="S4.F7.5.1.16" class="ltx_text" style="background-color:#FFFFFF;">,</span><span id="S4.F7.5.1.17" class="ltx_text" style="background-color:#FFFFFF;"> 6</span>
<br class="ltx_break">
<span id="S4.F7.5.1.18" class="ltx_text" style="background-color:#FFFFFF;">’s</span><span id="S4.F7.5.1.19" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.5.1.20" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.21" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.22" class="ltx_text" style="background-color:#FFFFFF;">04</span><span id="S4.F7.5.1.23" class="ltx_text" style="background-color:#FFFFFF;">Jul</span><span id="S4.F7.5.1.24" class="ltx_text" style="background-color:#FFFFFF;">65</span><span id="S4.F7.5.1.25" class="ltx_text" style="background-color:#FFFFFF;"> to</span><span id="S4.F7.5.1.26" class="ltx_text" style="background-color:#FFFFFF;"> 01</span><span id="S4.F7.5.1.27" class="ltx_text" style="background-color:#FFFFFF;">Aug</span><span id="S4.F7.5.1.28" class="ltx_text" style="background-color:#E8EC66;">65</span><span id="S4.F7.5.1.29" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.30" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.31" class="ltx_text" style="background-color:#FFFFFF;">1</span><span id="S4.F7.5.1.32" class="ltx_text" style="background-color:#FFFFFF;">/</span><span id="S4.F7.5.1.33" class="ltx_text" style="background-color:#FFFFFF;">3</span><span id="S4.F7.5.1.34" class="ltx_text" style="background-color:#FFFFFF;"> un</span>
<br class="ltx_break">
<span id="S4.F7.5.1.35" class="ltx_text" style="background-color:#FFFFFF;"> at</span><span id="S4.F7.5.1.36" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.37" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.38" class="ltx_text" style="background-color:#FFFFFF;">2013</span><span id="S4.F7.5.1.39" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.5.1.40" class="ltx_text" style="background-color:#FFFFFF;">11</span><span id="S4.F7.5.1.41" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.5.1.42" class="ltx_text" style="background-color:#FFFFFF;">16</span><span id="S4.F7.5.1.43" class="ltx_text" style="background-color:#FFFFFF;"> 09</span><span id="S4.F7.5.1.44" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.45" class="ltx_text" style="background-color:#F4F666;">47</span><span id="S4.F7.5.1.46" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.47" class="ltx_text" style="background-color:#FFFFFF;">18</span><span id="S4.F7.5.1.48" class="ltx_text" style="background-color:#FFFFFF;"> (</span><span id="S4.F7.5.1.49" class="ltx_text" style="background-color:#FFFFFF;">25</span><span id="S4.F7.5.1.50" class="ltx_text" style="background-color:#FFFFFF;">)</span><span id="S4.F7.5.1.51" class="ltx_text" style="background-color:#FFFFFF;"> chat</span><span id="S4.F7.5.1.52" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.53" class="ltx_text" style="background-color:#FFFFFF;"> &lt;</span><span id="S4.F7.5.1.54" class="ltx_text" style="background-color:#FFFFFF;">Diamond</span><span id="S4.F7.5.1.55" class="ltx_text" style="background-color:#FFFFFF;">Card</span>
<br class="ltx_break">
<span id="S4.F7.5.1.56" class="ltx_text" style="background-color:#FFFFFF;">0000</span><span id="S4.F7.5.1.57" class="ltx_text" style="background-color:#FFFFFF;"> -</span><span id="S4.F7.5.1.58" class="ltx_text" style="background-color:#FFFFFF;"> 00</span><span id="S4.F7.5.1.59" class="ltx_text" style="background-color:#D9E066;">26</span><span id="S4.F7.5.1.60" class="ltx_text" style="background-color:#D6DE66;">00</span><span id="S4.F7.5.1.61" class="ltx_text" style="background-color:#FFFFFF;">76</span><span id="S4.F7.5.1.62" class="ltx_text" style="background-color:#FFFFFF;">]</span><span id="S4.F7.5.1.63" class="ltx_text" style="background-color:#FFFFFF;"> 119</span><span id="S4.F7.5.1.64" class="ltx_text" style="background-color:#FFFFFF;"> pages</span><span id="S4.F7.5.1.65" class="ltx_text" style="background-color:#FFFFFF;"> 5</span><span id="S4.F7.5.1.66" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.67" class="ltx_text" style="background-color:#FFFFFF;"> [</span><span id="S4.F7.5.1.68" class="ltx_text" style="background-color:#FFFFFF;">002</span><span id="S4.F7.5.1.69" class="ltx_text" style="background-color:#FFFFFF;">60</span>
<br class="ltx_break">
<span id="S4.F7.5.1.70" class="ltx_text" style="background-color:#FFFFFF;">,</span><span id="S4.F7.5.1.71" class="ltx_text" style="background-color:#FFFFFF;"> calm</span><span id="S4.F7.5.1.72" class="ltx_text" style="background-color:#FFFFFF;"> it</span><span id="S4.F7.5.1.73" class="ltx_text" style="background-color:#FFFFFF;">2013</span><span id="S4.F7.5.1.74" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.5.1.75" class="ltx_text" style="background-color:#FFFFFF;">11</span><span id="S4.F7.5.1.76" class="ltx_text" style="background-color:#FFFFFF;">-</span><span id="S4.F7.5.1.77" class="ltx_text" style="background-color:#FFFFFF;">11</span><span id="S4.F7.5.1.78" class="ltx_text" style="background-color:#FFFFFF;"> 20</span><span id="S4.F7.5.1.79" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.80" class="ltx_text" style="background-color:#F2F466;">00</span><span id="S4.F7.5.1.81" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.82" class="ltx_text" style="background-color:#FFFFFF;">43</span><span id="S4.F7.5.1.83" class="ltx_text" style="background-color:#FFFFFF;"> (</span><span id="S4.F7.5.1.84" class="ltx_text" style="background-color:#FFFFFF;">25</span><span id="S4.F7.5.1.85" class="ltx_text" style="background-color:#FFFFFF;">)</span><span id="S4.F7.5.1.86" class="ltx_text" style="background-color:#FFFFFF;"> chat</span><span id="S4.F7.5.1.87" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.88" class="ltx_text" style="background-color:#FFFFFF;"> It</span><span id="S4.F7.5.1.89" class="ltx_text" style="background-color:#FFFFFF;">’s</span><span id="S4.F7.5.1.90" class="ltx_text" style="background-color:#FFFFFF;"> vanity</span>
<br class="ltx_break">

Precision Eval Sequences:
<br class="ltx_break">

<span id="S4.F7.5.1.91" class="ltx_text" style="background-color:#FFFFFF;">1</span><span id="S4.F7.5.1.92" class="ltx_text" style="background-color:#FFFFFF;"> Tue</span><span id="S4.F7.5.1.93" class="ltx_text" style="background-color:#FFFFFF;"> Mar</span><span id="S4.F7.5.1.94" class="ltx_text" style="background-color:#FFFFFF;"> 13</span><span id="S4.F7.5.1.95" class="ltx_text" style="background-color:#FFFFFF;"> 22</span><span id="S4.F7.5.1.96" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.97" class="ltx_text" style="background-color:#FFFFFF;">37</span><span id="S4.F7.5.1.98" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.99" class="ltx_text" style="background-color:#FFFFFF;">59</span><span id="S4.F7.5.1.100" class="ltx_text" style="background-color:#FFFFFF;"> EST</span>
<span id="S4.F7.5.1.101" class="ltx_tabular">
<span id="S4.F7.5.1.101.1" class="ltx_tr">
<span id="S4.F7.5.1.101.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.5.1.101.1.1.1" class="ltx_text"><span id="S4.F7.5.1.101.1.1.1.1" class="ltx_text" style="background-color:#DCE366;"> 2001</span></span></span></span>
</span><span id="S4.F7.5.1.102" class="ltx_text" style="background-color:#FFFFFF;"> i</span><span id="S4.F7.5.1.103" class="ltx_text" style="background-color:#FFFFFF;">686</span><span id="S4.F7.5.1.104" class="ltx_text" style="background-color:#FFFFFF;"> Loc</span><span id="S4.F7.5.1.105" class="ltx_text" style="background-color:#FFFFFF;">ale</span><span id="S4.F7.5.1.106" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.107" class="ltx_text" style="background-color:#FFFFFF;"> L</span><span id="S4.F7.5.1.108" class="ltx_text" style="background-color:#FFFFFF;">ANG</span><span id="S4.F7.5.1.109" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.110" class="ltx_text" style="background-color:#FFFFFF;">C</span>
<br class="ltx_break">
<span id="S4.F7.5.1.111" class="ltx_text" style="background-color:#FFFFFF;">K</span><span id="S4.F7.5.1.112" class="ltx_text" style="background-color:#FFFFFF;">iw</span><span id="S4.F7.5.1.113" class="ltx_text" style="background-color:#FFFFFF;">i</span><span id="S4.F7.5.1.114" class="ltx_text" style="background-color:#FFFFFF;">76</span><span id="S4.F7.5.1.115" class="ltx_text" style="background-color:#FFFFFF;"> added</span><span id="S4.F7.5.1.116" class="ltx_text" style="background-color:#FFFFFF;"> 01</span><span id="S4.F7.5.1.117" class="ltx_text" style="background-color:#FFFFFF;">:</span><span id="S4.F7.5.1.118" class="ltx_text" style="background-color:#FFFFFF;">08</span><span id="S4.F7.5.1.119" class="ltx_text" style="background-color:#FFFFFF;"> -</span><span id="S4.F7.5.1.120" class="ltx_text" style="background-color:#FFFFFF;"> Apr</span>
<span id="S4.F7.5.1.121" class="ltx_tabular">
<span id="S4.F7.5.1.121.1" class="ltx_tr">
<span id="S4.F7.5.1.121.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.5.1.121.1.1.1" class="ltx_text"><span id="S4.F7.5.1.121.1.1.1.1" class="ltx_text" style="background-color:#DEE566;"> 28</span></span></span></span>
</span><span id="S4.F7.5.1.122" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.123" class="ltx_text" style="background-color:#FFFFFF;">\n</span><span id="S4.F7.5.1.124" class="ltx_text" style="background-color:#FFFFFF;">Che</span><span id="S4.F7.5.1.125" class="ltx_text" style="background-color:#FFFFFF;">ers</span><span id="S4.F7.5.1.126" class="ltx_text" style="background-color:#FFFFFF;"> Clive</span><span id="S4.F7.5.1.127" class="ltx_text" style="background-color:#FFFFFF;"> and</span><span id="S4.F7.5.1.128" class="ltx_text" style="background-color:#FFFFFF;"> hopefully</span><span id="S4.F7.5.1.129" class="ltx_text" style="background-color:#FFFFFF;"> you</span><span id="S4.F7.5.1.130" class="ltx_text" style="background-color:#FFFFFF;"> can</span><span id="S4.F7.5.1.131" class="ltx_text" style="background-color:#FFFFFF;"> see</span>
<br class="ltx_break">
<span id="S4.F7.5.1.132" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.5.1.133" class="ltx_text" style="background-color:#FFFFFF;">aspx</span><span id="S4.F7.5.1.134" class="ltx_text" style="background-color:#FFFFFF;">?</span><span id="S4.F7.5.1.135" class="ltx_text" style="background-color:#FFFFFF;">key</span><span id="S4.F7.5.1.136" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.137" class="ltx_text" style="background-color:#FFFFFF;">24</span><span id="S4.F7.5.1.138" class="ltx_text" style="background-color:#FFFFFF;">88</span><span id="S4.F7.5.1.139" class="ltx_text" style="background-color:#FFFFFF;">58</span><span id="S4.F7.5.1.140" class="ltx_text" style="background-color:#FFFFFF;">886</span><span id="S4.F7.5.1.141" class="ltx_text" style="background-color:#FFFFFF;">|</span>
<span id="S4.F7.5.1.142" class="ltx_tabular">
<span id="S4.F7.5.1.142.1" class="ltx_tr">
<span id="S4.F7.5.1.142.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.5.1.142.1.1.1" class="ltx_text"><span id="S4.F7.5.1.142.1.1.1.1" class="ltx_text" style="background-color:#FFFFFF;">0001</span></span></span></span>
</span><span id="S4.F7.5.1.143" class="ltx_text" style="background-color:#FFFFFF;">00</span><span id="S4.F7.5.1.144" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.145" class="ltx_text" style="background-color:#FFFFFF;">C</span><span id="S4.F7.5.1.146" class="ltx_text" style="background-color:#FFFFFF;">MC</span><span id="S4.F7.5.1.147" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.148" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.149" class="ltx_text" style="background-color:#FFFFFF;">PN</span><span id="S4.F7.5.1.150" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.151" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.152" class="ltx_text" style="background-color:#FFFFFF;">Is</span>
<br class="ltx_break">
<span id="S4.F7.5.1.153" class="ltx_text" style="background-color:#FFFFFF;">.</span><span id="S4.F7.5.1.154" class="ltx_text" style="background-color:#FFFFFF;">aspx</span><span id="S4.F7.5.1.155" class="ltx_text" style="background-color:#FFFFFF;">?</span><span id="S4.F7.5.1.156" class="ltx_text" style="background-color:#FFFFFF;">key</span><span id="S4.F7.5.1.157" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.158" class="ltx_text" style="background-color:#FFFFFF;">24</span><span id="S4.F7.5.1.159" class="ltx_text" style="background-color:#FFFFFF;">88</span><span id="S4.F7.5.1.160" class="ltx_text" style="background-color:#FFFFFF;">58</span><span id="S4.F7.5.1.161" class="ltx_text" style="background-color:#FFFFFF;">886</span><span id="S4.F7.5.1.162" class="ltx_text" style="background-color:#FFFFFF;">|</span>
<span id="S4.F7.5.1.163" class="ltx_tabular">
<span id="S4.F7.5.1.163.1" class="ltx_tr">
<span id="S4.F7.5.1.163.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.5.1.163.1.1.1" class="ltx_text"><span id="S4.F7.5.1.163.1.1.1.1" class="ltx_text" style="background-color:#FFFFFF;">0001</span></span></span></span>
</span><span id="S4.F7.5.1.164" class="ltx_text" style="background-color:#FFFFFF;">01</span><span id="S4.F7.5.1.165" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.166" class="ltx_text" style="background-color:#FFFFFF;">C</span><span id="S4.F7.5.1.167" class="ltx_text" style="background-color:#FFFFFF;">MC</span><span id="S4.F7.5.1.168" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.169" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.170" class="ltx_text" style="background-color:#FFFFFF;">PN</span><span id="S4.F7.5.1.171" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.172" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.173" class="ltx_text" style="background-color:#FFFFFF;">Is</span>
<br class="ltx_break">
<span id="S4.F7.5.1.174" class="ltx_text" style="background-color:#FFFFFF;">key</span><span id="S4.F7.5.1.175" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.176" class="ltx_text" style="background-color:#FFFFFF;">24</span><span id="S4.F7.5.1.177" class="ltx_text" style="background-color:#FFFFFF;">88</span><span id="S4.F7.5.1.178" class="ltx_text" style="background-color:#FFFFFF;">58</span><span id="S4.F7.5.1.179" class="ltx_text" style="background-color:#FFFFFF;">886</span><span id="S4.F7.5.1.180" class="ltx_text" style="background-color:#FFFFFF;">|</span>
<span id="S4.F7.5.1.181" class="ltx_tabular">
<span id="S4.F7.5.1.181.1" class="ltx_tr">
<span id="S4.F7.5.1.181.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.5.1.181.1.1.1" class="ltx_text"><span id="S4.F7.5.1.181.1.1.1.1" class="ltx_text" style="background-color:#FFFFFF;">0001</span></span></span></span>
</span><span id="S4.F7.5.1.182" class="ltx_text" style="background-color:#FFFFFF;">02</span><span id="S4.F7.5.1.183" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.184" class="ltx_text" style="background-color:#FFFFFF;">C</span><span id="S4.F7.5.1.185" class="ltx_text" style="background-color:#FFFFFF;">MC</span><span id="S4.F7.5.1.186" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.187" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.188" class="ltx_text" style="background-color:#FFFFFF;">PN</span><span id="S4.F7.5.1.189" class="ltx_text" style="background-color:#FFFFFF;">=</span><span id="S4.F7.5.1.190" class="ltx_text" style="background-color:#FFFFFF;">&amp;</span><span id="S4.F7.5.1.191" class="ltx_text" style="background-color:#FFFFFF;">Is</span>
<br class="ltx_break"></span></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div class="ltx_pagination ltx_figure_panel ltx_role_end_2_columns"></div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F7.8.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="S4.F7.9.2" class="ltx_text" style="font-size:90%;">Qualitative examples of latents with high and low precision/recall N2G explanations.
<br class="ltx_break">Key: <span id="S4.F7.9.2.1" class="ltx_text" style="background-color:#93E066;">Green</span> = Ground truth feature activation, 
<span id="S4.F7.9.2.2" class="ltx_tabular">
<span id="S4.F7.9.2.2.1" class="ltx_tr">
<span id="S4.F7.9.2.2.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left"><span id="S4.F7.9.2.2.1.1.1" class="ltx_text">Underline</span></span></span>
</span> = N2G predicted feature activation</span></figcaption>
</figure>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p">Unfortunately, precision is extremely expensive to evaluate when the simulations are using GPT-4 as in <cite class="ltx_cite ltx_citemacro_citet">Bills et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite>. As an initial exploration, we focus on an improved version of Neuron to Graph (N2G) <cite class="ltx_cite ltx_citemacro_citep">(Foote et al., <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, a substantially less expressive but much cheaper method that outputs explanations in the form of collections of n-grams with wildcards.
In the future, we would like to explore ways to make it more tractable to approximate precision for arbitrary English explanations.</p>
</div>
<div id="S4.SS3.p3" class="ltx_para">
<p id="S4.SS3.p3.1" class="ltx_p">To construct a N2G explanation, we start with some sequences that activate the latent. For each one, we find the shortest suffix that still activates the latent.<span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span>with at least half the original activation strength</span></span></span> We then check whether any position in the n-gram can be replaced by a padding token, to insert wildcard tokens. We also check whether the explanation should be dependent on absolute position by checking whether inserting a padding token at the beginning matters.
We use a random sample of up to 16 nonzero activations to build the graph, and another 16 as true positives for computing recall.</p>
</div>
<div id="S4.SS3.p4" class="ltx_para">
<p id="S4.SS3.p4.1" class="ltx_p">Results for GPT-2 small are found in <a href="#A5.F25.sf1" title="25(a) ‣ Figure 25 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">25(a)</span></a> and <a href="#A5.F25.sf2" title="In Figure 25 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref"><span class="ltx_text ltx_ref_tag">25(b)</span></a>. Note that dense token patterns are trivial to explain, thus <math id="S4.SS3.p4.1.m1.2" class="ltx_Math" alttext="n=2048,k=512" display="inline"><semantics id="S4.SS3.p4.1.m1.2a"><mrow id="S4.SS3.p4.1.m1.2.2.2" xref="S4.SS3.p4.1.m1.2.2.3.cmml"><mrow id="S4.SS3.p4.1.m1.1.1.1.1" xref="S4.SS3.p4.1.m1.1.1.1.1.cmml"><mi id="S4.SS3.p4.1.m1.1.1.1.1.2" xref="S4.SS3.p4.1.m1.1.1.1.1.2.cmml">n</mi><mo id="S4.SS3.p4.1.m1.1.1.1.1.1" xref="S4.SS3.p4.1.m1.1.1.1.1.1.cmml">=</mo><mn id="S4.SS3.p4.1.m1.1.1.1.1.3" xref="S4.SS3.p4.1.m1.1.1.1.1.3.cmml">2048</mn></mrow><mo id="S4.SS3.p4.1.m1.2.2.2.3" xref="S4.SS3.p4.1.m1.2.2.3a.cmml">,</mo><mrow id="S4.SS3.p4.1.m1.2.2.2.2" xref="S4.SS3.p4.1.m1.2.2.2.2.cmml"><mi id="S4.SS3.p4.1.m1.2.2.2.2.2" xref="S4.SS3.p4.1.m1.2.2.2.2.2.cmml">k</mi><mo id="S4.SS3.p4.1.m1.2.2.2.2.1" xref="S4.SS3.p4.1.m1.2.2.2.2.1.cmml">=</mo><mn id="S4.SS3.p4.1.m1.2.2.2.2.3" xref="S4.SS3.p4.1.m1.2.2.2.2.3.cmml">512</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p4.1.m1.2b"><apply id="S4.SS3.p4.1.m1.2.2.3.cmml" xref="S4.SS3.p4.1.m1.2.2.2"><csymbol cd="ambiguous" id="S4.SS3.p4.1.m1.2.2.3a.cmml" xref="S4.SS3.p4.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S4.SS3.p4.1.m1.1.1.1.1.cmml" xref="S4.SS3.p4.1.m1.1.1.1.1"><eq id="S4.SS3.p4.1.m1.1.1.1.1.1.cmml" xref="S4.SS3.p4.1.m1.1.1.1.1.1"></eq><ci id="S4.SS3.p4.1.m1.1.1.1.1.2.cmml" xref="S4.SS3.p4.1.m1.1.1.1.1.2">𝑛</ci><cn type="integer" id="S4.SS3.p4.1.m1.1.1.1.1.3.cmml" xref="S4.SS3.p4.1.m1.1.1.1.1.3">2048</cn></apply><apply id="S4.SS3.p4.1.m1.2.2.2.2.cmml" xref="S4.SS3.p4.1.m1.2.2.2.2"><eq id="S4.SS3.p4.1.m1.2.2.2.2.1.cmml" xref="S4.SS3.p4.1.m1.2.2.2.2.1"></eq><ci id="S4.SS3.p4.1.m1.2.2.2.2.2.cmml" xref="S4.SS3.p4.1.m1.2.2.2.2.2">𝑘</ci><cn type="integer" id="S4.SS3.p4.1.m1.2.2.2.2.3.cmml" xref="S4.SS3.p4.1.m1.2.2.2.2.3">512</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p4.1.m1.2c">n=2048,k=512</annotation></semantics></math> latents are easy to explain on average since many latents activate extremely densely (see <a href="#A5.SS5" title="E.5 Solutions with dense latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.5</span></a>)<span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note">13</span>This highlights an issue with our precision/recall metrics, which care only about binarized values. We also propose a more expensive metric which uses simulated values <a href="#S4.SS4" title="4.4 Explanation reconstruction ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.4</span></a> and addresses this issue. </span></span></span>.
In general, autoencoders with more total latents and fewer active latents are easiest to model with N2G.</p>
</div>
<div id="S4.SS3.p5" class="ltx_para">
<p id="S4.SS3.p5.2" class="ltx_p">We also obtain evidence that TopK models have fewer spurious positive activations than their ReLU counterparts. N2G explanations have significantly better recall (&gt;1.5x) and only slightly worse precision (&gt;0.9x) for TopK models with the same <math id="S4.SS3.p5.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS3.p5.1.m1.1a"><mi id="S4.SS3.p5.1.m1.1.1" xref="S4.SS3.p5.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p5.1.m1.1b"><ci id="S4.SS3.p5.1.m1.1.1.cmml" xref="S4.SS3.p5.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p5.1.m1.1c">n</annotation></semantics></math> (resulting in better F1 scores) and similar L<sub id="S4.SS3.p5.2.1" class="ltx_sub">0</sub> (<a href="#A5.F24" title="Figure 24 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 24</span></a>).</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Explanation reconstruction</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">When our goal is for a model’s activations to be interpretable, one question we can ask is: how much performance do we sacrifice if we use only the parts of the model that we can interpret?</p>
</div>
<div id="S4.SS4.p2" class="ltx_para">
<p id="S4.SS4.p2.1" class="ltx_p">Our downstream loss metric measures how much of the performance we’re capturing (but our features could be uninterpretable), and our explanation based metric measures how monosemantic our features are (but they might not explain most of the model). This suggests combining our downstream loss and explanation metrics, by using our explanations to simulate autoencoder latents, and then checking downstream loss after decoding. This metric also has the advantage that it values both recall and precision in a way that is principled, and also values recall more for latents that activate more densely.</p>
</div>
<div id="S4.SS4.p3" class="ltx_para">
<p id="S4.SS4.p3.3" class="ltx_p">We tried this with N2G explanations. N2G produces a simulated value based on the node in the trie, but we scale this value to minimize variance explained. Specifically, we compute <math id="S4.SS4.p3.1.m1.2" class="ltx_Math" alttext="E[sa]/E[s^{2}]" display="inline"><semantics id="S4.SS4.p3.1.m1.2a"><mrow id="S4.SS4.p3.1.m1.2.2" xref="S4.SS4.p3.1.m1.2.2.cmml"><mrow id="S4.SS4.p3.1.m1.1.1.1" xref="S4.SS4.p3.1.m1.1.1.1.cmml"><mrow id="S4.SS4.p3.1.m1.1.1.1.1" xref="S4.SS4.p3.1.m1.1.1.1.1.cmml"><mi id="S4.SS4.p3.1.m1.1.1.1.1.3" xref="S4.SS4.p3.1.m1.1.1.1.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p3.1.m1.1.1.1.1.2" xref="S4.SS4.p3.1.m1.1.1.1.1.2.cmml">​</mo><mrow id="S4.SS4.p3.1.m1.1.1.1.1.1.1" xref="S4.SS4.p3.1.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.SS4.p3.1.m1.1.1.1.1.1.1.2" xref="S4.SS4.p3.1.m1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.2" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.1" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.3" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.3.cmml">a</mi></mrow><mo stretchy="false" id="S4.SS4.p3.1.m1.1.1.1.1.1.1.3" xref="S4.SS4.p3.1.m1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S4.SS4.p3.1.m1.1.1.1.2" xref="S4.SS4.p3.1.m1.1.1.1.2.cmml">/</mo><mi id="S4.SS4.p3.1.m1.1.1.1.3" xref="S4.SS4.p3.1.m1.1.1.1.3.cmml">E</mi></mrow><mo lspace="0em" rspace="0em" id="S4.SS4.p3.1.m1.2.2.3" xref="S4.SS4.p3.1.m1.2.2.3.cmml">​</mo><mrow id="S4.SS4.p3.1.m1.2.2.2.1" xref="S4.SS4.p3.1.m1.2.2.2.2.cmml"><mo stretchy="false" id="S4.SS4.p3.1.m1.2.2.2.1.2" xref="S4.SS4.p3.1.m1.2.2.2.2.1.cmml">[</mo><msup id="S4.SS4.p3.1.m1.2.2.2.1.1" xref="S4.SS4.p3.1.m1.2.2.2.1.1.cmml"><mi id="S4.SS4.p3.1.m1.2.2.2.1.1.2" xref="S4.SS4.p3.1.m1.2.2.2.1.1.2.cmml">s</mi><mn id="S4.SS4.p3.1.m1.2.2.2.1.1.3" xref="S4.SS4.p3.1.m1.2.2.2.1.1.3.cmml">2</mn></msup><mo stretchy="false" id="S4.SS4.p3.1.m1.2.2.2.1.3" xref="S4.SS4.p3.1.m1.2.2.2.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p3.1.m1.2b"><apply id="S4.SS4.p3.1.m1.2.2.cmml" xref="S4.SS4.p3.1.m1.2.2"><times id="S4.SS4.p3.1.m1.2.2.3.cmml" xref="S4.SS4.p3.1.m1.2.2.3"></times><apply id="S4.SS4.p3.1.m1.1.1.1.cmml" xref="S4.SS4.p3.1.m1.1.1.1"><divide id="S4.SS4.p3.1.m1.1.1.1.2.cmml" xref="S4.SS4.p3.1.m1.1.1.1.2"></divide><apply id="S4.SS4.p3.1.m1.1.1.1.1.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1"><times id="S4.SS4.p3.1.m1.1.1.1.1.2.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.2"></times><ci id="S4.SS4.p3.1.m1.1.1.1.1.3.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.3">𝐸</ci><apply id="S4.SS4.p3.1.m1.1.1.1.1.1.2.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="S4.SS4.p3.1.m1.1.1.1.1.1.2.1.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1"><times id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.1"></times><ci id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S4.SS4.p3.1.m1.1.1.1.1.1.1.1.3">𝑎</ci></apply></apply></apply><ci id="S4.SS4.p3.1.m1.1.1.1.3.cmml" xref="S4.SS4.p3.1.m1.1.1.1.3">𝐸</ci></apply><apply id="S4.SS4.p3.1.m1.2.2.2.2.cmml" xref="S4.SS4.p3.1.m1.2.2.2.1"><csymbol cd="latexml" id="S4.SS4.p3.1.m1.2.2.2.2.1.cmml" xref="S4.SS4.p3.1.m1.2.2.2.1.2">delimited-[]</csymbol><apply id="S4.SS4.p3.1.m1.2.2.2.1.1.cmml" xref="S4.SS4.p3.1.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS4.p3.1.m1.2.2.2.1.1.1.cmml" xref="S4.SS4.p3.1.m1.2.2.2.1.1">superscript</csymbol><ci id="S4.SS4.p3.1.m1.2.2.2.1.1.2.cmml" xref="S4.SS4.p3.1.m1.2.2.2.1.1.2">𝑠</ci><cn type="integer" id="S4.SS4.p3.1.m1.2.2.2.1.1.3.cmml" xref="S4.SS4.p3.1.m1.2.2.2.1.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p3.1.m1.2c">E[sa]/E[s^{2}]</annotation></semantics></math>, where <math id="S4.SS4.p3.2.m2.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S4.SS4.p3.2.m2.1a"><mi id="S4.SS4.p3.2.m2.1.1" xref="S4.SS4.p3.2.m2.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p3.2.m2.1b"><ci id="S4.SS4.p3.2.m2.1.1.cmml" xref="S4.SS4.p3.2.m2.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p3.2.m2.1c">s</annotation></semantics></math> is the simulated value and <math id="S4.SS4.p3.3.m3.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S4.SS4.p3.3.m3.1a"><mi id="S4.SS4.p3.3.m3.1.1" xref="S4.SS4.p3.3.m3.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p3.3.m3.1b"><ci id="S4.SS4.p3.3.m3.1.1.cmml" xref="S4.SS4.p3.3.m3.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p3.3.m3.1c">a</annotation></semantics></math> is the true value, and we estimate this quantity over a training set of tokens. Results for GPT-2 are shown in <a href="#S4.F8" title="Figure 8 ‣ 4.4 Explanation reconstruction ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 8</span></a>. We find that we can explain more of GPT-2 small than just explaining bigrams, and that larger and sparser autoencoders result in better downstream loss.</p>
</div>
<figure id="S4.F8" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_explain_recons_loss.png" id="S4.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="389" height="309" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F8.6.3.1" class="ltx_text" style="font-size:90%;">Figure 8</span>: </span><span id="S4.F8.4.2" class="ltx_text" style="font-size:90%;">Downstream loss on GPT-2 with various residual stream ablations at layer 8. N2G explanations of autoencoder latents improves downstream loss with larger <math id="S4.F8.3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.F8.3.1.m1.1b"><mi id="S4.F8.3.1.m1.1.1" xref="S4.F8.3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.F8.3.1.m1.1c"><ci id="S4.F8.3.1.m1.1.1.cmml" xref="S4.F8.3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F8.3.1.m1.1d">n</annotation></semantics></math> and smaller <math id="S4.F8.4.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.F8.4.2.m2.1b"><mi id="S4.F8.4.2.m2.1.1" xref="S4.F8.4.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.F8.4.2.m2.1c"><ci id="S4.F8.4.2.m2.1.1.cmml" xref="S4.F8.4.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F8.4.2.m2.1d">k</annotation></semantics></math>.</span></figcaption>
</figure>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Sparsity of ablation effects</h3>

<div id="S4.SS5.p1" class="ltx_para">
<p id="S4.SS5.p1.1" class="ltx_p">If the underlying computations learned by a language model are sparse,
one hypothesis is that natural features are not only sparse in terms of activations,
but also in terms of downstream effects <cite class="ltx_cite ltx_citemacro_citep">(Olah et al., <a href="#bib.bib47" title="" class="ltx_ref">2024</a>)</cite>. Anecdotally, we observed that ablation effects often are interpretable (see our visualizer).
Therefore, we developed a metric to measure the sparsity of downstream effects on the output logits.</p>
</div>
<div id="S4.SS5.p2" class="ltx_para">
<p id="S4.SS5.p2.6" class="ltx_p">At a particular token index, we obtain the latents at the residual stream, and proceed to ablate each autoencoder latent one by one, and compare the resulting logits before and after ablation. This process leads to <math id="S4.SS5.p2.1.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.SS5.p2.1.m1.1a"><mi id="S4.SS5.p2.1.m1.1.1" xref="S4.SS5.p2.1.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.1.m1.1b"><ci id="S4.SS5.p2.1.m1.1.1.cmml" xref="S4.SS5.p2.1.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.1.m1.1c">V</annotation></semantics></math> logit differences per ablation and affected token, where <math id="S4.SS5.p2.2.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.SS5.p2.2.m2.1a"><mi id="S4.SS5.p2.2.m2.1.1" xref="S4.SS5.p2.2.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.2.m2.1b"><ci id="S4.SS5.p2.2.m2.1.1.cmml" xref="S4.SS5.p2.2.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.2.m2.1c">V</annotation></semantics></math> is the size of the token vocabulary.
Because a constant difference at every logit does not affect the post-softmax probabilities,
we subtract at each token the median logit difference value. Finally, we concatenate these vectors together across some set of <math id="S4.SS5.p2.3.m3.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S4.SS5.p2.3.m3.1a"><mi id="S4.SS5.p2.3.m3.1.1" xref="S4.SS5.p2.3.m3.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.3.m3.1b"><ci id="S4.SS5.p2.3.m3.1.1.cmml" xref="S4.SS5.p2.3.m3.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.3.m3.1c">T</annotation></semantics></math> future tokens (at the ablated index or later) to obtain a vector of <math id="S4.SS5.p2.4.m4.1" class="ltx_Math" alttext="V\cdot T" display="inline"><semantics id="S4.SS5.p2.4.m4.1a"><mrow id="S4.SS5.p2.4.m4.1.1" xref="S4.SS5.p2.4.m4.1.1.cmml"><mi id="S4.SS5.p2.4.m4.1.1.2" xref="S4.SS5.p2.4.m4.1.1.2.cmml">V</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS5.p2.4.m4.1.1.1" xref="S4.SS5.p2.4.m4.1.1.1.cmml">⋅</mo><mi id="S4.SS5.p2.4.m4.1.1.3" xref="S4.SS5.p2.4.m4.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.4.m4.1b"><apply id="S4.SS5.p2.4.m4.1.1.cmml" xref="S4.SS5.p2.4.m4.1.1"><ci id="S4.SS5.p2.4.m4.1.1.1.cmml" xref="S4.SS5.p2.4.m4.1.1.1">⋅</ci><ci id="S4.SS5.p2.4.m4.1.1.2.cmml" xref="S4.SS5.p2.4.m4.1.1.2">𝑉</ci><ci id="S4.SS5.p2.4.m4.1.1.3.cmml" xref="S4.SS5.p2.4.m4.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.4.m4.1c">V\cdot T</annotation></semantics></math> total numbers.
We then measure the sparsity of this vector via <math id="S4.SS5.p2.5.m5.1" class="ltx_Math" alttext="(\frac{\text{L}_{1}}{\text{L}_{2}})^{2}" display="inline"><semantics id="S4.SS5.p2.5.m5.1a"><msup id="S4.SS5.p2.5.m5.1.2" xref="S4.SS5.p2.5.m5.1.2.cmml"><mrow id="S4.SS5.p2.5.m5.1.2.2.2" xref="S4.SS5.p2.5.m5.1.1.cmml"><mo stretchy="false" id="S4.SS5.p2.5.m5.1.2.2.2.1" xref="S4.SS5.p2.5.m5.1.1.cmml">(</mo><mfrac id="S4.SS5.p2.5.m5.1.1" xref="S4.SS5.p2.5.m5.1.1.cmml"><msub id="S4.SS5.p2.5.m5.1.1.2" xref="S4.SS5.p2.5.m5.1.1.2.cmml"><mtext id="S4.SS5.p2.5.m5.1.1.2.2" xref="S4.SS5.p2.5.m5.1.1.2.2a.cmml">L</mtext><mn id="S4.SS5.p2.5.m5.1.1.2.3" xref="S4.SS5.p2.5.m5.1.1.2.3.cmml">1</mn></msub><msub id="S4.SS5.p2.5.m5.1.1.3" xref="S4.SS5.p2.5.m5.1.1.3.cmml"><mtext id="S4.SS5.p2.5.m5.1.1.3.2" xref="S4.SS5.p2.5.m5.1.1.3.2a.cmml">L</mtext><mn id="S4.SS5.p2.5.m5.1.1.3.3" xref="S4.SS5.p2.5.m5.1.1.3.3.cmml">2</mn></msub></mfrac><mo stretchy="false" id="S4.SS5.p2.5.m5.1.2.2.2.2" xref="S4.SS5.p2.5.m5.1.1.cmml">)</mo></mrow><mn id="S4.SS5.p2.5.m5.1.2.3" xref="S4.SS5.p2.5.m5.1.2.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.5.m5.1b"><apply id="S4.SS5.p2.5.m5.1.2.cmml" xref="S4.SS5.p2.5.m5.1.2"><csymbol cd="ambiguous" id="S4.SS5.p2.5.m5.1.2.1.cmml" xref="S4.SS5.p2.5.m5.1.2">superscript</csymbol><apply id="S4.SS5.p2.5.m5.1.1.cmml" xref="S4.SS5.p2.5.m5.1.2.2.2"><divide id="S4.SS5.p2.5.m5.1.1.1.cmml" xref="S4.SS5.p2.5.m5.1.2.2.2"></divide><apply id="S4.SS5.p2.5.m5.1.1.2.cmml" xref="S4.SS5.p2.5.m5.1.1.2"><csymbol cd="ambiguous" id="S4.SS5.p2.5.m5.1.1.2.1.cmml" xref="S4.SS5.p2.5.m5.1.1.2">subscript</csymbol><ci id="S4.SS5.p2.5.m5.1.1.2.2a.cmml" xref="S4.SS5.p2.5.m5.1.1.2.2"><mtext mathsize="70%" id="S4.SS5.p2.5.m5.1.1.2.2.cmml" xref="S4.SS5.p2.5.m5.1.1.2.2">L</mtext></ci><cn type="integer" id="S4.SS5.p2.5.m5.1.1.2.3.cmml" xref="S4.SS5.p2.5.m5.1.1.2.3">1</cn></apply><apply id="S4.SS5.p2.5.m5.1.1.3.cmml" xref="S4.SS5.p2.5.m5.1.1.3"><csymbol cd="ambiguous" id="S4.SS5.p2.5.m5.1.1.3.1.cmml" xref="S4.SS5.p2.5.m5.1.1.3">subscript</csymbol><ci id="S4.SS5.p2.5.m5.1.1.3.2a.cmml" xref="S4.SS5.p2.5.m5.1.1.3.2"><mtext mathsize="70%" id="S4.SS5.p2.5.m5.1.1.3.2.cmml" xref="S4.SS5.p2.5.m5.1.1.3.2">L</mtext></ci><cn type="integer" id="S4.SS5.p2.5.m5.1.1.3.3.cmml" xref="S4.SS5.p2.5.m5.1.1.3.3">2</cn></apply></apply><cn type="integer" id="S4.SS5.p2.5.m5.1.2.3.cmml" xref="S4.SS5.p2.5.m5.1.2.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.5.m5.1c">(\frac{\text{L}_{1}}{\text{L}_{2}})^{2}</annotation></semantics></math>,
which corresponds to an “effective number of vocab tokens affected”. We normalize by <math id="S4.SS5.p2.6.m6.1" class="ltx_Math" alttext="V\cdot T" display="inline"><semantics id="S4.SS5.p2.6.m6.1a"><mrow id="S4.SS5.p2.6.m6.1.1" xref="S4.SS5.p2.6.m6.1.1.cmml"><mi id="S4.SS5.p2.6.m6.1.1.2" xref="S4.SS5.p2.6.m6.1.1.2.cmml">V</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS5.p2.6.m6.1.1.1" xref="S4.SS5.p2.6.m6.1.1.1.cmml">⋅</mo><mi id="S4.SS5.p2.6.m6.1.1.3" xref="S4.SS5.p2.6.m6.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.6.m6.1b"><apply id="S4.SS5.p2.6.m6.1.1.cmml" xref="S4.SS5.p2.6.m6.1.1"><ci id="S4.SS5.p2.6.m6.1.1.1.cmml" xref="S4.SS5.p2.6.m6.1.1.1">⋅</ci><ci id="S4.SS5.p2.6.m6.1.1.2.cmml" xref="S4.SS5.p2.6.m6.1.1.2">𝑉</ci><ci id="S4.SS5.p2.6.m6.1.1.3.cmml" xref="S4.SS5.p2.6.m6.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.6.m6.1c">V\cdot T</annotation></semantics></math> to have a fraction between 0 and 1, with smaller values corresponding to sparser effects.</p>
</div>
<div id="S4.SS5.p3" class="ltx_para">
<p id="S4.SS5.p3.7" class="ltx_p">We perform this for various autoencoders trained on the post-MLP residual stream at layer 8 in GPT-2 small, with <math id="S4.SS5.p3.1.m1.1" class="ltx_Math" alttext="T=16" display="inline"><semantics id="S4.SS5.p3.1.m1.1a"><mrow id="S4.SS5.p3.1.m1.1.1" xref="S4.SS5.p3.1.m1.1.1.cmml"><mi id="S4.SS5.p3.1.m1.1.1.2" xref="S4.SS5.p3.1.m1.1.1.2.cmml">T</mi><mo id="S4.SS5.p3.1.m1.1.1.1" xref="S4.SS5.p3.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS5.p3.1.m1.1.1.3" xref="S4.SS5.p3.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.1.m1.1b"><apply id="S4.SS5.p3.1.m1.1.1.cmml" xref="S4.SS5.p3.1.m1.1.1"><eq id="S4.SS5.p3.1.m1.1.1.1.cmml" xref="S4.SS5.p3.1.m1.1.1.1"></eq><ci id="S4.SS5.p3.1.m1.1.1.2.cmml" xref="S4.SS5.p3.1.m1.1.1.2">𝑇</ci><cn type="integer" id="S4.SS5.p3.1.m1.1.1.3.cmml" xref="S4.SS5.p3.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.1.m1.1c">T=16</annotation></semantics></math>. Results are shown in <a href="#S4.F6.sf2" title="6(b) ‣ Figure 6 ‣ 4.1 Downstream loss ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">6(b)</span></a>. Promisingly, models trained with larger <math id="S4.SS5.p3.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS5.p3.2.m2.1a"><mi id="S4.SS5.p3.2.m2.1.1" xref="S4.SS5.p3.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.2.m2.1b"><ci id="S4.SS5.p3.2.m2.1.1.cmml" xref="S4.SS5.p3.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.2.m2.1c">k</annotation></semantics></math> have latents with sparser effects. However, the trend reverses at <math id="S4.SS5.p3.3.m3.1" class="ltx_Math" alttext="k=512" display="inline"><semantics id="S4.SS5.p3.3.m3.1a"><mrow id="S4.SS5.p3.3.m3.1.1" xref="S4.SS5.p3.3.m3.1.1.cmml"><mi id="S4.SS5.p3.3.m3.1.1.2" xref="S4.SS5.p3.3.m3.1.1.2.cmml">k</mi><mo id="S4.SS5.p3.3.m3.1.1.1" xref="S4.SS5.p3.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS5.p3.3.m3.1.1.3" xref="S4.SS5.p3.3.m3.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.3.m3.1b"><apply id="S4.SS5.p3.3.m3.1.1.cmml" xref="S4.SS5.p3.3.m3.1.1"><eq id="S4.SS5.p3.3.m3.1.1.1.cmml" xref="S4.SS5.p3.3.m3.1.1.1"></eq><ci id="S4.SS5.p3.3.m3.1.1.2.cmml" xref="S4.SS5.p3.3.m3.1.1.2">𝑘</ci><cn type="integer" id="S4.SS5.p3.3.m3.1.1.3.cmml" xref="S4.SS5.p3.3.m3.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.3.m3.1c">k=512</annotation></semantics></math>, indicating that as <math id="S4.SS5.p3.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS5.p3.4.m4.1a"><mi id="S4.SS5.p3.4.m4.1.1" xref="S4.SS5.p3.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.4.m4.1b"><ci id="S4.SS5.p3.4.m4.1.1.cmml" xref="S4.SS5.p3.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.4.m4.1c">k</annotation></semantics></math> approaches <math id="S4.SS5.p3.5.m5.1" class="ltx_Math" alttext="d_{\text{model}}=768" display="inline"><semantics id="S4.SS5.p3.5.m5.1a"><mrow id="S4.SS5.p3.5.m5.1.1" xref="S4.SS5.p3.5.m5.1.1.cmml"><msub id="S4.SS5.p3.5.m5.1.1.2" xref="S4.SS5.p3.5.m5.1.1.2.cmml"><mi id="S4.SS5.p3.5.m5.1.1.2.2" xref="S4.SS5.p3.5.m5.1.1.2.2.cmml">d</mi><mtext id="S4.SS5.p3.5.m5.1.1.2.3" xref="S4.SS5.p3.5.m5.1.1.2.3a.cmml">model</mtext></msub><mo id="S4.SS5.p3.5.m5.1.1.1" xref="S4.SS5.p3.5.m5.1.1.1.cmml">=</mo><mn id="S4.SS5.p3.5.m5.1.1.3" xref="S4.SS5.p3.5.m5.1.1.3.cmml">768</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.5.m5.1b"><apply id="S4.SS5.p3.5.m5.1.1.cmml" xref="S4.SS5.p3.5.m5.1.1"><eq id="S4.SS5.p3.5.m5.1.1.1.cmml" xref="S4.SS5.p3.5.m5.1.1.1"></eq><apply id="S4.SS5.p3.5.m5.1.1.2.cmml" xref="S4.SS5.p3.5.m5.1.1.2"><csymbol cd="ambiguous" id="S4.SS5.p3.5.m5.1.1.2.1.cmml" xref="S4.SS5.p3.5.m5.1.1.2">subscript</csymbol><ci id="S4.SS5.p3.5.m5.1.1.2.2.cmml" xref="S4.SS5.p3.5.m5.1.1.2.2">𝑑</ci><ci id="S4.SS5.p3.5.m5.1.1.2.3a.cmml" xref="S4.SS5.p3.5.m5.1.1.2.3"><mtext mathsize="70%" id="S4.SS5.p3.5.m5.1.1.2.3.cmml" xref="S4.SS5.p3.5.m5.1.1.2.3">model</mtext></ci></apply><cn type="integer" id="S4.SS5.p3.5.m5.1.1.3.cmml" xref="S4.SS5.p3.5.m5.1.1.3">768</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.5.m5.1c">d_{\text{model}}=768</annotation></semantics></math>, the autoencoder learns latents with less interpretable effects.
Note that latents are sparse in an absolute sense, having a <math id="S4.SS5.p3.6.m6.1" class="ltx_Math" alttext="(\frac{\text{L}_{1}}{\text{L}_{2}})^{2}" display="inline"><semantics id="S4.SS5.p3.6.m6.1a"><msup id="S4.SS5.p3.6.m6.1.2" xref="S4.SS5.p3.6.m6.1.2.cmml"><mrow id="S4.SS5.p3.6.m6.1.2.2.2" xref="S4.SS5.p3.6.m6.1.1.cmml"><mo stretchy="false" id="S4.SS5.p3.6.m6.1.2.2.2.1" xref="S4.SS5.p3.6.m6.1.1.cmml">(</mo><mfrac id="S4.SS5.p3.6.m6.1.1" xref="S4.SS5.p3.6.m6.1.1.cmml"><msub id="S4.SS5.p3.6.m6.1.1.2" xref="S4.SS5.p3.6.m6.1.1.2.cmml"><mtext id="S4.SS5.p3.6.m6.1.1.2.2" xref="S4.SS5.p3.6.m6.1.1.2.2a.cmml">L</mtext><mn id="S4.SS5.p3.6.m6.1.1.2.3" xref="S4.SS5.p3.6.m6.1.1.2.3.cmml">1</mn></msub><msub id="S4.SS5.p3.6.m6.1.1.3" xref="S4.SS5.p3.6.m6.1.1.3.cmml"><mtext id="S4.SS5.p3.6.m6.1.1.3.2" xref="S4.SS5.p3.6.m6.1.1.3.2a.cmml">L</mtext><mn id="S4.SS5.p3.6.m6.1.1.3.3" xref="S4.SS5.p3.6.m6.1.1.3.3.cmml">2</mn></msub></mfrac><mo stretchy="false" id="S4.SS5.p3.6.m6.1.2.2.2.2" xref="S4.SS5.p3.6.m6.1.1.cmml">)</mo></mrow><mn id="S4.SS5.p3.6.m6.1.2.3" xref="S4.SS5.p3.6.m6.1.2.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.6.m6.1b"><apply id="S4.SS5.p3.6.m6.1.2.cmml" xref="S4.SS5.p3.6.m6.1.2"><csymbol cd="ambiguous" id="S4.SS5.p3.6.m6.1.2.1.cmml" xref="S4.SS5.p3.6.m6.1.2">superscript</csymbol><apply id="S4.SS5.p3.6.m6.1.1.cmml" xref="S4.SS5.p3.6.m6.1.2.2.2"><divide id="S4.SS5.p3.6.m6.1.1.1.cmml" xref="S4.SS5.p3.6.m6.1.2.2.2"></divide><apply id="S4.SS5.p3.6.m6.1.1.2.cmml" xref="S4.SS5.p3.6.m6.1.1.2"><csymbol cd="ambiguous" id="S4.SS5.p3.6.m6.1.1.2.1.cmml" xref="S4.SS5.p3.6.m6.1.1.2">subscript</csymbol><ci id="S4.SS5.p3.6.m6.1.1.2.2a.cmml" xref="S4.SS5.p3.6.m6.1.1.2.2"><mtext mathsize="70%" id="S4.SS5.p3.6.m6.1.1.2.2.cmml" xref="S4.SS5.p3.6.m6.1.1.2.2">L</mtext></ci><cn type="integer" id="S4.SS5.p3.6.m6.1.1.2.3.cmml" xref="S4.SS5.p3.6.m6.1.1.2.3">1</cn></apply><apply id="S4.SS5.p3.6.m6.1.1.3.cmml" xref="S4.SS5.p3.6.m6.1.1.3"><csymbol cd="ambiguous" id="S4.SS5.p3.6.m6.1.1.3.1.cmml" xref="S4.SS5.p3.6.m6.1.1.3">subscript</csymbol><ci id="S4.SS5.p3.6.m6.1.1.3.2a.cmml" xref="S4.SS5.p3.6.m6.1.1.3.2"><mtext mathsize="70%" id="S4.SS5.p3.6.m6.1.1.3.2.cmml" xref="S4.SS5.p3.6.m6.1.1.3.2">L</mtext></ci><cn type="integer" id="S4.SS5.p3.6.m6.1.1.3.3.cmml" xref="S4.SS5.p3.6.m6.1.1.3.3">2</cn></apply></apply><cn type="integer" id="S4.SS5.p3.6.m6.1.2.3.cmml" xref="S4.SS5.p3.6.m6.1.2.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.6.m6.1c">(\frac{\text{L}_{1}}{\text{L}_{2}})^{2}</annotation></semantics></math> of 10-14%
, whereas ablating residual stream channels gives 60%
(slightly better than the theoretical value of <math id="S4.SS5.p3.7.m7.1" class="ltx_Math" alttext="\sim\frac{2}{\pi}" display="inline"><semantics id="S4.SS5.p3.7.m7.1a"><mrow id="S4.SS5.p3.7.m7.1.1" xref="S4.SS5.p3.7.m7.1.1.cmml"><mi id="S4.SS5.p3.7.m7.1.1.2" xref="S4.SS5.p3.7.m7.1.1.2.cmml"></mi><mo id="S4.SS5.p3.7.m7.1.1.1" xref="S4.SS5.p3.7.m7.1.1.1.cmml">∼</mo><mfrac id="S4.SS5.p3.7.m7.1.1.3" xref="S4.SS5.p3.7.m7.1.1.3.cmml"><mn id="S4.SS5.p3.7.m7.1.1.3.2" xref="S4.SS5.p3.7.m7.1.1.3.2.cmml">2</mn><mi id="S4.SS5.p3.7.m7.1.1.3.3" xref="S4.SS5.p3.7.m7.1.1.3.3.cmml">π</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.7.m7.1b"><apply id="S4.SS5.p3.7.m7.1.1.cmml" xref="S4.SS5.p3.7.m7.1.1"><csymbol cd="latexml" id="S4.SS5.p3.7.m7.1.1.1.cmml" xref="S4.SS5.p3.7.m7.1.1.1">similar-to</csymbol><csymbol cd="latexml" id="S4.SS5.p3.7.m7.1.1.2.cmml" xref="S4.SS5.p3.7.m7.1.1.2">absent</csymbol><apply id="S4.SS5.p3.7.m7.1.1.3.cmml" xref="S4.SS5.p3.7.m7.1.1.3"><divide id="S4.SS5.p3.7.m7.1.1.3.1.cmml" xref="S4.SS5.p3.7.m7.1.1.3"></divide><cn type="integer" id="S4.SS5.p3.7.m7.1.1.3.2.cmml" xref="S4.SS5.p3.7.m7.1.1.3.2">2</cn><ci id="S4.SS5.p3.7.m7.1.1.3.3.cmml" xref="S4.SS5.p3.7.m7.1.1.3.3">𝜋</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.7.m7.1c">\sim\frac{2}{\pi}</annotation></semantics></math> for random vectors).</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Understanding the TopK activation function</h2>

<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>TopK prevents activation shrinkage</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.3" class="ltx_p">A major drawback of the L<sub id="S5.SS1.p1.3.1" class="ltx_sub">1</sub> penalty is that it tends to shrink all activations toward zero <cite class="ltx_cite ltx_citemacro_citep">(Tibshirani, <a href="#bib.bib65" title="" class="ltx_ref">1996</a>)</cite>.
Our proposed TopK activation function prevents activation shrinkage, as it entirely removes the need for an L<sub id="S5.SS1.p1.3.2" class="ltx_sub">1</sub> penalty.
To empirically measure the magnitude of activation shrinkage, we consider whether different (and potentially larger) activations would result in better reconstruction given a fixed decoder. We first run the encoder to obtain a set of activated latents, save the sparsity mask, and then optimize only the nonzero values to minimize MSE.<span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note">14</span>unlike the “inference-time optimization” procedure in <cite class="ltx_cite ltx_citemacro_cite">Nanda et al. (<a href="#bib.bib46" title="" class="ltx_ref">2024</a>)</cite></span></span></span> This refinement method has been proposed multiple times such as in <math id="S5.SS1.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS1.p1.3.m3.1a"><mi id="S5.SS1.p1.3.m3.1.1" xref="S5.SS1.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.3.m3.1b"><ci id="S5.SS1.p1.3.m3.1.1.cmml" xref="S5.SS1.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.3.m3.1c">k</annotation></semantics></math>-SVD <cite class="ltx_cite ltx_citemacro_citep">(Aharon et al., <a href="#bib.bib1" title="" class="ltx_ref">2006</a>)</cite>, the relaxed Lasso <cite class="ltx_cite ltx_citemacro_citep">(Meinshausen, <a href="#bib.bib43" title="" class="ltx_ref">2007</a>)</cite>, or ITI <cite class="ltx_cite ltx_citemacro_citep">(Maleki, <a href="#bib.bib37" title="" class="ltx_ref">2009</a>)</cite>. We solve for the optimal activations with a positivity constraint using projected gradient descent.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">This refinement procedure tends to increase activations in ReLU models on average, but not in TopK models (<a href="#S5.F9" title="Figure 9 ‣ 5.1 TopK prevents activation shrinkage ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a>a), which indicates that TopK is not impacted by activation shrinkage. The magnitude of the refinement is also smaller for TopK models than for ReLU models. In both ReLU and TopK models, the refinement procedure noticeably improves the reconstruction MSE (<a href="#S5.F9" title="Figure 9 ‣ 5.1 TopK prevents activation shrinkage ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a>b), and the downstream next-token-prediction cross-entropy (<a href="#S5.F9" title="Figure 9 ‣ 5.1 TopK prevents activation shrinkage ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a>c). However, this refinement only closes part of the gap between ReLU and TopK models.</p>
</div>
<figure id="S5.F9" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3"><img src="/html/2406.04093/assets/x9.png" id="S5.F9.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="147" height="113" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_3"><img src="/html/2406.04093/assets/x10.png" id="S5.F9.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="147" height="111" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_3"><img src="/html/2406.04093/assets/x11.png" id="S5.F9.g3" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="147" height="111" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F9.5.2.1" class="ltx_text" style="font-size:90%;">Figure 9</span>: </span><span id="S5.F9.2.1" class="ltx_text" style="font-size:90%;">Latent activations can be refined to improve reconstruction from a frozen set of latents. For ReLU autoencoders, the refinement is biased toward positive values, consistent with compensating for the shrinkage caused by the L<sub id="S5.F9.2.1.1" class="ltx_sub">1</sub> penalty. For TopK autoencoders, the refinement is not biased, and also smaller in magnitude. The refinement only closes part of the gap between ReLU and TopK.</span></figcaption>
</figure>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Comparison with other activation functions</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.6" class="ltx_p">Other recent works on sparse autoencoders have proposed different ways to address the L<sub id="S5.SS2.p1.6.1" class="ltx_sub">1</sub> activation shrinkage, and Pareto improve the L<sub id="S5.SS2.p1.6.2" class="ltx_sub">0</sub>-MSE frontier <cite class="ltx_cite ltx_citemacro_citep">(Wright and Sharkey, <a href="#bib.bib68" title="" class="ltx_ref">2024</a>; Taggart, <a href="#bib.bib62" title="" class="ltx_ref">2024</a>; Rajamanoharan et al., <a href="#bib.bib52" title="" class="ltx_ref">2024</a>)</cite>.
<cite class="ltx_cite ltx_citemacro_citet">Wright and Sharkey (<a href="#bib.bib68" title="" class="ltx_ref">2024</a>)</cite> propose to fine-tune a scaling parameter per latent, to correct for the L<sub id="S5.SS2.p1.6.3" class="ltx_sub">1</sub> activation shrinkage.
In Gated sparse autoencoders <cite class="ltx_cite ltx_citemacro_citep">(Rajamanoharan et al., <a href="#bib.bib52" title="" class="ltx_ref">2024</a>)</cite>, the selection of which latents are active is separate from the estimation of the activation magnitudes. This separation allows autoencoders to better estimate the activation magnitude, and avoid the L<sub id="S5.SS2.p1.6.4" class="ltx_sub">1</sub> activation shrinkage.
Another approach is to replace the ReLU activation function with a ProLU <cite class="ltx_cite ltx_citemacro_citep">(Taggart, <a href="#bib.bib62" title="" class="ltx_ref">2024</a>)</cite> (also known as TRec <cite class="ltx_cite ltx_citemacro_citep">(Konda et al., <a href="#bib.bib28" title="" class="ltx_ref">2014</a>)</cite>, or JumpReLU <cite class="ltx_cite ltx_citemacro_citep">(Erichson et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite>), which sets all values below a positive threshold to zero <math id="S5.SS2.p1.5.m5.2" class="ltx_Math" alttext="J_{\theta}(x)=x\cdot\mathbf{1}_{(x&gt;\theta)}" display="inline"><semantics id="S5.SS2.p1.5.m5.2a"><mrow id="S5.SS2.p1.5.m5.2.3" xref="S5.SS2.p1.5.m5.2.3.cmml"><mrow id="S5.SS2.p1.5.m5.2.3.2" xref="S5.SS2.p1.5.m5.2.3.2.cmml"><msub id="S5.SS2.p1.5.m5.2.3.2.2" xref="S5.SS2.p1.5.m5.2.3.2.2.cmml"><mi id="S5.SS2.p1.5.m5.2.3.2.2.2" xref="S5.SS2.p1.5.m5.2.3.2.2.2.cmml">J</mi><mi id="S5.SS2.p1.5.m5.2.3.2.2.3" xref="S5.SS2.p1.5.m5.2.3.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S5.SS2.p1.5.m5.2.3.2.1" xref="S5.SS2.p1.5.m5.2.3.2.1.cmml">​</mo><mrow id="S5.SS2.p1.5.m5.2.3.2.3.2" xref="S5.SS2.p1.5.m5.2.3.2.cmml"><mo stretchy="false" id="S5.SS2.p1.5.m5.2.3.2.3.2.1" xref="S5.SS2.p1.5.m5.2.3.2.cmml">(</mo><mi id="S5.SS2.p1.5.m5.2.2" xref="S5.SS2.p1.5.m5.2.2.cmml">x</mi><mo stretchy="false" id="S5.SS2.p1.5.m5.2.3.2.3.2.2" xref="S5.SS2.p1.5.m5.2.3.2.cmml">)</mo></mrow></mrow><mo id="S5.SS2.p1.5.m5.2.3.1" xref="S5.SS2.p1.5.m5.2.3.1.cmml">=</mo><mrow id="S5.SS2.p1.5.m5.2.3.3" xref="S5.SS2.p1.5.m5.2.3.3.cmml"><mi id="S5.SS2.p1.5.m5.2.3.3.2" xref="S5.SS2.p1.5.m5.2.3.3.2.cmml">x</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS2.p1.5.m5.2.3.3.1" xref="S5.SS2.p1.5.m5.2.3.3.1.cmml">⋅</mo><msub id="S5.SS2.p1.5.m5.2.3.3.3" xref="S5.SS2.p1.5.m5.2.3.3.3.cmml"><mn id="S5.SS2.p1.5.m5.2.3.3.3.2" xref="S5.SS2.p1.5.m5.2.3.3.3.2.cmml">𝟏</mn><mrow id="S5.SS2.p1.5.m5.1.1.1.1" xref="S5.SS2.p1.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS2.p1.5.m5.1.1.1.1.2" xref="S5.SS2.p1.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS2.p1.5.m5.1.1.1.1.1" xref="S5.SS2.p1.5.m5.1.1.1.1.1.cmml"><mi id="S5.SS2.p1.5.m5.1.1.1.1.1.2" xref="S5.SS2.p1.5.m5.1.1.1.1.1.2.cmml">x</mi><mo id="S5.SS2.p1.5.m5.1.1.1.1.1.1" xref="S5.SS2.p1.5.m5.1.1.1.1.1.1.cmml">&gt;</mo><mi id="S5.SS2.p1.5.m5.1.1.1.1.1.3" xref="S5.SS2.p1.5.m5.1.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S5.SS2.p1.5.m5.1.1.1.1.3" xref="S5.SS2.p1.5.m5.1.1.1.1.1.cmml">)</mo></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.5.m5.2b"><apply id="S5.SS2.p1.5.m5.2.3.cmml" xref="S5.SS2.p1.5.m5.2.3"><eq id="S5.SS2.p1.5.m5.2.3.1.cmml" xref="S5.SS2.p1.5.m5.2.3.1"></eq><apply id="S5.SS2.p1.5.m5.2.3.2.cmml" xref="S5.SS2.p1.5.m5.2.3.2"><times id="S5.SS2.p1.5.m5.2.3.2.1.cmml" xref="S5.SS2.p1.5.m5.2.3.2.1"></times><apply id="S5.SS2.p1.5.m5.2.3.2.2.cmml" xref="S5.SS2.p1.5.m5.2.3.2.2"><csymbol cd="ambiguous" id="S5.SS2.p1.5.m5.2.3.2.2.1.cmml" xref="S5.SS2.p1.5.m5.2.3.2.2">subscript</csymbol><ci id="S5.SS2.p1.5.m5.2.3.2.2.2.cmml" xref="S5.SS2.p1.5.m5.2.3.2.2.2">𝐽</ci><ci id="S5.SS2.p1.5.m5.2.3.2.2.3.cmml" xref="S5.SS2.p1.5.m5.2.3.2.2.3">𝜃</ci></apply><ci id="S5.SS2.p1.5.m5.2.2.cmml" xref="S5.SS2.p1.5.m5.2.2">𝑥</ci></apply><apply id="S5.SS2.p1.5.m5.2.3.3.cmml" xref="S5.SS2.p1.5.m5.2.3.3"><ci id="S5.SS2.p1.5.m5.2.3.3.1.cmml" xref="S5.SS2.p1.5.m5.2.3.3.1">⋅</ci><ci id="S5.SS2.p1.5.m5.2.3.3.2.cmml" xref="S5.SS2.p1.5.m5.2.3.3.2">𝑥</ci><apply id="S5.SS2.p1.5.m5.2.3.3.3.cmml" xref="S5.SS2.p1.5.m5.2.3.3.3"><csymbol cd="ambiguous" id="S5.SS2.p1.5.m5.2.3.3.3.1.cmml" xref="S5.SS2.p1.5.m5.2.3.3.3">subscript</csymbol><cn type="integer" id="S5.SS2.p1.5.m5.2.3.3.3.2.cmml" xref="S5.SS2.p1.5.m5.2.3.3.3.2">1</cn><apply id="S5.SS2.p1.5.m5.1.1.1.1.1.cmml" xref="S5.SS2.p1.5.m5.1.1.1.1"><gt id="S5.SS2.p1.5.m5.1.1.1.1.1.1.cmml" xref="S5.SS2.p1.5.m5.1.1.1.1.1.1"></gt><ci id="S5.SS2.p1.5.m5.1.1.1.1.1.2.cmml" xref="S5.SS2.p1.5.m5.1.1.1.1.1.2">𝑥</ci><ci id="S5.SS2.p1.5.m5.1.1.1.1.1.3.cmml" xref="S5.SS2.p1.5.m5.1.1.1.1.1.3">𝜃</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.5.m5.2c">J_{\theta}(x)=x\cdot\mathbf{1}_{(x&gt;\theta)}</annotation></semantics></math>. Because the parameter <math id="S5.SS2.p1.6.m6.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S5.SS2.p1.6.m6.1a"><mi id="S5.SS2.p1.6.m6.1.1" xref="S5.SS2.p1.6.m6.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.6.m6.1b"><ci id="S5.SS2.p1.6.m6.1.1.cmml" xref="S5.SS2.p1.6.m6.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.6.m6.1c">\theta</annotation></semantics></math> is non-differentiable, it requires a approximate gradient such as a ReLU equivalent (ProLU-ReLU) or a straight-through estimator (ProLU-STE) <cite class="ltx_cite ltx_citemacro_citep">(Taggart, <a href="#bib.bib62" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.1" class="ltx_p">We compared these different approaches in terms of reconstruction MSE, number of active latents L<sub id="S5.SS2.p2.1.1" class="ltx_sub">0</sub>, and downstream cross-entropy loss (<a href="#S2.F2" title="Figure 2 ‣ 2.3 TopK activation function ‣ 2 Methods ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 2</span></a> and <a href="#S3.F5" title="Figure 5 ‣ 3.2 Subject model size 𝐿_𝑠⁢(𝑁) ‣ 3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>). We find that they significantly improve the reconstruction-sparsity Pareto frontier, with TopK having the best performance overall.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Progressive recovery</h3>

<figure id="S5.F10" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/x12.png" id="S5.F10.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="185" height="144" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/x13.png" id="S5.F10.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="185" height="144" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/x14.png" id="S5.F10.g3" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="185" height="144" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/x15.png" id="S5.F10.g4" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="185" height="144" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F10.14.7.1" class="ltx_text" style="font-size:90%;">Figure 10</span>: </span><span id="S5.F10.12.6" class="ltx_text" style="font-size:90%;">Sparsity levels can be changed at test time by replacing the activation function with either TopK(<math id="S5.F10.7.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F10.7.1.m1.1b"><mi id="S5.F10.7.1.m1.1.1" xref="S5.F10.7.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F10.7.1.m1.1c"><ci id="S5.F10.7.1.m1.1.1.cmml" xref="S5.F10.7.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.7.1.m1.1d">k</annotation></semantics></math>) or JumpReLU(<math id="S5.F10.8.2.m2.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S5.F10.8.2.m2.1b"><mi id="S5.F10.8.2.m2.1.1" xref="S5.F10.8.2.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S5.F10.8.2.m2.1c"><ci id="S5.F10.8.2.m2.1.1.cmml" xref="S5.F10.8.2.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.8.2.m2.1d">\theta</annotation></semantics></math>), for a given value <math id="S5.F10.9.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F10.9.3.m3.1b"><mi id="S5.F10.9.3.m3.1.1" xref="S5.F10.9.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F10.9.3.m3.1c"><ci id="S5.F10.9.3.m3.1.1.cmml" xref="S5.F10.9.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.9.3.m3.1d">k</annotation></semantics></math> or <math id="S5.F10.10.4.m4.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S5.F10.10.4.m4.1b"><mi id="S5.F10.10.4.m4.1.1" xref="S5.F10.10.4.m4.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S5.F10.10.4.m4.1c"><ci id="S5.F10.10.4.m4.1.1.cmml" xref="S5.F10.10.4.m4.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.10.4.m4.1d">\theta</annotation></semantics></math>. TopK tends to overfit to the value of <math id="S5.F10.11.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F10.11.5.m5.1b"><mi id="S5.F10.11.5.m5.1.1" xref="S5.F10.11.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F10.11.5.m5.1c"><ci id="S5.F10.11.5.m5.1.1.cmml" xref="S5.F10.11.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.11.5.m5.1d">k</annotation></semantics></math> used during training, but using Multi-TopK improves generalization to larger <math id="S5.F10.12.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F10.12.6.m6.1b"><mi id="S5.F10.12.6.m6.1.1" xref="S5.F10.12.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F10.12.6.m6.1c"><ci id="S5.F10.12.6.m6.1.1.cmml" xref="S5.F10.12.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.12.6.m6.1d">k</annotation></semantics></math>.</span></figcaption>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.4" class="ltx_p">In a progressive code, a partial transmission still allows reconstructing the signal with reasonable fidelity <cite class="ltx_cite ltx_citemacro_citep">(Skodras et al., <a href="#bib.bib59" title="" class="ltx_ref">2001</a>)</cite>. For autoencoders, learning a progressive code means that ordering latents by activation magnitude gives a way to progressively recover the original vector. To study this property, we replace the autoencoder activation function (after training) by a TopK(<math id="S5.SS3.p1.1.m1.1" class="ltx_Math" alttext="k^{\prime}" display="inline"><semantics id="S5.SS3.p1.1.m1.1a"><msup id="S5.SS3.p1.1.m1.1.1" xref="S5.SS3.p1.1.m1.1.1.cmml"><mi id="S5.SS3.p1.1.m1.1.1.2" xref="S5.SS3.p1.1.m1.1.1.2.cmml">k</mi><mo id="S5.SS3.p1.1.m1.1.1.3" xref="S5.SS3.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><apply id="S5.SS3.p1.1.m1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS3.p1.1.m1.1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1">superscript</csymbol><ci id="S5.SS3.p1.1.m1.1.1.2.cmml" xref="S5.SS3.p1.1.m1.1.1.2">𝑘</ci><ci id="S5.SS3.p1.1.m1.1.1.3.cmml" xref="S5.SS3.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">k^{\prime}</annotation></semantics></math>) activation function where <math id="S5.SS3.p1.2.m2.1" class="ltx_Math" alttext="k^{\prime}" display="inline"><semantics id="S5.SS3.p1.2.m2.1a"><msup id="S5.SS3.p1.2.m2.1.1" xref="S5.SS3.p1.2.m2.1.1.cmml"><mi id="S5.SS3.p1.2.m2.1.1.2" xref="S5.SS3.p1.2.m2.1.1.2.cmml">k</mi><mo id="S5.SS3.p1.2.m2.1.1.3" xref="S5.SS3.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.2.m2.1b"><apply id="S5.SS3.p1.2.m2.1.1.cmml" xref="S5.SS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS3.p1.2.m2.1.1.1.cmml" xref="S5.SS3.p1.2.m2.1.1">superscript</csymbol><ci id="S5.SS3.p1.2.m2.1.1.2.cmml" xref="S5.SS3.p1.2.m2.1.1.2">𝑘</ci><ci id="S5.SS3.p1.2.m2.1.1.3.cmml" xref="S5.SS3.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.2.m2.1c">k^{\prime}</annotation></semantics></math> is different than during training. We then evaluate each value of <math id="S5.SS3.p1.3.m3.1" class="ltx_Math" alttext="k^{\prime}" display="inline"><semantics id="S5.SS3.p1.3.m3.1a"><msup id="S5.SS3.p1.3.m3.1.1" xref="S5.SS3.p1.3.m3.1.1.cmml"><mi id="S5.SS3.p1.3.m3.1.1.2" xref="S5.SS3.p1.3.m3.1.1.2.cmml">k</mi><mo id="S5.SS3.p1.3.m3.1.1.3" xref="S5.SS3.p1.3.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.3.m3.1b"><apply id="S5.SS3.p1.3.m3.1.1.cmml" xref="S5.SS3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS3.p1.3.m3.1.1.1.cmml" xref="S5.SS3.p1.3.m3.1.1">superscript</csymbol><ci id="S5.SS3.p1.3.m3.1.1.2.cmml" xref="S5.SS3.p1.3.m3.1.1.2">𝑘</ci><ci id="S5.SS3.p1.3.m3.1.1.3.cmml" xref="S5.SS3.p1.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.3.m3.1c">k^{\prime}</annotation></semantics></math> by placing it in the L<sub id="S5.SS3.p1.4.1" class="ltx_sub">0</sub>-MSE plane (<a href="#S5.F10" title="Figure 10 ‣ 5.3 Progressive recovery ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a>).</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.4" class="ltx_p">We find that training with TopK only gives a progressive code up to the value of <math id="S5.SS3.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.p2.1.m1.1a"><mi id="S5.SS3.p2.1.m1.1.1" xref="S5.SS3.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.1.m1.1b"><ci id="S5.SS3.p2.1.m1.1.1.cmml" xref="S5.SS3.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.1.m1.1c">k</annotation></semantics></math> used during training. MSE keeps improving for values slightly over <math id="S5.SS3.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.p2.2.m2.1a"><mi id="S5.SS3.p2.2.m2.1.1" xref="S5.SS3.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.2.m2.1b"><ci id="S5.SS3.p2.2.m2.1.1.cmml" xref="S5.SS3.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.2.m2.1c">k</annotation></semantics></math> (a result also described in <cite class="ltx_cite ltx_citemacro_citep">(Makhzani and Frey, <a href="#bib.bib36" title="" class="ltx_ref">2013</a>)</cite>), then gets substantially worse as <math id="S5.SS3.p2.3.m3.1" class="ltx_Math" alttext="k^{\prime}" display="inline"><semantics id="S5.SS3.p2.3.m3.1a"><msup id="S5.SS3.p2.3.m3.1.1" xref="S5.SS3.p2.3.m3.1.1.cmml"><mi id="S5.SS3.p2.3.m3.1.1.2" xref="S5.SS3.p2.3.m3.1.1.2.cmml">k</mi><mo id="S5.SS3.p2.3.m3.1.1.3" xref="S5.SS3.p2.3.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.3.m3.1b"><apply id="S5.SS3.p2.3.m3.1.1.cmml" xref="S5.SS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS3.p2.3.m3.1.1.1.cmml" xref="S5.SS3.p2.3.m3.1.1">superscript</csymbol><ci id="S5.SS3.p2.3.m3.1.1.2.cmml" xref="S5.SS3.p2.3.m3.1.1.2">𝑘</ci><ci id="S5.SS3.p2.3.m3.1.1.3.cmml" xref="S5.SS3.p2.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.3.m3.1c">k^{\prime}</annotation></semantics></math> increases (note that the effect on downstream loss is more muted). This can be interpreted as some sort of overfitting to the value <math id="S5.SS3.p2.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.p2.4.m4.1a"><mi id="S5.SS3.p2.4.m4.1.1" xref="S5.SS3.p2.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.4.m4.1b"><ci id="S5.SS3.p2.4.m4.1.1.cmml" xref="S5.SS3.p2.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.4.m4.1c">k</annotation></semantics></math>.</p>
</div>
<section id="S5.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.3.1 </span>Multi-TopK</h4>

<div id="S5.SS3.SSS1.p1" class="ltx_para">
<p id="S5.SS3.SSS1.p1.4" class="ltx_p">To mitigate this issue, we sum multiple TopK losses with different values of <math id="S5.SS3.SSS1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.SSS1.p1.1.m1.1a"><mi id="S5.SS3.SSS1.p1.1.m1.1.1" xref="S5.SS3.SSS1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.SSS1.p1.1.m1.1b"><ci id="S5.SS3.SSS1.p1.1.m1.1.1.cmml" xref="S5.SS3.SSS1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.SSS1.p1.1.m1.1c">k</annotation></semantics></math> (Multi-TopK). For example, using <math id="S5.SS3.SSS1.p1.2.m2.2" class="ltx_Math" alttext="\mathcal{L}(k)+\mathcal{L}(4k)/8" display="inline"><semantics id="S5.SS3.SSS1.p1.2.m2.2a"><mrow id="S5.SS3.SSS1.p1.2.m2.2.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.cmml"><mrow id="S5.SS3.SSS1.p1.2.m2.2.2.3" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS3.SSS1.p1.2.m2.2.2.3.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S5.SS3.SSS1.p1.2.m2.2.2.3.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.1.cmml">​</mo><mrow id="S5.SS3.SSS1.p1.2.m2.2.2.3.3.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.cmml"><mo stretchy="false" id="S5.SS3.SSS1.p1.2.m2.2.2.3.3.2.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.cmml">(</mo><mi id="S5.SS3.SSS1.p1.2.m2.1.1" xref="S5.SS3.SSS1.p1.2.m2.1.1.cmml">k</mi><mo stretchy="false" id="S5.SS3.SSS1.p1.2.m2.2.2.3.3.2.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S5.SS3.SSS1.p1.2.m2.2.2.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.2.cmml">+</mo><mrow id="S5.SS3.SSS1.p1.2.m2.2.2.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.cmml"><mrow id="S5.SS3.SSS1.p1.2.m2.2.2.1.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.3" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.3.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.2.cmml">​</mo><mrow id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.cmml"><mn id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.1" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.3" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.3.cmml">k</mi></mrow><mo stretchy="false" id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.3" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S5.SS3.SSS1.p1.2.m2.2.2.1.2" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.2.cmml">/</mo><mn id="S5.SS3.SSS1.p1.2.m2.2.2.1.3" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.3.cmml">8</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.SSS1.p1.2.m2.2b"><apply id="S5.SS3.SSS1.p1.2.m2.2.2.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2"><plus id="S5.SS3.SSS1.p1.2.m2.2.2.2.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.2"></plus><apply id="S5.SS3.SSS1.p1.2.m2.2.2.3.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.3"><times id="S5.SS3.SSS1.p1.2.m2.2.2.3.1.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.1"></times><ci id="S5.SS3.SSS1.p1.2.m2.2.2.3.2.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.3.2">ℒ</ci><ci id="S5.SS3.SSS1.p1.2.m2.1.1.cmml" xref="S5.SS3.SSS1.p1.2.m2.1.1">𝑘</ci></apply><apply id="S5.SS3.SSS1.p1.2.m2.2.2.1.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1"><divide id="S5.SS3.SSS1.p1.2.m2.2.2.1.2.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.2"></divide><apply id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1"><times id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.2.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.2"></times><ci id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.3.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.3">ℒ</ci><apply id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1"><times id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.1.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.1"></times><cn type="integer" id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.2.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.2">4</cn><ci id="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.3.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.1.1.1.1.3">𝑘</ci></apply></apply><cn type="integer" id="S5.SS3.SSS1.p1.2.m2.2.2.1.3.cmml" xref="S5.SS3.SSS1.p1.2.m2.2.2.1.3">8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.SSS1.p1.2.m2.2c">\mathcal{L}(k)+\mathcal{L}(4k)/8</annotation></semantics></math> is enough to obtain a progressive code over all <math id="S5.SS3.SSS1.p1.3.m3.1" class="ltx_Math" alttext="k^{\prime}" display="inline"><semantics id="S5.SS3.SSS1.p1.3.m3.1a"><msup id="S5.SS3.SSS1.p1.3.m3.1.1" xref="S5.SS3.SSS1.p1.3.m3.1.1.cmml"><mi id="S5.SS3.SSS1.p1.3.m3.1.1.2" xref="S5.SS3.SSS1.p1.3.m3.1.1.2.cmml">k</mi><mo id="S5.SS3.SSS1.p1.3.m3.1.1.3" xref="S5.SS3.SSS1.p1.3.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.SSS1.p1.3.m3.1b"><apply id="S5.SS3.SSS1.p1.3.m3.1.1.cmml" xref="S5.SS3.SSS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS3.SSS1.p1.3.m3.1.1.1.cmml" xref="S5.SS3.SSS1.p1.3.m3.1.1">superscript</csymbol><ci id="S5.SS3.SSS1.p1.3.m3.1.1.2.cmml" xref="S5.SS3.SSS1.p1.3.m3.1.1.2">𝑘</ci><ci id="S5.SS3.SSS1.p1.3.m3.1.1.3.cmml" xref="S5.SS3.SSS1.p1.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.SSS1.p1.3.m3.1c">k^{\prime}</annotation></semantics></math>
(note however that training with Multi-TopK does slightly worse than TopK at <math id="S5.SS3.SSS1.p1.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.SSS1.p1.4.m4.1a"><mi id="S5.SS3.SSS1.p1.4.m4.1.1" xref="S5.SS3.SSS1.p1.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.SSS1.p1.4.m4.1b"><ci id="S5.SS3.SSS1.p1.4.m4.1.1.cmml" xref="S5.SS3.SSS1.p1.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.SSS1.p1.4.m4.1c">k</annotation></semantics></math>).
Training with the baseline ReLU only gives a progressive code up to a value that corresponds to using all positive latents.</p>
</div>
</section>
<section id="S5.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.3.2 </span>Fixed sparsity versus fixed threshold</h4>

<div id="S5.SS3.SSS2.p1" class="ltx_para">
<p id="S5.SS3.SSS2.p1.2" class="ltx_p">At test time, the activation function can also be replaced by a JumpReLU activation, which activates above a fixed threshold <math id="S5.SS3.SSS2.p1.1.m1.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S5.SS3.SSS2.p1.1.m1.1a"><mi id="S5.SS3.SSS2.p1.1.m1.1.1" xref="S5.SS3.SSS2.p1.1.m1.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.SSS2.p1.1.m1.1b"><ci id="S5.SS3.SSS2.p1.1.m1.1.1.cmml" xref="S5.SS3.SSS2.p1.1.m1.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.SSS2.p1.1.m1.1c">\theta</annotation></semantics></math>, <math id="S5.SS3.SSS2.p1.2.m2.2" class="ltx_Math" alttext="J_{\theta}(x)=x\cdot\mathbf{1}_{(x&gt;\theta)}" display="inline"><semantics id="S5.SS3.SSS2.p1.2.m2.2a"><mrow id="S5.SS3.SSS2.p1.2.m2.2.3" xref="S5.SS3.SSS2.p1.2.m2.2.3.cmml"><mrow id="S5.SS3.SSS2.p1.2.m2.2.3.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.cmml"><msub id="S5.SS3.SSS2.p1.2.m2.2.3.2.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2.cmml"><mi id="S5.SS3.SSS2.p1.2.m2.2.3.2.2.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2.2.cmml">J</mi><mi id="S5.SS3.SSS2.p1.2.m2.2.3.2.2.3" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S5.SS3.SSS2.p1.2.m2.2.3.2.1" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.1.cmml">​</mo><mrow id="S5.SS3.SSS2.p1.2.m2.2.3.2.3.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.cmml"><mo stretchy="false" id="S5.SS3.SSS2.p1.2.m2.2.3.2.3.2.1" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.cmml">(</mo><mi id="S5.SS3.SSS2.p1.2.m2.2.2" xref="S5.SS3.SSS2.p1.2.m2.2.2.cmml">x</mi><mo stretchy="false" id="S5.SS3.SSS2.p1.2.m2.2.3.2.3.2.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.cmml">)</mo></mrow></mrow><mo id="S5.SS3.SSS2.p1.2.m2.2.3.1" xref="S5.SS3.SSS2.p1.2.m2.2.3.1.cmml">=</mo><mrow id="S5.SS3.SSS2.p1.2.m2.2.3.3" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.cmml"><mi id="S5.SS3.SSS2.p1.2.m2.2.3.3.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.2.cmml">x</mi><mo lspace="0.222em" rspace="0.222em" id="S5.SS3.SSS2.p1.2.m2.2.3.3.1" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.1.cmml">⋅</mo><msub id="S5.SS3.SSS2.p1.2.m2.2.3.3.3" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.3.cmml"><mn id="S5.SS3.SSS2.p1.2.m2.2.3.3.3.2" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.3.2.cmml">𝟏</mn><mrow id="S5.SS3.SSS2.p1.2.m2.1.1.1.1" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.2" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.cmml"><mi id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.2" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.2.cmml">x</mi><mo id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.1" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.1.cmml">&gt;</mo><mi id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.3" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.3" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.SSS2.p1.2.m2.2b"><apply id="S5.SS3.SSS2.p1.2.m2.2.3.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3"><eq id="S5.SS3.SSS2.p1.2.m2.2.3.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.1"></eq><apply id="S5.SS3.SSS2.p1.2.m2.2.3.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.2"><times id="S5.SS3.SSS2.p1.2.m2.2.3.2.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.1"></times><apply id="S5.SS3.SSS2.p1.2.m2.2.3.2.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2"><csymbol cd="ambiguous" id="S5.SS3.SSS2.p1.2.m2.2.3.2.2.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2">subscript</csymbol><ci id="S5.SS3.SSS2.p1.2.m2.2.3.2.2.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2.2">𝐽</ci><ci id="S5.SS3.SSS2.p1.2.m2.2.3.2.2.3.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.2.2.3">𝜃</ci></apply><ci id="S5.SS3.SSS2.p1.2.m2.2.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.2">𝑥</ci></apply><apply id="S5.SS3.SSS2.p1.2.m2.2.3.3.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.3"><ci id="S5.SS3.SSS2.p1.2.m2.2.3.3.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.1">⋅</ci><ci id="S5.SS3.SSS2.p1.2.m2.2.3.3.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.2">𝑥</ci><apply id="S5.SS3.SSS2.p1.2.m2.2.3.3.3.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.3"><csymbol cd="ambiguous" id="S5.SS3.SSS2.p1.2.m2.2.3.3.3.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.3">subscript</csymbol><cn type="integer" id="S5.SS3.SSS2.p1.2.m2.2.3.3.3.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.2.3.3.3.2">1</cn><apply id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1"><gt id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.1.cmml" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.1"></gt><ci id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.2.cmml" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.2">𝑥</ci><ci id="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.3.cmml" xref="S5.SS3.SSS2.p1.2.m2.1.1.1.1.1.3">𝜃</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.SSS2.p1.2.m2.2c">J_{\theta}(x)=x\cdot\mathbf{1}_{(x&gt;\theta)}</annotation></semantics></math>. In contrast to TopK, JumpReLU leads to a selection of active latents where the number of active latents can vary across tokens.
Results for replacing the activation function at test-time with a JumpReLU are shown in dashed lines in <a href="#S5.F10" title="Figure 10 ‣ 5.3 Progressive recovery ‣ 5 Understanding the TopK activation function ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a>.</p>
</div>
<div id="S5.SS3.SSS2.p2" class="ltx_para">
<p id="S5.SS3.SSS2.p2.4" class="ltx_p">For autoencoders trained with TopK, the test-time TopK and JumpReLU curves are superimposed only for values corresponding to an L<sub id="S5.SS3.SSS2.p2.4.1" class="ltx_sub">0</sub> below the training L<sub id="S5.SS3.SSS2.p2.4.2" class="ltx_sub">0</sub>, otherwise the JumpReLU activation is worse than the TopK activation. This discrepancy disappears with Multi-TopK, where both curves are nearly superimposed, which means that the model can be used with either a fixed or a dynamic number of latents per token without loss in reconstruction.
The two curves are also superimposed for autoencoders trained with ReLU.
Interestingly, it is sometimes more efficient to train a ReLU model with a low L<sub id="S5.SS3.SSS2.p2.4.3" class="ltx_sub">1</sub> penalty and to use a TopK or JumpReLU at test time, than to use a higher L<sub id="S5.SS3.SSS2.p2.4.4" class="ltx_sub">1</sub> penalty that would give a similar sparsity level (a result independently described in <cite class="ltx_cite ltx_citemacro_citet">Nanda et al. (<a href="#bib.bib46" title="" class="ltx_ref">2024</a>)</cite>).</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Limitations and Future Directions</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We believe many improvements can be made to our autoencoders.</p>
</div>
<div id="S6.p2" class="ltx_para">
<ul id="S6.I1" class="ltx_itemize">
<li id="S6.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i1.p1" class="ltx_para">
<p id="S6.I1.i1.p1.3" class="ltx_p">TopK forces every token to use exactly <math id="S6.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S6.I1.i1.p1.1.m1.1a"><mi id="S6.I1.i1.p1.1.m1.1.1" xref="S6.I1.i1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S6.I1.i1.p1.1.m1.1b"><ci id="S6.I1.i1.p1.1.m1.1.1.cmml" xref="S6.I1.i1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.I1.i1.p1.1.m1.1c">k</annotation></semantics></math> latents, which is likely suboptimal. Ideally we would constrain <math id="S6.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="\mathbb{E}[L_{0}]" display="inline"><semantics id="S6.I1.i1.p1.2.m2.1a"><mrow id="S6.I1.i1.p1.2.m2.1.1" xref="S6.I1.i1.p1.2.m2.1.1.cmml"><mi id="S6.I1.i1.p1.2.m2.1.1.3" xref="S6.I1.i1.p1.2.m2.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="S6.I1.i1.p1.2.m2.1.1.2" xref="S6.I1.i1.p1.2.m2.1.1.2.cmml">​</mo><mrow id="S6.I1.i1.p1.2.m2.1.1.1.1" xref="S6.I1.i1.p1.2.m2.1.1.1.2.cmml"><mo stretchy="false" id="S6.I1.i1.p1.2.m2.1.1.1.1.2" xref="S6.I1.i1.p1.2.m2.1.1.1.2.1.cmml">[</mo><msub id="S6.I1.i1.p1.2.m2.1.1.1.1.1" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1.cmml"><mi id="S6.I1.i1.p1.2.m2.1.1.1.1.1.2" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1.2.cmml">L</mi><mn id="S6.I1.i1.p1.2.m2.1.1.1.1.1.3" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1.3.cmml">0</mn></msub><mo stretchy="false" id="S6.I1.i1.p1.2.m2.1.1.1.1.3" xref="S6.I1.i1.p1.2.m2.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.I1.i1.p1.2.m2.1b"><apply id="S6.I1.i1.p1.2.m2.1.1.cmml" xref="S6.I1.i1.p1.2.m2.1.1"><times id="S6.I1.i1.p1.2.m2.1.1.2.cmml" xref="S6.I1.i1.p1.2.m2.1.1.2"></times><ci id="S6.I1.i1.p1.2.m2.1.1.3.cmml" xref="S6.I1.i1.p1.2.m2.1.1.3">𝔼</ci><apply id="S6.I1.i1.p1.2.m2.1.1.1.2.cmml" xref="S6.I1.i1.p1.2.m2.1.1.1.1"><csymbol cd="latexml" id="S6.I1.i1.p1.2.m2.1.1.1.2.1.cmml" xref="S6.I1.i1.p1.2.m2.1.1.1.1.2">delimited-[]</csymbol><apply id="S6.I1.i1.p1.2.m2.1.1.1.1.1.cmml" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S6.I1.i1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1">subscript</csymbol><ci id="S6.I1.i1.p1.2.m2.1.1.1.1.1.2.cmml" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1.2">𝐿</ci><cn type="integer" id="S6.I1.i1.p1.2.m2.1.1.1.1.1.3.cmml" xref="S6.I1.i1.p1.2.m2.1.1.1.1.1.3">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.I1.i1.p1.2.m2.1c">\mathbb{E}[L_{0}]</annotation></semantics></math> rather than <math id="S6.I1.i1.p1.3.m3.1" class="ltx_Math" alttext="L_{0}" display="inline"><semantics id="S6.I1.i1.p1.3.m3.1a"><msub id="S6.I1.i1.p1.3.m3.1.1" xref="S6.I1.i1.p1.3.m3.1.1.cmml"><mi id="S6.I1.i1.p1.3.m3.1.1.2" xref="S6.I1.i1.p1.3.m3.1.1.2.cmml">L</mi><mn id="S6.I1.i1.p1.3.m3.1.1.3" xref="S6.I1.i1.p1.3.m3.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S6.I1.i1.p1.3.m3.1b"><apply id="S6.I1.i1.p1.3.m3.1.1.cmml" xref="S6.I1.i1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S6.I1.i1.p1.3.m3.1.1.1.cmml" xref="S6.I1.i1.p1.3.m3.1.1">subscript</csymbol><ci id="S6.I1.i1.p1.3.m3.1.1.2.cmml" xref="S6.I1.i1.p1.3.m3.1.1.2">𝐿</ci><cn type="integer" id="S6.I1.i1.p1.3.m3.1.1.3.cmml" xref="S6.I1.i1.p1.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.I1.i1.p1.3.m3.1c">L_{0}</annotation></semantics></math>.</p>
</div>
</li>
<li id="S6.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i2.p1" class="ltx_para">
<p id="S6.I1.i2.p1.1" class="ltx_p">The optimization can likely be greatly improved, for example with learning rate scheduling,<span id="footnote15" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">15</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">15</sup><span class="ltx_tag ltx_tag_note">15</span>Anecdotally, we also found that lowering learning rates helped with decreasing dead latents.</span></span></span> better optimizers, and better aux losses for preventing dead latents.</p>
</div>
</li>
<li id="S6.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i3.p1" class="ltx_para">
<p id="S6.I1.i3.p1.1" class="ltx_p">Much more could be done to understand what metrics best track relevance to downstream applications, and to study those applications themselves. Applications include: finding vectors for steering behavior, doing anomaly detection, identifying circuits, and more.</p>
</div>
</li>
<li id="S6.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i4.p1" class="ltx_para">
<p id="S6.I1.i4.p1.1" class="ltx_p">We’re excited about work in the direction of combining MoE <cite class="ltx_cite ltx_citemacro_citep">(Shazeer et al., <a href="#bib.bib56" title="" class="ltx_ref">2017</a>)</cite> and autoencoders, which would substantially improve the asymptotic cost of autoencoder training, and enable much larger autoencoders.</p>
</div>
</li>
<li id="S6.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i5.p1" class="ltx_para">
<p id="S6.I1.i5.p1.1" class="ltx_p">A large fraction of the random activations of features we find, especially in GPT-4, are not yet adequately monosemantic. We believe that with improved techniques and greater scale<span id="footnote16" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">16</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">16</sup><span class="ltx_tag ltx_tag_note">16</span>both in number of latents and in training tokens</span></span></span> this is potentially surmountable.</p>
</div>
</li>
<li id="S6.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i6.p1" class="ltx_para">
<p id="S6.I1.i6.p1.1" class="ltx_p">Our probe based metric is quite noisy, which could be improved by having a greater breadth of tasks and higher quality tasks.</p>
</div>
</li>
<li id="S6.I1.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i7.p1" class="ltx_para">
<p id="S6.I1.i7.p1.1" class="ltx_p">While we use n2g for its computational efficiency, it is only able to capture very simple patterns. We believe there is a lot of room for improvement in terms of more expressive explanation methods that are also cheap enough to simulate to estimate explanation precision.</p>
</div>
</li>
<li id="S6.I1.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i8.p1" class="ltx_para">
<p id="S6.I1.i8.p1.1" class="ltx_p">A context length of 64 tokens is potentially too few tokens to exhibit the most interesting behaviors of GPT-4.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Related work</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.3" class="ltx_p">Sparse coding on an over-complete dictionary was introduced by <cite class="ltx_cite ltx_citemacro_cite">Mallat and Zhang (<a href="#bib.bib38" title="" class="ltx_ref">1993</a>)</cite>.
<cite class="ltx_cite ltx_citemacro_cite">Olshausen and Field (<a href="#bib.bib48" title="" class="ltx_ref">1996</a>)</cite> refined the idea by proposing to learn the dictionary from the data, without supervision. This approach has been particularly influential in image processing, as seen for example in <cite class="ltx_cite ltx_citemacro_citep">(Mairal et al., <a href="#bib.bib34" title="" class="ltx_ref">2014</a>)</cite>.
Later, <cite class="ltx_cite ltx_citemacro_cite">Hinton and Salakhutdinov (<a href="#bib.bib21" title="" class="ltx_ref">2006</a>)</cite> proposed the autoencoder architecture to perform dimensionality reduction.
Combining these concepts, sparse autoencoders were developed <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib30" title="" class="ltx_ref">2007</a>; Le et al., <a href="#bib.bib29" title="" class="ltx_ref">2013</a>; Konda et al., <a href="#bib.bib28" title="" class="ltx_ref">2014</a>)</cite> to train autoencoders with sparsity priors, such as the L<sub id="S7.p1.3.1" class="ltx_sub">1</sub> penalty, to extract sparse features.
<cite class="ltx_cite ltx_citemacro_cite">Makhzani and Frey (<a href="#bib.bib36" title="" class="ltx_ref">2013</a>)</cite> refined this concept by introducing <math id="S7.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S7.p1.2.m2.1a"><mi id="S7.p1.2.m2.1.1" xref="S7.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S7.p1.2.m2.1b"><ci id="S7.p1.2.m2.1.1.cmml" xref="S7.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.p1.2.m2.1c">k</annotation></semantics></math>-sparse autoencoders, which use a TopK activation function instead of the L<sub id="S7.p1.3.2" class="ltx_sub">1</sub> penalty. <cite class="ltx_cite ltx_citemacro_cite">Makelov et al. (<a href="#bib.bib35" title="" class="ltx_ref">2024</a>)</cite> evaluates autoencoders using a metric that measures recovery of features from previously discovered circuits.</p>
</div>
<div id="S7.p2" class="ltx_para">
<p id="S7.p2.2" class="ltx_p">More recently, sparse autoencoders were applied to language models <cite class="ltx_cite ltx_citemacro_citep">(Yun et al., <a href="#bib.bib70" title="" class="ltx_ref">2021</a>; Lee Sharkey, <a href="#bib.bib31" title="" class="ltx_ref">2022</a>; Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>; Cunningham et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite>,
and multiple sparse autoencoders were trained on small open-source language models <cite class="ltx_cite ltx_citemacro_citep">(Marks, <a href="#bib.bib39" title="" class="ltx_ref">2023</a>; Bloom, <a href="#bib.bib5" title="" class="ltx_ref">2024</a>; Mossing et al., <a href="#bib.bib45" title="" class="ltx_ref">2024</a>)</cite>.
<cite class="ltx_cite ltx_citemacro_cite">Marks et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite> showed that the resulting features from sparse autoencoders can find sparse circuits in language models.
<cite class="ltx_cite ltx_citemacro_cite">Wright and Sharkey (<a href="#bib.bib68" title="" class="ltx_ref">2024</a>)</cite> pointed out that sparse autoencoders are subject to activation shrinking from L<sub id="S7.p2.2.1" class="ltx_sub">1</sub> penalties, a property of L<sub id="S7.p2.2.2" class="ltx_sub">1</sub> penalties first described in <cite class="ltx_cite ltx_citemacro_cite">Tibshirani (<a href="#bib.bib65" title="" class="ltx_ref">1996</a>)</cite>.
<cite class="ltx_cite ltx_citemacro_cite">Taggart (<a href="#bib.bib62" title="" class="ltx_ref">2024</a>)</cite> and <cite class="ltx_cite ltx_citemacro_cite">Rajamanoharan et al. (<a href="#bib.bib52" title="" class="ltx_ref">2024</a>)</cite> proposed to use different activation functions to address activation shrinkage in sparse autoencoders. <cite class="ltx_cite ltx_citemacro_citet">Braun et al. (<a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite> proposed to train sparse autoencoders on downstream KL instead of reconstruction MSE.</p>
</div>
<div id="S7.p3" class="ltx_para">
<p id="S7.p3.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_cite">Kaplan et al. (<a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite> studied scaling laws for language models which examine how loss varies with various hyperparameters. <cite class="ltx_cite ltx_citemacro_cite">Clark et al. (<a href="#bib.bib10" title="" class="ltx_ref">2022</a>)</cite> explore scaling laws related to sparsity using a bilinear fit.
<cite class="ltx_cite ltx_citemacro_cite">Lindsey et al. (<a href="#bib.bib33" title="" class="ltx_ref">2024</a>)</cite> studied scaling laws specifically for autoencoders, defining the loss as a specific balance of reconstruction and sparsity (rather than simply reconstruction, while holding sparsity fixed).</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgments and Disclosure of Funding</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We are deeply grateful to Jan Leike and Ilya Sutskever for leading the Superalignment team and creating the research environment which made this work possible. We thank David Farhi for supporting our work after their departure.</p>
</div>
<div id="Sx1.p2" class="ltx_para">
<p id="Sx1.p2.1" class="ltx_p">We thank Cathy Yeh for explorations in finding features. We thank Carroll Wainwright for initial insights into clustering latents.</p>
</div>
<div id="Sx1.p3" class="ltx_para">
<p id="Sx1.p3.1" class="ltx_p">We thank Steven Bills, Dan Mossing, Cathy Yeh, William Saunders, Boaz Barak, Jakub Pachocki, and Jan Leike for many discussions about autoencoders and their applications.
We thank Manas Joglekar for some improvements on an earlier version of our activation loading. We thank William Saunders for an early autoencoder visualization implementation.</p>
</div>
<div id="Sx1.p4" class="ltx_para">
<p id="Sx1.p4.1" class="ltx_p">We thank Trenton Bricken, Dan Mossing, and Neil Chowdhury for feedback on an earlier version of this manuscript.</p>
</div>
<div id="Sx1.p5" class="ltx_para">
<p id="Sx1.p5.1" class="ltx_p">We thank Trenton Bricken, Lawrence Chan, Hoagy Cunningham, Adam Goucher, Ryan Greenblatt, Tristan Hume, Jan Hendrik Kirchner, Jake Mendel, Neel Nanda, Noa Nabeshima, Chris Olah, Logan Riggs, Fabien Roger, Buck Shlegeris, John Schulman, Lee Sharkey, Glen Taggart, Adly Templeton, Vikrant Varma for valuable discussions.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aharon et al. [2006]</span>
<span class="ltx_bibblock">
Michal Aharon, Michael Elad, and Alfred Bruckstein.

</span>
<span class="ltx_bibblock">K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on signal processing</em>, 54(11):4311–4322, 2006.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Baehrens et al. [2010]</span>
<span class="ltx_bibblock">
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller.

</span>
<span class="ltx_bibblock">How to explain individual classification decisions.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research</em>, 11:1803–1831, 2010.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bills et al. [2023]</span>
<span class="ltx_bibblock">
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders.

</span>
<span class="ltx_bibblock">Language models can explain neurons in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">OpenAI Blog</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. [2020]</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Thirty-Fourth AAAI Conference on Artificial Intelligence</em>, 2020.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bloom [2024]</span>
<span class="ltx_bibblock">
Joseph Bloom.

</span>
<span class="ltx_bibblock">Open source sparse autoencoders for all residual stream layers of gpt2-small.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">AI Alignment Forum</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.lesswrong.com/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.lesswrong.com/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bolukbasi et al. [2021]</span>
<span class="ltx_bibblock">
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg.

</span>
<span class="ltx_bibblock">An interpretability illusion for BERT.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.07143</em>, 2021.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Braun et al. [2024]</span>
<span class="ltx_bibblock">
Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey.

</span>
<span class="ltx_bibblock">Identifying functionally important features with end-to-end sparse dictionary learning.

</span>
<span class="ltx_bibblock">2024.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bricken et al. [2023]</span>
<span class="ltx_bibblock">
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah.

</span>
<span class="ltx_bibblock">Towards monosemanticity: Decomposing language models with dictionary learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2023.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2023/monosemantic-features/index.html.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cjadams et al. [2017]</span>
<span class="ltx_bibblock">
Cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, Nithum, and Will Cukierski.

</span>
<span class="ltx_bibblock">Toxic comment classification challenge.

</span>
<span class="ltx_bibblock">Kaggle, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. [2022]</span>
<span class="ltx_bibblock">
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al.

</span>
<span class="ltx_bibblock">Unified scaling laws for routed language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 4057–4086. PMLR, 2022.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Conerly et al. [2024]</span>
<span class="ltx_bibblock">
Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan.

</span>
<span class="ltx_bibblock">Update on how we train saes.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2024.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2024/april-update/index.html#training-saes.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cunningham et al. [2023]</span>
<span class="ltx_bibblock">
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey.

</span>
<span class="ltx_bibblock">Sparse autoencoders find highly interpretable features in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.08600</em>, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elhage et al. [2022]</span>
<span class="ltx_bibblock">
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al.

</span>
<span class="ltx_bibblock">Toy models of superposition.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2209.10652</em>, 2022.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Erichson et al. [2019]</span>
<span class="ltx_bibblock">
N Benjamin Erichson, Zhewei Yao, and Michael W Mahoney.

</span>
<span class="ltx_bibblock">JumpReLU: A retrofit defense strategy for adversarial attacks.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1904.03750</em>, 2019.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Foote et al. [2023]</span>
<span class="ltx_bibblock">
Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez.

</span>
<span class="ltx_bibblock">Neuron to graph: Interpreting language model neurons at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.19911</em>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goh [2016]</span>
<span class="ltx_bibblock">
Gabriel Goh.

</span>
<span class="ltx_bibblock">Decoding the thought vector, 2016.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://gabgoh.github.io/ThoughtVectors/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://gabgoh.github.io/ThoughtVectors/</a>.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-24.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Antonio Gulli.

</span>
<span class="ltx_bibblock">Ag’s corpus of news articles.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html</a>.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-21.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gurnee et al. [2023]</span>
<span class="ltx_bibblock">
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas.

</span>
<span class="ltx_bibblock">Finding neurons in a haystack: Case studies with sparse probing.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.01610</em>, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. [2020]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Aligning ai with shared human values.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2008.02275</em>, 2020.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Henighan et al. [2020]</span>
<span class="ltx_bibblock">
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al.

</span>
<span class="ltx_bibblock">Scaling laws for autoregressive generative modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2010.14701</em>, 2020.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton and Salakhutdinov [2006]</span>
<span class="ltx_bibblock">
Geoffrey E Hinton and Ruslan R Salakhutdinov.

</span>
<span class="ltx_bibblock">Reducing the dimensionality of data with neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">science</em>, 313(5786):504–507, 2006.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. [2022]</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.15556</em>, 2022.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. [2019]</span>
<span class="ltx_bibblock">
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al.

</span>
<span class="ltx_bibblock">Gpipe: Efficient training of giant neural networks using pipeline parallelism.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 32, 2019.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jermyn and Templeton [2024]</span>
<span class="ltx_bibblock">
Adam Jermyn and Adly Templeton.

</span>
<span class="ltx_bibblock">Ghost grads: An improvement on resampling.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2024.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2024/jan-update/index.html#dict-learning-resampling.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. [2020]</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2001.08361</em>, 2020.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kingma and Ba [2014]</span>
<span class="ltx_bibblock">
Diederik P Kingma and Jimmy Ba.

</span>
<span class="ltx_bibblock">Adam: A method for stochastic optimization.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1412.6980</em>, 2014.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Koehn [2005]</span>
<span class="ltx_bibblock">
Philipp Koehn.

</span>
<span class="ltx_bibblock">Europarl: A parallel corpus for statistical machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of Machine Translation Summit X: Papers</em>, pages 79–86, Phuket, Thailand, September 13-15 2005.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2005.mtsummit-papers.11" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2005.mtsummit-papers.11</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Konda et al. [2014]</span>
<span class="ltx_bibblock">
Kishore Konda, Roland Memisevic, and David Krueger.

</span>
<span class="ltx_bibblock">Zero-bias autoencoders and the benefits of co-adapting features.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1402.3337</em>, 2014.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Le et al. [2013]</span>
<span class="ltx_bibblock">
Quoc V Le, Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S Corrado, Jeff Dean, and Andrew Y Ng.

</span>
<span class="ltx_bibblock">Building high-level features using large scale unsupervised learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">2013 IEEE international conference on acoustics, speech and signal processing</em>, pages 8595–8598. IEEE, 2013.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. [2007]</span>
<span class="ltx_bibblock">
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng.

</span>
<span class="ltx_bibblock">Sparse deep belief net model for visual area v2.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 20, 2007.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee Sharkey [2022]</span>
<span class="ltx_bibblock">
Beren Millidge Lee Sharkey, Dan Braun.

</span>
<span class="ltx_bibblock">Taking features out of superposition with sparse autoencoders.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">AI Alignment Forum</em>, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition</a>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. [2021]</span>
<span class="ltx_bibblock">
Stephanie Lin, Jacob Hilton, and Owain Evans.

</span>
<span class="ltx_bibblock">Truthfulqa: Measuring how models mimic human falsehoods.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2109.07958</em>, 2021.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lindsey et al. [2024]</span>
<span class="ltx_bibblock">
Jack Lindsey, Tom Conerly, Adly Templeton, Jonathan Marcus, and Tom Henighan.

</span>
<span class="ltx_bibblock">Scaling laws for dictionary learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2024.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2024/april-update/index.html#scaling-laws.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mairal et al. [2014]</span>
<span class="ltx_bibblock">
Julien Mairal, Francis Bach, Jean Ponce, et al.

</span>
<span class="ltx_bibblock">Sparse modeling for image and vision processing.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Foundations and Trends® in Computer Graphics and Vision</em>, 8(2-3):85–283, 2014.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Makelov et al. [2024]</span>
<span class="ltx_bibblock">
Aleksandar Makelov, George Lange, and Neel Nanda.

</span>
<span class="ltx_bibblock">Towards principled evaluations of sparse autoencoders for interpretability and control, 2024.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Makhzani and Frey [2013]</span>
<span class="ltx_bibblock">
Alireza Makhzani and Brendan Frey.

</span>
<span class="ltx_bibblock">K-sparse autoencoders.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1312.5663</em>, 2013.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maleki [2009]</span>
<span class="ltx_bibblock">
Arian Maleki.

</span>
<span class="ltx_bibblock">Coherence analysis of iterative thresholding algorithms.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton)</em>, pages 236–243. IEEE, 2009.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mallat and Zhang [1993]</span>
<span class="ltx_bibblock">
Stéphane G Mallat and Zhifeng Zhang.

</span>
<span class="ltx_bibblock">Matching pursuits with time-frequency dictionaries.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on signal processing</em>, 41(12):3397–3415, 1993.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Marks [2023]</span>
<span class="ltx_bibblock">
Sam Marks.

</span>
<span class="ltx_bibblock">Some open-source dictionaries and dictionary learning infrastructure.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">AI Alignment Forum</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.alignmentforum.org/posts/AaoWLcmpY3LKvtdyq/some-open-source-dictionaries-and-dictionary-learning" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.alignmentforum.org/posts/AaoWLcmpY3LKvtdyq/some-open-source-dictionaries-and-dictionary-learning</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Marks et al. [2024]</span>
<span class="ltx_bibblock">
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller.

</span>
<span class="ltx_bibblock">Sparse feature circuits: Discovering and editing interpretable causal graphs in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.19647</em>, 2024.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McAuley and Leskovec [2013]</span>
<span class="ltx_bibblock">
Julian McAuley and Jure Leskovec.

</span>
<span class="ltx_bibblock">Hidden factors and hidden topics: understanding rating dimensions with review text.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 7th ACM conference on Recommender systems</em>, pages 165–172, 2013.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McInnes et al. [2018]</span>
<span class="ltx_bibblock">
Leland McInnes, John Healy, and James Melville.

</span>
<span class="ltx_bibblock">Umap: Uniform manifold approximation and projection for dimension reduction.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1802.03426</em>, 2018.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meinshausen [2007]</span>
<span class="ltx_bibblock">
Nicolai Meinshausen.

</span>
<span class="ltx_bibblock">Relaxed lasso.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Computational Statistics &amp; Data Analysis</em>, 52(1):374–393, 2007.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mihaylov et al. [2018]</span>
<span class="ltx_bibblock">
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.

</span>
<span class="ltx_bibblock">Can a suit of armor conduct electricity? a new dataset for open book question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">EMNLP</em>, 2018.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mossing et al. [2024]</span>
<span class="ltx_bibblock">
Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders.

</span>
<span class="ltx_bibblock">Transformer debugger.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/openai/transformer-debugger" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/openai/transformer-debugger</a>, 2024.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nanda et al. [2024]</span>
<span class="ltx_bibblock">
Neel Nanda, Arthur Conmy, Lewis Smith, Senthooran Rajamanoharan, Tom Lieberum, János Kramár, and Vikrant Varma.

</span>
<span class="ltx_bibblock">Progress update #1 from the gdm mech interp team: Full update.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">AI Alignment Forum</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/progress-update-1-from-the-gdm-mech-interp-team-full-update" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/progress-update-1-from-the-gdm-mech-interp-team-full-update</a>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Olah et al. [2024]</span>
<span class="ltx_bibblock">
Chris Olah, Adly Templeton, Trenton Bricken, and Adam Jermyn.

</span>
<span class="ltx_bibblock">Open problem: Attribution dictionary learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2024.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2024/april-update/index.html#attr-dl.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Olshausen and Field [1996]</span>
<span class="ltx_bibblock">
Bruno A Olshausen and David J Field.

</span>
<span class="ltx_bibblock">Emergence of simple-cell receptive field properties by learning a sparse code for natural images.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Nature</em>, 381(6583):607–609, 1996.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI [2019]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-2 output dataset.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/openai/gpt-2-output-dataset/tree/master" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/openai/gpt-2-output-dataset/tree/master</a>, 2019.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-21.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI [2023]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">GPT-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">arXiv prepreint arXiv:2303.08774</em>, 2023.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. [2019]</span>
<span class="ltx_bibblock">
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">OpenAI blog</em>, 1(8):9, 2019.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajamanoharan et al. [2024]</span>
<span class="ltx_bibblock">
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda.

</span>
<span class="ltx_bibblock">Improving dictionary learning with gated sparse autoencoders.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.16014</em>, 2024.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rogers et al. [2020]</span>
<span class="ltx_bibblock">
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky.

</span>
<span class="ltx_bibblock">Getting closer to ai complete question answering: A set of prerequisite real tasks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI conference on artificial intelligence</em>, volume 34, pages 8722–8731, 2020.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ruppert [1988]</span>
<span class="ltx_bibblock">
David Ruppert.

</span>
<span class="ltx_bibblock">Efficient estimations from a slowly convergent robbins-monro process.

</span>
<span class="ltx_bibblock">Technical report, Cornell University Operations Research and Industrial Engineering, 1988.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sap et al. [2019]</span>
<span class="ltx_bibblock">
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.

</span>
<span class="ltx_bibblock">Socialiqa: Commonsense reasoning about social interactions.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1904.09728</em>, 2019.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer et al. [2017]</span>
<span class="ltx_bibblock">
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.

</span>
<span class="ltx_bibblock">Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1701.06538</em>, 2017.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shoeybi et al. [2019]</span>
<span class="ltx_bibblock">
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.

</span>
<span class="ltx_bibblock">Megatron-lm: Training multi-billion parameter language models using model parallelism.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1909.08053</em>, 2019.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Simonyan et al. [2013]</span>
<span class="ltx_bibblock">
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.

</span>
<span class="ltx_bibblock">Deep inside convolutional networks: Visualising image classification models and saliency maps.

</span>
<span class="ltx_bibblock"><em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1312.6034</em>, 2013.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Skodras et al. [2001]</span>
<span class="ltx_bibblock">
Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi.

</span>
<span class="ltx_bibblock">The jpeg 2000 still image compression standard.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">IEEE Signal processing magazine</em>, 18(5):36–58, 2001.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. [2024]</span>
<span class="ltx_bibblock">
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu.

</span>
<span class="ltx_bibblock">Massive activations in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.17762</em>, 2024.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tafjord et al. [2019]</span>
<span class="ltx_bibblock">
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark.

</span>
<span class="ltx_bibblock">Quartz: An open-domain dataset of qualitative relationship questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1909.03553</em>, 2019.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taggart [2024]</span>
<span class="ltx_bibblock">
Glen Taggart.

</span>
<span class="ltx_bibblock">ProLU: A nonlinearity for sparse autoencoders.

</span>
<span class="ltx_bibblock"><em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">AI Alignment Forum</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-pareto-improvement-for-sparse-autoencoders" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-pareto-improvement-for-sparse-autoencoders</a>.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Talmor et al. [2022]</span>
<span class="ltx_bibblock">
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant.

</span>
<span class="ltx_bibblock">Commonsenseqa 2.0: Exposing the limits of ai through gamification.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2201.05320</em>, 2022.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Templeton et al. [2024]</span>
<span class="ltx_bibblock">
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan.

</span>
<span class="ltx_bibblock">Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.

</span>
<span class="ltx_bibblock"><em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html</a>.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tibshirani [1996]</span>
<span class="ltx_bibblock">
Robert Tibshirani.

</span>
<span class="ltx_bibblock">Regression shrinkage and selection via the lasso.

</span>
<span class="ltx_bibblock"><em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">Journal of the Royal Statistical Society Series B: Statistical Methodology</em>, 58(1):267–288, 1996.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2018]</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.

</span>
<span class="ltx_bibblock">Glue: A multi-task benchmark and analysis platform for natural language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib66.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1804.07461</em>, 2018.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Welbl et al. [2017]</span>
<span class="ltx_bibblock">
Johannes Welbl, Nelson F. Liu, Matt Gardner, Gabor Angeli, Rik Koncel-Kedziorski, Emily Bender, Kyle Richardson, Peter Clark, and Nate Kushman.

</span>
<span class="ltx_bibblock">Crowdsourcing multiple choice science questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1707.06209</em>, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1707.06209" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1707.06209</a>.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wright and Sharkey [2024]</span>
<span class="ltx_bibblock">
Benjamin Wright and Lee Sharkey.

</span>
<span class="ltx_bibblock">Addressing feature suppression in SAEs.

</span>
<span class="ltx_bibblock"><em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">AI Alignment Forum</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.alignmentforum.org/posts/3JuSjTZyMzaSeTxKk/addressing-feature-suppression-in-saes" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.alignmentforum.org/posts/3JuSjTZyMzaSeTxKk/addressing-feature-suppression-in-saes</a>.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. [2023]</span>
<span class="ltx_bibblock">
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.

</span>
<span class="ltx_bibblock">Efficient streaming language models with attention sinks.

</span>
<span class="ltx_bibblock"><em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.17453</em>, 2023.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yun et al. [2021]</span>
<span class="ltx_bibblock">
Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun.

</span>
<span class="ltx_bibblock">Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors.

</span>
<span class="ltx_bibblock"><em id="bib.bib70.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2103.15949</em>, 2021.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. [2023]</span>
<span class="ltx_bibblock">
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al.

</span>
<span class="ltx_bibblock">Pytorch fsdp: experiences on scaling fully sharded data parallel.

</span>
<span class="ltx_bibblock"><em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.11277</em>, 2023.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2019]</span>
<span class="ltx_bibblock">
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth.

</span>
<span class="ltx_bibblock">“going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib72.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</em>, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1909.03065" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1909.03065</a>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Optimization</h2>

<figure id="A1.F11" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_LofN.png" id="A1.F11.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="598" height="197" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2406.04093/assets/figures/fig_gpt4_LofN.png" id="A1.F11.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="598" height="197" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A1.F11.4.2.1" class="ltx_text" style="font-size:90%;">Figure 11</span>: </span><span id="A1.F11.2.1" class="ltx_text" style="font-size:90%;">Token budget, MSE, and learning rate power laws, averaged across values of <math id="A1.F11.2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.F11.2.1.m1.1b"><mi id="A1.F11.2.1.m1.1.1" xref="A1.F11.2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.F11.2.1.m1.1c"><ci id="A1.F11.2.1.m1.1.1.cmml" xref="A1.F11.2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F11.2.1.m1.1d">k</annotation></semantics></math>. First row is GPT-2, second row is GPT-4. Note that individual fits are noisy (especially for GPT-4) since learning rate sweeps are coarse, and token budget depends on learning rate.</span></figcaption>
</figure>
<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Initialization</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.1" class="ltx_p">We initialize our autoencoders as follows:</p>
</div>
<div id="A1.SS1.p2" class="ltx_para">
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.1" class="ltx_p">We initialize the bias <math id="A1.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="b_{pre}" display="inline"><semantics id="A1.I1.i1.p1.1.m1.1a"><msub id="A1.I1.i1.p1.1.m1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.2" xref="A1.I1.i1.p1.1.m1.1.1.2.cmml">b</mi><mrow id="A1.I1.i1.p1.1.m1.1.1.3" xref="A1.I1.i1.p1.1.m1.1.1.3.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.3.2" xref="A1.I1.i1.p1.1.m1.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A1.I1.i1.p1.1.m1.1.1.3.1" xref="A1.I1.i1.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="A1.I1.i1.p1.1.m1.1.1.3.3" xref="A1.I1.i1.p1.1.m1.1.1.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="A1.I1.i1.p1.1.m1.1.1.3.1a" xref="A1.I1.i1.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="A1.I1.i1.p1.1.m1.1.1.3.4" xref="A1.I1.i1.p1.1.m1.1.1.3.4.cmml">e</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.1.m1.1b"><apply id="A1.I1.i1.p1.1.m1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.I1.i1.p1.1.m1.1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1">subscript</csymbol><ci id="A1.I1.i1.p1.1.m1.1.1.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2">𝑏</ci><apply id="A1.I1.i1.p1.1.m1.1.1.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3"><times id="A1.I1.i1.p1.1.m1.1.1.3.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.1"></times><ci id="A1.I1.i1.p1.1.m1.1.1.3.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2">𝑝</ci><ci id="A1.I1.i1.p1.1.m1.1.1.3.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3">𝑟</ci><ci id="A1.I1.i1.p1.1.m1.1.1.3.4.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.4">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.1.m1.1c">b_{pre}</annotation></semantics></math> to be the geometric median of a sample set of data points, following <cite class="ltx_cite ltx_citemacro_citet">Bricken et al. [<a href="#bib.bib8" title="" class="ltx_ref">2023</a>]</cite>.</p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i2.p1" class="ltx_para">
<p id="A1.I1.i2.p1.1" class="ltx_p">We initialize the encoder directions parallel to the respective decoder directions, so that the corresponding latent read/write directions are the same<span id="footnote17" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">17</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">17</sup><span class="ltx_tag ltx_tag_note">17</span>This is done only at initialization; we do not tie the parameters as in <cite class="ltx_cite ltx_citemacro_citet">Cunningham et al. [<a href="#bib.bib12" title="" class="ltx_ref">2023</a>]</cite>. This strategy is also presented in concurrent work <cite class="ltx_cite ltx_citemacro_citep">[Conerly et al., <a href="#bib.bib11" title="" class="ltx_ref">2024</a>]</cite>.</span></span></span> Directions are chosen uniformly randomly.</p>
</div>
</li>
<li id="A1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i3.p1" class="ltx_para">
<p id="A1.I1.i3.p1.1" class="ltx_p">We scale decoder latent directions to be unit norm at initialization (and also after each training step), following <cite class="ltx_cite ltx_citemacro_citet">Bricken et al. [<a href="#bib.bib8" title="" class="ltx_ref">2023</a>]</cite>.</p>
</div>
</li>
<li id="A1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i4.p1" class="ltx_para">
<p id="A1.I1.i4.p1.1" class="ltx_p">For baseline models we use torch default initialization for encoder magnitudes. For TopK models, we initialized the magnitude of the encoder such that the magnitude of reconstructed vectors match that of the inputs. However, in our ablations we find this has no effect or a weak negative effect (<a href="#A3.F16" title="Figure 16 ‣ C.2 Initialization ‣ Appendix C Training ablations ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 16</span></a>).<span id="footnote18" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">18</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">18</sup><span class="ltx_tag ltx_tag_note">18</span>Note that the scaling factor has nontrivial interaction with <math id="footnote18.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="footnote18.m1.1b"><mi id="footnote18.m1.1.1" xref="footnote18.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="footnote18.m1.1c"><ci id="footnote18.m1.1.1.cmml" xref="footnote18.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote18.m1.1d">n</annotation></semantics></math>, and scales between <math id="footnote18.m2.1" class="ltx_Math" alttext="\Theta(1/\sqrt{k})" display="inline"><semantics id="footnote18.m2.1b"><mrow id="footnote18.m2.1.1" xref="footnote18.m2.1.1.cmml"><mi mathvariant="normal" id="footnote18.m2.1.1.3" xref="footnote18.m2.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="footnote18.m2.1.1.2" xref="footnote18.m2.1.1.2.cmml">​</mo><mrow id="footnote18.m2.1.1.1.1" xref="footnote18.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="footnote18.m2.1.1.1.1.2" xref="footnote18.m2.1.1.1.1.1.cmml">(</mo><mrow id="footnote18.m2.1.1.1.1.1" xref="footnote18.m2.1.1.1.1.1.cmml"><mn id="footnote18.m2.1.1.1.1.1.2" xref="footnote18.m2.1.1.1.1.1.2.cmml">1</mn><mo id="footnote18.m2.1.1.1.1.1.1" xref="footnote18.m2.1.1.1.1.1.1.cmml">/</mo><msqrt id="footnote18.m2.1.1.1.1.1.3" xref="footnote18.m2.1.1.1.1.1.3.cmml"><mi id="footnote18.m2.1.1.1.1.1.3.2" xref="footnote18.m2.1.1.1.1.1.3.2.cmml">k</mi></msqrt></mrow><mo stretchy="false" id="footnote18.m2.1.1.1.1.3" xref="footnote18.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote18.m2.1c"><apply id="footnote18.m2.1.1.cmml" xref="footnote18.m2.1.1"><times id="footnote18.m2.1.1.2.cmml" xref="footnote18.m2.1.1.2"></times><ci id="footnote18.m2.1.1.3.cmml" xref="footnote18.m2.1.1.3">Θ</ci><apply id="footnote18.m2.1.1.1.1.1.cmml" xref="footnote18.m2.1.1.1.1"><divide id="footnote18.m2.1.1.1.1.1.1.cmml" xref="footnote18.m2.1.1.1.1.1.1"></divide><cn type="integer" id="footnote18.m2.1.1.1.1.1.2.cmml" xref="footnote18.m2.1.1.1.1.1.2">1</cn><apply id="footnote18.m2.1.1.1.1.1.3.cmml" xref="footnote18.m2.1.1.1.1.1.3"><root id="footnote18.m2.1.1.1.1.1.3a.cmml" xref="footnote18.m2.1.1.1.1.1.3"></root><ci id="footnote18.m2.1.1.1.1.1.3.2.cmml" xref="footnote18.m2.1.1.1.1.1.3.2">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote18.m2.1d">\Theta(1/\sqrt{k})</annotation></semantics></math> and <math id="footnote18.m3.1" class="ltx_Math" alttext="\Theta(1/k)" display="inline"><semantics id="footnote18.m3.1b"><mrow id="footnote18.m3.1.1" xref="footnote18.m3.1.1.cmml"><mi mathvariant="normal" id="footnote18.m3.1.1.3" xref="footnote18.m3.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="footnote18.m3.1.1.2" xref="footnote18.m3.1.1.2.cmml">​</mo><mrow id="footnote18.m3.1.1.1.1" xref="footnote18.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="footnote18.m3.1.1.1.1.2" xref="footnote18.m3.1.1.1.1.1.cmml">(</mo><mrow id="footnote18.m3.1.1.1.1.1" xref="footnote18.m3.1.1.1.1.1.cmml"><mn id="footnote18.m3.1.1.1.1.1.2" xref="footnote18.m3.1.1.1.1.1.2.cmml">1</mn><mo id="footnote18.m3.1.1.1.1.1.1" xref="footnote18.m3.1.1.1.1.1.1.cmml">/</mo><mi id="footnote18.m3.1.1.1.1.1.3" xref="footnote18.m3.1.1.1.1.1.3.cmml">k</mi></mrow><mo stretchy="false" id="footnote18.m3.1.1.1.1.3" xref="footnote18.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote18.m3.1c"><apply id="footnote18.m3.1.1.cmml" xref="footnote18.m3.1.1"><times id="footnote18.m3.1.1.2.cmml" xref="footnote18.m3.1.1.2"></times><ci id="footnote18.m3.1.1.3.cmml" xref="footnote18.m3.1.1.3">Θ</ci><apply id="footnote18.m3.1.1.1.1.1.cmml" xref="footnote18.m3.1.1.1.1"><divide id="footnote18.m3.1.1.1.1.1.1.cmml" xref="footnote18.m3.1.1.1.1.1.1"></divide><cn type="integer" id="footnote18.m3.1.1.1.1.1.2.cmml" xref="footnote18.m3.1.1.1.1.1.2">1</cn><ci id="footnote18.m3.1.1.1.1.1.3.cmml" xref="footnote18.m3.1.1.1.1.1.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote18.m3.1d">\Theta(1/k)</annotation></semantics></math>. This scheme has the advantage that is optimal at init in the infinite width limit. We did not try simpler schemes like scaling by <math id="footnote18.m4.1" class="ltx_Math" alttext="\Theta(1/\sqrt{k})" display="inline"><semantics id="footnote18.m4.1b"><mrow id="footnote18.m4.1.1" xref="footnote18.m4.1.1.cmml"><mi mathvariant="normal" id="footnote18.m4.1.1.3" xref="footnote18.m4.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="footnote18.m4.1.1.2" xref="footnote18.m4.1.1.2.cmml">​</mo><mrow id="footnote18.m4.1.1.1.1" xref="footnote18.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="footnote18.m4.1.1.1.1.2" xref="footnote18.m4.1.1.1.1.1.cmml">(</mo><mrow id="footnote18.m4.1.1.1.1.1" xref="footnote18.m4.1.1.1.1.1.cmml"><mn id="footnote18.m4.1.1.1.1.1.2" xref="footnote18.m4.1.1.1.1.1.2.cmml">1</mn><mo id="footnote18.m4.1.1.1.1.1.1" xref="footnote18.m4.1.1.1.1.1.1.cmml">/</mo><msqrt id="footnote18.m4.1.1.1.1.1.3" xref="footnote18.m4.1.1.1.1.1.3.cmml"><mi id="footnote18.m4.1.1.1.1.1.3.2" xref="footnote18.m4.1.1.1.1.1.3.2.cmml">k</mi></msqrt></mrow><mo stretchy="false" id="footnote18.m4.1.1.1.1.3" xref="footnote18.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote18.m4.1c"><apply id="footnote18.m4.1.1.cmml" xref="footnote18.m4.1.1"><times id="footnote18.m4.1.1.2.cmml" xref="footnote18.m4.1.1.2"></times><ci id="footnote18.m4.1.1.3.cmml" xref="footnote18.m4.1.1.3">Θ</ci><apply id="footnote18.m4.1.1.1.1.1.cmml" xref="footnote18.m4.1.1.1.1"><divide id="footnote18.m4.1.1.1.1.1.1.cmml" xref="footnote18.m4.1.1.1.1.1.1"></divide><cn type="integer" id="footnote18.m4.1.1.1.1.1.2.cmml" xref="footnote18.m4.1.1.1.1.1.2">1</cn><apply id="footnote18.m4.1.1.1.1.1.3.cmml" xref="footnote18.m4.1.1.1.1.1.3"><root id="footnote18.m4.1.1.1.1.1.3a.cmml" xref="footnote18.m4.1.1.1.1.1.3"></root><ci id="footnote18.m4.1.1.1.1.1.3.2.cmml" xref="footnote18.m4.1.1.1.1.1.3.2">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote18.m4.1d">\Theta(1/\sqrt{k})</annotation></semantics></math>.</span></span></span></p>
</div>
</li>
</ul>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Auxiliary loss</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.9" class="ltx_p">We define an auxiliary loss (AuxK) similar to “ghost grads” <cite class="ltx_cite ltx_citemacro_citep">[Jermyn and Templeton, <a href="#bib.bib24" title="" class="ltx_ref">2024</a>]</cite> that models the reconstruction error using the top-<math id="A1.SS2.p1.1.m1.1" class="ltx_Math" alttext="k_{\text{aux}}" display="inline"><semantics id="A1.SS2.p1.1.m1.1a"><msub id="A1.SS2.p1.1.m1.1.1" xref="A1.SS2.p1.1.m1.1.1.cmml"><mi id="A1.SS2.p1.1.m1.1.1.2" xref="A1.SS2.p1.1.m1.1.1.2.cmml">k</mi><mtext id="A1.SS2.p1.1.m1.1.1.3" xref="A1.SS2.p1.1.m1.1.1.3a.cmml">aux</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.1.m1.1b"><apply id="A1.SS2.p1.1.m1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2">𝑘</ci><ci id="A1.SS2.p1.1.m1.1.1.3a.cmml" xref="A1.SS2.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A1.SS2.p1.1.m1.1.1.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3">aux</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.1.m1.1c">k_{\text{aux}}</annotation></semantics></math> dead latents (typically <math id="A1.SS2.p1.2.m2.1" class="ltx_Math" alttext="k_{\text{aux}}=512" display="inline"><semantics id="A1.SS2.p1.2.m2.1a"><mrow id="A1.SS2.p1.2.m2.1.1" xref="A1.SS2.p1.2.m2.1.1.cmml"><msub id="A1.SS2.p1.2.m2.1.1.2" xref="A1.SS2.p1.2.m2.1.1.2.cmml"><mi id="A1.SS2.p1.2.m2.1.1.2.2" xref="A1.SS2.p1.2.m2.1.1.2.2.cmml">k</mi><mtext id="A1.SS2.p1.2.m2.1.1.2.3" xref="A1.SS2.p1.2.m2.1.1.2.3a.cmml">aux</mtext></msub><mo id="A1.SS2.p1.2.m2.1.1.1" xref="A1.SS2.p1.2.m2.1.1.1.cmml">=</mo><mn id="A1.SS2.p1.2.m2.1.1.3" xref="A1.SS2.p1.2.m2.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.2.m2.1b"><apply id="A1.SS2.p1.2.m2.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1"><eq id="A1.SS2.p1.2.m2.1.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1.1"></eq><apply id="A1.SS2.p1.2.m2.1.1.2.cmml" xref="A1.SS2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.2.m2.1.1.2.1.cmml" xref="A1.SS2.p1.2.m2.1.1.2">subscript</csymbol><ci id="A1.SS2.p1.2.m2.1.1.2.2.cmml" xref="A1.SS2.p1.2.m2.1.1.2.2">𝑘</ci><ci id="A1.SS2.p1.2.m2.1.1.2.3a.cmml" xref="A1.SS2.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="A1.SS2.p1.2.m2.1.1.2.3.cmml" xref="A1.SS2.p1.2.m2.1.1.2.3">aux</mtext></ci></apply><cn type="integer" id="A1.SS2.p1.2.m2.1.1.3.cmml" xref="A1.SS2.p1.2.m2.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.2.m2.1c">k_{\text{aux}}=512</annotation></semantics></math>). Latents are flagged as dead during training if they have not activated for some predetermined number of tokens (typically 10 million). Then, given the reconstruction error of the main model <math id="A1.SS2.p1.3.m3.1" class="ltx_Math" alttext="e=x-\hat{x}" display="inline"><semantics id="A1.SS2.p1.3.m3.1a"><mrow id="A1.SS2.p1.3.m3.1.1" xref="A1.SS2.p1.3.m3.1.1.cmml"><mi id="A1.SS2.p1.3.m3.1.1.2" xref="A1.SS2.p1.3.m3.1.1.2.cmml">e</mi><mo id="A1.SS2.p1.3.m3.1.1.1" xref="A1.SS2.p1.3.m3.1.1.1.cmml">=</mo><mrow id="A1.SS2.p1.3.m3.1.1.3" xref="A1.SS2.p1.3.m3.1.1.3.cmml"><mi id="A1.SS2.p1.3.m3.1.1.3.2" xref="A1.SS2.p1.3.m3.1.1.3.2.cmml">x</mi><mo id="A1.SS2.p1.3.m3.1.1.3.1" xref="A1.SS2.p1.3.m3.1.1.3.1.cmml">−</mo><mover accent="true" id="A1.SS2.p1.3.m3.1.1.3.3" xref="A1.SS2.p1.3.m3.1.1.3.3.cmml"><mi id="A1.SS2.p1.3.m3.1.1.3.3.2" xref="A1.SS2.p1.3.m3.1.1.3.3.2.cmml">x</mi><mo id="A1.SS2.p1.3.m3.1.1.3.3.1" xref="A1.SS2.p1.3.m3.1.1.3.3.1.cmml">^</mo></mover></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.3.m3.1b"><apply id="A1.SS2.p1.3.m3.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1"><eq id="A1.SS2.p1.3.m3.1.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1.1"></eq><ci id="A1.SS2.p1.3.m3.1.1.2.cmml" xref="A1.SS2.p1.3.m3.1.1.2">𝑒</ci><apply id="A1.SS2.p1.3.m3.1.1.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3"><minus id="A1.SS2.p1.3.m3.1.1.3.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.1"></minus><ci id="A1.SS2.p1.3.m3.1.1.3.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2">𝑥</ci><apply id="A1.SS2.p1.3.m3.1.1.3.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3"><ci id="A1.SS2.p1.3.m3.1.1.3.3.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1">^</ci><ci id="A1.SS2.p1.3.m3.1.1.3.3.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.3.m3.1c">e=x-\hat{x}</annotation></semantics></math>, we define the auxiliary loss <math id="A1.SS2.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{L}_{\text{aux}}=||e-\hat{e}||^{2}_{2}" display="inline"><semantics id="A1.SS2.p1.4.m4.1a"><mrow id="A1.SS2.p1.4.m4.1.1" xref="A1.SS2.p1.4.m4.1.1.cmml"><msub id="A1.SS2.p1.4.m4.1.1.3" xref="A1.SS2.p1.4.m4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.p1.4.m4.1.1.3.2" xref="A1.SS2.p1.4.m4.1.1.3.2.cmml">ℒ</mi><mtext id="A1.SS2.p1.4.m4.1.1.3.3" xref="A1.SS2.p1.4.m4.1.1.3.3a.cmml">aux</mtext></msub><mo id="A1.SS2.p1.4.m4.1.1.2" xref="A1.SS2.p1.4.m4.1.1.2.cmml">=</mo><msubsup id="A1.SS2.p1.4.m4.1.1.1" xref="A1.SS2.p1.4.m4.1.1.1.cmml"><mrow id="A1.SS2.p1.4.m4.1.1.1.1.1.1" xref="A1.SS2.p1.4.m4.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS2.p1.4.m4.1.1.1.1.1.1.2" xref="A1.SS2.p1.4.m4.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.cmml"><mi id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.2" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.2.cmml">e</mi><mo id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.1" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.cmml"><mi id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.2" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.2.cmml">e</mi><mo id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.1" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.1.cmml">^</mo></mover></mrow><mo stretchy="false" id="A1.SS2.p1.4.m4.1.1.1.1.1.1.3" xref="A1.SS2.p1.4.m4.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="A1.SS2.p1.4.m4.1.1.1.3" xref="A1.SS2.p1.4.m4.1.1.1.3.cmml">2</mn><mn id="A1.SS2.p1.4.m4.1.1.1.1.3" xref="A1.SS2.p1.4.m4.1.1.1.1.3.cmml">2</mn></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.4.m4.1b"><apply id="A1.SS2.p1.4.m4.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1"><eq id="A1.SS2.p1.4.m4.1.1.2.cmml" xref="A1.SS2.p1.4.m4.1.1.2"></eq><apply id="A1.SS2.p1.4.m4.1.1.3.cmml" xref="A1.SS2.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.1.1.3.1.cmml" xref="A1.SS2.p1.4.m4.1.1.3">subscript</csymbol><ci id="A1.SS2.p1.4.m4.1.1.3.2.cmml" xref="A1.SS2.p1.4.m4.1.1.3.2">ℒ</ci><ci id="A1.SS2.p1.4.m4.1.1.3.3a.cmml" xref="A1.SS2.p1.4.m4.1.1.3.3"><mtext mathsize="70%" id="A1.SS2.p1.4.m4.1.1.3.3.cmml" xref="A1.SS2.p1.4.m4.1.1.3.3">aux</mtext></ci></apply><apply id="A1.SS2.p1.4.m4.1.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.1.1.1.2.cmml" xref="A1.SS2.p1.4.m4.1.1.1">subscript</csymbol><apply id="A1.SS2.p1.4.m4.1.1.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.1.1.1.1.2.cmml" xref="A1.SS2.p1.4.m4.1.1.1">superscript</csymbol><apply id="A1.SS2.p1.4.m4.1.1.1.1.1.2.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS2.p1.4.m4.1.1.1.1.1.2.1.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.2">norm</csymbol><apply id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1"><minus id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.1"></minus><ci id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.2.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.2">𝑒</ci><apply id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3"><ci id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.1.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.1">^</ci><ci id="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.2.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.1.1.1.3.2">𝑒</ci></apply></apply></apply><cn type="integer" id="A1.SS2.p1.4.m4.1.1.1.1.3.cmml" xref="A1.SS2.p1.4.m4.1.1.1.1.3">2</cn></apply><cn type="integer" id="A1.SS2.p1.4.m4.1.1.1.3.cmml" xref="A1.SS2.p1.4.m4.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.4.m4.1c">\mathcal{L}_{\text{aux}}=||e-\hat{e}||^{2}_{2}</annotation></semantics></math>, where <math id="A1.SS2.p1.5.m5.1" class="ltx_Math" alttext="\hat{e}=W_{\text{dec}}z" display="inline"><semantics id="A1.SS2.p1.5.m5.1a"><mrow id="A1.SS2.p1.5.m5.1.1" xref="A1.SS2.p1.5.m5.1.1.cmml"><mover accent="true" id="A1.SS2.p1.5.m5.1.1.2" xref="A1.SS2.p1.5.m5.1.1.2.cmml"><mi id="A1.SS2.p1.5.m5.1.1.2.2" xref="A1.SS2.p1.5.m5.1.1.2.2.cmml">e</mi><mo id="A1.SS2.p1.5.m5.1.1.2.1" xref="A1.SS2.p1.5.m5.1.1.2.1.cmml">^</mo></mover><mo id="A1.SS2.p1.5.m5.1.1.1" xref="A1.SS2.p1.5.m5.1.1.1.cmml">=</mo><mrow id="A1.SS2.p1.5.m5.1.1.3" xref="A1.SS2.p1.5.m5.1.1.3.cmml"><msub id="A1.SS2.p1.5.m5.1.1.3.2" xref="A1.SS2.p1.5.m5.1.1.3.2.cmml"><mi id="A1.SS2.p1.5.m5.1.1.3.2.2" xref="A1.SS2.p1.5.m5.1.1.3.2.2.cmml">W</mi><mtext id="A1.SS2.p1.5.m5.1.1.3.2.3" xref="A1.SS2.p1.5.m5.1.1.3.2.3a.cmml">dec</mtext></msub><mo lspace="0em" rspace="0em" id="A1.SS2.p1.5.m5.1.1.3.1" xref="A1.SS2.p1.5.m5.1.1.3.1.cmml">​</mo><mi id="A1.SS2.p1.5.m5.1.1.3.3" xref="A1.SS2.p1.5.m5.1.1.3.3.cmml">z</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.5.m5.1b"><apply id="A1.SS2.p1.5.m5.1.1.cmml" xref="A1.SS2.p1.5.m5.1.1"><eq id="A1.SS2.p1.5.m5.1.1.1.cmml" xref="A1.SS2.p1.5.m5.1.1.1"></eq><apply id="A1.SS2.p1.5.m5.1.1.2.cmml" xref="A1.SS2.p1.5.m5.1.1.2"><ci id="A1.SS2.p1.5.m5.1.1.2.1.cmml" xref="A1.SS2.p1.5.m5.1.1.2.1">^</ci><ci id="A1.SS2.p1.5.m5.1.1.2.2.cmml" xref="A1.SS2.p1.5.m5.1.1.2.2">𝑒</ci></apply><apply id="A1.SS2.p1.5.m5.1.1.3.cmml" xref="A1.SS2.p1.5.m5.1.1.3"><times id="A1.SS2.p1.5.m5.1.1.3.1.cmml" xref="A1.SS2.p1.5.m5.1.1.3.1"></times><apply id="A1.SS2.p1.5.m5.1.1.3.2.cmml" xref="A1.SS2.p1.5.m5.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.p1.5.m5.1.1.3.2.1.cmml" xref="A1.SS2.p1.5.m5.1.1.3.2">subscript</csymbol><ci id="A1.SS2.p1.5.m5.1.1.3.2.2.cmml" xref="A1.SS2.p1.5.m5.1.1.3.2.2">𝑊</ci><ci id="A1.SS2.p1.5.m5.1.1.3.2.3a.cmml" xref="A1.SS2.p1.5.m5.1.1.3.2.3"><mtext mathsize="70%" id="A1.SS2.p1.5.m5.1.1.3.2.3.cmml" xref="A1.SS2.p1.5.m5.1.1.3.2.3">dec</mtext></ci></apply><ci id="A1.SS2.p1.5.m5.1.1.3.3.cmml" xref="A1.SS2.p1.5.m5.1.1.3.3">𝑧</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.5.m5.1c">\hat{e}=W_{\text{dec}}z</annotation></semantics></math> is the reconstruction using the top-<math id="A1.SS2.p1.6.m6.1" class="ltx_Math" alttext="k_{\text{aux}}" display="inline"><semantics id="A1.SS2.p1.6.m6.1a"><msub id="A1.SS2.p1.6.m6.1.1" xref="A1.SS2.p1.6.m6.1.1.cmml"><mi id="A1.SS2.p1.6.m6.1.1.2" xref="A1.SS2.p1.6.m6.1.1.2.cmml">k</mi><mtext id="A1.SS2.p1.6.m6.1.1.3" xref="A1.SS2.p1.6.m6.1.1.3a.cmml">aux</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.6.m6.1b"><apply id="A1.SS2.p1.6.m6.1.1.cmml" xref="A1.SS2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.6.m6.1.1.1.cmml" xref="A1.SS2.p1.6.m6.1.1">subscript</csymbol><ci id="A1.SS2.p1.6.m6.1.1.2.cmml" xref="A1.SS2.p1.6.m6.1.1.2">𝑘</ci><ci id="A1.SS2.p1.6.m6.1.1.3a.cmml" xref="A1.SS2.p1.6.m6.1.1.3"><mtext mathsize="70%" id="A1.SS2.p1.6.m6.1.1.3.cmml" xref="A1.SS2.p1.6.m6.1.1.3">aux</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.6.m6.1c">k_{\text{aux}}</annotation></semantics></math> dead latents.
The full loss is then defined as <math id="A1.SS2.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{L}+\alpha\mathcal{L}_{\text{aux}}" display="inline"><semantics id="A1.SS2.p1.7.m7.1a"><mrow id="A1.SS2.p1.7.m7.1.1" xref="A1.SS2.p1.7.m7.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.p1.7.m7.1.1.2" xref="A1.SS2.p1.7.m7.1.1.2.cmml">ℒ</mi><mo id="A1.SS2.p1.7.m7.1.1.1" xref="A1.SS2.p1.7.m7.1.1.1.cmml">+</mo><mrow id="A1.SS2.p1.7.m7.1.1.3" xref="A1.SS2.p1.7.m7.1.1.3.cmml"><mi id="A1.SS2.p1.7.m7.1.1.3.2" xref="A1.SS2.p1.7.m7.1.1.3.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.SS2.p1.7.m7.1.1.3.1" xref="A1.SS2.p1.7.m7.1.1.3.1.cmml">​</mo><msub id="A1.SS2.p1.7.m7.1.1.3.3" xref="A1.SS2.p1.7.m7.1.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.p1.7.m7.1.1.3.3.2" xref="A1.SS2.p1.7.m7.1.1.3.3.2.cmml">ℒ</mi><mtext id="A1.SS2.p1.7.m7.1.1.3.3.3" xref="A1.SS2.p1.7.m7.1.1.3.3.3a.cmml">aux</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.7.m7.1b"><apply id="A1.SS2.p1.7.m7.1.1.cmml" xref="A1.SS2.p1.7.m7.1.1"><plus id="A1.SS2.p1.7.m7.1.1.1.cmml" xref="A1.SS2.p1.7.m7.1.1.1"></plus><ci id="A1.SS2.p1.7.m7.1.1.2.cmml" xref="A1.SS2.p1.7.m7.1.1.2">ℒ</ci><apply id="A1.SS2.p1.7.m7.1.1.3.cmml" xref="A1.SS2.p1.7.m7.1.1.3"><times id="A1.SS2.p1.7.m7.1.1.3.1.cmml" xref="A1.SS2.p1.7.m7.1.1.3.1"></times><ci id="A1.SS2.p1.7.m7.1.1.3.2.cmml" xref="A1.SS2.p1.7.m7.1.1.3.2">𝛼</ci><apply id="A1.SS2.p1.7.m7.1.1.3.3.cmml" xref="A1.SS2.p1.7.m7.1.1.3.3"><csymbol cd="ambiguous" id="A1.SS2.p1.7.m7.1.1.3.3.1.cmml" xref="A1.SS2.p1.7.m7.1.1.3.3">subscript</csymbol><ci id="A1.SS2.p1.7.m7.1.1.3.3.2.cmml" xref="A1.SS2.p1.7.m7.1.1.3.3.2">ℒ</ci><ci id="A1.SS2.p1.7.m7.1.1.3.3.3a.cmml" xref="A1.SS2.p1.7.m7.1.1.3.3.3"><mtext mathsize="70%" id="A1.SS2.p1.7.m7.1.1.3.3.3.cmml" xref="A1.SS2.p1.7.m7.1.1.3.3.3">aux</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.7.m7.1c">\mathcal{L}+\alpha\mathcal{L}_{\text{aux}}</annotation></semantics></math>, where <math id="A1.SS2.p1.8.m8.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="A1.SS2.p1.8.m8.1a"><mi id="A1.SS2.p1.8.m8.1.1" xref="A1.SS2.p1.8.m8.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.8.m8.1b"><ci id="A1.SS2.p1.8.m8.1.1.cmml" xref="A1.SS2.p1.8.m8.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.8.m8.1c">\alpha</annotation></semantics></math> is a small coefficient (typically <math id="A1.SS2.p1.9.m9.1" class="ltx_Math" alttext="1/32" display="inline"><semantics id="A1.SS2.p1.9.m9.1a"><mrow id="A1.SS2.p1.9.m9.1.1" xref="A1.SS2.p1.9.m9.1.1.cmml"><mn id="A1.SS2.p1.9.m9.1.1.2" xref="A1.SS2.p1.9.m9.1.1.2.cmml">1</mn><mo id="A1.SS2.p1.9.m9.1.1.1" xref="A1.SS2.p1.9.m9.1.1.1.cmml">/</mo><mn id="A1.SS2.p1.9.m9.1.1.3" xref="A1.SS2.p1.9.m9.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.9.m9.1b"><apply id="A1.SS2.p1.9.m9.1.1.cmml" xref="A1.SS2.p1.9.m9.1.1"><divide id="A1.SS2.p1.9.m9.1.1.1.cmml" xref="A1.SS2.p1.9.m9.1.1.1"></divide><cn type="integer" id="A1.SS2.p1.9.m9.1.1.2.cmml" xref="A1.SS2.p1.9.m9.1.1.2">1</cn><cn type="integer" id="A1.SS2.p1.9.m9.1.1.3.cmml" xref="A1.SS2.p1.9.m9.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.9.m9.1c">1/32</annotation></semantics></math>).
Because the encoder forward pass can be shared (and dominates decoder cost and encoder backwards cost, see <a href="#A4" title="Appendix D Systems ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix D</span></a>), adding this auxiliary loss only increases the computational cost by about 10%.</p>
</div>
<div id="A1.SS2.p2" class="ltx_para">
<p id="A1.SS2.p2.1" class="ltx_p">We found that the AuxK loss very occasionally NaNs at large scale, and zero it when it is NaN to prevent training run collapse.</p>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Optimizer</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.4" class="ltx_p">We use the Adam optimizer <cite class="ltx_cite ltx_citemacro_citep">[Kingma and Ba, <a href="#bib.bib26" title="" class="ltx_ref">2014</a>]</cite> with <math id="A1.SS3.p1.1.m1.1" class="ltx_Math" alttext="\beta_{1}=0.9" display="inline"><semantics id="A1.SS3.p1.1.m1.1a"><mrow id="A1.SS3.p1.1.m1.1.1" xref="A1.SS3.p1.1.m1.1.1.cmml"><msub id="A1.SS3.p1.1.m1.1.1.2" xref="A1.SS3.p1.1.m1.1.1.2.cmml"><mi id="A1.SS3.p1.1.m1.1.1.2.2" xref="A1.SS3.p1.1.m1.1.1.2.2.cmml">β</mi><mn id="A1.SS3.p1.1.m1.1.1.2.3" xref="A1.SS3.p1.1.m1.1.1.2.3.cmml">1</mn></msub><mo id="A1.SS3.p1.1.m1.1.1.1" xref="A1.SS3.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.SS3.p1.1.m1.1.1.3" xref="A1.SS3.p1.1.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.1.m1.1b"><apply id="A1.SS3.p1.1.m1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1"><eq id="A1.SS3.p1.1.m1.1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1.1"></eq><apply id="A1.SS3.p1.1.m1.1.1.2.cmml" xref="A1.SS3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS3.p1.1.m1.1.1.2.1.cmml" xref="A1.SS3.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.SS3.p1.1.m1.1.1.2.2.cmml" xref="A1.SS3.p1.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="A1.SS3.p1.1.m1.1.1.2.3.cmml" xref="A1.SS3.p1.1.m1.1.1.2.3">1</cn></apply><cn type="float" id="A1.SS3.p1.1.m1.1.1.3.cmml" xref="A1.SS3.p1.1.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.1.m1.1c">\beta_{1}=0.9</annotation></semantics></math> and <math id="A1.SS3.p1.2.m2.1" class="ltx_Math" alttext="\beta_{2}=0.999" display="inline"><semantics id="A1.SS3.p1.2.m2.1a"><mrow id="A1.SS3.p1.2.m2.1.1" xref="A1.SS3.p1.2.m2.1.1.cmml"><msub id="A1.SS3.p1.2.m2.1.1.2" xref="A1.SS3.p1.2.m2.1.1.2.cmml"><mi id="A1.SS3.p1.2.m2.1.1.2.2" xref="A1.SS3.p1.2.m2.1.1.2.2.cmml">β</mi><mn id="A1.SS3.p1.2.m2.1.1.2.3" xref="A1.SS3.p1.2.m2.1.1.2.3.cmml">2</mn></msub><mo id="A1.SS3.p1.2.m2.1.1.1" xref="A1.SS3.p1.2.m2.1.1.1.cmml">=</mo><mn id="A1.SS3.p1.2.m2.1.1.3" xref="A1.SS3.p1.2.m2.1.1.3.cmml">0.999</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.2.m2.1b"><apply id="A1.SS3.p1.2.m2.1.1.cmml" xref="A1.SS3.p1.2.m2.1.1"><eq id="A1.SS3.p1.2.m2.1.1.1.cmml" xref="A1.SS3.p1.2.m2.1.1.1"></eq><apply id="A1.SS3.p1.2.m2.1.1.2.cmml" xref="A1.SS3.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A1.SS3.p1.2.m2.1.1.2.1.cmml" xref="A1.SS3.p1.2.m2.1.1.2">subscript</csymbol><ci id="A1.SS3.p1.2.m2.1.1.2.2.cmml" xref="A1.SS3.p1.2.m2.1.1.2.2">𝛽</ci><cn type="integer" id="A1.SS3.p1.2.m2.1.1.2.3.cmml" xref="A1.SS3.p1.2.m2.1.1.2.3">2</cn></apply><cn type="float" id="A1.SS3.p1.2.m2.1.1.3.cmml" xref="A1.SS3.p1.2.m2.1.1.3">0.999</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.2.m2.1c">\beta_{2}=0.999</annotation></semantics></math>, and a constant learning rate. We tried several learning rate decay schedules but did not find consistent improvements in token budget to convergence. We also did not find major benefits from tuning <math id="A1.SS3.p1.3.m3.1" class="ltx_Math" alttext="\beta_{1}" display="inline"><semantics id="A1.SS3.p1.3.m3.1a"><msub id="A1.SS3.p1.3.m3.1.1" xref="A1.SS3.p1.3.m3.1.1.cmml"><mi id="A1.SS3.p1.3.m3.1.1.2" xref="A1.SS3.p1.3.m3.1.1.2.cmml">β</mi><mn id="A1.SS3.p1.3.m3.1.1.3" xref="A1.SS3.p1.3.m3.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.3.m3.1b"><apply id="A1.SS3.p1.3.m3.1.1.cmml" xref="A1.SS3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.3.m3.1.1.1.cmml" xref="A1.SS3.p1.3.m3.1.1">subscript</csymbol><ci id="A1.SS3.p1.3.m3.1.1.2.cmml" xref="A1.SS3.p1.3.m3.1.1.2">𝛽</ci><cn type="integer" id="A1.SS3.p1.3.m3.1.1.3.cmml" xref="A1.SS3.p1.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.3.m3.1c">\beta_{1}</annotation></semantics></math> and <math id="A1.SS3.p1.4.m4.1" class="ltx_Math" alttext="\beta_{2}" display="inline"><semantics id="A1.SS3.p1.4.m4.1a"><msub id="A1.SS3.p1.4.m4.1.1" xref="A1.SS3.p1.4.m4.1.1.cmml"><mi id="A1.SS3.p1.4.m4.1.1.2" xref="A1.SS3.p1.4.m4.1.1.2.cmml">β</mi><mn id="A1.SS3.p1.4.m4.1.1.3" xref="A1.SS3.p1.4.m4.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.4.m4.1b"><apply id="A1.SS3.p1.4.m4.1.1.cmml" xref="A1.SS3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.4.m4.1.1.1.cmml" xref="A1.SS3.p1.4.m4.1.1">subscript</csymbol><ci id="A1.SS3.p1.4.m4.1.1.2.cmml" xref="A1.SS3.p1.4.m4.1.1.2">𝛽</ci><cn type="integer" id="A1.SS3.p1.4.m4.1.1.3.cmml" xref="A1.SS3.p1.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.4.m4.1c">\beta_{2}</annotation></semantics></math>.</p>
</div>
<div id="A1.SS3.p2" class="ltx_para">
<p id="A1.SS3.p2.1" class="ltx_p">We project away gradient information parallel to the decoder vectors, to account for interaction between Adam and decoder normalization, as described in <cite class="ltx_cite ltx_citemacro_citet">Bricken et al. [<a href="#bib.bib8" title="" class="ltx_ref">2023</a>]</cite>.</p>
</div>
<section id="A1.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">A.3.1 </span>Adam epsilon</h4>

<div id="A1.SS3.SSS1.p1" class="ltx_para">
<p id="A1.SS3.SSS1.p1.2" class="ltx_p">By convention, we average the gradient across the batch dimension. As a result, the root mean square (RMS) of the gradient can often be very small, causing Adam to no longer be loss scale invariant.We find that by setting epsilon sufficiently small, these issues are prevented, and that <math id="A1.SS3.SSS1.p1.1.m1.1" class="ltx_Math" alttext="\varepsilon" display="inline"><semantics id="A1.SS3.SSS1.p1.1.m1.1a"><mi id="A1.SS3.SSS1.p1.1.m1.1.1" xref="A1.SS3.SSS1.p1.1.m1.1.1.cmml">ε</mi><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS1.p1.1.m1.1b"><ci id="A1.SS3.SSS1.p1.1.m1.1.1.cmml" xref="A1.SS3.SSS1.p1.1.m1.1.1">𝜀</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS1.p1.1.m1.1c">\varepsilon</annotation></semantics></math> is otherwise not very sensitive and does not result in significant benefit to tune further. We use <math id="A1.SS3.SSS1.p1.2.m2.1" class="ltx_Math" alttext="\varepsilon=6.25\times 10^{-10}" display="inline"><semantics id="A1.SS3.SSS1.p1.2.m2.1a"><mrow id="A1.SS3.SSS1.p1.2.m2.1.1" xref="A1.SS3.SSS1.p1.2.m2.1.1.cmml"><mi id="A1.SS3.SSS1.p1.2.m2.1.1.2" xref="A1.SS3.SSS1.p1.2.m2.1.1.2.cmml">ε</mi><mo id="A1.SS3.SSS1.p1.2.m2.1.1.1" xref="A1.SS3.SSS1.p1.2.m2.1.1.1.cmml">=</mo><mrow id="A1.SS3.SSS1.p1.2.m2.1.1.3" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.cmml"><mn id="A1.SS3.SSS1.p1.2.m2.1.1.3.2" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.2.cmml">6.25</mn><mo lspace="0.222em" rspace="0.222em" id="A1.SS3.SSS1.p1.2.m2.1.1.3.1" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.1.cmml">×</mo><msup id="A1.SS3.SSS1.p1.2.m2.1.1.3.3" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.cmml"><mn id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.2" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.2.cmml">10</mn><mrow id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.cmml"><mo id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3a" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.cmml">−</mo><mn id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.2" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.2.cmml">10</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS1.p1.2.m2.1b"><apply id="A1.SS3.SSS1.p1.2.m2.1.1.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1"><eq id="A1.SS3.SSS1.p1.2.m2.1.1.1.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.1"></eq><ci id="A1.SS3.SSS1.p1.2.m2.1.1.2.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.2">𝜀</ci><apply id="A1.SS3.SSS1.p1.2.m2.1.1.3.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3"><times id="A1.SS3.SSS1.p1.2.m2.1.1.3.1.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.1"></times><cn type="float" id="A1.SS3.SSS1.p1.2.m2.1.1.3.2.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.2">6.25</cn><apply id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3"><csymbol cd="ambiguous" id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.1.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3">superscript</csymbol><cn type="integer" id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.2.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.2">10</cn><apply id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3"><minus id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.1.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3"></minus><cn type="integer" id="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.2.cmml" xref="A1.SS3.SSS1.p1.2.m2.1.1.3.3.3.2">10</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS1.p1.2.m2.1c">\varepsilon=6.25\times 10^{-10}</annotation></semantics></math> in many experiments in this paper, though we reduced it further for some of the largest runs to be safe.</p>
</div>
</section>
<section id="A1.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">A.3.2 </span>Gradient clipping</h4>

<div id="A1.SS3.SSS2.p1" class="ltx_para">
<p id="A1.SS3.SSS2.p1.2" class="ltx_p">When scaling the GPT-4 autoencoders, we found that gradient clipping was necessary to prevent instability and divergence at higher learning rates. We found that gradient clipping substantially affected <math id="A1.SS3.SSS2.p1.1.m1.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="A1.SS3.SSS2.p1.1.m1.1a"><mrow id="A1.SS3.SSS2.p1.1.m1.1.2" xref="A1.SS3.SSS2.p1.1.m1.1.2.cmml"><mi id="A1.SS3.SSS2.p1.1.m1.1.2.2" xref="A1.SS3.SSS2.p1.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.SS3.SSS2.p1.1.m1.1.2.1" xref="A1.SS3.SSS2.p1.1.m1.1.2.1.cmml">​</mo><mrow id="A1.SS3.SSS2.p1.1.m1.1.2.3.2" xref="A1.SS3.SSS2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="A1.SS3.SSS2.p1.1.m1.1.2.3.2.1" xref="A1.SS3.SSS2.p1.1.m1.1.2.cmml">(</mo><mi id="A1.SS3.SSS2.p1.1.m1.1.1" xref="A1.SS3.SSS2.p1.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="A1.SS3.SSS2.p1.1.m1.1.2.3.2.2" xref="A1.SS3.SSS2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS2.p1.1.m1.1b"><apply id="A1.SS3.SSS2.p1.1.m1.1.2.cmml" xref="A1.SS3.SSS2.p1.1.m1.1.2"><times id="A1.SS3.SSS2.p1.1.m1.1.2.1.cmml" xref="A1.SS3.SSS2.p1.1.m1.1.2.1"></times><ci id="A1.SS3.SSS2.p1.1.m1.1.2.2.cmml" xref="A1.SS3.SSS2.p1.1.m1.1.2.2">𝐿</ci><ci id="A1.SS3.SSS2.p1.1.m1.1.1.cmml" xref="A1.SS3.SSS2.p1.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS2.p1.1.m1.1c">L(C)</annotation></semantics></math> but not <math id="A1.SS3.SSS2.p1.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A1.SS3.SSS2.p1.2.m2.1a"><mrow id="A1.SS3.SSS2.p1.2.m2.1.2" xref="A1.SS3.SSS2.p1.2.m2.1.2.cmml"><mi id="A1.SS3.SSS2.p1.2.m2.1.2.2" xref="A1.SS3.SSS2.p1.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.SS3.SSS2.p1.2.m2.1.2.1" xref="A1.SS3.SSS2.p1.2.m2.1.2.1.cmml">​</mo><mrow id="A1.SS3.SSS2.p1.2.m2.1.2.3.2" xref="A1.SS3.SSS2.p1.2.m2.1.2.cmml"><mo stretchy="false" id="A1.SS3.SSS2.p1.2.m2.1.2.3.2.1" xref="A1.SS3.SSS2.p1.2.m2.1.2.cmml">(</mo><mi id="A1.SS3.SSS2.p1.2.m2.1.1" xref="A1.SS3.SSS2.p1.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="A1.SS3.SSS2.p1.2.m2.1.2.3.2.2" xref="A1.SS3.SSS2.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS2.p1.2.m2.1b"><apply id="A1.SS3.SSS2.p1.2.m2.1.2.cmml" xref="A1.SS3.SSS2.p1.2.m2.1.2"><times id="A1.SS3.SSS2.p1.2.m2.1.2.1.cmml" xref="A1.SS3.SSS2.p1.2.m2.1.2.1"></times><ci id="A1.SS3.SSS2.p1.2.m2.1.2.2.cmml" xref="A1.SS3.SSS2.p1.2.m2.1.2.2">𝐿</ci><ci id="A1.SS3.SSS2.p1.2.m2.1.1.cmml" xref="A1.SS3.SSS2.p1.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS2.p1.2.m2.1c">L(N)</annotation></semantics></math>. We did not use gradient clipping for the GPT-2 small runs.</p>
</div>
</section>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Batch size</h3>

<div id="A1.SS4.p1" class="ltx_para">
<p id="A1.SS4.p1.1" class="ltx_p">Larger batch sizes are critical for allowing much greater parallelism. Prior work tends to use batch sizes like 2048 or 4096 tokens <cite class="ltx_cite ltx_citemacro_citep">[Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>, Conerly et al., <a href="#bib.bib11" title="" class="ltx_ref">2024</a>, Rajamanoharan et al., <a href="#bib.bib52" title="" class="ltx_ref">2024</a>]</cite>. To gain the benefits of parallelism, we use a batch size of 131,072 tokens for most of our experiments.</p>
</div>
<div id="A1.SS4.p2" class="ltx_para">
<p id="A1.SS4.p2.2" class="ltx_p">While batch size affects <math id="A1.SS4.p2.1.m1.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="A1.SS4.p2.1.m1.1a"><mrow id="A1.SS4.p2.1.m1.1.2" xref="A1.SS4.p2.1.m1.1.2.cmml"><mi id="A1.SS4.p2.1.m1.1.2.2" xref="A1.SS4.p2.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.SS4.p2.1.m1.1.2.1" xref="A1.SS4.p2.1.m1.1.2.1.cmml">​</mo><mrow id="A1.SS4.p2.1.m1.1.2.3.2" xref="A1.SS4.p2.1.m1.1.2.cmml"><mo stretchy="false" id="A1.SS4.p2.1.m1.1.2.3.2.1" xref="A1.SS4.p2.1.m1.1.2.cmml">(</mo><mi id="A1.SS4.p2.1.m1.1.1" xref="A1.SS4.p2.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="A1.SS4.p2.1.m1.1.2.3.2.2" xref="A1.SS4.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.1.m1.1b"><apply id="A1.SS4.p2.1.m1.1.2.cmml" xref="A1.SS4.p2.1.m1.1.2"><times id="A1.SS4.p2.1.m1.1.2.1.cmml" xref="A1.SS4.p2.1.m1.1.2.1"></times><ci id="A1.SS4.p2.1.m1.1.2.2.cmml" xref="A1.SS4.p2.1.m1.1.2.2">𝐿</ci><ci id="A1.SS4.p2.1.m1.1.1.cmml" xref="A1.SS4.p2.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.1.m1.1c">L(C)</annotation></semantics></math> substantially, we find that the <math id="A1.SS4.p2.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A1.SS4.p2.2.m2.1a"><mrow id="A1.SS4.p2.2.m2.1.2" xref="A1.SS4.p2.2.m2.1.2.cmml"><mi id="A1.SS4.p2.2.m2.1.2.2" xref="A1.SS4.p2.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.SS4.p2.2.m2.1.2.1" xref="A1.SS4.p2.2.m2.1.2.1.cmml">​</mo><mrow id="A1.SS4.p2.2.m2.1.2.3.2" xref="A1.SS4.p2.2.m2.1.2.cmml"><mo stretchy="false" id="A1.SS4.p2.2.m2.1.2.3.2.1" xref="A1.SS4.p2.2.m2.1.2.cmml">(</mo><mi id="A1.SS4.p2.2.m2.1.1" xref="A1.SS4.p2.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="A1.SS4.p2.2.m2.1.2.3.2.2" xref="A1.SS4.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.2.m2.1b"><apply id="A1.SS4.p2.2.m2.1.2.cmml" xref="A1.SS4.p2.2.m2.1.2"><times id="A1.SS4.p2.2.m2.1.2.1.cmml" xref="A1.SS4.p2.2.m2.1.2.1"></times><ci id="A1.SS4.p2.2.m2.1.2.2.cmml" xref="A1.SS4.p2.2.m2.1.2.2">𝐿</ci><ci id="A1.SS4.p2.2.m2.1.1.cmml" xref="A1.SS4.p2.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.2.m2.1c">L(N)</annotation></semantics></math> loss does not depend strongly on batch size when optimization hyperparameters are set appropriately (<a href="#A1.F12" title="Figure 12 ‣ A.4 Batch size ‣ Appendix A Optimization ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 12</span></a>).</p>
</div>
<figure id="A1.F12" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_bs.png" id="A1.F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="196" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A1.F12.4.2.1" class="ltx_text" style="font-size:90%;">Figure 12</span>: </span><span id="A1.F12.2.1" class="ltx_text" style="font-size:90%;">With correct hyperparameter settings, different batch sizes converge to the same <math id="A1.F12.2.1.m1.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A1.F12.2.1.m1.1b"><mrow id="A1.F12.2.1.m1.1.2" xref="A1.F12.2.1.m1.1.2.cmml"><mi id="A1.F12.2.1.m1.1.2.2" xref="A1.F12.2.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.F12.2.1.m1.1.2.1" xref="A1.F12.2.1.m1.1.2.1.cmml">​</mo><mrow id="A1.F12.2.1.m1.1.2.3.2" xref="A1.F12.2.1.m1.1.2.cmml"><mo stretchy="false" id="A1.F12.2.1.m1.1.2.3.2.1" xref="A1.F12.2.1.m1.1.2.cmml">(</mo><mi id="A1.F12.2.1.m1.1.1" xref="A1.F12.2.1.m1.1.1.cmml">N</mi><mo stretchy="false" id="A1.F12.2.1.m1.1.2.3.2.2" xref="A1.F12.2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.F12.2.1.m1.1c"><apply id="A1.F12.2.1.m1.1.2.cmml" xref="A1.F12.2.1.m1.1.2"><times id="A1.F12.2.1.m1.1.2.1.cmml" xref="A1.F12.2.1.m1.1.2.1"></times><ci id="A1.F12.2.1.m1.1.2.2.cmml" xref="A1.F12.2.1.m1.1.2.2">𝐿</ci><ci id="A1.F12.2.1.m1.1.1.cmml" xref="A1.F12.2.1.m1.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F12.2.1.m1.1d">L(N)</annotation></semantics></math> loss (gpt2small).</span></figcaption>
</figure>
</section>
<section id="A1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.5 </span>Weight averaging</h3>

<div id="A1.SS5.p1" class="ltx_para">
<p id="A1.SS5.p1.1" class="ltx_p">We find that keeping an exponential moving average (EMA) <cite class="ltx_cite ltx_citemacro_cite">Ruppert [<a href="#bib.bib54" title="" class="ltx_ref">1988</a>]</cite> of the weights slightly reduces sensitivity to learning rate by allowing slightly higher learning rates to be tolerated. Due to its low cost, we use EMA in all experiments. We use an EMA coefficient of 0.999, and did not find a substantial benefit to tuning it.</p>
</div>
<div id="A1.SS5.p2" class="ltx_para">
<p id="A1.SS5.p2.1" class="ltx_p">We use a bias-correction similar to that used in <cite class="ltx_cite ltx_citemacro_citet">Kingma and Ba [<a href="#bib.bib26" title="" class="ltx_ref">2014</a>]</cite>. Despite this, the early steps of EMA are still generally worse than the original model. Thus for the <math id="A1.SS5.p2.1.m1.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="A1.SS5.p2.1.m1.1a"><mrow id="A1.SS5.p2.1.m1.1.2" xref="A1.SS5.p2.1.m1.1.2.cmml"><mi id="A1.SS5.p2.1.m1.1.2.2" xref="A1.SS5.p2.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.SS5.p2.1.m1.1.2.1" xref="A1.SS5.p2.1.m1.1.2.1.cmml">​</mo><mrow id="A1.SS5.p2.1.m1.1.2.3.2" xref="A1.SS5.p2.1.m1.1.2.cmml"><mo stretchy="false" id="A1.SS5.p2.1.m1.1.2.3.2.1" xref="A1.SS5.p2.1.m1.1.2.cmml">(</mo><mi id="A1.SS5.p2.1.m1.1.1" xref="A1.SS5.p2.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="A1.SS5.p2.1.m1.1.2.3.2.2" xref="A1.SS5.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS5.p2.1.m1.1b"><apply id="A1.SS5.p2.1.m1.1.2.cmml" xref="A1.SS5.p2.1.m1.1.2"><times id="A1.SS5.p2.1.m1.1.2.1.cmml" xref="A1.SS5.p2.1.m1.1.2.1"></times><ci id="A1.SS5.p2.1.m1.1.2.2.cmml" xref="A1.SS5.p2.1.m1.1.2.2">𝐿</ci><ci id="A1.SS5.p2.1.m1.1.1.cmml" xref="A1.SS5.p2.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS5.p2.1.m1.1c">L(C)</annotation></semantics></math> experiments, we take the min of the EMA model’s and non-averaged model’s validation losses.</p>
</div>
</section>
<section id="A1.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.6 </span>Other details</h3>

<div id="A1.SS6.p1" class="ltx_para">
<ul id="A1.I2" class="ltx_itemize">
<li id="A1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i1.p1" class="ltx_para">
<p id="A1.I2.i1.p1.1" class="ltx_p">For the main MSE loss, we compute an MSE normalization constant once at the beginning of training, and do not do any loss normalization per batch.</p>
</div>
</li>
<li id="A1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i2.p1" class="ltx_para">
<p id="A1.I2.i2.p1.1" class="ltx_p">For the AuxK MSE loss, we compute the normalization per token, because the scale of the error changes throughout training.</p>
</div>
</li>
<li id="A1.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i3.p1" class="ltx_para">
<p id="A1.I2.i3.p1.1" class="ltx_p">In theory, the <math id="A1.I2.i3.p1.1.m1.1" class="ltx_Math" alttext="b_{\text{pre}}" display="inline"><semantics id="A1.I2.i3.p1.1.m1.1a"><msub id="A1.I2.i3.p1.1.m1.1.1" xref="A1.I2.i3.p1.1.m1.1.1.cmml"><mi id="A1.I2.i3.p1.1.m1.1.1.2" xref="A1.I2.i3.p1.1.m1.1.1.2.cmml">b</mi><mtext id="A1.I2.i3.p1.1.m1.1.1.3" xref="A1.I2.i3.p1.1.m1.1.1.3a.cmml">pre</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.I2.i3.p1.1.m1.1b"><apply id="A1.I2.i3.p1.1.m1.1.1.cmml" xref="A1.I2.i3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.I2.i3.p1.1.m1.1.1.1.cmml" xref="A1.I2.i3.p1.1.m1.1.1">subscript</csymbol><ci id="A1.I2.i3.p1.1.m1.1.1.2.cmml" xref="A1.I2.i3.p1.1.m1.1.1.2">𝑏</ci><ci id="A1.I2.i3.p1.1.m1.1.1.3a.cmml" xref="A1.I2.i3.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A1.I2.i3.p1.1.m1.1.1.3.cmml" xref="A1.I2.i3.p1.1.m1.1.1.3">pre</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i3.p1.1.m1.1c">b_{\text{pre}}</annotation></semantics></math> lr should be scaled linearly with the norm of the data to make the autoencoder completely invariant to input scale. In practice, we find it to tolerate an extremely wide range of values with little impact on quality.</p>
</div>
</li>
<li id="A1.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i4.p1" class="ltx_para">
<p id="A1.I2.i4.p1.1" class="ltx_p">Anecdotally, we noticed that when decaying the learning rate of an autoencoder previously training at the L(C) loss, the number of dead latents would decrease.</p>
</div>
</li>
</ul>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Other training details</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">Unless otherwise noted, autoencoders were trained on the residual activation directly after the layernorm (with layernorm weights folded into the attention weights), since this corresponds to how residual stream activations are used. This also causes importance of input vectors to be uniform, rather than weighted by norm<span id="footnote19" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">19</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">19</sup><span class="ltx_tag ltx_tag_note">19</span>We believe one of the main impacts of normalizing is that it the first token positions are downweighted in importance (see <a href="#A6.SS2" title="F.2 Impact of token position ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section F.2</span></a> for more discussion).</span></span></span>.</p>
</div>
<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>TopK training details</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.3" class="ltx_p">We select <math id="A2.SS1.p1.1.m1.1" class="ltx_Math" alttext="k_{\text{aux}}" display="inline"><semantics id="A2.SS1.p1.1.m1.1a"><msub id="A2.SS1.p1.1.m1.1.1" xref="A2.SS1.p1.1.m1.1.1.cmml"><mi id="A2.SS1.p1.1.m1.1.1.2" xref="A2.SS1.p1.1.m1.1.1.2.cmml">k</mi><mtext id="A2.SS1.p1.1.m1.1.1.3" xref="A2.SS1.p1.1.m1.1.1.3a.cmml">aux</mtext></msub><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.1.m1.1b"><apply id="A2.SS1.p1.1.m1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A2.SS1.p1.1.m1.1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="A2.SS1.p1.1.m1.1.1.2.cmml" xref="A2.SS1.p1.1.m1.1.1.2">𝑘</ci><ci id="A2.SS1.p1.1.m1.1.1.3a.cmml" xref="A2.SS1.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A2.SS1.p1.1.m1.1.1.3.cmml" xref="A2.SS1.p1.1.m1.1.1.3">aux</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.1.m1.1c">k_{\text{aux}}</annotation></semantics></math> as a power of two close to <math id="A2.SS1.p1.2.m2.1" class="ltx_Math" alttext="\frac{d_{model}}{2}" display="inline"><semantics id="A2.SS1.p1.2.m2.1a"><mfrac id="A2.SS1.p1.2.m2.1.1" xref="A2.SS1.p1.2.m2.1.1.cmml"><msub id="A2.SS1.p1.2.m2.1.1.2" xref="A2.SS1.p1.2.m2.1.1.2.cmml"><mi id="A2.SS1.p1.2.m2.1.1.2.2" xref="A2.SS1.p1.2.m2.1.1.2.2.cmml">d</mi><mrow id="A2.SS1.p1.2.m2.1.1.2.3" xref="A2.SS1.p1.2.m2.1.1.2.3.cmml"><mi id="A2.SS1.p1.2.m2.1.1.2.3.2" xref="A2.SS1.p1.2.m2.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p1.2.m2.1.1.2.3.1" xref="A2.SS1.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="A2.SS1.p1.2.m2.1.1.2.3.3" xref="A2.SS1.p1.2.m2.1.1.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p1.2.m2.1.1.2.3.1a" xref="A2.SS1.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="A2.SS1.p1.2.m2.1.1.2.3.4" xref="A2.SS1.p1.2.m2.1.1.2.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p1.2.m2.1.1.2.3.1b" xref="A2.SS1.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="A2.SS1.p1.2.m2.1.1.2.3.5" xref="A2.SS1.p1.2.m2.1.1.2.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p1.2.m2.1.1.2.3.1c" xref="A2.SS1.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="A2.SS1.p1.2.m2.1.1.2.3.6" xref="A2.SS1.p1.2.m2.1.1.2.3.6.cmml">l</mi></mrow></msub><mn id="A2.SS1.p1.2.m2.1.1.3" xref="A2.SS1.p1.2.m2.1.1.3.cmml">2</mn></mfrac><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.2.m2.1b"><apply id="A2.SS1.p1.2.m2.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1"><divide id="A2.SS1.p1.2.m2.1.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1"></divide><apply id="A2.SS1.p1.2.m2.1.1.2.cmml" xref="A2.SS1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A2.SS1.p1.2.m2.1.1.2.1.cmml" xref="A2.SS1.p1.2.m2.1.1.2">subscript</csymbol><ci id="A2.SS1.p1.2.m2.1.1.2.2.cmml" xref="A2.SS1.p1.2.m2.1.1.2.2">𝑑</ci><apply id="A2.SS1.p1.2.m2.1.1.2.3.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3"><times id="A2.SS1.p1.2.m2.1.1.2.3.1.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3.1"></times><ci id="A2.SS1.p1.2.m2.1.1.2.3.2.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3.2">𝑚</ci><ci id="A2.SS1.p1.2.m2.1.1.2.3.3.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3.3">𝑜</ci><ci id="A2.SS1.p1.2.m2.1.1.2.3.4.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3.4">𝑑</ci><ci id="A2.SS1.p1.2.m2.1.1.2.3.5.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3.5">𝑒</ci><ci id="A2.SS1.p1.2.m2.1.1.2.3.6.cmml" xref="A2.SS1.p1.2.m2.1.1.2.3.6">𝑙</ci></apply></apply><cn type="integer" id="A2.SS1.p1.2.m2.1.1.3.cmml" xref="A2.SS1.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.2.m2.1c">\frac{d_{model}}{2}</annotation></semantics></math> (e.g. 512 for GPT-2 small). We typically select <math id="A2.SS1.p1.3.m3.1" class="ltx_Math" alttext="\alpha=1/32" display="inline"><semantics id="A2.SS1.p1.3.m3.1a"><mrow id="A2.SS1.p1.3.m3.1.1" xref="A2.SS1.p1.3.m3.1.1.cmml"><mi id="A2.SS1.p1.3.m3.1.1.2" xref="A2.SS1.p1.3.m3.1.1.2.cmml">α</mi><mo id="A2.SS1.p1.3.m3.1.1.1" xref="A2.SS1.p1.3.m3.1.1.1.cmml">=</mo><mrow id="A2.SS1.p1.3.m3.1.1.3" xref="A2.SS1.p1.3.m3.1.1.3.cmml"><mn id="A2.SS1.p1.3.m3.1.1.3.2" xref="A2.SS1.p1.3.m3.1.1.3.2.cmml">1</mn><mo id="A2.SS1.p1.3.m3.1.1.3.1" xref="A2.SS1.p1.3.m3.1.1.3.1.cmml">/</mo><mn id="A2.SS1.p1.3.m3.1.1.3.3" xref="A2.SS1.p1.3.m3.1.1.3.3.cmml">32</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.3.m3.1b"><apply id="A2.SS1.p1.3.m3.1.1.cmml" xref="A2.SS1.p1.3.m3.1.1"><eq id="A2.SS1.p1.3.m3.1.1.1.cmml" xref="A2.SS1.p1.3.m3.1.1.1"></eq><ci id="A2.SS1.p1.3.m3.1.1.2.cmml" xref="A2.SS1.p1.3.m3.1.1.2">𝛼</ci><apply id="A2.SS1.p1.3.m3.1.1.3.cmml" xref="A2.SS1.p1.3.m3.1.1.3"><divide id="A2.SS1.p1.3.m3.1.1.3.1.cmml" xref="A2.SS1.p1.3.m3.1.1.3.1"></divide><cn type="integer" id="A2.SS1.p1.3.m3.1.1.3.2.cmml" xref="A2.SS1.p1.3.m3.1.1.3.2">1</cn><cn type="integer" id="A2.SS1.p1.3.m3.1.1.3.3.cmml" xref="A2.SS1.p1.3.m3.1.1.3.3">32</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.3.m3.1c">\alpha=1/32</annotation></semantics></math>. We find that the training is generally not extremely sensitive to the choice of these hyperparameters.</p>
</div>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.1" class="ltx_p">We find empirically that using AuxK eliminates almost all dead latents by the end of training.</p>
</div>
<div id="A2.SS1.p3" class="ltx_para">
<p id="A2.SS1.p3.2" class="ltx_p">Unfortunately, because of compute constraints, we were unable to train our 16M latent autoencoder to <math id="A2.SS1.p3.1.m1.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A2.SS1.p3.1.m1.1a"><mrow id="A2.SS1.p3.1.m1.1.2" xref="A2.SS1.p3.1.m1.1.2.cmml"><mi id="A2.SS1.p3.1.m1.1.2.2" xref="A2.SS1.p3.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.1.2.1" xref="A2.SS1.p3.1.m1.1.2.1.cmml">​</mo><mrow id="A2.SS1.p3.1.m1.1.2.3.2" xref="A2.SS1.p3.1.m1.1.2.cmml"><mo stretchy="false" id="A2.SS1.p3.1.m1.1.2.3.2.1" xref="A2.SS1.p3.1.m1.1.2.cmml">(</mo><mi id="A2.SS1.p3.1.m1.1.1" xref="A2.SS1.p3.1.m1.1.1.cmml">N</mi><mo stretchy="false" id="A2.SS1.p3.1.m1.1.2.3.2.2" xref="A2.SS1.p3.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.1.m1.1b"><apply id="A2.SS1.p3.1.m1.1.2.cmml" xref="A2.SS1.p3.1.m1.1.2"><times id="A2.SS1.p3.1.m1.1.2.1.cmml" xref="A2.SS1.p3.1.m1.1.2.1"></times><ci id="A2.SS1.p3.1.m1.1.2.2.cmml" xref="A2.SS1.p3.1.m1.1.2.2">𝐿</ci><ci id="A2.SS1.p3.1.m1.1.1.cmml" xref="A2.SS1.p3.1.m1.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.1.m1.1c">L(N)</annotation></semantics></math>, which made it not possible to include the 16M as part of a consistent <math id="A2.SS1.p3.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A2.SS1.p3.2.m2.1a"><mrow id="A2.SS1.p3.2.m2.1.2" xref="A2.SS1.p3.2.m2.1.2.cmml"><mi id="A2.SS1.p3.2.m2.1.2.2" xref="A2.SS1.p3.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.2.m2.1.2.1" xref="A2.SS1.p3.2.m2.1.2.1.cmml">​</mo><mrow id="A2.SS1.p3.2.m2.1.2.3.2" xref="A2.SS1.p3.2.m2.1.2.cmml"><mo stretchy="false" id="A2.SS1.p3.2.m2.1.2.3.2.1" xref="A2.SS1.p3.2.m2.1.2.cmml">(</mo><mi id="A2.SS1.p3.2.m2.1.1" xref="A2.SS1.p3.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="A2.SS1.p3.2.m2.1.2.3.2.2" xref="A2.SS1.p3.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.2.m2.1b"><apply id="A2.SS1.p3.2.m2.1.2.cmml" xref="A2.SS1.p3.2.m2.1.2"><times id="A2.SS1.p3.2.m2.1.2.1.cmml" xref="A2.SS1.p3.2.m2.1.2.1"></times><ci id="A2.SS1.p3.2.m2.1.2.2.cmml" xref="A2.SS1.p3.2.m2.1.2.2">𝐿</ci><ci id="A2.SS1.p3.2.m2.1.1.cmml" xref="A2.SS1.p3.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.2.m2.1c">L(N)</annotation></semantics></math> series.</p>
</div>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Baseline hyperparameters</h3>

<div id="A2.SS2.p1" class="ltx_para">
<p id="A2.SS2.p1.1" class="ltx_p">Baseline ReLU autoencoders were trained on GPT-2 small, layer 8.
We sweep learning rate in [5e-5, 1e-4, 2e-4, 4e-4], L<sub id="A2.SS2.p1.1.1" class="ltx_sub">1</sub> coefficient in [1.7e-3, 3.1e-3, 5e-3, 1e-2, 1.7e-2] and train for 8 epochs of 6.4 billion tokens at a batch size of 131072. We try different resampling periods in [12.5k, 25k] steps, and choose to resample 4 times throughout training. We consider a feature dead if it does not activate for 10 million tokens.</p>
</div>
<div id="A2.SS2.p2" class="ltx_para">
<p id="A2.SS2.p2.1" class="ltx_p">For Gated SAE <cite class="ltx_cite ltx_citemacro_citep">[Rajamanoharan et al., <a href="#bib.bib52" title="" class="ltx_ref">2024</a>]</cite>, we sweep L<sub id="A2.SS2.p2.1.1" class="ltx_sub">1</sub> coefficient in [1e-3, 2.5e-3, 5e-3, 1e-2, 2e-2], learning rate in [2.5e-5, 5e-5, 1e-4], train for 6 epochs of 6.4 billion tokens at a batch size of 131072. We resample 4 times throughout training.</p>
</div>
<div id="A2.SS2.p3" class="ltx_para">
<p id="A2.SS2.p3.3" class="ltx_p">For ProLU autoencoders <cite class="ltx_cite ltx_citemacro_citep">[Taggart, <a href="#bib.bib62" title="" class="ltx_ref">2024</a>]</cite>, we sweep L<sub id="A2.SS2.p3.3.1" class="ltx_sub">1</sub> coefficient in [5e-4, 1e-3, 2.5e-3, 5e-3, 1e-2, 2e-2], learning rate in [2.5e-5, 5e-5, 1e-4], train for 6 epochs of 6.4 billion tokens at a batch size of 131072. We resample 4 times throughout training. For the ProLU gradient, we try both ProLU-STE and ProLU-ReLU. Note that, consistent with the original work, ProLU-STE autoencoders all have <math id="A2.SS2.p3.2.m2.1" class="ltx_Math" alttext="L_{0}&lt;25" display="inline"><semantics id="A2.SS2.p3.2.m2.1a"><mrow id="A2.SS2.p3.2.m2.1.1" xref="A2.SS2.p3.2.m2.1.1.cmml"><msub id="A2.SS2.p3.2.m2.1.1.2" xref="A2.SS2.p3.2.m2.1.1.2.cmml"><mi id="A2.SS2.p3.2.m2.1.1.2.2" xref="A2.SS2.p3.2.m2.1.1.2.2.cmml">L</mi><mn id="A2.SS2.p3.2.m2.1.1.2.3" xref="A2.SS2.p3.2.m2.1.1.2.3.cmml">0</mn></msub><mo id="A2.SS2.p3.2.m2.1.1.1" xref="A2.SS2.p3.2.m2.1.1.1.cmml">&lt;</mo><mn id="A2.SS2.p3.2.m2.1.1.3" xref="A2.SS2.p3.2.m2.1.1.3.cmml">25</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS2.p3.2.m2.1b"><apply id="A2.SS2.p3.2.m2.1.1.cmml" xref="A2.SS2.p3.2.m2.1.1"><lt id="A2.SS2.p3.2.m2.1.1.1.cmml" xref="A2.SS2.p3.2.m2.1.1.1"></lt><apply id="A2.SS2.p3.2.m2.1.1.2.cmml" xref="A2.SS2.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="A2.SS2.p3.2.m2.1.1.2.1.cmml" xref="A2.SS2.p3.2.m2.1.1.2">subscript</csymbol><ci id="A2.SS2.p3.2.m2.1.1.2.2.cmml" xref="A2.SS2.p3.2.m2.1.1.2.2">𝐿</ci><cn type="integer" id="A2.SS2.p3.2.m2.1.1.2.3.cmml" xref="A2.SS2.p3.2.m2.1.1.2.3">0</cn></apply><cn type="integer" id="A2.SS2.p3.2.m2.1.1.3.cmml" xref="A2.SS2.p3.2.m2.1.1.3">25</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p3.2.m2.1c">L_{0}&lt;25</annotation></semantics></math>, even for small L<sub id="A2.SS2.p3.3.2" class="ltx_sub">1</sub> coefficients.</p>
</div>
<div id="A2.SS2.p4" class="ltx_para">
<p id="A2.SS2.p4.2" class="ltx_p">We used similar settings and sweeps for autoencoders trained on GPT-4.
Differences include: replacing the resampling of dead latents with a L<sub id="A2.SS2.p4.2.1" class="ltx_sub">1</sub> coefficient warm-up over 5% of training <cite class="ltx_cite ltx_citemacro_citep">[Conerly et al., <a href="#bib.bib11" title="" class="ltx_ref">2024</a>]</cite>; removing the decoder unit-norm constraint and adding the decoder norm in the L<sub id="A2.SS2.p4.2.2" class="ltx_sub">1</sub> penalty <cite class="ltx_cite ltx_citemacro_citep">[Conerly et al., <a href="#bib.bib11" title="" class="ltx_ref">2024</a>]</cite>.</p>
</div>
<div id="A2.SS2.p5" class="ltx_para">
<p id="A2.SS2.p5.1" class="ltx_p">Our baselines generally have few dead latents, similar or less than our TopK models (see <a href="#A2.F14" title="Figure 14 ‣ B.2 Baseline hyperparameters ‣ Appendix B Other training details ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 14</span></a>).</p>
</div>
<figure id="A2.F13" class="ltx_figure"><img src="/html/2406.04093/assets/x16.png" id="A2.F13.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="226" height="155" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F13.4.2.1" class="ltx_text" style="font-size:90%;">Figure 13</span>: </span><span id="A2.F13.2.1" class="ltx_text" style="font-size:90%;">For a fixed sparsity level (<math id="A2.F13.2.1.m1.1" class="ltx_Math" alttext="L_{0}=128" display="inline"><semantics id="A2.F13.2.1.m1.1b"><mrow id="A2.F13.2.1.m1.1.1" xref="A2.F13.2.1.m1.1.1.cmml"><msub id="A2.F13.2.1.m1.1.1.2" xref="A2.F13.2.1.m1.1.1.2.cmml"><mi id="A2.F13.2.1.m1.1.1.2.2" xref="A2.F13.2.1.m1.1.1.2.2.cmml">L</mi><mn id="A2.F13.2.1.m1.1.1.2.3" xref="A2.F13.2.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="A2.F13.2.1.m1.1.1.1" xref="A2.F13.2.1.m1.1.1.1.cmml">=</mo><mn id="A2.F13.2.1.m1.1.1.3" xref="A2.F13.2.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.F13.2.1.m1.1c"><apply id="A2.F13.2.1.m1.1.1.cmml" xref="A2.F13.2.1.m1.1.1"><eq id="A2.F13.2.1.m1.1.1.1.cmml" xref="A2.F13.2.1.m1.1.1.1"></eq><apply id="A2.F13.2.1.m1.1.1.2.cmml" xref="A2.F13.2.1.m1.1.1.2"><csymbol cd="ambiguous" id="A2.F13.2.1.m1.1.1.2.1.cmml" xref="A2.F13.2.1.m1.1.1.2">subscript</csymbol><ci id="A2.F13.2.1.m1.1.1.2.2.cmml" xref="A2.F13.2.1.m1.1.1.2.2">𝐿</ci><cn type="integer" id="A2.F13.2.1.m1.1.1.2.3.cmml" xref="A2.F13.2.1.m1.1.1.2.3">0</cn></apply><cn type="integer" id="A2.F13.2.1.m1.1.1.3.cmml" xref="A2.F13.2.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.2.1.m1.1d">L_{0}=128</annotation></semantics></math>), a given MSE leads to a lower downstream-loss for TopK than for other activations functions. (GPT-4). We are less confident about these runs than the corresponding ones for GPT-2, yet the results are consistent (see <a href="#S3.F5" title="Figure 5 ‣ 3.2 Subject model size 𝐿_𝑠⁢(𝑁) ‣ 3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 5</span></a>).</span></figcaption>
</figure>
<figure id="A2.F14" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F14.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.04093/assets/placeholders/dead_ratio_gpt2.png" id="A2.F14.sf1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="452" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A2.F14.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="A2.F14.sf1.3.2" class="ltx_text" style="font-size:90%;">Dead latent ratios for GPT-2</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F14.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.04093/assets/placeholders/dead_ratio_gpt4.png" id="A2.F14.sf2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="451" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A2.F14.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="A2.F14.sf2.3.2" class="ltx_text" style="font-size:90%;">Dead latent ratios for GPT-4</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F14.2.1.1" class="ltx_text" style="font-size:90%;">Figure 14</span>: </span><span id="A2.F14.3.2" class="ltx_text" style="font-size:90%;">Our baselines generally have few dead latents, similar or less than our TopK models.</span></figcaption>
</figure>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Training ablations</h2>

<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Dead latent prevention</h3>

<figure id="A3.F15" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_2M_dead_maxk_init.png" id="A3.F15.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="212" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F15.2.1.1" class="ltx_text" style="font-size:90%;">Figure 15</span>: </span><span id="A3.F15.3.2" class="ltx_text" style="font-size:90%;">Methods that reduce the number of dead latents (gpt2sm 2M, k=32). With AuxK and/or tied initialization, number of dead latents generally decreases over the course of training, after an early spike.</span></figcaption>
</figure>
<div id="A3.SS1.p1" class="ltx_para">
<p id="A3.SS1.p1.1" class="ltx_p">We find that the reduction in dead latents is mostly due to a combination of the AuxK loss and the tied initialization scheme.</p>
</div>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>Initialization</h3>

<figure id="A3.F16" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_ablation_init.png" id="A3.F16.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="195" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F16.2.1.1" class="ltx_text" style="font-size:90%;">Figure 16</span>: </span><span id="A3.F16.3.2" class="ltx_text" style="font-size:90%;">Initialization ablation (gpt2sm 128k, k=32).</span></figcaption>
</figure>
<div id="A3.SS2.p1" class="ltx_para">
<p id="A3.SS2.p1.1" class="ltx_p">We find that tied initialization substantially improves MSE, and that our encoder initialization scheme has no effect when tied initialization is being used, and hurts slightly on its own.</p>
</div>
</section>
<section id="A3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.3 </span><math id="A3.SS3.1.m1.1" class="ltx_Math" alttext="b_{\textrm{enc}}" display="inline"><semantics id="A3.SS3.1.m1.1b"><msub id="A3.SS3.1.m1.1.1" xref="A3.SS3.1.m1.1.1.cmml"><mi id="A3.SS3.1.m1.1.1.2" xref="A3.SS3.1.m1.1.1.2.cmml">b</mi><mtext id="A3.SS3.1.m1.1.1.3" xref="A3.SS3.1.m1.1.1.3a.cmml">enc</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS3.1.m1.1c"><apply id="A3.SS3.1.m1.1.1.cmml" xref="A3.SS3.1.m1.1.1"><csymbol cd="ambiguous" id="A3.SS3.1.m1.1.1.1.cmml" xref="A3.SS3.1.m1.1.1">subscript</csymbol><ci id="A3.SS3.1.m1.1.1.2.cmml" xref="A3.SS3.1.m1.1.1.2">𝑏</ci><ci id="A3.SS3.1.m1.1.1.3a.cmml" xref="A3.SS3.1.m1.1.1.3"><mtext mathsize="70%" id="A3.SS3.1.m1.1.1.3.cmml" xref="A3.SS3.1.m1.1.1.3">enc</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.1.m1.1d">b_{\textrm{enc}}</annotation></semantics></math>
</h3>

<figure id="A3.F17" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_ablation_lbias.png" id="A3.F17.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="195" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F17.4.1.1" class="ltx_text" style="font-size:90%;">Figure 17</span>: </span><math id="A3.F17.2.m1.1" class="ltx_Math" alttext="b_{\textrm{enc}}" display="inline"><semantics id="A3.F17.2.m1.1b"><msub id="A3.F17.2.m1.1.1" xref="A3.F17.2.m1.1.1.cmml"><mi mathsize="90%" id="A3.F17.2.m1.1.1.2" xref="A3.F17.2.m1.1.1.2.cmml">b</mi><mtext mathsize="90%" id="A3.F17.2.m1.1.1.3" xref="A3.F17.2.m1.1.1.3a.cmml">enc</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.F17.2.m1.1c"><apply id="A3.F17.2.m1.1.1.cmml" xref="A3.F17.2.m1.1.1"><csymbol cd="ambiguous" id="A3.F17.2.m1.1.1.1.cmml" xref="A3.F17.2.m1.1.1">subscript</csymbol><ci id="A3.F17.2.m1.1.1.2.cmml" xref="A3.F17.2.m1.1.1.2">𝑏</ci><ci id="A3.F17.2.m1.1.1.3a.cmml" xref="A3.F17.2.m1.1.1.3"><mtext mathsize="63%" id="A3.F17.2.m1.1.1.3.cmml" xref="A3.F17.2.m1.1.1.3">enc</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.2.m1.1d">b_{\textrm{enc}}</annotation></semantics></math><span id="A3.F17.5.2" class="ltx_text" style="font-size:90%;"> does not strongly affect loss (gpt2sm 128k, k=32).</span></figcaption>
</figure>
<div id="A3.SS3.p1" class="ltx_para">
<p id="A3.SS3.p1.4" class="ltx_p">We find that <math id="A3.SS3.p1.1.m1.1" class="ltx_Math" alttext="b_{\textrm{enc}}" display="inline"><semantics id="A3.SS3.p1.1.m1.1a"><msub id="A3.SS3.p1.1.m1.1.1" xref="A3.SS3.p1.1.m1.1.1.cmml"><mi id="A3.SS3.p1.1.m1.1.1.2" xref="A3.SS3.p1.1.m1.1.1.2.cmml">b</mi><mtext id="A3.SS3.p1.1.m1.1.1.3" xref="A3.SS3.p1.1.m1.1.1.3a.cmml">enc</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS3.p1.1.m1.1b"><apply id="A3.SS3.p1.1.m1.1.1.cmml" xref="A3.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.SS3.p1.1.m1.1.1.1.cmml" xref="A3.SS3.p1.1.m1.1.1">subscript</csymbol><ci id="A3.SS3.p1.1.m1.1.1.2.cmml" xref="A3.SS3.p1.1.m1.1.1.2">𝑏</ci><ci id="A3.SS3.p1.1.m1.1.1.3a.cmml" xref="A3.SS3.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A3.SS3.p1.1.m1.1.1.3.cmml" xref="A3.SS3.p1.1.m1.1.1.3">enc</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.p1.1.m1.1c">b_{\textrm{enc}}</annotation></semantics></math> does not affect the MSE at convergence. With <math id="A3.SS3.p1.2.m2.1" class="ltx_Math" alttext="b_{\textrm{enc}}" display="inline"><semantics id="A3.SS3.p1.2.m2.1a"><msub id="A3.SS3.p1.2.m2.1.1" xref="A3.SS3.p1.2.m2.1.1.cmml"><mi id="A3.SS3.p1.2.m2.1.1.2" xref="A3.SS3.p1.2.m2.1.1.2.cmml">b</mi><mtext id="A3.SS3.p1.2.m2.1.1.3" xref="A3.SS3.p1.2.m2.1.1.3a.cmml">enc</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS3.p1.2.m2.1b"><apply id="A3.SS3.p1.2.m2.1.1.cmml" xref="A3.SS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A3.SS3.p1.2.m2.1.1.1.cmml" xref="A3.SS3.p1.2.m2.1.1">subscript</csymbol><ci id="A3.SS3.p1.2.m2.1.1.2.cmml" xref="A3.SS3.p1.2.m2.1.1.2">𝑏</ci><ci id="A3.SS3.p1.2.m2.1.1.3a.cmml" xref="A3.SS3.p1.2.m2.1.1.3"><mtext mathsize="70%" id="A3.SS3.p1.2.m2.1.1.3.cmml" xref="A3.SS3.p1.2.m2.1.1.3">enc</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.p1.2.m2.1c">b_{\textrm{enc}}</annotation></semantics></math> removed, the autencoder is equivalent to a JumpReLU where the threshold is dynamically chosen per example such that exactly <math id="A3.SS3.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS3.p1.3.m3.1a"><mi id="A3.SS3.p1.3.m3.1.1" xref="A3.SS3.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS3.p1.3.m3.1b"><ci id="A3.SS3.p1.3.m3.1.1.cmml" xref="A3.SS3.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.p1.3.m3.1c">k</annotation></semantics></math> latents are active. However, convergence is slightly slower without <math id="A3.SS3.p1.4.m4.1" class="ltx_Math" alttext="b_{\textrm{enc}}" display="inline"><semantics id="A3.SS3.p1.4.m4.1a"><msub id="A3.SS3.p1.4.m4.1.1" xref="A3.SS3.p1.4.m4.1.1.cmml"><mi id="A3.SS3.p1.4.m4.1.1.2" xref="A3.SS3.p1.4.m4.1.1.2.cmml">b</mi><mtext id="A3.SS3.p1.4.m4.1.1.3" xref="A3.SS3.p1.4.m4.1.1.3a.cmml">enc</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS3.p1.4.m4.1b"><apply id="A3.SS3.p1.4.m4.1.1.cmml" xref="A3.SS3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A3.SS3.p1.4.m4.1.1.1.cmml" xref="A3.SS3.p1.4.m4.1.1">subscript</csymbol><ci id="A3.SS3.p1.4.m4.1.1.2.cmml" xref="A3.SS3.p1.4.m4.1.1.2">𝑏</ci><ci id="A3.SS3.p1.4.m4.1.1.3a.cmml" xref="A3.SS3.p1.4.m4.1.1.3"><mtext mathsize="70%" id="A3.SS3.p1.4.m4.1.1.3.cmml" xref="A3.SS3.p1.4.m4.1.1.3">enc</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.p1.4.m4.1c">b_{\textrm{enc}}</annotation></semantics></math>. We believe this may be confounded by encoder learning rate but did not investigate this further.</p>
</div>
</section>
<section id="A3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.4 </span>Decoder normalization</h3>

<div id="A3.SS4.p1" class="ltx_para">
<p id="A3.SS4.p1.1" class="ltx_p">After each step we renormalize columns of the decoder to be unit-norm, following <cite class="ltx_cite ltx_citemacro_citet">Bricken et al. [<a href="#bib.bib8" title="" class="ltx_ref">2023</a>]</cite>. This normalization (or a modified L1 term, as in <cite class="ltx_cite ltx_citemacro_citet">Conerly et al. [<a href="#bib.bib11" title="" class="ltx_ref">2024</a>]</cite>) is necessary for L1 autoencoders, because otherwise the L1 loss can be gamed by making the latents arbitrarily small. For TopK autoencoders, the normalization is optional. However, we find that it still improves MSE, so we still use it in all of our experiments.</p>
</div>
<figure id="A3.F18" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_ablation_decodernorm.png" id="A3.F18.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="195" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F18.2.1.1" class="ltx_text" style="font-size:90%;">Figure 18</span>: </span><span id="A3.F18.3.2" class="ltx_text" style="font-size:90%;">The decoder normalization slightly improves loss (gpt2sm 128k, k=32).</span></figcaption>
</figure>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Systems</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">Scaling autoencoders to the largest scales in this paper would not be feasible without our systems improvements. Model parallelism is necessary once parameters cannot fit on one GPU. A naive implementation can be an order of magnitude slower than our optimized implementation at the very largest scales.</p>
</div>
<section id="A4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.1 </span>Parallelism</h3>

<div id="A4.SS1.p1" class="ltx_para">
<p id="A4.SS1.p1.1" class="ltx_p">We use standard data parallel and tensor sharding <cite class="ltx_cite ltx_citemacro_citep">[Shoeybi et al., <a href="#bib.bib57" title="" class="ltx_ref">2019</a>]</cite>, with an additional allgather for the TopK forward pass to determine which <math id="A4.SS1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A4.SS1.p1.1.m1.1a"><mi id="A4.SS1.p1.1.m1.1.1" xref="A4.SS1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A4.SS1.p1.1.m1.1b"><ci id="A4.SS1.p1.1.m1.1.1.cmml" xref="A4.SS1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.SS1.p1.1.m1.1c">k</annotation></semantics></math> latents should are in the global top k. To minimize the cost of this allgather, we truncate to a capacity factor of 2 per shard—further improvements are possible but would require modifications to NCCL. For the largest (16 million) latent autoencoder, we use 512-way sharding. Large batch sizes (<a href="#A1.SS4" title="A.4 Batch size ‣ Appendix A Optimization ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section A.4</span></a>) are very important for reducing the parallelization overhead.</p>
</div>
<div id="A4.SS1.p2" class="ltx_para">
<p id="A4.SS1.p2.1" class="ltx_p">The very small number of layers creates a challenge for parallelism - it makes pipeline parallelism <cite class="ltx_cite ltx_citemacro_citep">[Huang et al., <a href="#bib.bib23" title="" class="ltx_ref">2019</a>]</cite>
and FSDP <cite class="ltx_cite ltx_citemacro_citep">[Zhao et al., <a href="#bib.bib71" title="" class="ltx_ref">2023</a>]</cite> inapplicable. Additionally, opportunities for communications overlap are limited because of the small number of layers, though we do overlap host to device transfers and encoder data parallel comms for a small improvement.</p>
</div>
</section>
<section id="A4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.2 </span>Kernels</h3>

<div id="A4.SS2.p1" class="ltx_para">
<p id="A4.SS2.p1.1" class="ltx_p">We can take advantage of the extreme sparsity of latents to perform most operations using substantially less compute and memory than naively doing dense matrix multiplication. This was important when scaling to large numbers of latents, both via directly increasing throughput and reducing memory usage.</p>
</div>
<div id="A4.SS2.p2" class="ltx_para">
<p id="A4.SS2.p2.1" class="ltx_p">We use two main kernels:</p>
<ul id="A4.I1" class="ltx_itemize">
<li id="A4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i1.p1" class="ltx_para">
<p id="A4.I1.i1.p1.1" class="ltx_p"><span id="A4.I1.i1.p1.1.1" class="ltx_text ltx_font_typewriter">DenseSparseMatmul</span>: a multiplication between a dense and sparse matrix</p>
</div>
</li>
<li id="A4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i2.p1" class="ltx_para">
<p id="A4.I1.i2.p1.1" class="ltx_p"><span id="A4.I1.i2.p1.1.1" class="ltx_text ltx_font_typewriter">MatmulAtSparseIndices</span>: a multiplication of two dense matrices evaluated at a set of sparse indices</p>
</div>
</li>
</ul>
</div>
<div id="A4.SS2.p3" class="ltx_para">
<p id="A4.SS2.p3.1" class="ltx_p">Then, we have the following optimizations:</p>
<ol id="A4.I2" class="ltx_enumerate">
<li id="A4.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="A4.I2.i1.p1" class="ltx_para">
<p id="A4.I2.i1.p1.1" class="ltx_p">The decoder forward pass uses <span id="A4.I2.i1.p1.1.1" class="ltx_text ltx_font_typewriter">DenseSparseMatmul</span></p>
</div>
</li>
<li id="A4.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="A4.I2.i2.p1" class="ltx_para">
<p id="A4.I2.i2.p1.1" class="ltx_p">The decoder gradient uses <span id="A4.I2.i2.p1.1.1" class="ltx_text ltx_font_typewriter">DenseSparseMatmul</span></p>
</div>
</li>
<li id="A4.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="A4.I2.i3.p1" class="ltx_para">
<p id="A4.I2.i3.p1.1" class="ltx_p">The latent gradient uses <span id="A4.I2.i3.p1.1.1" class="ltx_text ltx_font_typewriter">MatmulAtSparseIndices</span></p>
</div>
</li>
<li id="A4.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="A4.I2.i4.p1" class="ltx_para">
<p id="A4.I2.i4.p1.1" class="ltx_p">The encoder gradient uses <span id="A4.I2.i4.p1.1.1" class="ltx_text ltx_font_typewriter">DenseSparseMatmul</span></p>
</div>
</li>
<li id="A4.I2.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5.</span> 
<div id="A4.I2.i5.p1" class="ltx_para">
<p id="A4.I2.i5.p1.1" class="ltx_p">The pre-bias gradient uses a trick of summing pre-activation gradient across the batch dimension before multiplying with the encoder weights.</p>
</div>
</li>
</ol>
</div>
<div id="A4.SS2.p4" class="ltx_para">
<p id="A4.SS2.p4.1" class="ltx_p">Theoretically, this gives a compute efficiency improvement of up to 6x in the limit of sparsity, since the encoder forward pass is the only remaining dense operation. In practice, we indeed find the encoder forward pass is much of the compute, and the pre-activations are much of the memory.</p>
</div>
<div id="A4.SS2.p5" class="ltx_para">
<p id="A4.SS2.p5.1" class="ltx_p">To ensure that reads are coalesced, the decoder weight matrix must also be stored transposed from the typical layout. We also use many other kernels for fusing various operations for reducing memory and memory bandwidth usage.</p>
</div>
</section>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Qualitative results</h2>

<section id="A5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.1 </span>Subjective latent quality</h3>

<div id="A5.SS1.p1" class="ltx_para">
<p id="A5.SS1.p1.1" class="ltx_p">Throughout the project, we stumbled upon many subjectively interesting latents. The majority of latents in our GPT-2 small autoencoders seemed interpretable, even on random positive activations. Furthermore, the ablations typically had predictable effects based on the activation conditions. For example, some features that are potentially part of interesting circuits in GPT-2:</p>
</div>
<div id="A5.SS1.p2" class="ltx_para">
<ul id="A5.I1" class="ltx_itemize">
<li id="A5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i1.p1" class="ltx_para">
<p id="A5.I1.i1.p1.1" class="ltx_p">An unexpected token breaking a repetition pattern (A B C D … A B X!). This upvotes future pattern breaks (A B Y!), but also upvotes continuation of the pattern right after the break token (A B X <math id="A5.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A5.I1.i1.p1.1.m1.1a"><mo stretchy="false" id="A5.I1.i1.p1.1.m1.1.1" xref="A5.I1.i1.p1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A5.I1.i1.p1.1.m1.1b"><ci id="A5.I1.i1.p1.1.m1.1.1.cmml" xref="A5.I1.i1.p1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i1.p1.1.m1.1c">\rightarrow</annotation></semantics></math> D).</p>
</div>
</li>
<li id="A5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i2.p1" class="ltx_para">
<p id="A5.I1.i2.p1.1" class="ltx_p">Text within quotes, especially activating when within two nested sets of quotes. Upvotes tokens that close the quotes like " or ’, as well as tokens which close multiple sets of quotes at once, such as "’ and ’".</p>
</div>
</li>
<li id="A5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i3.p1" class="ltx_para">
<p id="A5.I1.i3.p1.1" class="ltx_p">Copying/induction of capitalized phrases (A B … A <math id="A5.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A5.I1.i3.p1.1.m1.1a"><mo stretchy="false" id="A5.I1.i3.p1.1.m1.1.1" xref="A5.I1.i3.p1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A5.I1.i3.p1.1.m1.1b"><ci id="A5.I1.i3.p1.1.m1.1.1.cmml" xref="A5.I1.i3.p1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i3.p1.1.m1.1c">\rightarrow</annotation></semantics></math> B)</p>
</div>
</li>
</ul>
</div>
<div id="A5.SS1.p3" class="ltx_para">
<p id="A5.SS1.p3.1" class="ltx_p">In GPT-4, we tended to find more complex features, including ones that activate on the same concept in multiple languages, or on complex technical concepts such as algebraic rings.</p>
</div>
<div id="A5.SS1.p4" class="ltx_para">
<p id="A5.SS1.p4.1" class="ltx_p">You can explore for yourself at <a target="_blank" href="https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html" title="" class="ltx_ref ltx_href">our viewer</a>.</p>
</div>
</section>
<section id="A5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.2 </span>Finding features</h3>

<div id="A5.SS2.p1" class="ltx_para">
<p id="A5.SS2.p1.1" class="ltx_p">Typical features are not of particular interest, but having an autoencoder also lets one easily find relevant features. Specifically, one can use gradient-based attribution to quickly compute how relevant latents are to behaviors of interest <cite class="ltx_cite ltx_citemacro_citep">[Baehrens et al., <a href="#bib.bib2" title="" class="ltx_ref">2010</a>, Simonyan et al., <a href="#bib.bib58" title="" class="ltx_ref">2013</a>]</cite>.</p>
</div>
<div id="A5.SS2.p2" class="ltx_para">
<p id="A5.SS2.p2.1" class="ltx_p">Following the methodologies of <cite class="ltx_cite ltx_citemacro_cite">Templeton et al. [<a href="#bib.bib64" title="" class="ltx_ref">2024</a>]</cite> and <cite class="ltx_cite ltx_citemacro_cite">Mossing et al. [<a href="#bib.bib45" title="" class="ltx_ref">2024</a>]</cite>, we found features by using a hand-written prompt, with a set of "positive" and "negative" token predictions, and back-propagating from logit differences to latent values. We then consider latents sorted by activation times gradient value, and inspect the latents.</p>
</div>
<div id="A5.SS2.p3" class="ltx_para">
<p id="A5.SS2.p3.2" class="ltx_p">We tried this methodology with a <math id="A5.SS2.p3.1.m1.1" class="ltx_Math" alttext="n=2^{17}=131072" display="inline"><semantics id="A5.SS2.p3.1.m1.1a"><mrow id="A5.SS2.p3.1.m1.1.1" xref="A5.SS2.p3.1.m1.1.1.cmml"><mi id="A5.SS2.p3.1.m1.1.1.2" xref="A5.SS2.p3.1.m1.1.1.2.cmml">n</mi><mo id="A5.SS2.p3.1.m1.1.1.3" xref="A5.SS2.p3.1.m1.1.1.3.cmml">=</mo><msup id="A5.SS2.p3.1.m1.1.1.4" xref="A5.SS2.p3.1.m1.1.1.4.cmml"><mn id="A5.SS2.p3.1.m1.1.1.4.2" xref="A5.SS2.p3.1.m1.1.1.4.2.cmml">2</mn><mn id="A5.SS2.p3.1.m1.1.1.4.3" xref="A5.SS2.p3.1.m1.1.1.4.3.cmml">17</mn></msup><mo id="A5.SS2.p3.1.m1.1.1.5" xref="A5.SS2.p3.1.m1.1.1.5.cmml">=</mo><mn id="A5.SS2.p3.1.m1.1.1.6" xref="A5.SS2.p3.1.m1.1.1.6.cmml">131072</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.p3.1.m1.1b"><apply id="A5.SS2.p3.1.m1.1.1.cmml" xref="A5.SS2.p3.1.m1.1.1"><and id="A5.SS2.p3.1.m1.1.1a.cmml" xref="A5.SS2.p3.1.m1.1.1"></and><apply id="A5.SS2.p3.1.m1.1.1b.cmml" xref="A5.SS2.p3.1.m1.1.1"><eq id="A5.SS2.p3.1.m1.1.1.3.cmml" xref="A5.SS2.p3.1.m1.1.1.3"></eq><ci id="A5.SS2.p3.1.m1.1.1.2.cmml" xref="A5.SS2.p3.1.m1.1.1.2">𝑛</ci><apply id="A5.SS2.p3.1.m1.1.1.4.cmml" xref="A5.SS2.p3.1.m1.1.1.4"><csymbol cd="ambiguous" id="A5.SS2.p3.1.m1.1.1.4.1.cmml" xref="A5.SS2.p3.1.m1.1.1.4">superscript</csymbol><cn type="integer" id="A5.SS2.p3.1.m1.1.1.4.2.cmml" xref="A5.SS2.p3.1.m1.1.1.4.2">2</cn><cn type="integer" id="A5.SS2.p3.1.m1.1.1.4.3.cmml" xref="A5.SS2.p3.1.m1.1.1.4.3">17</cn></apply></apply><apply id="A5.SS2.p3.1.m1.1.1c.cmml" xref="A5.SS2.p3.1.m1.1.1"><eq id="A5.SS2.p3.1.m1.1.1.5.cmml" xref="A5.SS2.p3.1.m1.1.1.5"></eq><share href="#A5.SS2.p3.1.m1.1.1.4.cmml" id="A5.SS2.p3.1.m1.1.1d.cmml" xref="A5.SS2.p3.1.m1.1.1"></share><cn type="integer" id="A5.SS2.p3.1.m1.1.1.6.cmml" xref="A5.SS2.p3.1.m1.1.1.6">131072</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.p3.1.m1.1c">n=2^{17}=131072</annotation></semantics></math>, <math id="A5.SS2.p3.2.m2.1" class="ltx_Math" alttext="k=32" display="inline"><semantics id="A5.SS2.p3.2.m2.1a"><mrow id="A5.SS2.p3.2.m2.1.1" xref="A5.SS2.p3.2.m2.1.1.cmml"><mi id="A5.SS2.p3.2.m2.1.1.2" xref="A5.SS2.p3.2.m2.1.1.2.cmml">k</mi><mo id="A5.SS2.p3.2.m2.1.1.1" xref="A5.SS2.p3.2.m2.1.1.1.cmml">=</mo><mn id="A5.SS2.p3.2.m2.1.1.3" xref="A5.SS2.p3.2.m2.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.p3.2.m2.1b"><apply id="A5.SS2.p3.2.m2.1.1.cmml" xref="A5.SS2.p3.2.m2.1.1"><eq id="A5.SS2.p3.2.m2.1.1.1.cmml" xref="A5.SS2.p3.2.m2.1.1.1"></eq><ci id="A5.SS2.p3.2.m2.1.1.2.cmml" xref="A5.SS2.p3.2.m2.1.1.2">𝑘</ci><cn type="integer" id="A5.SS2.p3.2.m2.1.1.3.cmml" xref="A5.SS2.p3.2.m2.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.p3.2.m2.1c">k=32</annotation></semantics></math> GPT-2 autoencoder and were able to quickly find a number of safety relevant latents, such as ones corresponding to profanity or child sexual content. Clamping these latents appeared to have causal effect on the samples. For example, clamping the profanity latent to negative values results in significantly less profanity (some profanity can still be observed in situations where it copies from the prompt).</p>
</div>
</section>
<section id="A5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.3 </span>Latent activation distributions</h3>

<div id="A5.SS3.p1" class="ltx_para">
<p id="A5.SS3.p1.1" class="ltx_p">We anecdotally found that latent activation distributions often have multiple modes, especially in early layers.</p>
</div>
</section>
<section id="A5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.4 </span>Latent density and importance curves</h3>

<figure id="A5.F19" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepn_2m_hist_feature_density.png" id="A5.F19.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="293" height="202" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepn_2m_hist_mean_acts_squared.png" id="A5.F19.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="293" height="202" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F19.2.1.1" class="ltx_text" style="font-size:90%;">Figure 19</span>: </span><span id="A5.F19.3.2" class="ltx_text" style="font-size:90%;">Distributions of latent densities, and average squared activation. Note that we do not observe multiple density modes, as observed in <cite class="ltx_cite ltx_citemacro_citep">[Bricken et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>]</cite>. Counts are sampled over 1.5e7 total tokens. Note that because latents that activate every 1e7 tokens are considered dead during training (and thus receive AuxK gradient updates), 1e-7 is in some sense the minimum density, though the AuxK loss term coefficient may allow it to be lower.</span></figcaption>
</figure>
<div id="A5.SS4.p1" class="ltx_para">
<p id="A5.SS4.p1.1" class="ltx_p">We find that log of latent density is approximately Gaussian (<a href="#A5.F19" title="Figure 19 ‣ E.4 Latent density and importance curves ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 19</span></a>). If we define the importance of a feature to be the expected squared activation (an approximation of its marginal impact on MSE), then log importance looks a bit more like a Laplace distribution. Modal density and modal feature importance both decrease with number of total latents (for a fixed <math id="A5.SS4.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS4.p1.1.m1.1a"><mi id="A5.SS4.p1.1.m1.1.1" xref="A5.SS4.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS4.p1.1.m1.1b"><ci id="A5.SS4.p1.1.m1.1.1.cmml" xref="A5.SS4.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS4.p1.1.m1.1c">k</annotation></semantics></math>), as expected.</p>
</div>
</section>
<section id="A5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.5 </span>Solutions with dense latents</h3>

<div id="A5.SS5.p1" class="ltx_para">
<p id="A5.SS5.p1.6" class="ltx_p">One option for a sparse autoencoder is to simply learn the <math id="A5.SS5.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p1.1.m1.1a"><mi id="A5.SS5.p1.1.m1.1.1" xref="A5.SS5.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.1.m1.1b"><ci id="A5.SS5.p1.1.m1.1.1.cmml" xref="A5.SS5.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.1.m1.1c">k</annotation></semantics></math> directions that explain the most variance. As <math id="A5.SS5.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p1.2.m2.1a"><mi id="A5.SS5.p1.2.m2.1.1" xref="A5.SS5.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.2.m2.1b"><ci id="A5.SS5.p1.2.m2.1.1.cmml" xref="A5.SS5.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.2.m2.1c">k</annotation></semantics></math> approaches <math id="A5.SS5.p1.3.m3.1" class="ltx_Math" alttext="d_{model}" display="inline"><semantics id="A5.SS5.p1.3.m3.1a"><msub id="A5.SS5.p1.3.m3.1.1" xref="A5.SS5.p1.3.m3.1.1.cmml"><mi id="A5.SS5.p1.3.m3.1.1.2" xref="A5.SS5.p1.3.m3.1.1.2.cmml">d</mi><mrow id="A5.SS5.p1.3.m3.1.1.3" xref="A5.SS5.p1.3.m3.1.1.3.cmml"><mi id="A5.SS5.p1.3.m3.1.1.3.2" xref="A5.SS5.p1.3.m3.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.3.m3.1.1.3.1" xref="A5.SS5.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.3.m3.1.1.3.3" xref="A5.SS5.p1.3.m3.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.3.m3.1.1.3.1a" xref="A5.SS5.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.3.m3.1.1.3.4" xref="A5.SS5.p1.3.m3.1.1.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.3.m3.1.1.3.1b" xref="A5.SS5.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.3.m3.1.1.3.5" xref="A5.SS5.p1.3.m3.1.1.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.3.m3.1.1.3.1c" xref="A5.SS5.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.3.m3.1.1.3.6" xref="A5.SS5.p1.3.m3.1.1.3.6.cmml">l</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.3.m3.1b"><apply id="A5.SS5.p1.3.m3.1.1.cmml" xref="A5.SS5.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS5.p1.3.m3.1.1.1.cmml" xref="A5.SS5.p1.3.m3.1.1">subscript</csymbol><ci id="A5.SS5.p1.3.m3.1.1.2.cmml" xref="A5.SS5.p1.3.m3.1.1.2">𝑑</ci><apply id="A5.SS5.p1.3.m3.1.1.3.cmml" xref="A5.SS5.p1.3.m3.1.1.3"><times id="A5.SS5.p1.3.m3.1.1.3.1.cmml" xref="A5.SS5.p1.3.m3.1.1.3.1"></times><ci id="A5.SS5.p1.3.m3.1.1.3.2.cmml" xref="A5.SS5.p1.3.m3.1.1.3.2">𝑚</ci><ci id="A5.SS5.p1.3.m3.1.1.3.3.cmml" xref="A5.SS5.p1.3.m3.1.1.3.3">𝑜</ci><ci id="A5.SS5.p1.3.m3.1.1.3.4.cmml" xref="A5.SS5.p1.3.m3.1.1.3.4">𝑑</ci><ci id="A5.SS5.p1.3.m3.1.1.3.5.cmml" xref="A5.SS5.p1.3.m3.1.1.3.5">𝑒</ci><ci id="A5.SS5.p1.3.m3.1.1.3.6.cmml" xref="A5.SS5.p1.3.m3.1.1.3.6">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.3.m3.1c">d_{model}</annotation></semantics></math>, this “principal component”-like solution may become competitive with solutions where each latent is used sparsely. In order to check for such solutions, we can simply measure the average density of the <math id="A5.SS5.p1.4.m4.1" class="ltx_Math" alttext="d_{model}" display="inline"><semantics id="A5.SS5.p1.4.m4.1a"><msub id="A5.SS5.p1.4.m4.1.1" xref="A5.SS5.p1.4.m4.1.1.cmml"><mi id="A5.SS5.p1.4.m4.1.1.2" xref="A5.SS5.p1.4.m4.1.1.2.cmml">d</mi><mrow id="A5.SS5.p1.4.m4.1.1.3" xref="A5.SS5.p1.4.m4.1.1.3.cmml"><mi id="A5.SS5.p1.4.m4.1.1.3.2" xref="A5.SS5.p1.4.m4.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.4.m4.1.1.3.1" xref="A5.SS5.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.4.m4.1.1.3.3" xref="A5.SS5.p1.4.m4.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.4.m4.1.1.3.1a" xref="A5.SS5.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.4.m4.1.1.3.4" xref="A5.SS5.p1.4.m4.1.1.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.4.m4.1.1.3.1b" xref="A5.SS5.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.4.m4.1.1.3.5" xref="A5.SS5.p1.4.m4.1.1.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p1.4.m4.1.1.3.1c" xref="A5.SS5.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="A5.SS5.p1.4.m4.1.1.3.6" xref="A5.SS5.p1.4.m4.1.1.3.6.cmml">l</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.4.m4.1b"><apply id="A5.SS5.p1.4.m4.1.1.cmml" xref="A5.SS5.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A5.SS5.p1.4.m4.1.1.1.cmml" xref="A5.SS5.p1.4.m4.1.1">subscript</csymbol><ci id="A5.SS5.p1.4.m4.1.1.2.cmml" xref="A5.SS5.p1.4.m4.1.1.2">𝑑</ci><apply id="A5.SS5.p1.4.m4.1.1.3.cmml" xref="A5.SS5.p1.4.m4.1.1.3"><times id="A5.SS5.p1.4.m4.1.1.3.1.cmml" xref="A5.SS5.p1.4.m4.1.1.3.1"></times><ci id="A5.SS5.p1.4.m4.1.1.3.2.cmml" xref="A5.SS5.p1.4.m4.1.1.3.2">𝑚</ci><ci id="A5.SS5.p1.4.m4.1.1.3.3.cmml" xref="A5.SS5.p1.4.m4.1.1.3.3">𝑜</ci><ci id="A5.SS5.p1.4.m4.1.1.3.4.cmml" xref="A5.SS5.p1.4.m4.1.1.3.4">𝑑</ci><ci id="A5.SS5.p1.4.m4.1.1.3.5.cmml" xref="A5.SS5.p1.4.m4.1.1.3.5">𝑒</ci><ci id="A5.SS5.p1.4.m4.1.1.3.6.cmml" xref="A5.SS5.p1.4.m4.1.1.3.6">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.4.m4.1c">d_{model}</annotation></semantics></math> densest latents. Using this metric, we find that for some hyperparameter settings, GPT-2 small autoencoders find solutions with many dense latents (<a href="#A5.F20" title="Figure 20 ‣ E.5 Solutions with dense latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 20</span></a>), beginning around <math id="A5.SS5.p1.5.m5.1" class="ltx_Math" alttext="k=256" display="inline"><semantics id="A5.SS5.p1.5.m5.1a"><mrow id="A5.SS5.p1.5.m5.1.1" xref="A5.SS5.p1.5.m5.1.1.cmml"><mi id="A5.SS5.p1.5.m5.1.1.2" xref="A5.SS5.p1.5.m5.1.1.2.cmml">k</mi><mo id="A5.SS5.p1.5.m5.1.1.1" xref="A5.SS5.p1.5.m5.1.1.1.cmml">=</mo><mn id="A5.SS5.p1.5.m5.1.1.3" xref="A5.SS5.p1.5.m5.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.5.m5.1b"><apply id="A5.SS5.p1.5.m5.1.1.cmml" xref="A5.SS5.p1.5.m5.1.1"><eq id="A5.SS5.p1.5.m5.1.1.1.cmml" xref="A5.SS5.p1.5.m5.1.1.1"></eq><ci id="A5.SS5.p1.5.m5.1.1.2.cmml" xref="A5.SS5.p1.5.m5.1.1.2">𝑘</ci><cn type="integer" id="A5.SS5.p1.5.m5.1.1.3.cmml" xref="A5.SS5.p1.5.m5.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.5.m5.1c">k=256</annotation></semantics></math> but especially for <math id="A5.SS5.p1.6.m6.1" class="ltx_Math" alttext="k=512" display="inline"><semantics id="A5.SS5.p1.6.m6.1a"><mrow id="A5.SS5.p1.6.m6.1.1" xref="A5.SS5.p1.6.m6.1.1.cmml"><mi id="A5.SS5.p1.6.m6.1.1.2" xref="A5.SS5.p1.6.m6.1.1.2.cmml">k</mi><mo id="A5.SS5.p1.6.m6.1.1.1" xref="A5.SS5.p1.6.m6.1.1.1.cmml">=</mo><mn id="A5.SS5.p1.6.m6.1.1.3" xref="A5.SS5.p1.6.m6.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.6.m6.1b"><apply id="A5.SS5.p1.6.m6.1.1.cmml" xref="A5.SS5.p1.6.m6.1.1"><eq id="A5.SS5.p1.6.m6.1.1.1.cmml" xref="A5.SS5.p1.6.m6.1.1.1"></eq><ci id="A5.SS5.p1.6.m6.1.1.2.cmml" xref="A5.SS5.p1.6.m6.1.1.2">𝑘</ci><cn type="integer" id="A5.SS5.p1.6.m6.1.1.3.cmml" xref="A5.SS5.p1.6.m6.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.6.m6.1c">k=512</annotation></semantics></math>.</p>
</div>
<div id="A5.SS5.p2" class="ltx_para">
<p id="A5.SS5.p2.3" class="ltx_p">This coincides with when the scaling laws from <a href="#S3" title="3 Scaling laws ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3</span></a> begin to break - many of our trends between <math id="A5.SS5.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p2.1.m1.1a"><mi id="A5.SS5.p2.1.m1.1.1" xref="A5.SS5.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.1.m1.1b"><ci id="A5.SS5.p2.1.m1.1.1.cmml" xref="A5.SS5.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.1.m1.1c">k</annotation></semantics></math> and reconstruction loss bend significantly. Also, MSE becomes significantly less sensitive to <math id="A5.SS5.p2.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A5.SS5.p2.2.m2.1a"><mi id="A5.SS5.p2.2.m2.1.1" xref="A5.SS5.p2.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.2.m2.1b"><ci id="A5.SS5.p2.2.m2.1.1.cmml" xref="A5.SS5.p2.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.2.m2.1c">n</annotation></semantics></math>, at <math id="A5.SS5.p2.3.m3.1" class="ltx_Math" alttext="k=512" display="inline"><semantics id="A5.SS5.p2.3.m3.1a"><mrow id="A5.SS5.p2.3.m3.1.1" xref="A5.SS5.p2.3.m3.1.1.cmml"><mi id="A5.SS5.p2.3.m3.1.1.2" xref="A5.SS5.p2.3.m3.1.1.2.cmml">k</mi><mo id="A5.SS5.p2.3.m3.1.1.1" xref="A5.SS5.p2.3.m3.1.1.1.cmml">=</mo><mn id="A5.SS5.p2.3.m3.1.1.3" xref="A5.SS5.p2.3.m3.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.3.m3.1b"><apply id="A5.SS5.p2.3.m3.1.1.cmml" xref="A5.SS5.p2.3.m3.1.1"><eq id="A5.SS5.p2.3.m3.1.1.1.cmml" xref="A5.SS5.p2.3.m3.1.1.1"></eq><ci id="A5.SS5.p2.3.m3.1.1.2.cmml" xref="A5.SS5.p2.3.m3.1.1.2">𝑘</ci><cn type="integer" id="A5.SS5.p2.3.m3.1.1.3.cmml" xref="A5.SS5.p2.3.m3.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.3.m3.1c">k=512</annotation></semantics></math>.</p>
</div>
<figure id="A5.F20" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_memorization.png" id="A5.F20.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="419" height="233" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F20.13.6.1" class="ltx_text" style="font-size:90%;">Figure 20</span>: </span><span id="A5.F20.10.5" class="ltx_text" style="font-size:90%;">Average density of the <math id="A5.F20.6.1.m1.1" class="ltx_Math" alttext="d_{model}" display="inline"><semantics id="A5.F20.6.1.m1.1b"><msub id="A5.F20.6.1.m1.1.1" xref="A5.F20.6.1.m1.1.1.cmml"><mi id="A5.F20.6.1.m1.1.1.2" xref="A5.F20.6.1.m1.1.1.2.cmml">d</mi><mrow id="A5.F20.6.1.m1.1.1.3" xref="A5.F20.6.1.m1.1.1.3.cmml"><mi id="A5.F20.6.1.m1.1.1.3.2" xref="A5.F20.6.1.m1.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.F20.6.1.m1.1.1.3.1" xref="A5.F20.6.1.m1.1.1.3.1.cmml">​</mo><mi id="A5.F20.6.1.m1.1.1.3.3" xref="A5.F20.6.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A5.F20.6.1.m1.1.1.3.1b" xref="A5.F20.6.1.m1.1.1.3.1.cmml">​</mo><mi id="A5.F20.6.1.m1.1.1.3.4" xref="A5.F20.6.1.m1.1.1.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="A5.F20.6.1.m1.1.1.3.1c" xref="A5.F20.6.1.m1.1.1.3.1.cmml">​</mo><mi id="A5.F20.6.1.m1.1.1.3.5" xref="A5.F20.6.1.m1.1.1.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="A5.F20.6.1.m1.1.1.3.1d" xref="A5.F20.6.1.m1.1.1.3.1.cmml">​</mo><mi id="A5.F20.6.1.m1.1.1.3.6" xref="A5.F20.6.1.m1.1.1.3.6.cmml">l</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.F20.6.1.m1.1c"><apply id="A5.F20.6.1.m1.1.1.cmml" xref="A5.F20.6.1.m1.1.1"><csymbol cd="ambiguous" id="A5.F20.6.1.m1.1.1.1.cmml" xref="A5.F20.6.1.m1.1.1">subscript</csymbol><ci id="A5.F20.6.1.m1.1.1.2.cmml" xref="A5.F20.6.1.m1.1.1.2">𝑑</ci><apply id="A5.F20.6.1.m1.1.1.3.cmml" xref="A5.F20.6.1.m1.1.1.3"><times id="A5.F20.6.1.m1.1.1.3.1.cmml" xref="A5.F20.6.1.m1.1.1.3.1"></times><ci id="A5.F20.6.1.m1.1.1.3.2.cmml" xref="A5.F20.6.1.m1.1.1.3.2">𝑚</ci><ci id="A5.F20.6.1.m1.1.1.3.3.cmml" xref="A5.F20.6.1.m1.1.1.3.3">𝑜</ci><ci id="A5.F20.6.1.m1.1.1.3.4.cmml" xref="A5.F20.6.1.m1.1.1.3.4">𝑑</ci><ci id="A5.F20.6.1.m1.1.1.3.5.cmml" xref="A5.F20.6.1.m1.1.1.3.5">𝑒</ci><ci id="A5.F20.6.1.m1.1.1.3.6.cmml" xref="A5.F20.6.1.m1.1.1.3.6">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F20.6.1.m1.1d">d_{model}</annotation></semantics></math> most-dense features, divided by L<sub id="A5.F20.10.5.1" class="ltx_sub">0</sub>, for different autoencoders. When <math id="A5.F20.8.3.m3.1" class="ltx_Math" alttext="k=512" display="inline"><semantics id="A5.F20.8.3.m3.1b"><mrow id="A5.F20.8.3.m3.1.1" xref="A5.F20.8.3.m3.1.1.cmml"><mi id="A5.F20.8.3.m3.1.1.2" xref="A5.F20.8.3.m3.1.1.2.cmml">k</mi><mo id="A5.F20.8.3.m3.1.1.1" xref="A5.F20.8.3.m3.1.1.1.cmml">=</mo><mn id="A5.F20.8.3.m3.1.1.3" xref="A5.F20.8.3.m3.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.F20.8.3.m3.1c"><apply id="A5.F20.8.3.m3.1.1.cmml" xref="A5.F20.8.3.m3.1.1"><eq id="A5.F20.8.3.m3.1.1.1.cmml" xref="A5.F20.8.3.m3.1.1.1"></eq><ci id="A5.F20.8.3.m3.1.1.2.cmml" xref="A5.F20.8.3.m3.1.1.2">𝑘</ci><cn type="integer" id="A5.F20.8.3.m3.1.1.3.cmml" xref="A5.F20.8.3.m3.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F20.8.3.m3.1d">k=512</annotation></semantics></math>, the learned autoencoders have many dense features. This corresponds to when ablations stop having sparse effects <a href="#S4.SS5" title="4.5 Sparsity of ablation effects ‣ 4 Evaluation ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.5</span></a>, and anecdotally corresponds to noticeably less interpretable features. For <math id="A5.F20.9.4.m4.1" class="ltx_Math" alttext="k=256" display="inline"><semantics id="A5.F20.9.4.m4.1b"><mrow id="A5.F20.9.4.m4.1.1" xref="A5.F20.9.4.m4.1.1.cmml"><mi id="A5.F20.9.4.m4.1.1.2" xref="A5.F20.9.4.m4.1.1.2.cmml">k</mi><mo id="A5.F20.9.4.m4.1.1.1" xref="A5.F20.9.4.m4.1.1.1.cmml">=</mo><mn id="A5.F20.9.4.m4.1.1.3" xref="A5.F20.9.4.m4.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.F20.9.4.m4.1c"><apply id="A5.F20.9.4.m4.1.1.cmml" xref="A5.F20.9.4.m4.1.1"><eq id="A5.F20.9.4.m4.1.1.1.cmml" xref="A5.F20.9.4.m4.1.1.1"></eq><ci id="A5.F20.9.4.m4.1.1.2.cmml" xref="A5.F20.9.4.m4.1.1.2">𝑘</ci><cn type="integer" id="A5.F20.9.4.m4.1.1.3.cmml" xref="A5.F20.9.4.m4.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F20.9.4.m4.1d">k=256</annotation></semantics></math>, <math id="A5.F20.10.5.m5.1" class="ltx_Math" alttext="n=2048" display="inline"><semantics id="A5.F20.10.5.m5.1b"><mrow id="A5.F20.10.5.m5.1.1" xref="A5.F20.10.5.m5.1.1.cmml"><mi id="A5.F20.10.5.m5.1.1.2" xref="A5.F20.10.5.m5.1.1.2.cmml">n</mi><mo id="A5.F20.10.5.m5.1.1.1" xref="A5.F20.10.5.m5.1.1.1.cmml">=</mo><mn id="A5.F20.10.5.m5.1.1.3" xref="A5.F20.10.5.m5.1.1.3.cmml">2048</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.F20.10.5.m5.1c"><apply id="A5.F20.10.5.m5.1.1.cmml" xref="A5.F20.10.5.m5.1.1"><eq id="A5.F20.10.5.m5.1.1.1.cmml" xref="A5.F20.10.5.m5.1.1.1"></eq><ci id="A5.F20.10.5.m5.1.1.2.cmml" xref="A5.F20.10.5.m5.1.1.2">𝑛</ci><cn type="integer" id="A5.F20.10.5.m5.1.1.3.cmml" xref="A5.F20.10.5.m5.1.1.3">2048</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F20.10.5.m5.1d">n=2048</annotation></semantics></math>, there is perhaps an intermediate regime.</span></figcaption>
</figure>
<figure id="A5.F21" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_explanation_score.png" id="A5.F21.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="167" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F21.6.3.1" class="ltx_text" style="font-size:90%;">Figure 21</span>: </span><span id="A5.F21.4.2" class="ltx_text" style="font-size:90%;">Explanation scores for GPT-2 small autoencoders of different <math id="A5.F21.3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A5.F21.3.1.m1.1b"><mi id="A5.F21.3.1.m1.1.1" xref="A5.F21.3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A5.F21.3.1.m1.1c"><ci id="A5.F21.3.1.m1.1.1.cmml" xref="A5.F21.3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.F21.3.1.m1.1d">n</annotation></semantics></math> and <math id="A5.F21.4.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.F21.4.2.m2.1b"><mi id="A5.F21.4.2.m2.1.1" xref="A5.F21.4.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.F21.4.2.m2.1c"><ci id="A5.F21.4.2.m2.1.1.cmml" xref="A5.F21.4.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.F21.4.2.m2.1d">k</annotation></semantics></math>, evaluated on 400 randomly chosen latents per autoencoder. It is hard to read off trends, but the explanation score is able to somewhat detect the dense solutions region.</span></figcaption>
</figure>
</section>
<section id="A5.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.6 </span>Recurring dense features in GPT-2 small</h3>

<div id="A5.SS6.p1" class="ltx_para">
<p id="A5.SS6.p1.1" class="ltx_p">We manually examined the densest latents across various GPT-2 small layer 8 autoencoders, trained in different ways (e.g. differing numbers of total latents).</p>
</div>
<div id="A5.SS6.p2" class="ltx_para">
<p id="A5.SS6.p2.2" class="ltx_p">The two densest latents are always the same feature: the latent simply activates more and more later in the context (<math id="A5.SS6.p2.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="A5.SS6.p2.1.m1.1a"><mo id="A5.SS6.p2.1.m1.1.1" xref="A5.SS6.p2.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.1.m1.1b"><csymbol cd="latexml" id="A5.SS6.p2.1.m1.1.1.cmml" xref="A5.SS6.p2.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.1.m1.1c">\sim</annotation></semantics></math> 40% active), and one that activates more and more earlier in the context excluding the first position (<math id="A5.SS6.p2.2.m2.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="A5.SS6.p2.2.m2.1a"><mo id="A5.SS6.p2.2.m2.1.1" xref="A5.SS6.p2.2.m2.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="A5.SS6.p2.2.m2.1b"><csymbol cd="latexml" id="A5.SS6.p2.2.m2.1.1.cmml" xref="A5.SS6.p2.2.m2.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p2.2.m2.1c">\sim</annotation></semantics></math>35% active). Both features look like they want to activate more than they do, with TopK probably preventing it from activating with lower values.</p>
</div>
<div id="A5.SS6.p3" class="ltx_para">
<p id="A5.SS6.p3.1" class="ltx_p">The third densest latent is always a first-token-position feature (<math id="A5.SS6.p3.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="A5.SS6.p3.1.m1.1a"><mo id="A5.SS6.p3.1.m1.1.1" xref="A5.SS6.p3.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="A5.SS6.p3.1.m1.1b"><csymbol cd="latexml" id="A5.SS6.p3.1.m1.1.1.cmml" xref="A5.SS6.p3.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p3.1.m1.1c">\sim</annotation></semantics></math>30% active), which has a modal activation value in a narrow range between 14.6-14.8. Most of its activation values are significantly smaller values, at tokens after the first position; the large value is always at the first token. These smaller values appear uninterpretable; we conjecture these are simply interference with the first position direction. (Sometimes there are two of these latents, the second with smaller activation values.)</p>
</div>
<div id="A5.SS6.p4" class="ltx_para">
<p id="A5.SS6.p4.1" class="ltx_p">Finally, there is a recurring “repetition” feature that is <math id="A5.SS6.p4.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="A5.SS6.p4.1.m1.1a"><mo id="A5.SS6.p4.1.m1.1.1" xref="A5.SS6.p4.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="A5.SS6.p4.1.m1.1b"><csymbol cd="latexml" id="A5.SS6.p4.1.m1.1.1.cmml" xref="A5.SS6.p4.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p4.1.m1.1c">\sim</annotation></semantics></math>20% dense. Its top activations are mostly highly repetitive sequences, such as series of dates, chapter indices, numbers, punctuations, repeated exact phrases, or other repetitive things such as Chess PGN notation. However, like the first-token-position latents, random activations of this latent are typically appear unrelated and uninterpretable.</p>
</div>
<div id="A5.SS6.p5" class="ltx_para">
<p id="A5.SS6.p5.1" class="ltx_p">Often in the top ten densest latents, we find opposing latents, which have decoder cosine similarity close to <math id="A5.SS6.p5.1.m1.1" class="ltx_Math" alttext="-1" display="inline"><semantics id="A5.SS6.p5.1.m1.1a"><mrow id="A5.SS6.p5.1.m1.1.1" xref="A5.SS6.p5.1.m1.1.1.cmml"><mo id="A5.SS6.p5.1.m1.1.1a" xref="A5.SS6.p5.1.m1.1.1.cmml">−</mo><mn id="A5.SS6.p5.1.m1.1.1.2" xref="A5.SS6.p5.1.m1.1.1.2.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS6.p5.1.m1.1b"><apply id="A5.SS6.p5.1.m1.1.1.cmml" xref="A5.SS6.p5.1.m1.1.1"><minus id="A5.SS6.p5.1.m1.1.1.1.cmml" xref="A5.SS6.p5.1.m1.1.1"></minus><cn type="integer" id="A5.SS6.p5.1.m1.1.1.2.cmml" xref="A5.SS6.p5.1.m1.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS6.p5.1.m1.1c">-1</annotation></semantics></math>. In particular, the first-token-position feature and the repetition latent both seems to always have an opposite latent. The less dense of the two opposite latents always seems to appear uninterpretable. We conjecture that these are symptoms of optimization failure - the opposite latents cancel out spurious activations in the denser latent.</p>
</div>
</section>
<section id="A5.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.7 </span>Clustering latents</h3>

<div id="A5.SS7.p1" class="ltx_para">
<p id="A5.SS7.p1.12" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citep">[Elhage et al., <a href="#bib.bib13" title="" class="ltx_ref">2022</a>]</cite> discuss how underlying features may lie in distinct sub-spaces.
If such sub-spaces exists, we hypothesize that the set of latent encoding vectors <math id="A5.SS7.p1.1.m1.1" class="ltx_Math" alttext="W\in\mathbb{R}^{n\times d}" display="inline"><semantics id="A5.SS7.p1.1.m1.1a"><mrow id="A5.SS7.p1.1.m1.1.1" xref="A5.SS7.p1.1.m1.1.1.cmml"><mi id="A5.SS7.p1.1.m1.1.1.2" xref="A5.SS7.p1.1.m1.1.1.2.cmml">W</mi><mo id="A5.SS7.p1.1.m1.1.1.1" xref="A5.SS7.p1.1.m1.1.1.1.cmml">∈</mo><msup id="A5.SS7.p1.1.m1.1.1.3" xref="A5.SS7.p1.1.m1.1.1.3.cmml"><mi id="A5.SS7.p1.1.m1.1.1.3.2" xref="A5.SS7.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="A5.SS7.p1.1.m1.1.1.3.3" xref="A5.SS7.p1.1.m1.1.1.3.3.cmml"><mi id="A5.SS7.p1.1.m1.1.1.3.3.2" xref="A5.SS7.p1.1.m1.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="A5.SS7.p1.1.m1.1.1.3.3.1" xref="A5.SS7.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="A5.SS7.p1.1.m1.1.1.3.3.3" xref="A5.SS7.p1.1.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.1.m1.1b"><apply id="A5.SS7.p1.1.m1.1.1.cmml" xref="A5.SS7.p1.1.m1.1.1"><in id="A5.SS7.p1.1.m1.1.1.1.cmml" xref="A5.SS7.p1.1.m1.1.1.1"></in><ci id="A5.SS7.p1.1.m1.1.1.2.cmml" xref="A5.SS7.p1.1.m1.1.1.2">𝑊</ci><apply id="A5.SS7.p1.1.m1.1.1.3.cmml" xref="A5.SS7.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A5.SS7.p1.1.m1.1.1.3.1.cmml" xref="A5.SS7.p1.1.m1.1.1.3">superscript</csymbol><ci id="A5.SS7.p1.1.m1.1.1.3.2.cmml" xref="A5.SS7.p1.1.m1.1.1.3.2">ℝ</ci><apply id="A5.SS7.p1.1.m1.1.1.3.3.cmml" xref="A5.SS7.p1.1.m1.1.1.3.3"><times id="A5.SS7.p1.1.m1.1.1.3.3.1.cmml" xref="A5.SS7.p1.1.m1.1.1.3.3.1"></times><ci id="A5.SS7.p1.1.m1.1.1.3.3.2.cmml" xref="A5.SS7.p1.1.m1.1.1.3.3.2">𝑛</ci><ci id="A5.SS7.p1.1.m1.1.1.3.3.3.cmml" xref="A5.SS7.p1.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.1.m1.1c">W\in\mathbb{R}^{n\times d}</annotation></semantics></math> can be written as a block-diagonal matrix <math id="A5.SS7.p1.2.m2.1" class="ltx_Math" alttext="W^{\prime}=PWR" display="inline"><semantics id="A5.SS7.p1.2.m2.1a"><mrow id="A5.SS7.p1.2.m2.1.1" xref="A5.SS7.p1.2.m2.1.1.cmml"><msup id="A5.SS7.p1.2.m2.1.1.2" xref="A5.SS7.p1.2.m2.1.1.2.cmml"><mi id="A5.SS7.p1.2.m2.1.1.2.2" xref="A5.SS7.p1.2.m2.1.1.2.2.cmml">W</mi><mo id="A5.SS7.p1.2.m2.1.1.2.3" xref="A5.SS7.p1.2.m2.1.1.2.3.cmml">′</mo></msup><mo id="A5.SS7.p1.2.m2.1.1.1" xref="A5.SS7.p1.2.m2.1.1.1.cmml">=</mo><mrow id="A5.SS7.p1.2.m2.1.1.3" xref="A5.SS7.p1.2.m2.1.1.3.cmml"><mi id="A5.SS7.p1.2.m2.1.1.3.2" xref="A5.SS7.p1.2.m2.1.1.3.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="A5.SS7.p1.2.m2.1.1.3.1" xref="A5.SS7.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="A5.SS7.p1.2.m2.1.1.3.3" xref="A5.SS7.p1.2.m2.1.1.3.3.cmml">W</mi><mo lspace="0em" rspace="0em" id="A5.SS7.p1.2.m2.1.1.3.1a" xref="A5.SS7.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="A5.SS7.p1.2.m2.1.1.3.4" xref="A5.SS7.p1.2.m2.1.1.3.4.cmml">R</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.2.m2.1b"><apply id="A5.SS7.p1.2.m2.1.1.cmml" xref="A5.SS7.p1.2.m2.1.1"><eq id="A5.SS7.p1.2.m2.1.1.1.cmml" xref="A5.SS7.p1.2.m2.1.1.1"></eq><apply id="A5.SS7.p1.2.m2.1.1.2.cmml" xref="A5.SS7.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A5.SS7.p1.2.m2.1.1.2.1.cmml" xref="A5.SS7.p1.2.m2.1.1.2">superscript</csymbol><ci id="A5.SS7.p1.2.m2.1.1.2.2.cmml" xref="A5.SS7.p1.2.m2.1.1.2.2">𝑊</ci><ci id="A5.SS7.p1.2.m2.1.1.2.3.cmml" xref="A5.SS7.p1.2.m2.1.1.2.3">′</ci></apply><apply id="A5.SS7.p1.2.m2.1.1.3.cmml" xref="A5.SS7.p1.2.m2.1.1.3"><times id="A5.SS7.p1.2.m2.1.1.3.1.cmml" xref="A5.SS7.p1.2.m2.1.1.3.1"></times><ci id="A5.SS7.p1.2.m2.1.1.3.2.cmml" xref="A5.SS7.p1.2.m2.1.1.3.2">𝑃</ci><ci id="A5.SS7.p1.2.m2.1.1.3.3.cmml" xref="A5.SS7.p1.2.m2.1.1.3.3">𝑊</ci><ci id="A5.SS7.p1.2.m2.1.1.3.4.cmml" xref="A5.SS7.p1.2.m2.1.1.3.4">𝑅</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.2.m2.1c">W^{\prime}=PWR</annotation></semantics></math>, where <math id="A5.SS7.p1.3.m3.1" class="ltx_Math" alttext="P\in\mathbb{R}^{n\times n}" display="inline"><semantics id="A5.SS7.p1.3.m3.1a"><mrow id="A5.SS7.p1.3.m3.1.1" xref="A5.SS7.p1.3.m3.1.1.cmml"><mi id="A5.SS7.p1.3.m3.1.1.2" xref="A5.SS7.p1.3.m3.1.1.2.cmml">P</mi><mo id="A5.SS7.p1.3.m3.1.1.1" xref="A5.SS7.p1.3.m3.1.1.1.cmml">∈</mo><msup id="A5.SS7.p1.3.m3.1.1.3" xref="A5.SS7.p1.3.m3.1.1.3.cmml"><mi id="A5.SS7.p1.3.m3.1.1.3.2" xref="A5.SS7.p1.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="A5.SS7.p1.3.m3.1.1.3.3" xref="A5.SS7.p1.3.m3.1.1.3.3.cmml"><mi id="A5.SS7.p1.3.m3.1.1.3.3.2" xref="A5.SS7.p1.3.m3.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="A5.SS7.p1.3.m3.1.1.3.3.1" xref="A5.SS7.p1.3.m3.1.1.3.3.1.cmml">×</mo><mi id="A5.SS7.p1.3.m3.1.1.3.3.3" xref="A5.SS7.p1.3.m3.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.3.m3.1b"><apply id="A5.SS7.p1.3.m3.1.1.cmml" xref="A5.SS7.p1.3.m3.1.1"><in id="A5.SS7.p1.3.m3.1.1.1.cmml" xref="A5.SS7.p1.3.m3.1.1.1"></in><ci id="A5.SS7.p1.3.m3.1.1.2.cmml" xref="A5.SS7.p1.3.m3.1.1.2">𝑃</ci><apply id="A5.SS7.p1.3.m3.1.1.3.cmml" xref="A5.SS7.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A5.SS7.p1.3.m3.1.1.3.1.cmml" xref="A5.SS7.p1.3.m3.1.1.3">superscript</csymbol><ci id="A5.SS7.p1.3.m3.1.1.3.2.cmml" xref="A5.SS7.p1.3.m3.1.1.3.2">ℝ</ci><apply id="A5.SS7.p1.3.m3.1.1.3.3.cmml" xref="A5.SS7.p1.3.m3.1.1.3.3"><times id="A5.SS7.p1.3.m3.1.1.3.3.1.cmml" xref="A5.SS7.p1.3.m3.1.1.3.3.1"></times><ci id="A5.SS7.p1.3.m3.1.1.3.3.2.cmml" xref="A5.SS7.p1.3.m3.1.1.3.3.2">𝑛</ci><ci id="A5.SS7.p1.3.m3.1.1.3.3.3.cmml" xref="A5.SS7.p1.3.m3.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.3.m3.1c">P\in\mathbb{R}^{n\times n}</annotation></semantics></math> is a permutation matrix, and <math id="A5.SS7.p1.4.m4.1" class="ltx_Math" alttext="R\in\mathbb{R}^{d\times d}" display="inline"><semantics id="A5.SS7.p1.4.m4.1a"><mrow id="A5.SS7.p1.4.m4.1.1" xref="A5.SS7.p1.4.m4.1.1.cmml"><mi id="A5.SS7.p1.4.m4.1.1.2" xref="A5.SS7.p1.4.m4.1.1.2.cmml">R</mi><mo id="A5.SS7.p1.4.m4.1.1.1" xref="A5.SS7.p1.4.m4.1.1.1.cmml">∈</mo><msup id="A5.SS7.p1.4.m4.1.1.3" xref="A5.SS7.p1.4.m4.1.1.3.cmml"><mi id="A5.SS7.p1.4.m4.1.1.3.2" xref="A5.SS7.p1.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="A5.SS7.p1.4.m4.1.1.3.3" xref="A5.SS7.p1.4.m4.1.1.3.3.cmml"><mi id="A5.SS7.p1.4.m4.1.1.3.3.2" xref="A5.SS7.p1.4.m4.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="A5.SS7.p1.4.m4.1.1.3.3.1" xref="A5.SS7.p1.4.m4.1.1.3.3.1.cmml">×</mo><mi id="A5.SS7.p1.4.m4.1.1.3.3.3" xref="A5.SS7.p1.4.m4.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.4.m4.1b"><apply id="A5.SS7.p1.4.m4.1.1.cmml" xref="A5.SS7.p1.4.m4.1.1"><in id="A5.SS7.p1.4.m4.1.1.1.cmml" xref="A5.SS7.p1.4.m4.1.1.1"></in><ci id="A5.SS7.p1.4.m4.1.1.2.cmml" xref="A5.SS7.p1.4.m4.1.1.2">𝑅</ci><apply id="A5.SS7.p1.4.m4.1.1.3.cmml" xref="A5.SS7.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="A5.SS7.p1.4.m4.1.1.3.1.cmml" xref="A5.SS7.p1.4.m4.1.1.3">superscript</csymbol><ci id="A5.SS7.p1.4.m4.1.1.3.2.cmml" xref="A5.SS7.p1.4.m4.1.1.3.2">ℝ</ci><apply id="A5.SS7.p1.4.m4.1.1.3.3.cmml" xref="A5.SS7.p1.4.m4.1.1.3.3"><times id="A5.SS7.p1.4.m4.1.1.3.3.1.cmml" xref="A5.SS7.p1.4.m4.1.1.3.3.1"></times><ci id="A5.SS7.p1.4.m4.1.1.3.3.2.cmml" xref="A5.SS7.p1.4.m4.1.1.3.3.2">𝑑</ci><ci id="A5.SS7.p1.4.m4.1.1.3.3.3.cmml" xref="A5.SS7.p1.4.m4.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.4.m4.1c">R\in\mathbb{R}^{d\times d}</annotation></semantics></math> is orthogonal.
We can then use the singular vector decomposition (SVD) to write <math id="A5.SS7.p1.5.m5.1" class="ltx_Math" alttext="W=U\Sigma V^{\top}" display="inline"><semantics id="A5.SS7.p1.5.m5.1a"><mrow id="A5.SS7.p1.5.m5.1.1" xref="A5.SS7.p1.5.m5.1.1.cmml"><mi id="A5.SS7.p1.5.m5.1.1.2" xref="A5.SS7.p1.5.m5.1.1.2.cmml">W</mi><mo id="A5.SS7.p1.5.m5.1.1.1" xref="A5.SS7.p1.5.m5.1.1.1.cmml">=</mo><mrow id="A5.SS7.p1.5.m5.1.1.3" xref="A5.SS7.p1.5.m5.1.1.3.cmml"><mi id="A5.SS7.p1.5.m5.1.1.3.2" xref="A5.SS7.p1.5.m5.1.1.3.2.cmml">U</mi><mo lspace="0em" rspace="0em" id="A5.SS7.p1.5.m5.1.1.3.1" xref="A5.SS7.p1.5.m5.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="A5.SS7.p1.5.m5.1.1.3.3" xref="A5.SS7.p1.5.m5.1.1.3.3.cmml">Σ</mi><mo lspace="0em" rspace="0em" id="A5.SS7.p1.5.m5.1.1.3.1a" xref="A5.SS7.p1.5.m5.1.1.3.1.cmml">​</mo><msup id="A5.SS7.p1.5.m5.1.1.3.4" xref="A5.SS7.p1.5.m5.1.1.3.4.cmml"><mi id="A5.SS7.p1.5.m5.1.1.3.4.2" xref="A5.SS7.p1.5.m5.1.1.3.4.2.cmml">V</mi><mo id="A5.SS7.p1.5.m5.1.1.3.4.3" xref="A5.SS7.p1.5.m5.1.1.3.4.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.5.m5.1b"><apply id="A5.SS7.p1.5.m5.1.1.cmml" xref="A5.SS7.p1.5.m5.1.1"><eq id="A5.SS7.p1.5.m5.1.1.1.cmml" xref="A5.SS7.p1.5.m5.1.1.1"></eq><ci id="A5.SS7.p1.5.m5.1.1.2.cmml" xref="A5.SS7.p1.5.m5.1.1.2">𝑊</ci><apply id="A5.SS7.p1.5.m5.1.1.3.cmml" xref="A5.SS7.p1.5.m5.1.1.3"><times id="A5.SS7.p1.5.m5.1.1.3.1.cmml" xref="A5.SS7.p1.5.m5.1.1.3.1"></times><ci id="A5.SS7.p1.5.m5.1.1.3.2.cmml" xref="A5.SS7.p1.5.m5.1.1.3.2">𝑈</ci><ci id="A5.SS7.p1.5.m5.1.1.3.3.cmml" xref="A5.SS7.p1.5.m5.1.1.3.3">Σ</ci><apply id="A5.SS7.p1.5.m5.1.1.3.4.cmml" xref="A5.SS7.p1.5.m5.1.1.3.4"><csymbol cd="ambiguous" id="A5.SS7.p1.5.m5.1.1.3.4.1.cmml" xref="A5.SS7.p1.5.m5.1.1.3.4">superscript</csymbol><ci id="A5.SS7.p1.5.m5.1.1.3.4.2.cmml" xref="A5.SS7.p1.5.m5.1.1.3.4.2">𝑉</ci><csymbol cd="latexml" id="A5.SS7.p1.5.m5.1.1.3.4.3.cmml" xref="A5.SS7.p1.5.m5.1.1.3.4.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.5.m5.1c">W=U\Sigma V^{\top}</annotation></semantics></math> and <math id="A5.SS7.p1.6.m6.2" class="ltx_Math" alttext="W^{\prime}=U^{\prime}\Sigma^{\prime}V^{\prime\top}" display="inline"><semantics id="A5.SS7.p1.6.m6.2a"><mrow id="A5.SS7.p1.6.m6.2.3" xref="A5.SS7.p1.6.m6.2.3.cmml"><msup id="A5.SS7.p1.6.m6.2.3.2" xref="A5.SS7.p1.6.m6.2.3.2.cmml"><mi id="A5.SS7.p1.6.m6.2.3.2.2" xref="A5.SS7.p1.6.m6.2.3.2.2.cmml">W</mi><mo id="A5.SS7.p1.6.m6.2.3.2.3" xref="A5.SS7.p1.6.m6.2.3.2.3.cmml">′</mo></msup><mo id="A5.SS7.p1.6.m6.2.3.1" xref="A5.SS7.p1.6.m6.2.3.1.cmml">=</mo><mrow id="A5.SS7.p1.6.m6.2.3.3" xref="A5.SS7.p1.6.m6.2.3.3.cmml"><msup id="A5.SS7.p1.6.m6.2.3.3.2" xref="A5.SS7.p1.6.m6.2.3.3.2.cmml"><mi id="A5.SS7.p1.6.m6.2.3.3.2.2" xref="A5.SS7.p1.6.m6.2.3.3.2.2.cmml">U</mi><mo id="A5.SS7.p1.6.m6.2.3.3.2.3" xref="A5.SS7.p1.6.m6.2.3.3.2.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.6.m6.2.3.3.1" xref="A5.SS7.p1.6.m6.2.3.3.1.cmml">​</mo><msup id="A5.SS7.p1.6.m6.2.3.3.3" xref="A5.SS7.p1.6.m6.2.3.3.3.cmml"><mi mathvariant="normal" id="A5.SS7.p1.6.m6.2.3.3.3.2" xref="A5.SS7.p1.6.m6.2.3.3.3.2.cmml">Σ</mi><mo id="A5.SS7.p1.6.m6.2.3.3.3.3" xref="A5.SS7.p1.6.m6.2.3.3.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.6.m6.2.3.3.1a" xref="A5.SS7.p1.6.m6.2.3.3.1.cmml">​</mo><msup id="A5.SS7.p1.6.m6.2.3.3.4" xref="A5.SS7.p1.6.m6.2.3.3.4.cmml"><mi id="A5.SS7.p1.6.m6.2.3.3.4.2" xref="A5.SS7.p1.6.m6.2.3.3.4.2.cmml">V</mi><mrow id="A5.SS7.p1.6.m6.2.2.2.2" xref="A5.SS7.p1.6.m6.2.2.2.3.cmml"><mo mathsize="142%" id="A5.SS7.p1.6.m6.2.2.2.2.1" xref="A5.SS7.p1.6.m6.2.2.2.2.1.cmml">′</mo><mo lspace="0.222em" id="A5.SS7.p1.6.m6.2.2.2.2.2" xref="A5.SS7.p1.6.m6.2.2.2.3.cmml">⁣</mo><mo id="A5.SS7.p1.6.m6.1.1.1.1" xref="A5.SS7.p1.6.m6.1.1.1.1.cmml">⊤</mo></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.6.m6.2b"><apply id="A5.SS7.p1.6.m6.2.3.cmml" xref="A5.SS7.p1.6.m6.2.3"><eq id="A5.SS7.p1.6.m6.2.3.1.cmml" xref="A5.SS7.p1.6.m6.2.3.1"></eq><apply id="A5.SS7.p1.6.m6.2.3.2.cmml" xref="A5.SS7.p1.6.m6.2.3.2"><csymbol cd="ambiguous" id="A5.SS7.p1.6.m6.2.3.2.1.cmml" xref="A5.SS7.p1.6.m6.2.3.2">superscript</csymbol><ci id="A5.SS7.p1.6.m6.2.3.2.2.cmml" xref="A5.SS7.p1.6.m6.2.3.2.2">𝑊</ci><ci id="A5.SS7.p1.6.m6.2.3.2.3.cmml" xref="A5.SS7.p1.6.m6.2.3.2.3">′</ci></apply><apply id="A5.SS7.p1.6.m6.2.3.3.cmml" xref="A5.SS7.p1.6.m6.2.3.3"><times id="A5.SS7.p1.6.m6.2.3.3.1.cmml" xref="A5.SS7.p1.6.m6.2.3.3.1"></times><apply id="A5.SS7.p1.6.m6.2.3.3.2.cmml" xref="A5.SS7.p1.6.m6.2.3.3.2"><csymbol cd="ambiguous" id="A5.SS7.p1.6.m6.2.3.3.2.1.cmml" xref="A5.SS7.p1.6.m6.2.3.3.2">superscript</csymbol><ci id="A5.SS7.p1.6.m6.2.3.3.2.2.cmml" xref="A5.SS7.p1.6.m6.2.3.3.2.2">𝑈</ci><ci id="A5.SS7.p1.6.m6.2.3.3.2.3.cmml" xref="A5.SS7.p1.6.m6.2.3.3.2.3">′</ci></apply><apply id="A5.SS7.p1.6.m6.2.3.3.3.cmml" xref="A5.SS7.p1.6.m6.2.3.3.3"><csymbol cd="ambiguous" id="A5.SS7.p1.6.m6.2.3.3.3.1.cmml" xref="A5.SS7.p1.6.m6.2.3.3.3">superscript</csymbol><ci id="A5.SS7.p1.6.m6.2.3.3.3.2.cmml" xref="A5.SS7.p1.6.m6.2.3.3.3.2">Σ</ci><ci id="A5.SS7.p1.6.m6.2.3.3.3.3.cmml" xref="A5.SS7.p1.6.m6.2.3.3.3.3">′</ci></apply><apply id="A5.SS7.p1.6.m6.2.3.3.4.cmml" xref="A5.SS7.p1.6.m6.2.3.3.4"><csymbol cd="ambiguous" id="A5.SS7.p1.6.m6.2.3.3.4.1.cmml" xref="A5.SS7.p1.6.m6.2.3.3.4">superscript</csymbol><ci id="A5.SS7.p1.6.m6.2.3.3.4.2.cmml" xref="A5.SS7.p1.6.m6.2.3.3.4.2">𝑉</ci><list id="A5.SS7.p1.6.m6.2.2.2.3.cmml" xref="A5.SS7.p1.6.m6.2.2.2.2"><ci id="A5.SS7.p1.6.m6.2.2.2.2.1.cmml" xref="A5.SS7.p1.6.m6.2.2.2.2.1">′</ci><csymbol cd="latexml" id="A5.SS7.p1.6.m6.1.1.1.1.cmml" xref="A5.SS7.p1.6.m6.1.1.1.1">top</csymbol></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.6.m6.2c">W^{\prime}=U^{\prime}\Sigma^{\prime}V^{\prime\top}</annotation></semantics></math>, noting that <math id="A5.SS7.p1.7.m7.1" class="ltx_Math" alttext="U^{\prime}" display="inline"><semantics id="A5.SS7.p1.7.m7.1a"><msup id="A5.SS7.p1.7.m7.1.1" xref="A5.SS7.p1.7.m7.1.1.cmml"><mi id="A5.SS7.p1.7.m7.1.1.2" xref="A5.SS7.p1.7.m7.1.1.2.cmml">U</mi><mo id="A5.SS7.p1.7.m7.1.1.3" xref="A5.SS7.p1.7.m7.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.7.m7.1b"><apply id="A5.SS7.p1.7.m7.1.1.cmml" xref="A5.SS7.p1.7.m7.1.1"><csymbol cd="ambiguous" id="A5.SS7.p1.7.m7.1.1.1.cmml" xref="A5.SS7.p1.7.m7.1.1">superscript</csymbol><ci id="A5.SS7.p1.7.m7.1.1.2.cmml" xref="A5.SS7.p1.7.m7.1.1.2">𝑈</ci><ci id="A5.SS7.p1.7.m7.1.1.3.cmml" xref="A5.SS7.p1.7.m7.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.7.m7.1c">U^{\prime}</annotation></semantics></math> is also block diagonal. Finally, we write <math id="A5.SS7.p1.8.m8.2" class="ltx_Math" alttext="W=P^{\top}U^{\prime}\Sigma^{\prime}V^{\prime\top}R^{\top}=U\Sigma V^{\top}" display="inline"><semantics id="A5.SS7.p1.8.m8.2a"><mrow id="A5.SS7.p1.8.m8.2.3" xref="A5.SS7.p1.8.m8.2.3.cmml"><mi id="A5.SS7.p1.8.m8.2.3.2" xref="A5.SS7.p1.8.m8.2.3.2.cmml">W</mi><mo id="A5.SS7.p1.8.m8.2.3.3" xref="A5.SS7.p1.8.m8.2.3.3.cmml">=</mo><mrow id="A5.SS7.p1.8.m8.2.3.4" xref="A5.SS7.p1.8.m8.2.3.4.cmml"><msup id="A5.SS7.p1.8.m8.2.3.4.2" xref="A5.SS7.p1.8.m8.2.3.4.2.cmml"><mi id="A5.SS7.p1.8.m8.2.3.4.2.2" xref="A5.SS7.p1.8.m8.2.3.4.2.2.cmml">P</mi><mo id="A5.SS7.p1.8.m8.2.3.4.2.3" xref="A5.SS7.p1.8.m8.2.3.4.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.8.m8.2.3.4.1" xref="A5.SS7.p1.8.m8.2.3.4.1.cmml">​</mo><msup id="A5.SS7.p1.8.m8.2.3.4.3" xref="A5.SS7.p1.8.m8.2.3.4.3.cmml"><mi id="A5.SS7.p1.8.m8.2.3.4.3.2" xref="A5.SS7.p1.8.m8.2.3.4.3.2.cmml">U</mi><mo id="A5.SS7.p1.8.m8.2.3.4.3.3" xref="A5.SS7.p1.8.m8.2.3.4.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.8.m8.2.3.4.1a" xref="A5.SS7.p1.8.m8.2.3.4.1.cmml">​</mo><msup id="A5.SS7.p1.8.m8.2.3.4.4" xref="A5.SS7.p1.8.m8.2.3.4.4.cmml"><mi mathvariant="normal" id="A5.SS7.p1.8.m8.2.3.4.4.2" xref="A5.SS7.p1.8.m8.2.3.4.4.2.cmml">Σ</mi><mo id="A5.SS7.p1.8.m8.2.3.4.4.3" xref="A5.SS7.p1.8.m8.2.3.4.4.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.8.m8.2.3.4.1b" xref="A5.SS7.p1.8.m8.2.3.4.1.cmml">​</mo><msup id="A5.SS7.p1.8.m8.2.3.4.5" xref="A5.SS7.p1.8.m8.2.3.4.5.cmml"><mi id="A5.SS7.p1.8.m8.2.3.4.5.2" xref="A5.SS7.p1.8.m8.2.3.4.5.2.cmml">V</mi><mrow id="A5.SS7.p1.8.m8.2.2.2.2" xref="A5.SS7.p1.8.m8.2.2.2.3.cmml"><mo mathsize="142%" id="A5.SS7.p1.8.m8.2.2.2.2.1" xref="A5.SS7.p1.8.m8.2.2.2.2.1.cmml">′</mo><mo lspace="0.222em" id="A5.SS7.p1.8.m8.2.2.2.2.2" xref="A5.SS7.p1.8.m8.2.2.2.3.cmml">⁣</mo><mo id="A5.SS7.p1.8.m8.1.1.1.1" xref="A5.SS7.p1.8.m8.1.1.1.1.cmml">⊤</mo></mrow></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.8.m8.2.3.4.1c" xref="A5.SS7.p1.8.m8.2.3.4.1.cmml">​</mo><msup id="A5.SS7.p1.8.m8.2.3.4.6" xref="A5.SS7.p1.8.m8.2.3.4.6.cmml"><mi id="A5.SS7.p1.8.m8.2.3.4.6.2" xref="A5.SS7.p1.8.m8.2.3.4.6.2.cmml">R</mi><mo id="A5.SS7.p1.8.m8.2.3.4.6.3" xref="A5.SS7.p1.8.m8.2.3.4.6.3.cmml">⊤</mo></msup></mrow><mo id="A5.SS7.p1.8.m8.2.3.5" xref="A5.SS7.p1.8.m8.2.3.5.cmml">=</mo><mrow id="A5.SS7.p1.8.m8.2.3.6" xref="A5.SS7.p1.8.m8.2.3.6.cmml"><mi id="A5.SS7.p1.8.m8.2.3.6.2" xref="A5.SS7.p1.8.m8.2.3.6.2.cmml">U</mi><mo lspace="0em" rspace="0em" id="A5.SS7.p1.8.m8.2.3.6.1" xref="A5.SS7.p1.8.m8.2.3.6.1.cmml">​</mo><mi mathvariant="normal" id="A5.SS7.p1.8.m8.2.3.6.3" xref="A5.SS7.p1.8.m8.2.3.6.3.cmml">Σ</mi><mo lspace="0em" rspace="0em" id="A5.SS7.p1.8.m8.2.3.6.1a" xref="A5.SS7.p1.8.m8.2.3.6.1.cmml">​</mo><msup id="A5.SS7.p1.8.m8.2.3.6.4" xref="A5.SS7.p1.8.m8.2.3.6.4.cmml"><mi id="A5.SS7.p1.8.m8.2.3.6.4.2" xref="A5.SS7.p1.8.m8.2.3.6.4.2.cmml">V</mi><mo id="A5.SS7.p1.8.m8.2.3.6.4.3" xref="A5.SS7.p1.8.m8.2.3.6.4.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.8.m8.2b"><apply id="A5.SS7.p1.8.m8.2.3.cmml" xref="A5.SS7.p1.8.m8.2.3"><and id="A5.SS7.p1.8.m8.2.3a.cmml" xref="A5.SS7.p1.8.m8.2.3"></and><apply id="A5.SS7.p1.8.m8.2.3b.cmml" xref="A5.SS7.p1.8.m8.2.3"><eq id="A5.SS7.p1.8.m8.2.3.3.cmml" xref="A5.SS7.p1.8.m8.2.3.3"></eq><ci id="A5.SS7.p1.8.m8.2.3.2.cmml" xref="A5.SS7.p1.8.m8.2.3.2">𝑊</ci><apply id="A5.SS7.p1.8.m8.2.3.4.cmml" xref="A5.SS7.p1.8.m8.2.3.4"><times id="A5.SS7.p1.8.m8.2.3.4.1.cmml" xref="A5.SS7.p1.8.m8.2.3.4.1"></times><apply id="A5.SS7.p1.8.m8.2.3.4.2.cmml" xref="A5.SS7.p1.8.m8.2.3.4.2"><csymbol cd="ambiguous" id="A5.SS7.p1.8.m8.2.3.4.2.1.cmml" xref="A5.SS7.p1.8.m8.2.3.4.2">superscript</csymbol><ci id="A5.SS7.p1.8.m8.2.3.4.2.2.cmml" xref="A5.SS7.p1.8.m8.2.3.4.2.2">𝑃</ci><csymbol cd="latexml" id="A5.SS7.p1.8.m8.2.3.4.2.3.cmml" xref="A5.SS7.p1.8.m8.2.3.4.2.3">top</csymbol></apply><apply id="A5.SS7.p1.8.m8.2.3.4.3.cmml" xref="A5.SS7.p1.8.m8.2.3.4.3"><csymbol cd="ambiguous" id="A5.SS7.p1.8.m8.2.3.4.3.1.cmml" xref="A5.SS7.p1.8.m8.2.3.4.3">superscript</csymbol><ci id="A5.SS7.p1.8.m8.2.3.4.3.2.cmml" xref="A5.SS7.p1.8.m8.2.3.4.3.2">𝑈</ci><ci id="A5.SS7.p1.8.m8.2.3.4.3.3.cmml" xref="A5.SS7.p1.8.m8.2.3.4.3.3">′</ci></apply><apply id="A5.SS7.p1.8.m8.2.3.4.4.cmml" xref="A5.SS7.p1.8.m8.2.3.4.4"><csymbol cd="ambiguous" id="A5.SS7.p1.8.m8.2.3.4.4.1.cmml" xref="A5.SS7.p1.8.m8.2.3.4.4">superscript</csymbol><ci id="A5.SS7.p1.8.m8.2.3.4.4.2.cmml" xref="A5.SS7.p1.8.m8.2.3.4.4.2">Σ</ci><ci id="A5.SS7.p1.8.m8.2.3.4.4.3.cmml" xref="A5.SS7.p1.8.m8.2.3.4.4.3">′</ci></apply><apply id="A5.SS7.p1.8.m8.2.3.4.5.cmml" xref="A5.SS7.p1.8.m8.2.3.4.5"><csymbol cd="ambiguous" id="A5.SS7.p1.8.m8.2.3.4.5.1.cmml" xref="A5.SS7.p1.8.m8.2.3.4.5">superscript</csymbol><ci id="A5.SS7.p1.8.m8.2.3.4.5.2.cmml" xref="A5.SS7.p1.8.m8.2.3.4.5.2">𝑉</ci><list id="A5.SS7.p1.8.m8.2.2.2.3.cmml" xref="A5.SS7.p1.8.m8.2.2.2.2"><ci id="A5.SS7.p1.8.m8.2.2.2.2.1.cmml" xref="A5.SS7.p1.8.m8.2.2.2.2.1">′</ci><csymbol cd="latexml" id="A5.SS7.p1.8.m8.1.1.1.1.cmml" xref="A5.SS7.p1.8.m8.1.1.1.1">top</csymbol></list></apply><apply id="A5.SS7.p1.8.m8.2.3.4.6.cmml" xref="A5.SS7.p1.8.m8.2.3.4.6"><csymbol cd="ambiguous" id="A5.SS7.p1.8.m8.2.3.4.6.1.cmml" xref="A5.SS7.p1.8.m8.2.3.4.6">superscript</csymbol><ci id="A5.SS7.p1.8.m8.2.3.4.6.2.cmml" xref="A5.SS7.p1.8.m8.2.3.4.6.2">𝑅</ci><csymbol cd="latexml" id="A5.SS7.p1.8.m8.2.3.4.6.3.cmml" xref="A5.SS7.p1.8.m8.2.3.4.6.3">top</csymbol></apply></apply></apply><apply id="A5.SS7.p1.8.m8.2.3c.cmml" xref="A5.SS7.p1.8.m8.2.3"><eq id="A5.SS7.p1.8.m8.2.3.5.cmml" xref="A5.SS7.p1.8.m8.2.3.5"></eq><share href="#A5.SS7.p1.8.m8.2.3.4.cmml" id="A5.SS7.p1.8.m8.2.3d.cmml" xref="A5.SS7.p1.8.m8.2.3"></share><apply id="A5.SS7.p1.8.m8.2.3.6.cmml" xref="A5.SS7.p1.8.m8.2.3.6"><times id="A5.SS7.p1.8.m8.2.3.6.1.cmml" xref="A5.SS7.p1.8.m8.2.3.6.1"></times><ci id="A5.SS7.p1.8.m8.2.3.6.2.cmml" xref="A5.SS7.p1.8.m8.2.3.6.2">𝑈</ci><ci id="A5.SS7.p1.8.m8.2.3.6.3.cmml" xref="A5.SS7.p1.8.m8.2.3.6.3">Σ</ci><apply id="A5.SS7.p1.8.m8.2.3.6.4.cmml" xref="A5.SS7.p1.8.m8.2.3.6.4"><csymbol cd="ambiguous" id="A5.SS7.p1.8.m8.2.3.6.4.1.cmml" xref="A5.SS7.p1.8.m8.2.3.6.4">superscript</csymbol><ci id="A5.SS7.p1.8.m8.2.3.6.4.2.cmml" xref="A5.SS7.p1.8.m8.2.3.6.4.2">𝑉</ci><csymbol cd="latexml" id="A5.SS7.p1.8.m8.2.3.6.4.3.cmml" xref="A5.SS7.p1.8.m8.2.3.6.4.3">top</csymbol></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.8.m8.2c">W=P^{\top}U^{\prime}\Sigma^{\prime}V^{\prime\top}R^{\top}=U\Sigma V^{\top}</annotation></semantics></math>, and because the SVD is unique up to a column permutation <math id="A5.SS7.p1.9.m9.1" class="ltx_Math" alttext="P^{\prime}" display="inline"><semantics id="A5.SS7.p1.9.m9.1a"><msup id="A5.SS7.p1.9.m9.1.1" xref="A5.SS7.p1.9.m9.1.1.cmml"><mi id="A5.SS7.p1.9.m9.1.1.2" xref="A5.SS7.p1.9.m9.1.1.2.cmml">P</mi><mo id="A5.SS7.p1.9.m9.1.1.3" xref="A5.SS7.p1.9.m9.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.9.m9.1b"><apply id="A5.SS7.p1.9.m9.1.1.cmml" xref="A5.SS7.p1.9.m9.1.1"><csymbol cd="ambiguous" id="A5.SS7.p1.9.m9.1.1.1.cmml" xref="A5.SS7.p1.9.m9.1.1">superscript</csymbol><ci id="A5.SS7.p1.9.m9.1.1.2.cmml" xref="A5.SS7.p1.9.m9.1.1.2">𝑃</ci><ci id="A5.SS7.p1.9.m9.1.1.3.cmml" xref="A5.SS7.p1.9.m9.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.9.m9.1c">P^{\prime}</annotation></semantics></math>, we get <math id="A5.SS7.p1.10.m10.1" class="ltx_Math" alttext="U=P^{\top}U^{\prime}P^{\prime}" display="inline"><semantics id="A5.SS7.p1.10.m10.1a"><mrow id="A5.SS7.p1.10.m10.1.1" xref="A5.SS7.p1.10.m10.1.1.cmml"><mi id="A5.SS7.p1.10.m10.1.1.2" xref="A5.SS7.p1.10.m10.1.1.2.cmml">U</mi><mo id="A5.SS7.p1.10.m10.1.1.1" xref="A5.SS7.p1.10.m10.1.1.1.cmml">=</mo><mrow id="A5.SS7.p1.10.m10.1.1.3" xref="A5.SS7.p1.10.m10.1.1.3.cmml"><msup id="A5.SS7.p1.10.m10.1.1.3.2" xref="A5.SS7.p1.10.m10.1.1.3.2.cmml"><mi id="A5.SS7.p1.10.m10.1.1.3.2.2" xref="A5.SS7.p1.10.m10.1.1.3.2.2.cmml">P</mi><mo id="A5.SS7.p1.10.m10.1.1.3.2.3" xref="A5.SS7.p1.10.m10.1.1.3.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.10.m10.1.1.3.1" xref="A5.SS7.p1.10.m10.1.1.3.1.cmml">​</mo><msup id="A5.SS7.p1.10.m10.1.1.3.3" xref="A5.SS7.p1.10.m10.1.1.3.3.cmml"><mi id="A5.SS7.p1.10.m10.1.1.3.3.2" xref="A5.SS7.p1.10.m10.1.1.3.3.2.cmml">U</mi><mo id="A5.SS7.p1.10.m10.1.1.3.3.3" xref="A5.SS7.p1.10.m10.1.1.3.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A5.SS7.p1.10.m10.1.1.3.1a" xref="A5.SS7.p1.10.m10.1.1.3.1.cmml">​</mo><msup id="A5.SS7.p1.10.m10.1.1.3.4" xref="A5.SS7.p1.10.m10.1.1.3.4.cmml"><mi id="A5.SS7.p1.10.m10.1.1.3.4.2" xref="A5.SS7.p1.10.m10.1.1.3.4.2.cmml">P</mi><mo id="A5.SS7.p1.10.m10.1.1.3.4.3" xref="A5.SS7.p1.10.m10.1.1.3.4.3.cmml">′</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.10.m10.1b"><apply id="A5.SS7.p1.10.m10.1.1.cmml" xref="A5.SS7.p1.10.m10.1.1"><eq id="A5.SS7.p1.10.m10.1.1.1.cmml" xref="A5.SS7.p1.10.m10.1.1.1"></eq><ci id="A5.SS7.p1.10.m10.1.1.2.cmml" xref="A5.SS7.p1.10.m10.1.1.2">𝑈</ci><apply id="A5.SS7.p1.10.m10.1.1.3.cmml" xref="A5.SS7.p1.10.m10.1.1.3"><times id="A5.SS7.p1.10.m10.1.1.3.1.cmml" xref="A5.SS7.p1.10.m10.1.1.3.1"></times><apply id="A5.SS7.p1.10.m10.1.1.3.2.cmml" xref="A5.SS7.p1.10.m10.1.1.3.2"><csymbol cd="ambiguous" id="A5.SS7.p1.10.m10.1.1.3.2.1.cmml" xref="A5.SS7.p1.10.m10.1.1.3.2">superscript</csymbol><ci id="A5.SS7.p1.10.m10.1.1.3.2.2.cmml" xref="A5.SS7.p1.10.m10.1.1.3.2.2">𝑃</ci><csymbol cd="latexml" id="A5.SS7.p1.10.m10.1.1.3.2.3.cmml" xref="A5.SS7.p1.10.m10.1.1.3.2.3">top</csymbol></apply><apply id="A5.SS7.p1.10.m10.1.1.3.3.cmml" xref="A5.SS7.p1.10.m10.1.1.3.3"><csymbol cd="ambiguous" id="A5.SS7.p1.10.m10.1.1.3.3.1.cmml" xref="A5.SS7.p1.10.m10.1.1.3.3">superscript</csymbol><ci id="A5.SS7.p1.10.m10.1.1.3.3.2.cmml" xref="A5.SS7.p1.10.m10.1.1.3.3.2">𝑈</ci><ci id="A5.SS7.p1.10.m10.1.1.3.3.3.cmml" xref="A5.SS7.p1.10.m10.1.1.3.3.3">′</ci></apply><apply id="A5.SS7.p1.10.m10.1.1.3.4.cmml" xref="A5.SS7.p1.10.m10.1.1.3.4"><csymbol cd="ambiguous" id="A5.SS7.p1.10.m10.1.1.3.4.1.cmml" xref="A5.SS7.p1.10.m10.1.1.3.4">superscript</csymbol><ci id="A5.SS7.p1.10.m10.1.1.3.4.2.cmml" xref="A5.SS7.p1.10.m10.1.1.3.4.2">𝑃</ci><ci id="A5.SS7.p1.10.m10.1.1.3.4.3.cmml" xref="A5.SS7.p1.10.m10.1.1.3.4.3">′</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.10.m10.1c">U=P^{\top}U^{\prime}P^{\prime}</annotation></semantics></math>. In other words, if <math id="A5.SS7.p1.11.m11.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A5.SS7.p1.11.m11.1a"><mi id="A5.SS7.p1.11.m11.1.1" xref="A5.SS7.p1.11.m11.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.11.m11.1b"><ci id="A5.SS7.p1.11.m11.1.1.cmml" xref="A5.SS7.p1.11.m11.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.11.m11.1c">W</annotation></semantics></math> is block-diagonal in some unknown basis, <math id="A5.SS7.p1.12.m12.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A5.SS7.p1.12.m12.1a"><mi id="A5.SS7.p1.12.m12.1.1" xref="A5.SS7.p1.12.m12.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p1.12.m12.1b"><ci id="A5.SS7.p1.12.m12.1.1.cmml" xref="A5.SS7.p1.12.m12.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p1.12.m12.1c">U</annotation></semantics></math> is also block diagonal up to a permutation of rows and columns.</p>
</div>
<div id="A5.SS7.p2" class="ltx_para">
<p id="A5.SS7.p2.4" class="ltx_p">To find a good permutation of rows, we sorted the rows of <math id="A5.SS7.p2.1.m1.1" class="ltx_Math" alttext="U" display="inline"><semantics id="A5.SS7.p2.1.m1.1a"><mi id="A5.SS7.p2.1.m1.1.1" xref="A5.SS7.p2.1.m1.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p2.1.m1.1b"><ci id="A5.SS7.p2.1.m1.1.1.cmml" xref="A5.SS7.p2.1.m1.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p2.1.m1.1c">U</annotation></semantics></math> based on how similarly they project on all elements of the singular vector basis. Specifically, we normalized each row to unit norm <math id="A5.SS7.p2.2.m2.1" class="ltx_Math" alttext="\tilde{U_{i}}=U_{i}/||U_{i}||" display="inline"><semantics id="A5.SS7.p2.2.m2.1a"><mrow id="A5.SS7.p2.2.m2.1.1" xref="A5.SS7.p2.2.m2.1.1.cmml"><mover accent="true" id="A5.SS7.p2.2.m2.1.1.3" xref="A5.SS7.p2.2.m2.1.1.3.cmml"><msub id="A5.SS7.p2.2.m2.1.1.3.2" xref="A5.SS7.p2.2.m2.1.1.3.2.cmml"><mi id="A5.SS7.p2.2.m2.1.1.3.2.2" xref="A5.SS7.p2.2.m2.1.1.3.2.2.cmml">U</mi><mi id="A5.SS7.p2.2.m2.1.1.3.2.3" xref="A5.SS7.p2.2.m2.1.1.3.2.3.cmml">i</mi></msub><mo id="A5.SS7.p2.2.m2.1.1.3.1" xref="A5.SS7.p2.2.m2.1.1.3.1.cmml">~</mo></mover><mo id="A5.SS7.p2.2.m2.1.1.2" xref="A5.SS7.p2.2.m2.1.1.2.cmml">=</mo><mrow id="A5.SS7.p2.2.m2.1.1.1" xref="A5.SS7.p2.2.m2.1.1.1.cmml"><msub id="A5.SS7.p2.2.m2.1.1.1.3" xref="A5.SS7.p2.2.m2.1.1.1.3.cmml"><mi id="A5.SS7.p2.2.m2.1.1.1.3.2" xref="A5.SS7.p2.2.m2.1.1.1.3.2.cmml">U</mi><mi id="A5.SS7.p2.2.m2.1.1.1.3.3" xref="A5.SS7.p2.2.m2.1.1.1.3.3.cmml">i</mi></msub><mo id="A5.SS7.p2.2.m2.1.1.1.2" xref="A5.SS7.p2.2.m2.1.1.1.2.cmml">/</mo><mrow id="A5.SS7.p2.2.m2.1.1.1.1.1" xref="A5.SS7.p2.2.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="A5.SS7.p2.2.m2.1.1.1.1.1.2" xref="A5.SS7.p2.2.m2.1.1.1.1.2.1.cmml">‖</mo><msub id="A5.SS7.p2.2.m2.1.1.1.1.1.1" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1.cmml"><mi id="A5.SS7.p2.2.m2.1.1.1.1.1.1.2" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1.2.cmml">U</mi><mi id="A5.SS7.p2.2.m2.1.1.1.1.1.1.3" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A5.SS7.p2.2.m2.1.1.1.1.1.3" xref="A5.SS7.p2.2.m2.1.1.1.1.2.1.cmml">‖</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p2.2.m2.1b"><apply id="A5.SS7.p2.2.m2.1.1.cmml" xref="A5.SS7.p2.2.m2.1.1"><eq id="A5.SS7.p2.2.m2.1.1.2.cmml" xref="A5.SS7.p2.2.m2.1.1.2"></eq><apply id="A5.SS7.p2.2.m2.1.1.3.cmml" xref="A5.SS7.p2.2.m2.1.1.3"><ci id="A5.SS7.p2.2.m2.1.1.3.1.cmml" xref="A5.SS7.p2.2.m2.1.1.3.1">~</ci><apply id="A5.SS7.p2.2.m2.1.1.3.2.cmml" xref="A5.SS7.p2.2.m2.1.1.3.2"><csymbol cd="ambiguous" id="A5.SS7.p2.2.m2.1.1.3.2.1.cmml" xref="A5.SS7.p2.2.m2.1.1.3.2">subscript</csymbol><ci id="A5.SS7.p2.2.m2.1.1.3.2.2.cmml" xref="A5.SS7.p2.2.m2.1.1.3.2.2">𝑈</ci><ci id="A5.SS7.p2.2.m2.1.1.3.2.3.cmml" xref="A5.SS7.p2.2.m2.1.1.3.2.3">𝑖</ci></apply></apply><apply id="A5.SS7.p2.2.m2.1.1.1.cmml" xref="A5.SS7.p2.2.m2.1.1.1"><divide id="A5.SS7.p2.2.m2.1.1.1.2.cmml" xref="A5.SS7.p2.2.m2.1.1.1.2"></divide><apply id="A5.SS7.p2.2.m2.1.1.1.3.cmml" xref="A5.SS7.p2.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="A5.SS7.p2.2.m2.1.1.1.3.1.cmml" xref="A5.SS7.p2.2.m2.1.1.1.3">subscript</csymbol><ci id="A5.SS7.p2.2.m2.1.1.1.3.2.cmml" xref="A5.SS7.p2.2.m2.1.1.1.3.2">𝑈</ci><ci id="A5.SS7.p2.2.m2.1.1.1.3.3.cmml" xref="A5.SS7.p2.2.m2.1.1.1.3.3">𝑖</ci></apply><apply id="A5.SS7.p2.2.m2.1.1.1.1.2.cmml" xref="A5.SS7.p2.2.m2.1.1.1.1.1"><csymbol cd="latexml" id="A5.SS7.p2.2.m2.1.1.1.1.2.1.cmml" xref="A5.SS7.p2.2.m2.1.1.1.1.1.2">norm</csymbol><apply id="A5.SS7.p2.2.m2.1.1.1.1.1.1.cmml" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A5.SS7.p2.2.m2.1.1.1.1.1.1.1.cmml" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1">subscript</csymbol><ci id="A5.SS7.p2.2.m2.1.1.1.1.1.1.2.cmml" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1.2">𝑈</ci><ci id="A5.SS7.p2.2.m2.1.1.1.1.1.1.3.cmml" xref="A5.SS7.p2.2.m2.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p2.2.m2.1c">\tilde{U_{i}}=U_{i}/||U_{i}||</annotation></semantics></math> and considered the pairwise euclidean distances <math id="A5.SS7.p2.3.m3.3" class="ltx_Math" alttext="d_{i,j}=||\tilde{U_{i}}^{2}-\tilde{U_{j}}^{2}||" display="inline"><semantics id="A5.SS7.p2.3.m3.3a"><mrow id="A5.SS7.p2.3.m3.3.3" xref="A5.SS7.p2.3.m3.3.3.cmml"><msub id="A5.SS7.p2.3.m3.3.3.3" xref="A5.SS7.p2.3.m3.3.3.3.cmml"><mi id="A5.SS7.p2.3.m3.3.3.3.2" xref="A5.SS7.p2.3.m3.3.3.3.2.cmml">d</mi><mrow id="A5.SS7.p2.3.m3.2.2.2.4" xref="A5.SS7.p2.3.m3.2.2.2.3.cmml"><mi id="A5.SS7.p2.3.m3.1.1.1.1" xref="A5.SS7.p2.3.m3.1.1.1.1.cmml">i</mi><mo id="A5.SS7.p2.3.m3.2.2.2.4.1" xref="A5.SS7.p2.3.m3.2.2.2.3.cmml">,</mo><mi id="A5.SS7.p2.3.m3.2.2.2.2" xref="A5.SS7.p2.3.m3.2.2.2.2.cmml">j</mi></mrow></msub><mo id="A5.SS7.p2.3.m3.3.3.2" xref="A5.SS7.p2.3.m3.3.3.2.cmml">=</mo><mrow id="A5.SS7.p2.3.m3.3.3.1.1" xref="A5.SS7.p2.3.m3.3.3.1.2.cmml"><mo stretchy="false" id="A5.SS7.p2.3.m3.3.3.1.1.2" xref="A5.SS7.p2.3.m3.3.3.1.2.1.cmml">‖</mo><mrow id="A5.SS7.p2.3.m3.3.3.1.1.1" xref="A5.SS7.p2.3.m3.3.3.1.1.1.cmml"><msup id="A5.SS7.p2.3.m3.3.3.1.1.1.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.cmml"><mover accent="true" id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.cmml"><msub id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.cmml"><mi id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.2.cmml">U</mi><mi id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.3" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.3.cmml">i</mi></msub><mo id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.1" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.1.cmml">~</mo></mover><mn id="A5.SS7.p2.3.m3.3.3.1.1.1.2.3" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.3.cmml">2</mn></msup><mo id="A5.SS7.p2.3.m3.3.3.1.1.1.1" xref="A5.SS7.p2.3.m3.3.3.1.1.1.1.cmml">−</mo><msup id="A5.SS7.p2.3.m3.3.3.1.1.1.3" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.cmml"><mover accent="true" id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.cmml"><msub id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.cmml"><mi id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.2" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.2.cmml">U</mi><mi id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.3" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.3.cmml">j</mi></msub><mo id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.1" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.1.cmml">~</mo></mover><mn id="A5.SS7.p2.3.m3.3.3.1.1.1.3.3" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="A5.SS7.p2.3.m3.3.3.1.1.3" xref="A5.SS7.p2.3.m3.3.3.1.2.1.cmml">‖</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p2.3.m3.3b"><apply id="A5.SS7.p2.3.m3.3.3.cmml" xref="A5.SS7.p2.3.m3.3.3"><eq id="A5.SS7.p2.3.m3.3.3.2.cmml" xref="A5.SS7.p2.3.m3.3.3.2"></eq><apply id="A5.SS7.p2.3.m3.3.3.3.cmml" xref="A5.SS7.p2.3.m3.3.3.3"><csymbol cd="ambiguous" id="A5.SS7.p2.3.m3.3.3.3.1.cmml" xref="A5.SS7.p2.3.m3.3.3.3">subscript</csymbol><ci id="A5.SS7.p2.3.m3.3.3.3.2.cmml" xref="A5.SS7.p2.3.m3.3.3.3.2">𝑑</ci><list id="A5.SS7.p2.3.m3.2.2.2.3.cmml" xref="A5.SS7.p2.3.m3.2.2.2.4"><ci id="A5.SS7.p2.3.m3.1.1.1.1.cmml" xref="A5.SS7.p2.3.m3.1.1.1.1">𝑖</ci><ci id="A5.SS7.p2.3.m3.2.2.2.2.cmml" xref="A5.SS7.p2.3.m3.2.2.2.2">𝑗</ci></list></apply><apply id="A5.SS7.p2.3.m3.3.3.1.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1"><csymbol cd="latexml" id="A5.SS7.p2.3.m3.3.3.1.2.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.2">norm</csymbol><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1"><minus id="A5.SS7.p2.3.m3.3.3.1.1.1.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.1"></minus><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2"><csymbol cd="ambiguous" id="A5.SS7.p2.3.m3.3.3.1.1.1.2.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2">superscript</csymbol><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2"><ci id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.1">~</ci><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2"><csymbol cd="ambiguous" id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2">subscript</csymbol><ci id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.2">𝑈</ci><ci id="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.3.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.2.2.3">𝑖</ci></apply></apply><cn type="integer" id="A5.SS7.p2.3.m3.3.3.1.1.1.2.3.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.2.3">2</cn></apply><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.3.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3"><csymbol cd="ambiguous" id="A5.SS7.p2.3.m3.3.3.1.1.1.3.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3">superscript</csymbol><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2"><ci id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.1">~</ci><apply id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2"><csymbol cd="ambiguous" id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.1.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2">subscript</csymbol><ci id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.2.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.2">𝑈</ci><ci id="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.3.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.2.2.3">𝑗</ci></apply></apply><cn type="integer" id="A5.SS7.p2.3.m3.3.3.1.1.1.3.3.cmml" xref="A5.SS7.p2.3.m3.3.3.1.1.1.3.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p2.3.m3.3c">d_{i,j}=||\tilde{U_{i}}^{2}-\tilde{U_{j}}^{2}||</annotation></semantics></math>. These pairwise distances were then reduced to a single dimension with a UMAP algorithm <cite class="ltx_cite ltx_citemacro_citep">[McInnes et al., <a href="#bib.bib42" title="" class="ltx_ref">2018</a>]</cite>. The obtained 1-dimensional embedding was then used to order the projections <math id="A5.SS7.p2.4.m4.1" class="ltx_Math" alttext="\tilde{U_{i}}^{2}" display="inline"><semantics id="A5.SS7.p2.4.m4.1a"><msup id="A5.SS7.p2.4.m4.1.1" xref="A5.SS7.p2.4.m4.1.1.cmml"><mover accent="true" id="A5.SS7.p2.4.m4.1.1.2" xref="A5.SS7.p2.4.m4.1.1.2.cmml"><msub id="A5.SS7.p2.4.m4.1.1.2.2" xref="A5.SS7.p2.4.m4.1.1.2.2.cmml"><mi id="A5.SS7.p2.4.m4.1.1.2.2.2" xref="A5.SS7.p2.4.m4.1.1.2.2.2.cmml">U</mi><mi id="A5.SS7.p2.4.m4.1.1.2.2.3" xref="A5.SS7.p2.4.m4.1.1.2.2.3.cmml">i</mi></msub><mo id="A5.SS7.p2.4.m4.1.1.2.1" xref="A5.SS7.p2.4.m4.1.1.2.1.cmml">~</mo></mover><mn id="A5.SS7.p2.4.m4.1.1.3" xref="A5.SS7.p2.4.m4.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="A5.SS7.p2.4.m4.1b"><apply id="A5.SS7.p2.4.m4.1.1.cmml" xref="A5.SS7.p2.4.m4.1.1"><csymbol cd="ambiguous" id="A5.SS7.p2.4.m4.1.1.1.cmml" xref="A5.SS7.p2.4.m4.1.1">superscript</csymbol><apply id="A5.SS7.p2.4.m4.1.1.2.cmml" xref="A5.SS7.p2.4.m4.1.1.2"><ci id="A5.SS7.p2.4.m4.1.1.2.1.cmml" xref="A5.SS7.p2.4.m4.1.1.2.1">~</ci><apply id="A5.SS7.p2.4.m4.1.1.2.2.cmml" xref="A5.SS7.p2.4.m4.1.1.2.2"><csymbol cd="ambiguous" id="A5.SS7.p2.4.m4.1.1.2.2.1.cmml" xref="A5.SS7.p2.4.m4.1.1.2.2">subscript</csymbol><ci id="A5.SS7.p2.4.m4.1.1.2.2.2.cmml" xref="A5.SS7.p2.4.m4.1.1.2.2.2">𝑈</ci><ci id="A5.SS7.p2.4.m4.1.1.2.2.3.cmml" xref="A5.SS7.p2.4.m4.1.1.2.2.3">𝑖</ci></apply></apply><cn type="integer" id="A5.SS7.p2.4.m4.1.1.3.cmml" xref="A5.SS7.p2.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p2.4.m4.1c">\tilde{U_{i}}^{2}</annotation></semantics></math> (<a href="#A5.F22" title="Figure 22 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 22</span></a>a), which reveals two fuzzily separated sub-spaces. These two sub-spaces use respectively about 25% and 75% of the dimensions of the entire vector space.</p>
</div>
<div id="A5.SS7.p3" class="ltx_para">
<p id="A5.SS7.p3.1" class="ltx_p">Interestingly, ordering the columns by singular values is fairly consistent with these two sub-spaces. One reason for this result might by that latents projecting to the first sub-space have different encoder norms than latents projecting to the second sub-space (<a href="#A5.F22" title="Figure 22 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 22</span></a>b). This difference in norm can significantly guide the SVD to separate these two sub-spaces.</p>
</div>
<div id="A5.SS7.p4" class="ltx_para">
<p id="A5.SS7.p4.11" class="ltx_p">To further interpret these two sub-spaces, we manually looked at latents from each cluster. We found that latents from the smaller cluster tend to activate on relatively non-diverse vocabulary tokens. To quantify this insight, we first estimated <math id="A5.SS7.p4.1.m1.2" class="ltx_Math" alttext="A_{i,v}" display="inline"><semantics id="A5.SS7.p4.1.m1.2a"><msub id="A5.SS7.p4.1.m1.2.3" xref="A5.SS7.p4.1.m1.2.3.cmml"><mi id="A5.SS7.p4.1.m1.2.3.2" xref="A5.SS7.p4.1.m1.2.3.2.cmml">A</mi><mrow id="A5.SS7.p4.1.m1.2.2.2.4" xref="A5.SS7.p4.1.m1.2.2.2.3.cmml"><mi id="A5.SS7.p4.1.m1.1.1.1.1" xref="A5.SS7.p4.1.m1.1.1.1.1.cmml">i</mi><mo id="A5.SS7.p4.1.m1.2.2.2.4.1" xref="A5.SS7.p4.1.m1.2.2.2.3.cmml">,</mo><mi id="A5.SS7.p4.1.m1.2.2.2.2" xref="A5.SS7.p4.1.m1.2.2.2.2.cmml">v</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.1.m1.2b"><apply id="A5.SS7.p4.1.m1.2.3.cmml" xref="A5.SS7.p4.1.m1.2.3"><csymbol cd="ambiguous" id="A5.SS7.p4.1.m1.2.3.1.cmml" xref="A5.SS7.p4.1.m1.2.3">subscript</csymbol><ci id="A5.SS7.p4.1.m1.2.3.2.cmml" xref="A5.SS7.p4.1.m1.2.3.2">𝐴</ci><list id="A5.SS7.p4.1.m1.2.2.2.3.cmml" xref="A5.SS7.p4.1.m1.2.2.2.4"><ci id="A5.SS7.p4.1.m1.1.1.1.1.cmml" xref="A5.SS7.p4.1.m1.1.1.1.1">𝑖</ci><ci id="A5.SS7.p4.1.m1.2.2.2.2.cmml" xref="A5.SS7.p4.1.m1.2.2.2.2">𝑣</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.1.m1.2c">A_{i,v}</annotation></semantics></math>, the average squared activation of latent <math id="A5.SS7.p4.2.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A5.SS7.p4.2.m2.1a"><mi id="A5.SS7.p4.2.m2.1.1" xref="A5.SS7.p4.2.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.2.m2.1b"><ci id="A5.SS7.p4.2.m2.1.1.cmml" xref="A5.SS7.p4.2.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.2.m2.1c">i</annotation></semantics></math> on vocabulary token <math id="A5.SS7.p4.3.m3.1" class="ltx_Math" alttext="v" display="inline"><semantics id="A5.SS7.p4.3.m3.1a"><mi id="A5.SS7.p4.3.m3.1.1" xref="A5.SS7.p4.3.m3.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.3.m3.1b"><ci id="A5.SS7.p4.3.m3.1.1.cmml" xref="A5.SS7.p4.3.m3.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.3.m3.1c">v</annotation></semantics></math>. Then, we normalized the vectors <math id="A5.SS7.p4.4.m4.1" class="ltx_Math" alttext="A_{i}" display="inline"><semantics id="A5.SS7.p4.4.m4.1a"><msub id="A5.SS7.p4.4.m4.1.1" xref="A5.SS7.p4.4.m4.1.1.cmml"><mi id="A5.SS7.p4.4.m4.1.1.2" xref="A5.SS7.p4.4.m4.1.1.2.cmml">A</mi><mi id="A5.SS7.p4.4.m4.1.1.3" xref="A5.SS7.p4.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.4.m4.1b"><apply id="A5.SS7.p4.4.m4.1.1.cmml" xref="A5.SS7.p4.4.m4.1.1"><csymbol cd="ambiguous" id="A5.SS7.p4.4.m4.1.1.1.cmml" xref="A5.SS7.p4.4.m4.1.1">subscript</csymbol><ci id="A5.SS7.p4.4.m4.1.1.2.cmml" xref="A5.SS7.p4.4.m4.1.1.2">𝐴</ci><ci id="A5.SS7.p4.4.m4.1.1.3.cmml" xref="A5.SS7.p4.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.4.m4.1c">A_{i}</annotation></semantics></math> to sum to one, <math id="A5.SS7.p4.5.m5.6" class="ltx_Math" alttext="\tilde{A}_{i,v}=A_{i,v}/\sum_{w}A_{i,w}" display="inline"><semantics id="A5.SS7.p4.5.m5.6a"><mrow id="A5.SS7.p4.5.m5.6.7" xref="A5.SS7.p4.5.m5.6.7.cmml"><msub id="A5.SS7.p4.5.m5.6.7.2" xref="A5.SS7.p4.5.m5.6.7.2.cmml"><mover accent="true" id="A5.SS7.p4.5.m5.6.7.2.2" xref="A5.SS7.p4.5.m5.6.7.2.2.cmml"><mi id="A5.SS7.p4.5.m5.6.7.2.2.2" xref="A5.SS7.p4.5.m5.6.7.2.2.2.cmml">A</mi><mo id="A5.SS7.p4.5.m5.6.7.2.2.1" xref="A5.SS7.p4.5.m5.6.7.2.2.1.cmml">~</mo></mover><mrow id="A5.SS7.p4.5.m5.2.2.2.4" xref="A5.SS7.p4.5.m5.2.2.2.3.cmml"><mi id="A5.SS7.p4.5.m5.1.1.1.1" xref="A5.SS7.p4.5.m5.1.1.1.1.cmml">i</mi><mo id="A5.SS7.p4.5.m5.2.2.2.4.1" xref="A5.SS7.p4.5.m5.2.2.2.3.cmml">,</mo><mi id="A5.SS7.p4.5.m5.2.2.2.2" xref="A5.SS7.p4.5.m5.2.2.2.2.cmml">v</mi></mrow></msub><mo id="A5.SS7.p4.5.m5.6.7.1" xref="A5.SS7.p4.5.m5.6.7.1.cmml">=</mo><mrow id="A5.SS7.p4.5.m5.6.7.3" xref="A5.SS7.p4.5.m5.6.7.3.cmml"><msub id="A5.SS7.p4.5.m5.6.7.3.2" xref="A5.SS7.p4.5.m5.6.7.3.2.cmml"><mi id="A5.SS7.p4.5.m5.6.7.3.2.2" xref="A5.SS7.p4.5.m5.6.7.3.2.2.cmml">A</mi><mrow id="A5.SS7.p4.5.m5.4.4.2.4" xref="A5.SS7.p4.5.m5.4.4.2.3.cmml"><mi id="A5.SS7.p4.5.m5.3.3.1.1" xref="A5.SS7.p4.5.m5.3.3.1.1.cmml">i</mi><mo id="A5.SS7.p4.5.m5.4.4.2.4.1" xref="A5.SS7.p4.5.m5.4.4.2.3.cmml">,</mo><mi id="A5.SS7.p4.5.m5.4.4.2.2" xref="A5.SS7.p4.5.m5.4.4.2.2.cmml">v</mi></mrow></msub><mo rspace="0.055em" id="A5.SS7.p4.5.m5.6.7.3.1" xref="A5.SS7.p4.5.m5.6.7.3.1.cmml">/</mo><mrow id="A5.SS7.p4.5.m5.6.7.3.3" xref="A5.SS7.p4.5.m5.6.7.3.3.cmml"><msub id="A5.SS7.p4.5.m5.6.7.3.3.1" xref="A5.SS7.p4.5.m5.6.7.3.3.1.cmml"><mo id="A5.SS7.p4.5.m5.6.7.3.3.1.2" xref="A5.SS7.p4.5.m5.6.7.3.3.1.2.cmml">∑</mo><mi id="A5.SS7.p4.5.m5.6.7.3.3.1.3" xref="A5.SS7.p4.5.m5.6.7.3.3.1.3.cmml">w</mi></msub><msub id="A5.SS7.p4.5.m5.6.7.3.3.2" xref="A5.SS7.p4.5.m5.6.7.3.3.2.cmml"><mi id="A5.SS7.p4.5.m5.6.7.3.3.2.2" xref="A5.SS7.p4.5.m5.6.7.3.3.2.2.cmml">A</mi><mrow id="A5.SS7.p4.5.m5.6.6.2.4" xref="A5.SS7.p4.5.m5.6.6.2.3.cmml"><mi id="A5.SS7.p4.5.m5.5.5.1.1" xref="A5.SS7.p4.5.m5.5.5.1.1.cmml">i</mi><mo id="A5.SS7.p4.5.m5.6.6.2.4.1" xref="A5.SS7.p4.5.m5.6.6.2.3.cmml">,</mo><mi id="A5.SS7.p4.5.m5.6.6.2.2" xref="A5.SS7.p4.5.m5.6.6.2.2.cmml">w</mi></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.5.m5.6b"><apply id="A5.SS7.p4.5.m5.6.7.cmml" xref="A5.SS7.p4.5.m5.6.7"><eq id="A5.SS7.p4.5.m5.6.7.1.cmml" xref="A5.SS7.p4.5.m5.6.7.1"></eq><apply id="A5.SS7.p4.5.m5.6.7.2.cmml" xref="A5.SS7.p4.5.m5.6.7.2"><csymbol cd="ambiguous" id="A5.SS7.p4.5.m5.6.7.2.1.cmml" xref="A5.SS7.p4.5.m5.6.7.2">subscript</csymbol><apply id="A5.SS7.p4.5.m5.6.7.2.2.cmml" xref="A5.SS7.p4.5.m5.6.7.2.2"><ci id="A5.SS7.p4.5.m5.6.7.2.2.1.cmml" xref="A5.SS7.p4.5.m5.6.7.2.2.1">~</ci><ci id="A5.SS7.p4.5.m5.6.7.2.2.2.cmml" xref="A5.SS7.p4.5.m5.6.7.2.2.2">𝐴</ci></apply><list id="A5.SS7.p4.5.m5.2.2.2.3.cmml" xref="A5.SS7.p4.5.m5.2.2.2.4"><ci id="A5.SS7.p4.5.m5.1.1.1.1.cmml" xref="A5.SS7.p4.5.m5.1.1.1.1">𝑖</ci><ci id="A5.SS7.p4.5.m5.2.2.2.2.cmml" xref="A5.SS7.p4.5.m5.2.2.2.2">𝑣</ci></list></apply><apply id="A5.SS7.p4.5.m5.6.7.3.cmml" xref="A5.SS7.p4.5.m5.6.7.3"><divide id="A5.SS7.p4.5.m5.6.7.3.1.cmml" xref="A5.SS7.p4.5.m5.6.7.3.1"></divide><apply id="A5.SS7.p4.5.m5.6.7.3.2.cmml" xref="A5.SS7.p4.5.m5.6.7.3.2"><csymbol cd="ambiguous" id="A5.SS7.p4.5.m5.6.7.3.2.1.cmml" xref="A5.SS7.p4.5.m5.6.7.3.2">subscript</csymbol><ci id="A5.SS7.p4.5.m5.6.7.3.2.2.cmml" xref="A5.SS7.p4.5.m5.6.7.3.2.2">𝐴</ci><list id="A5.SS7.p4.5.m5.4.4.2.3.cmml" xref="A5.SS7.p4.5.m5.4.4.2.4"><ci id="A5.SS7.p4.5.m5.3.3.1.1.cmml" xref="A5.SS7.p4.5.m5.3.3.1.1">𝑖</ci><ci id="A5.SS7.p4.5.m5.4.4.2.2.cmml" xref="A5.SS7.p4.5.m5.4.4.2.2">𝑣</ci></list></apply><apply id="A5.SS7.p4.5.m5.6.7.3.3.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3"><apply id="A5.SS7.p4.5.m5.6.7.3.3.1.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.1"><csymbol cd="ambiguous" id="A5.SS7.p4.5.m5.6.7.3.3.1.1.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.1">subscript</csymbol><sum id="A5.SS7.p4.5.m5.6.7.3.3.1.2.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.1.2"></sum><ci id="A5.SS7.p4.5.m5.6.7.3.3.1.3.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.1.3">𝑤</ci></apply><apply id="A5.SS7.p4.5.m5.6.7.3.3.2.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.2"><csymbol cd="ambiguous" id="A5.SS7.p4.5.m5.6.7.3.3.2.1.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.2">subscript</csymbol><ci id="A5.SS7.p4.5.m5.6.7.3.3.2.2.cmml" xref="A5.SS7.p4.5.m5.6.7.3.3.2.2">𝐴</ci><list id="A5.SS7.p4.5.m5.6.6.2.3.cmml" xref="A5.SS7.p4.5.m5.6.6.2.4"><ci id="A5.SS7.p4.5.m5.5.5.1.1.cmml" xref="A5.SS7.p4.5.m5.5.5.1.1">𝑖</ci><ci id="A5.SS7.p4.5.m5.6.6.2.2.cmml" xref="A5.SS7.p4.5.m5.6.6.2.2">𝑤</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.5.m5.6c">\tilde{A}_{i,v}=A_{i,v}/\sum_{w}A_{i,w}</annotation></semantics></math> and computed the effective number of token <math id="A5.SS7.p4.6.m6.7" class="ltx_Math" alttext="m_{i}=\exp(\sum_{v}\tilde{A}_{i,v}\log(\tilde{A}_{i,v}))" display="inline"><semantics id="A5.SS7.p4.6.m6.7a"><mrow id="A5.SS7.p4.6.m6.7.7" xref="A5.SS7.p4.6.m6.7.7.cmml"><msub id="A5.SS7.p4.6.m6.7.7.3" xref="A5.SS7.p4.6.m6.7.7.3.cmml"><mi id="A5.SS7.p4.6.m6.7.7.3.2" xref="A5.SS7.p4.6.m6.7.7.3.2.cmml">m</mi><mi id="A5.SS7.p4.6.m6.7.7.3.3" xref="A5.SS7.p4.6.m6.7.7.3.3.cmml">i</mi></msub><mo id="A5.SS7.p4.6.m6.7.7.2" xref="A5.SS7.p4.6.m6.7.7.2.cmml">=</mo><mrow id="A5.SS7.p4.6.m6.7.7.1.1" xref="A5.SS7.p4.6.m6.7.7.1.2.cmml"><mi id="A5.SS7.p4.6.m6.6.6" xref="A5.SS7.p4.6.m6.6.6.cmml">exp</mi><mo id="A5.SS7.p4.6.m6.7.7.1.1a" xref="A5.SS7.p4.6.m6.7.7.1.2.cmml">⁡</mo><mrow id="A5.SS7.p4.6.m6.7.7.1.1.1" xref="A5.SS7.p4.6.m6.7.7.1.2.cmml"><mo stretchy="false" id="A5.SS7.p4.6.m6.7.7.1.1.1.2" xref="A5.SS7.p4.6.m6.7.7.1.2.cmml">(</mo><mrow id="A5.SS7.p4.6.m6.7.7.1.1.1.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.cmml"><msub id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.cmml"><mo lspace="0em" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.2.cmml">∑</mo><mi id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.3" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.3.cmml">v</mi></msub><mrow id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.cmml"><msub id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.cmml"><mover accent="true" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.cmml"><mi id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.2.cmml">A</mi><mo id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mrow id="A5.SS7.p4.6.m6.2.2.2.4" xref="A5.SS7.p4.6.m6.2.2.2.3.cmml"><mi id="A5.SS7.p4.6.m6.1.1.1.1" xref="A5.SS7.p4.6.m6.1.1.1.1.cmml">i</mi><mo id="A5.SS7.p4.6.m6.2.2.2.4.1" xref="A5.SS7.p4.6.m6.2.2.2.3.cmml">,</mo><mi id="A5.SS7.p4.6.m6.2.2.2.2" xref="A5.SS7.p4.6.m6.2.2.2.2.cmml">v</mi></mrow></msub><mo lspace="0.167em" rspace="0em" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.2.cmml">​</mo><mrow id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.2.cmml"><mi id="A5.SS7.p4.6.m6.5.5" xref="A5.SS7.p4.6.m6.5.5.cmml">log</mi><mo id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1a" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.2.cmml">(</mo><msub id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.2" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.2.cmml">A</mi><mo id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.1" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.1.cmml">~</mo></mover><mrow id="A5.SS7.p4.6.m6.4.4.2.4" xref="A5.SS7.p4.6.m6.4.4.2.3.cmml"><mi id="A5.SS7.p4.6.m6.3.3.1.1" xref="A5.SS7.p4.6.m6.3.3.1.1.cmml">i</mi><mo id="A5.SS7.p4.6.m6.4.4.2.4.1" xref="A5.SS7.p4.6.m6.4.4.2.3.cmml">,</mo><mi id="A5.SS7.p4.6.m6.4.4.2.2" xref="A5.SS7.p4.6.m6.4.4.2.2.cmml">v</mi></mrow></msub><mo stretchy="false" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.3" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><mo stretchy="false" id="A5.SS7.p4.6.m6.7.7.1.1.1.3" xref="A5.SS7.p4.6.m6.7.7.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.6.m6.7b"><apply id="A5.SS7.p4.6.m6.7.7.cmml" xref="A5.SS7.p4.6.m6.7.7"><eq id="A5.SS7.p4.6.m6.7.7.2.cmml" xref="A5.SS7.p4.6.m6.7.7.2"></eq><apply id="A5.SS7.p4.6.m6.7.7.3.cmml" xref="A5.SS7.p4.6.m6.7.7.3"><csymbol cd="ambiguous" id="A5.SS7.p4.6.m6.7.7.3.1.cmml" xref="A5.SS7.p4.6.m6.7.7.3">subscript</csymbol><ci id="A5.SS7.p4.6.m6.7.7.3.2.cmml" xref="A5.SS7.p4.6.m6.7.7.3.2">𝑚</ci><ci id="A5.SS7.p4.6.m6.7.7.3.3.cmml" xref="A5.SS7.p4.6.m6.7.7.3.3">𝑖</ci></apply><apply id="A5.SS7.p4.6.m6.7.7.1.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1"><exp id="A5.SS7.p4.6.m6.6.6.cmml" xref="A5.SS7.p4.6.m6.6.6"></exp><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1"><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2"><csymbol cd="ambiguous" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2">subscript</csymbol><sum id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.2"></sum><ci id="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.3.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.2.3">𝑣</ci></apply><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1"><times id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.2"></times><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3">subscript</csymbol><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2"><ci id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.1">~</ci><ci id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.3.2.2">𝐴</ci></apply><list id="A5.SS7.p4.6.m6.2.2.2.3.cmml" xref="A5.SS7.p4.6.m6.2.2.2.4"><ci id="A5.SS7.p4.6.m6.1.1.1.1.cmml" xref="A5.SS7.p4.6.m6.1.1.1.1">𝑖</ci><ci id="A5.SS7.p4.6.m6.2.2.2.2.cmml" xref="A5.SS7.p4.6.m6.2.2.2.2">𝑣</ci></list></apply><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1"><log id="A5.SS7.p4.6.m6.5.5.cmml" xref="A5.SS7.p4.6.m6.5.5"></log><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2"><ci id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.1">~</ci><ci id="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A5.SS7.p4.6.m6.7.7.1.1.1.1.1.1.1.1.1.2.2">𝐴</ci></apply><list id="A5.SS7.p4.6.m6.4.4.2.3.cmml" xref="A5.SS7.p4.6.m6.4.4.2.4"><ci id="A5.SS7.p4.6.m6.3.3.1.1.cmml" xref="A5.SS7.p4.6.m6.3.3.1.1">𝑖</ci><ci id="A5.SS7.p4.6.m6.4.4.2.2.cmml" xref="A5.SS7.p4.6.m6.4.4.2.2">𝑣</ci></list></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.6.m6.7c">m_{i}=\exp(\sum_{v}\tilde{A}_{i,v}\log(\tilde{A}_{i,v}))</annotation></semantics></math>. The effective number of token is a continuous metric with values in <math id="A5.SS7.p4.7.m7.2" class="ltx_Math" alttext="[1,n_{\text{vocab}}]" display="inline"><semantics id="A5.SS7.p4.7.m7.2a"><mrow id="A5.SS7.p4.7.m7.2.2.1" xref="A5.SS7.p4.7.m7.2.2.2.cmml"><mo stretchy="false" id="A5.SS7.p4.7.m7.2.2.1.2" xref="A5.SS7.p4.7.m7.2.2.2.cmml">[</mo><mn id="A5.SS7.p4.7.m7.1.1" xref="A5.SS7.p4.7.m7.1.1.cmml">1</mn><mo id="A5.SS7.p4.7.m7.2.2.1.3" xref="A5.SS7.p4.7.m7.2.2.2.cmml">,</mo><msub id="A5.SS7.p4.7.m7.2.2.1.1" xref="A5.SS7.p4.7.m7.2.2.1.1.cmml"><mi id="A5.SS7.p4.7.m7.2.2.1.1.2" xref="A5.SS7.p4.7.m7.2.2.1.1.2.cmml">n</mi><mtext id="A5.SS7.p4.7.m7.2.2.1.1.3" xref="A5.SS7.p4.7.m7.2.2.1.1.3a.cmml">vocab</mtext></msub><mo stretchy="false" id="A5.SS7.p4.7.m7.2.2.1.4" xref="A5.SS7.p4.7.m7.2.2.2.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.7.m7.2b"><interval closure="closed" id="A5.SS7.p4.7.m7.2.2.2.cmml" xref="A5.SS7.p4.7.m7.2.2.1"><cn type="integer" id="A5.SS7.p4.7.m7.1.1.cmml" xref="A5.SS7.p4.7.m7.1.1">1</cn><apply id="A5.SS7.p4.7.m7.2.2.1.1.cmml" xref="A5.SS7.p4.7.m7.2.2.1.1"><csymbol cd="ambiguous" id="A5.SS7.p4.7.m7.2.2.1.1.1.cmml" xref="A5.SS7.p4.7.m7.2.2.1.1">subscript</csymbol><ci id="A5.SS7.p4.7.m7.2.2.1.1.2.cmml" xref="A5.SS7.p4.7.m7.2.2.1.1.2">𝑛</ci><ci id="A5.SS7.p4.7.m7.2.2.1.1.3a.cmml" xref="A5.SS7.p4.7.m7.2.2.1.1.3"><mtext mathsize="70%" id="A5.SS7.p4.7.m7.2.2.1.1.3.cmml" xref="A5.SS7.p4.7.m7.2.2.1.1.3">vocab</mtext></ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.7.m7.2c">[1,n_{\text{vocab}}]</annotation></semantics></math>, and it is equal to <math id="A5.SS7.p4.8.m8.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS7.p4.8.m8.1a"><mi id="A5.SS7.p4.8.m8.1.1" xref="A5.SS7.p4.8.m8.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.8.m8.1b"><ci id="A5.SS7.p4.8.m8.1.1.cmml" xref="A5.SS7.p4.8.m8.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.8.m8.1c">k</annotation></semantics></math> when a latent activates equally on <math id="A5.SS7.p4.9.m9.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS7.p4.9.m9.1a"><mi id="A5.SS7.p4.9.m9.1.1" xref="A5.SS7.p4.9.m9.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.9.m9.1b"><ci id="A5.SS7.p4.9.m9.1.1.cmml" xref="A5.SS7.p4.9.m9.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.9.m9.1c">k</annotation></semantics></math> vocabulary tokens. With this metric, we confirmed quantitatively (<a href="#A5.F22" title="Figure 22 ‣ E.7 Clustering latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 22</span></a>c) that latents from the smaller cluster all activate on relatively low numbers of vocabulary tokens (less than <math id="A5.SS7.p4.10.m10.1" class="ltx_Math" alttext="100" display="inline"><semantics id="A5.SS7.p4.10.m10.1a"><mn id="A5.SS7.p4.10.m10.1.1" xref="A5.SS7.p4.10.m10.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.10.m10.1b"><cn type="integer" id="A5.SS7.p4.10.m10.1.1.cmml" xref="A5.SS7.p4.10.m10.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.10.m10.1c">100</annotation></semantics></math>), whereas latents from the larger cluster sometimes activate on a larger numbers of vocabulary tokens (up to <math id="A5.SS7.p4.11.m11.1" class="ltx_Math" alttext="1000" display="inline"><semantics id="A5.SS7.p4.11.m11.1a"><mn id="A5.SS7.p4.11.m11.1.1" xref="A5.SS7.p4.11.m11.1.1.cmml">1000</mn><annotation-xml encoding="MathML-Content" id="A5.SS7.p4.11.m11.1b"><cn type="integer" id="A5.SS7.p4.11.m11.1.1.cmml" xref="A5.SS7.p4.11.m11.1.1">1000</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS7.p4.11.m11.1c">1000</annotation></semantics></math>).</p>
</div>
<figure id="A5.F22" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3"><img src="/html/2406.04093/assets/x17.png" id="A5.F22.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="152" height="113" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_3"><img src="/html/2406.04093/assets/x18.png" id="A5.F22.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="142" height="115" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_3"><img src="/html/2406.04093/assets/x19.png" id="A5.F22.g3" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="143" height="109" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F22.2.1.1" class="ltx_text" style="font-size:90%;">Figure 22</span>: </span><span id="A5.F22.3.2" class="ltx_text" style="font-size:90%;">The residual stream seems composed of two separate sub-spaces. About 25% of latents mostly project on a sub-space using 25% of dimensions. These latents tend to have larger encoder norm, and to activate on a smaller number of vocabulary tokens. The remaining 75% of latents mostly project on the remaining 75% of dimensions, and can activate on a larger number of vocabulary tokens.</span></figcaption>
</figure>
<figure id="A5.F23" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8_relu_vs_topk_probe_vs_L0.png" id="A5.F23.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="479" height="318" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F23.2.1.1" class="ltx_text" style="font-size:90%;">Figure 23</span>: </span><span id="A5.F23.3.2" class="ltx_text" style="font-size:90%;">TopK beats ReLU not only on the sparsity-MSE frontier, but also on the sparsity-probe loss frontier. (lower is better)</span></figcaption>
</figure>
<figure id="A5.F24" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8_relu_vs_topk_n2g_f1_vs_L0.png" id="A5.F24.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="479" height="318" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8_relu_vs_topk_n2g_recall_vs_L0.png" id="A5.F24.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="269" height="179" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8_relu_vs_topk_n2g_precision_vs_L0.png" id="A5.F24.g3" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="269" height="179" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F24.2.1.1" class="ltx_text" style="font-size:90%;">Figure 24</span>: </span><span id="A5.F24.3.2" class="ltx_text" style="font-size:90%;">TopK beats ReLU on N2G F1 score. Its N2G explanations have noticeably higher recall, but worse precision. (higher is better)</span></figcaption>
</figure>
<figure id="A5.F25" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A5.F25.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_n2g_recall.png" id="A5.F25.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="265" height="150" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F25.sf1.4.2.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="A5.F25.sf1.2.1" class="ltx_text" style="font-size:90%;">Recall of N2G explanations
<br class="ltx_break"><math id="A5.F25.sf1.2.1.m1.2" class="ltx_Math" alttext="P(\text{n2g}&gt;0|\text{act}&gt;0)" display="inline"><semantics id="A5.F25.sf1.2.1.m1.2b"><mrow id="A5.F25.sf1.2.1.m1.2.2" xref="A5.F25.sf1.2.1.m1.2.2.cmml"><mi id="A5.F25.sf1.2.1.m1.2.2.3" xref="A5.F25.sf1.2.1.m1.2.2.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A5.F25.sf1.2.1.m1.2.2.2" xref="A5.F25.sf1.2.1.m1.2.2.2.cmml">​</mo><mrow id="A5.F25.sf1.2.1.m1.2.2.1.1" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.F25.sf1.2.1.m1.2.2.1.1.2" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.cmml">(</mo><mrow id="A5.F25.sf1.2.1.m1.2.2.1.1.1" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.cmml"><mtext id="A5.F25.sf1.2.1.m1.2.2.1.1.1.2" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.2a.cmml">n2g</mtext><mo id="A5.F25.sf1.2.1.m1.2.2.1.1.1.1" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.1.cmml">&gt;</mo><mrow id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.cmml"><mn id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.2" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.2.cmml">0</mn><mo lspace="0em" rspace="0em" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.1" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mrow id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.2" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.1.cmml"><mo stretchy="false" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.2.1" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">|</mo><mtext id="A5.F25.sf1.2.1.m1.1.1" xref="A5.F25.sf1.2.1.m1.1.1a.cmml">act</mtext><mo fence="true" lspace="0em" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.2.2" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">&gt;</mo></mrow><mo lspace="0em" rspace="0em" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.1b" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mn id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.4" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.4.cmml">0</mn></mrow></mrow><mo stretchy="false" id="A5.F25.sf1.2.1.m1.2.2.1.1.3" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.F25.sf1.2.1.m1.2c"><apply id="A5.F25.sf1.2.1.m1.2.2.cmml" xref="A5.F25.sf1.2.1.m1.2.2"><times id="A5.F25.sf1.2.1.m1.2.2.2.cmml" xref="A5.F25.sf1.2.1.m1.2.2.2"></times><ci id="A5.F25.sf1.2.1.m1.2.2.3.cmml" xref="A5.F25.sf1.2.1.m1.2.2.3">𝑃</ci><apply id="A5.F25.sf1.2.1.m1.2.2.1.1.1.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1"><gt id="A5.F25.sf1.2.1.m1.2.2.1.1.1.1.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.1"></gt><ci id="A5.F25.sf1.2.1.m1.2.2.1.1.1.2a.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.2"><mtext id="A5.F25.sf1.2.1.m1.2.2.1.1.1.2.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.2">n2g</mtext></ci><apply id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3"><times id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.1.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.1"></times><cn type="integer" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.2.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.2">0</cn><apply id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.1.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.2"><csymbol cd="latexml" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.1.1.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.3.2.1">ket</csymbol><ci id="A5.F25.sf1.2.1.m1.1.1a.cmml" xref="A5.F25.sf1.2.1.m1.1.1"><mtext id="A5.F25.sf1.2.1.m1.1.1.cmml" xref="A5.F25.sf1.2.1.m1.1.1">act</mtext></ci></apply><cn type="integer" id="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.4.cmml" xref="A5.F25.sf1.2.1.m1.2.2.1.1.1.3.4">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F25.sf1.2.1.m1.2d">P(\text{n2g}&gt;0|\text{act}&gt;0)</annotation></semantics></math></span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A5.F25.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_L8sweepnk_n2g_precision.png" id="A5.F25.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="265" height="150" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F25.sf2.4.2.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="A5.F25.sf2.2.1" class="ltx_text" style="font-size:90%;">Precision of N2G explanations
<br class="ltx_break"><math id="A5.F25.sf2.2.1.m1.2" class="ltx_Math" alttext="P(\text{act}&gt;0|\text{n2g}&gt;0)" display="inline"><semantics id="A5.F25.sf2.2.1.m1.2b"><mrow id="A5.F25.sf2.2.1.m1.2.2" xref="A5.F25.sf2.2.1.m1.2.2.cmml"><mi id="A5.F25.sf2.2.1.m1.2.2.3" xref="A5.F25.sf2.2.1.m1.2.2.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A5.F25.sf2.2.1.m1.2.2.2" xref="A5.F25.sf2.2.1.m1.2.2.2.cmml">​</mo><mrow id="A5.F25.sf2.2.1.m1.2.2.1.1" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.F25.sf2.2.1.m1.2.2.1.1.2" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.cmml">(</mo><mrow id="A5.F25.sf2.2.1.m1.2.2.1.1.1" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.cmml"><mtext id="A5.F25.sf2.2.1.m1.2.2.1.1.1.2" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.2a.cmml">act</mtext><mo id="A5.F25.sf2.2.1.m1.2.2.1.1.1.1" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.1.cmml">&gt;</mo><mrow id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.cmml"><mn id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.2" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.2.cmml">0</mn><mo lspace="0em" rspace="0em" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.1" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mrow id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.2" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.1.cmml"><mo stretchy="false" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.2.1" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">|</mo><mtext id="A5.F25.sf2.2.1.m1.1.1" xref="A5.F25.sf2.2.1.m1.1.1a.cmml">n2g</mtext><mo fence="true" lspace="0em" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.2.2" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">&gt;</mo></mrow><mo lspace="0em" rspace="0em" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.1b" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mn id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.4" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.4.cmml">0</mn></mrow></mrow><mo stretchy="false" id="A5.F25.sf2.2.1.m1.2.2.1.1.3" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.F25.sf2.2.1.m1.2c"><apply id="A5.F25.sf2.2.1.m1.2.2.cmml" xref="A5.F25.sf2.2.1.m1.2.2"><times id="A5.F25.sf2.2.1.m1.2.2.2.cmml" xref="A5.F25.sf2.2.1.m1.2.2.2"></times><ci id="A5.F25.sf2.2.1.m1.2.2.3.cmml" xref="A5.F25.sf2.2.1.m1.2.2.3">𝑃</ci><apply id="A5.F25.sf2.2.1.m1.2.2.1.1.1.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1"><gt id="A5.F25.sf2.2.1.m1.2.2.1.1.1.1.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.1"></gt><ci id="A5.F25.sf2.2.1.m1.2.2.1.1.1.2a.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.2"><mtext id="A5.F25.sf2.2.1.m1.2.2.1.1.1.2.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.2">act</mtext></ci><apply id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3"><times id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.1.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.1"></times><cn type="integer" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.2.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.2">0</cn><apply id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.1.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.2"><csymbol cd="latexml" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.1.1.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.3.2.1">ket</csymbol><ci id="A5.F25.sf2.2.1.m1.1.1a.cmml" xref="A5.F25.sf2.2.1.m1.1.1"><mtext id="A5.F25.sf2.2.1.m1.1.1.cmml" xref="A5.F25.sf2.2.1.m1.1.1">n2g</mtext></ci></apply><cn type="integer" id="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.4.cmml" xref="A5.F25.sf2.2.1.m1.2.2.1.1.1.3.4">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F25.sf2.2.1.m1.2d">P(\text{act}&gt;0|\text{n2g}&gt;0)</annotation></semantics></math></span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A5.F25.8.4.1" class="ltx_text" style="font-size:90%;">Figure 25</span>: </span><span id="A5.F25.6.3" class="ltx_text" style="font-size:90%;">Neuron2graph precision and recall. The average autoencoder latent is generally easier to explain as <math id="A5.F25.4.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.F25.4.1.m1.1b"><mi id="A5.F25.4.1.m1.1.1" xref="A5.F25.4.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.F25.4.1.m1.1c"><ci id="A5.F25.4.1.m1.1.1.cmml" xref="A5.F25.4.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.F25.4.1.m1.1d">k</annotation></semantics></math> decreases and <math id="A5.F25.5.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A5.F25.5.2.m2.1b"><mi id="A5.F25.5.2.m2.1.1" xref="A5.F25.5.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A5.F25.5.2.m2.1c"><ci id="A5.F25.5.2.m2.1.1.cmml" xref="A5.F25.5.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.F25.5.2.m2.1d">n</annotation></semantics></math> increases. However, <math id="A5.F25.6.3.m3.2" class="ltx_Math" alttext="n=2048,k=512" display="inline"><semantics id="A5.F25.6.3.m3.2b"><mrow id="A5.F25.6.3.m3.2.2.2" xref="A5.F25.6.3.m3.2.2.3.cmml"><mrow id="A5.F25.6.3.m3.1.1.1.1" xref="A5.F25.6.3.m3.1.1.1.1.cmml"><mi id="A5.F25.6.3.m3.1.1.1.1.2" xref="A5.F25.6.3.m3.1.1.1.1.2.cmml">n</mi><mo id="A5.F25.6.3.m3.1.1.1.1.1" xref="A5.F25.6.3.m3.1.1.1.1.1.cmml">=</mo><mn id="A5.F25.6.3.m3.1.1.1.1.3" xref="A5.F25.6.3.m3.1.1.1.1.3.cmml">2048</mn></mrow><mo id="A5.F25.6.3.m3.2.2.2.3" xref="A5.F25.6.3.m3.2.2.3a.cmml">,</mo><mrow id="A5.F25.6.3.m3.2.2.2.2" xref="A5.F25.6.3.m3.2.2.2.2.cmml"><mi id="A5.F25.6.3.m3.2.2.2.2.2" xref="A5.F25.6.3.m3.2.2.2.2.2.cmml">k</mi><mo id="A5.F25.6.3.m3.2.2.2.2.1" xref="A5.F25.6.3.m3.2.2.2.2.1.cmml">=</mo><mn id="A5.F25.6.3.m3.2.2.2.2.3" xref="A5.F25.6.3.m3.2.2.2.2.3.cmml">512</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.F25.6.3.m3.2c"><apply id="A5.F25.6.3.m3.2.2.3.cmml" xref="A5.F25.6.3.m3.2.2.2"><csymbol cd="ambiguous" id="A5.F25.6.3.m3.2.2.3a.cmml" xref="A5.F25.6.3.m3.2.2.2.3">formulae-sequence</csymbol><apply id="A5.F25.6.3.m3.1.1.1.1.cmml" xref="A5.F25.6.3.m3.1.1.1.1"><eq id="A5.F25.6.3.m3.1.1.1.1.1.cmml" xref="A5.F25.6.3.m3.1.1.1.1.1"></eq><ci id="A5.F25.6.3.m3.1.1.1.1.2.cmml" xref="A5.F25.6.3.m3.1.1.1.1.2">𝑛</ci><cn type="integer" id="A5.F25.6.3.m3.1.1.1.1.3.cmml" xref="A5.F25.6.3.m3.1.1.1.1.3">2048</cn></apply><apply id="A5.F25.6.3.m3.2.2.2.2.cmml" xref="A5.F25.6.3.m3.2.2.2.2"><eq id="A5.F25.6.3.m3.2.2.2.2.1.cmml" xref="A5.F25.6.3.m3.2.2.2.2.1"></eq><ci id="A5.F25.6.3.m3.2.2.2.2.2.cmml" xref="A5.F25.6.3.m3.2.2.2.2.2">𝑘</ci><cn type="integer" id="A5.F25.6.3.m3.2.2.2.2.3.cmml" xref="A5.F25.6.3.m3.2.2.2.2.3">512</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F25.6.3.m3.2d">n=2048,k=512</annotation></semantics></math> latents are easy to explain since many latents activate extremely densely (see <a href="#A5.SS5" title="E.5 Solutions with dense latents ‣ Appendix E Qualitative results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.5</span></a>).
</span></figcaption>
</figure>
<figure id="A5.F26" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_lofn_16M.png" id="A5.F26.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="479" height="314" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F26.6.3.1" class="ltx_text" style="font-size:90%;">Figure 26</span>: </span><span id="A5.F26.4.2" class="ltx_text" style="font-size:90%;">The <math id="A5.F26.3.1.m1.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A5.F26.3.1.m1.1b"><mrow id="A5.F26.3.1.m1.1.2" xref="A5.F26.3.1.m1.1.2.cmml"><mi id="A5.F26.3.1.m1.1.2.2" xref="A5.F26.3.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A5.F26.3.1.m1.1.2.1" xref="A5.F26.3.1.m1.1.2.1.cmml">​</mo><mrow id="A5.F26.3.1.m1.1.2.3.2" xref="A5.F26.3.1.m1.1.2.cmml"><mo stretchy="false" id="A5.F26.3.1.m1.1.2.3.2.1" xref="A5.F26.3.1.m1.1.2.cmml">(</mo><mi id="A5.F26.3.1.m1.1.1" xref="A5.F26.3.1.m1.1.1.cmml">N</mi><mo stretchy="false" id="A5.F26.3.1.m1.1.2.3.2.2" xref="A5.F26.3.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.F26.3.1.m1.1c"><apply id="A5.F26.3.1.m1.1.2.cmml" xref="A5.F26.3.1.m1.1.2"><times id="A5.F26.3.1.m1.1.2.1.cmml" xref="A5.F26.3.1.m1.1.2.1"></times><ci id="A5.F26.3.1.m1.1.2.2.cmml" xref="A5.F26.3.1.m1.1.2.2">𝐿</ci><ci id="A5.F26.3.1.m1.1.1.cmml" xref="A5.F26.3.1.m1.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F26.3.1.m1.1d">L(N)</annotation></semantics></math> scaling law, including the best 16M checkpoint, which we did not have time to train to the <math id="A5.F26.4.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A5.F26.4.2.m2.1b"><mrow id="A5.F26.4.2.m2.1.2" xref="A5.F26.4.2.m2.1.2.cmml"><mi id="A5.F26.4.2.m2.1.2.2" xref="A5.F26.4.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A5.F26.4.2.m2.1.2.1" xref="A5.F26.4.2.m2.1.2.1.cmml">​</mo><mrow id="A5.F26.4.2.m2.1.2.3.2" xref="A5.F26.4.2.m2.1.2.cmml"><mo stretchy="false" id="A5.F26.4.2.m2.1.2.3.2.1" xref="A5.F26.4.2.m2.1.2.cmml">(</mo><mi id="A5.F26.4.2.m2.1.1" xref="A5.F26.4.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="A5.F26.4.2.m2.1.2.3.2.2" xref="A5.F26.4.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.F26.4.2.m2.1c"><apply id="A5.F26.4.2.m2.1.2.cmml" xref="A5.F26.4.2.m2.1.2"><times id="A5.F26.4.2.m2.1.2.1.cmml" xref="A5.F26.4.2.m2.1.2.1"></times><ci id="A5.F26.4.2.m2.1.2.2.cmml" xref="A5.F26.4.2.m2.1.2.2">𝐿</ci><ci id="A5.F26.4.2.m2.1.1.cmml" xref="A5.F26.4.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F26.4.2.m2.1d">L(N)</annotation></semantics></math> token budget due to compute constraints.</span></figcaption>
</figure>
</section>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Miscellaneous small results</h2>

<section id="A6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.1 </span>Impact of different locations</h3>

<div id="A6.SS1.p1" class="ltx_para">
<p id="A6.SS1.p1.1" class="ltx_p">In a sweep across locations in GPT-2 small, we found that the optimal learning rate varies with layer and location type (MLP delta, attention delta, MLP post, attention post), but was within a factor of two.</p>
</div>
<div id="A6.SS1.p2" class="ltx_para">
<p id="A6.SS1.p2.1" class="ltx_p">Number of tokens needed for convergence is noticeably higher for earlier layers. MSE is lowest at early layers of the residual stream, increasing until the final layer, at which point it drops. Residual stream deltas have MSE peaking around layer 6, with attention delta falling sharply at the final layer.</p>
</div>
<figure id="A6.F27" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/placeholders/fig_gpt2sm_sweeploc_downstream_nmse_vs_loss_diff.png" id="A6.F27.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="269" height="180" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2406.04093/assets/placeholders/fig_gpt2sm_sweeploc_downstream_nmse_vs_loss_diff_first.png" id="A6.F27.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="269" height="180" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F27.4.1.1" class="ltx_text" style="font-size:90%;">Figure 27</span>: </span><span id="A6.F27.5.2" class="ltx_text ltx_font_bold" style="font-size:90%;">(a)<span id="A6.F27.5.2.1" class="ltx_text ltx_font_medium"> Normalized MSE gets worse later in the network, with the exception of the last two layers, where it improves. Later layers suffer worse loss differences when ablating to reconstruction, even at the final two layers. </span>(b)<span id="A6.F27.5.2.2" class="ltx_text ltx_font_medium"> First position token loss is more severely affected by ablation than other layers despite having lower normalized MSE. Overall loss difference has no clear relation with normalized MSE across layers. In very early layers and the final layer, where residual stream norm is also more normal (<a href="#A6.F30" title="Figure 30 ‣ F.2 Impact of token position ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 30</span></a>), we see a more typical loss difference and MSE. This is consistent with the hypothesis that the large norm component of the first token is primarily to serve attention operations at later tokens.
</span></span></figcaption>
</figure>
<div id="A6.SS1.p3" class="ltx_para">
<p id="A6.SS1.p3.1" class="ltx_p">When ablating to reconstructions, downstream loss and KL get strictly worse with layer. This is despite normalized MSE dropping at late layers. However, there appears to be an exception at final layers (layer 11/12 and especially 12/12 of GPT-2 small), which can have better normalized MSE than earlier layers, but more severe effects on downstream prediction (<a href="#A6.F27" title="Figure 27 ‣ F.1 Impact of different locations ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 27</span></a>).</p>
</div>
<div id="A6.SS1.p4" class="ltx_para">
<p id="A6.SS1.p4.1" class="ltx_p">We can also see that choice of layer affects different metrics differently (<a href="#A6.F28" title="Figure 28 ‣ F.1 Impact of different locations ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 28</span></a>). While earlier layers (unsurprisingly) have better N2G explanations, later layers do better on probe loss and sparsity.</p>
</div>
<div id="A6.SS1.p5" class="ltx_para">
<p id="A6.SS1.p5.1" class="ltx_p">In early results with autoencoders trained on the last layer of GPT-2 small, we found the results to be qualitatively worse than the layer 8 results, so we use layer 8 for all experiments going forwards.</p>
</div>
<figure id="A6.F28" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A6.F28.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_sweeploc_n2g_recall.png" id="A6.F28.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="257" height="60" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F28.sf1.4.2.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="A6.F28.sf1.2.1" class="ltx_text" style="font-size:90%;">Recall of N2G explanations
<br class="ltx_break"><math id="A6.F28.sf1.2.1.m1.2" class="ltx_Math" alttext="P(\text{n2g}&gt;0|\text{act}&gt;0)" display="inline"><semantics id="A6.F28.sf1.2.1.m1.2b"><mrow id="A6.F28.sf1.2.1.m1.2.2" xref="A6.F28.sf1.2.1.m1.2.2.cmml"><mi id="A6.F28.sf1.2.1.m1.2.2.3" xref="A6.F28.sf1.2.1.m1.2.2.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A6.F28.sf1.2.1.m1.2.2.2" xref="A6.F28.sf1.2.1.m1.2.2.2.cmml">​</mo><mrow id="A6.F28.sf1.2.1.m1.2.2.1.1" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="A6.F28.sf1.2.1.m1.2.2.1.1.2" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.cmml">(</mo><mrow id="A6.F28.sf1.2.1.m1.2.2.1.1.1" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.cmml"><mtext id="A6.F28.sf1.2.1.m1.2.2.1.1.1.2" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.2a.cmml">n2g</mtext><mo id="A6.F28.sf1.2.1.m1.2.2.1.1.1.1" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.1.cmml">&gt;</mo><mrow id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.cmml"><mn id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.2" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.2.cmml">0</mn><mo lspace="0em" rspace="0em" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.1" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mrow id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.2" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.1.cmml"><mo stretchy="false" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.2.1" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">|</mo><mtext id="A6.F28.sf1.2.1.m1.1.1" xref="A6.F28.sf1.2.1.m1.1.1a.cmml">act</mtext><mo fence="true" lspace="0em" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.2.2" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">&gt;</mo></mrow><mo lspace="0em" rspace="0em" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.1b" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mn id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.4" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.4.cmml">0</mn></mrow></mrow><mo stretchy="false" id="A6.F28.sf1.2.1.m1.2.2.1.1.3" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.F28.sf1.2.1.m1.2c"><apply id="A6.F28.sf1.2.1.m1.2.2.cmml" xref="A6.F28.sf1.2.1.m1.2.2"><times id="A6.F28.sf1.2.1.m1.2.2.2.cmml" xref="A6.F28.sf1.2.1.m1.2.2.2"></times><ci id="A6.F28.sf1.2.1.m1.2.2.3.cmml" xref="A6.F28.sf1.2.1.m1.2.2.3">𝑃</ci><apply id="A6.F28.sf1.2.1.m1.2.2.1.1.1.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1"><gt id="A6.F28.sf1.2.1.m1.2.2.1.1.1.1.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.1"></gt><ci id="A6.F28.sf1.2.1.m1.2.2.1.1.1.2a.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.2"><mtext id="A6.F28.sf1.2.1.m1.2.2.1.1.1.2.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.2">n2g</mtext></ci><apply id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3"><times id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.1.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.1"></times><cn type="integer" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.2.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.2">0</cn><apply id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.1.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.2"><csymbol cd="latexml" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.1.1.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.3.2.1">ket</csymbol><ci id="A6.F28.sf1.2.1.m1.1.1a.cmml" xref="A6.F28.sf1.2.1.m1.1.1"><mtext id="A6.F28.sf1.2.1.m1.1.1.cmml" xref="A6.F28.sf1.2.1.m1.1.1">act</mtext></ci></apply><cn type="integer" id="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.4.cmml" xref="A6.F28.sf1.2.1.m1.2.2.1.1.1.3.4">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.F28.sf1.2.1.m1.2d">P(\text{n2g}&gt;0|\text{act}&gt;0)</annotation></semantics></math></span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A6.F28.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_sweeploc_n2g_precision.png" id="A6.F28.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="254" height="60" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F28.sf2.4.2.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="A6.F28.sf2.2.1" class="ltx_text" style="font-size:90%;">Precision of N2G explanations
<br class="ltx_break"><math id="A6.F28.sf2.2.1.m1.2" class="ltx_Math" alttext="P(\text{act}&gt;0|\text{n2g}&gt;0)" display="inline"><semantics id="A6.F28.sf2.2.1.m1.2b"><mrow id="A6.F28.sf2.2.1.m1.2.2" xref="A6.F28.sf2.2.1.m1.2.2.cmml"><mi id="A6.F28.sf2.2.1.m1.2.2.3" xref="A6.F28.sf2.2.1.m1.2.2.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A6.F28.sf2.2.1.m1.2.2.2" xref="A6.F28.sf2.2.1.m1.2.2.2.cmml">​</mo><mrow id="A6.F28.sf2.2.1.m1.2.2.1.1" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="A6.F28.sf2.2.1.m1.2.2.1.1.2" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.cmml">(</mo><mrow id="A6.F28.sf2.2.1.m1.2.2.1.1.1" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.cmml"><mtext id="A6.F28.sf2.2.1.m1.2.2.1.1.1.2" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.2a.cmml">act</mtext><mo id="A6.F28.sf2.2.1.m1.2.2.1.1.1.1" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.1.cmml">&gt;</mo><mrow id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.cmml"><mn id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.2" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.2.cmml">0</mn><mo lspace="0em" rspace="0em" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.1" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mrow id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.2" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.1.cmml"><mo stretchy="false" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.2.1" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">|</mo><mtext id="A6.F28.sf2.2.1.m1.1.1" xref="A6.F28.sf2.2.1.m1.1.1a.cmml">n2g</mtext><mo fence="true" lspace="0em" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.2.2" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.1.1.cmml">&gt;</mo></mrow><mo lspace="0em" rspace="0em" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.1b" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mn id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.4" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.4.cmml">0</mn></mrow></mrow><mo stretchy="false" id="A6.F28.sf2.2.1.m1.2.2.1.1.3" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.F28.sf2.2.1.m1.2c"><apply id="A6.F28.sf2.2.1.m1.2.2.cmml" xref="A6.F28.sf2.2.1.m1.2.2"><times id="A6.F28.sf2.2.1.m1.2.2.2.cmml" xref="A6.F28.sf2.2.1.m1.2.2.2"></times><ci id="A6.F28.sf2.2.1.m1.2.2.3.cmml" xref="A6.F28.sf2.2.1.m1.2.2.3">𝑃</ci><apply id="A6.F28.sf2.2.1.m1.2.2.1.1.1.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1"><gt id="A6.F28.sf2.2.1.m1.2.2.1.1.1.1.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.1"></gt><ci id="A6.F28.sf2.2.1.m1.2.2.1.1.1.2a.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.2"><mtext id="A6.F28.sf2.2.1.m1.2.2.1.1.1.2.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.2">act</mtext></ci><apply id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3"><times id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.1.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.1"></times><cn type="integer" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.2.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.2">0</cn><apply id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.1.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.2"><csymbol cd="latexml" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.1.1.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.3.2.1">ket</csymbol><ci id="A6.F28.sf2.2.1.m1.1.1a.cmml" xref="A6.F28.sf2.2.1.m1.1.1"><mtext id="A6.F28.sf2.2.1.m1.1.1.cmml" xref="A6.F28.sf2.2.1.m1.1.1">n2g</mtext></ci></apply><cn type="integer" id="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.4.cmml" xref="A6.F28.sf2.2.1.m1.2.2.1.1.1.3.4">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.F28.sf2.2.1.m1.2d">P(\text{act}&gt;0|\text{n2g}&gt;0)</annotation></semantics></math></span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A6.F28.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_sweeploc_probe_grid.png" id="A6.F28.sf3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="261" height="60" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F28.sf3.2.1.1" class="ltx_text" style="font-size:90%;">(c)</span> </span><span id="A6.F28.sf3.3.2" class="ltx_text" style="font-size:90%;">Probe loss</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A6.F28.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_sweeploc_ablatevote_sparsity_l1l2.png" id="A6.F28.sf4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="254" height="60" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F28.sf4.2.1.1" class="ltx_text" style="font-size:90%;">(d)</span> </span><span id="A6.F28.sf4.3.2" class="ltx_text" style="font-size:90%;">Logit diff sparsity</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A6.F28.6.3.1" class="ltx_text" style="font-size:90%;">Figure 28</span>: </span><span id="A6.F28.4.2" class="ltx_text" style="font-size:90%;">Metrics as a function of layer, for GPT-2 small autoencoders with <math id="A6.F28.3.1.m1.1" class="ltx_Math" alttext="k=32" display="inline"><semantics id="A6.F28.3.1.m1.1b"><mrow id="A6.F28.3.1.m1.1.1" xref="A6.F28.3.1.m1.1.1.cmml"><mi id="A6.F28.3.1.m1.1.1.2" xref="A6.F28.3.1.m1.1.1.2.cmml">k</mi><mo id="A6.F28.3.1.m1.1.1.1" xref="A6.F28.3.1.m1.1.1.1.cmml">=</mo><mn id="A6.F28.3.1.m1.1.1.3" xref="A6.F28.3.1.m1.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.F28.3.1.m1.1c"><apply id="A6.F28.3.1.m1.1.1.cmml" xref="A6.F28.3.1.m1.1.1"><eq id="A6.F28.3.1.m1.1.1.1.cmml" xref="A6.F28.3.1.m1.1.1.1"></eq><ci id="A6.F28.3.1.m1.1.1.2.cmml" xref="A6.F28.3.1.m1.1.1.2">𝑘</ci><cn type="integer" id="A6.F28.3.1.m1.1.1.3.cmml" xref="A6.F28.3.1.m1.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.F28.3.1.m1.1d">k=32</annotation></semantics></math> and <math id="A6.F28.4.2.m2.1" class="ltx_Math" alttext="n=32768" display="inline"><semantics id="A6.F28.4.2.m2.1b"><mrow id="A6.F28.4.2.m2.1.1" xref="A6.F28.4.2.m2.1.1.cmml"><mi id="A6.F28.4.2.m2.1.1.2" xref="A6.F28.4.2.m2.1.1.2.cmml">n</mi><mo id="A6.F28.4.2.m2.1.1.1" xref="A6.F28.4.2.m2.1.1.1.cmml">=</mo><mn id="A6.F28.4.2.m2.1.1.3" xref="A6.F28.4.2.m2.1.1.3.cmml">32768</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.F28.4.2.m2.1c"><apply id="A6.F28.4.2.m2.1.1.cmml" xref="A6.F28.4.2.m2.1.1"><eq id="A6.F28.4.2.m2.1.1.1.cmml" xref="A6.F28.4.2.m2.1.1.1"></eq><ci id="A6.F28.4.2.m2.1.1.2.cmml" xref="A6.F28.4.2.m2.1.1.2">𝑛</ci><cn type="integer" id="A6.F28.4.2.m2.1.1.3.cmml" xref="A6.F28.4.2.m2.1.1.3">32768</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.F28.4.2.m2.1d">n=32768</annotation></semantics></math>. Earlier layers are easier to explain in terms of token patterns, but later layers are better for recovering features and have sparser logit diffs.
</span></figcaption>
</figure>
</section>
<section id="A6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.2 </span>Impact of token position</h3>

<figure id="A6.F29" class="ltx_figure"><img src="/html/2406.04093/assets/placeholders/fig_gpt2sm_downstream_nmse_by_token.png" id="A6.F29.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="225" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F29.2.1.1" class="ltx_text" style="font-size:90%;">Figure 29</span>: </span><span id="A6.F29.3.2" class="ltx_text" style="font-size:90%;">Later tokens are more difficult to reconstruct. (lower is better) </span></figcaption>
</figure>
<div id="A6.SS2.p1" class="ltx_para">
<p id="A6.SS2.p1.1" class="ltx_p">We find that tokens at later positions are harder to reconstruct (<a href="#A6.F29" title="Figure 29 ‣ F.2 Impact of token position ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 29</span></a>). We hypothesize that this is because the residual stream at later positions have more features. First positions are particularly egregiously easy to reconstruct, in terms of normalized MSE, but they also generally have residual stream norm more than an order of magnitude larger than other positions ( <a href="#A6.F30" title="Figure 30 ‣ F.2 Impact of token position ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 30</span></a> shows that in GPT-2 small, the exception is only at early layers and the final layer of GPT-2 small). This phenomenon was explained in <cite class="ltx_cite ltx_citemacro_citep">[Sun et al., <a href="#bib.bib60" title="" class="ltx_ref">2024</a>, Xiao et al., <a href="#bib.bib69" title="" class="ltx_ref">2023</a>]</cite>, which demonstrate that these activations serve as crucial attention resting states.</p>
</div>
<div id="A6.SS2.p2" class="ltx_para">
<p id="A6.SS2.p2.1" class="ltx_p">First token positions have significantly worse downstream loss and KL after ablating to autoencoder reconstruction at layers with these large norms (<a href="#A6.F27" title="Figure 27 ‣ F.1 Impact of different locations ‣ Appendix F Miscellaneous small results ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 27</span></a>), despite having better normalized MSE. This is consistent with the hypothesis that the large norm directions at the first position are important for loss on other tokens but not the current token. This position then potentially gets subtracted back out at the final layer as the model focuses on current-token prediction.</p>
</div>
<figure id="A6.F30" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_gpt2sm_norm_per_tok.png" id="A6.F30.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="200" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A6.F30.2.1.1" class="ltx_text" style="font-size:90%;">Figure 30</span>: </span><span id="A6.F30.3.2" class="ltx_text" style="font-size:90%;">Residual stream norms by context position. First token positions are more than an order of magnitude larger than other positions, except at the first and last layer for GPT-2 small.</span></figcaption>
</figure>
</section>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>Irreducible loss term</h2>

<div id="A7.p1" class="ltx_para">
<p id="A7.p1.1" class="ltx_p">In language models, the irreducible loss exists because text has some intrinsic unpredictableness—even with a perfect language model, the loss of predicting the next token cannot be zero.
Since an arbitrarily large autoencoder can in fact perfectly reconstruct the input, we initially expected there to be no irreducible loss term. However, we found the quality of the fit to be substantially less good without an irreducible loss.</p>
</div>
<div id="A7.p2" class="ltx_para">
<p id="A7.p2.1" class="ltx_p">While we don’t fully understand the reason behind the irreducible loss term, our hypothesis is that the activations are made of a spectrum of components with different amount of structure. We expect less strucutured data to also have a worse scaling exponent. At the most extreme, some amount of the activations could be completely unstructured gaussian noise. In synthetic experiments with unstructured noise (see <a href="#A7.F31" title="Figure 31 ‣ Appendix G Irreducible loss term ‣ Scaling and evaluating sparse autoencoders" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 31</span></a>), we find an <math id="A7.p2.1.m1.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A7.p2.1.m1.1a"><mrow id="A7.p2.1.m1.1.2" xref="A7.p2.1.m1.1.2.cmml"><mi id="A7.p2.1.m1.1.2.2" xref="A7.p2.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A7.p2.1.m1.1.2.1" xref="A7.p2.1.m1.1.2.1.cmml">​</mo><mrow id="A7.p2.1.m1.1.2.3.2" xref="A7.p2.1.m1.1.2.cmml"><mo stretchy="false" id="A7.p2.1.m1.1.2.3.2.1" xref="A7.p2.1.m1.1.2.cmml">(</mo><mi id="A7.p2.1.m1.1.1" xref="A7.p2.1.m1.1.1.cmml">N</mi><mo stretchy="false" id="A7.p2.1.m1.1.2.3.2.2" xref="A7.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A7.p2.1.m1.1b"><apply id="A7.p2.1.m1.1.2.cmml" xref="A7.p2.1.m1.1.2"><times id="A7.p2.1.m1.1.2.1.cmml" xref="A7.p2.1.m1.1.2.1"></times><ci id="A7.p2.1.m1.1.2.2.cmml" xref="A7.p2.1.m1.1.2.2">𝐿</ci><ci id="A7.p2.1.m1.1.1.cmml" xref="A7.p2.1.m1.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.p2.1.m1.1c">L(N)</annotation></semantics></math> exponent of -0.04 on 768-dimensional gaussian data, which is much shallower than the approximately -0.26 we see on GPT-2-small activations of a similar dimensionality.</p>
</div>
<figure id="A7.F31" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_random_scaling.png" id="A7.F31.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="299" height="196" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A7.F31.2.1.1" class="ltx_text" style="font-size:90%;">Figure 31</span>: </span><span id="A7.F31.3.2" class="ltx_text" style="font-size:90%;">L(N) scaling law for training on 768-dimensional random gaussian data with k=32</span></figcaption>
</figure>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Further probe based evaluations results</h2>

<figure id="A8.F32" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_probe_through_training.png" id="A8.F32.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="479" height="298" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A8.F32.2.1.1" class="ltx_text" style="font-size:90%;">Figure 32</span>: </span><span id="A8.F32.3.2" class="ltx_text" style="font-size:90%;">Probe eval scores through training for 128k, 1M, and 16M autoencoders. The baseline score of using the channels of the residual stream directly is 0.600.</span></figcaption>
</figure>
<figure id="A8.F33" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_probe_breakdown_by_task_zeronorm.png" id="A8.F33.g1" class="ltx_graphics ltx_centering ltx_img_square" width="479" height="459" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A8.F33.2.1.1" class="ltx_text" style="font-size:90%;">Figure 33</span>: </span><span id="A8.F33.3.2" class="ltx_text" style="font-size:90%;">Probe eval scores for the 16M autoencoder starting at the point where probe features start developing (around 10B tokens elapsed).</span></figcaption>
</figure>
<figure id="A8.F34" class="ltx_figure"><img src="/html/2406.04093/assets/figures/fig_probe_breakdown_by_task.png" id="A8.F34.g1" class="ltx_graphics ltx_centering ltx_img_square" width="598" height="591" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A8.F34.2.1.1" class="ltx_text" style="font-size:90%;">Figure 34</span>: </span><span id="A8.F34.3.2" class="ltx_text" style="font-size:90%;">Probe eval scores for the 16M autoencoder broken down by task. Some lines (europarl, bigrams, occupations, ag_news) are aggregations of multiple tasks.</span></figcaption>
</figure>
<figure id="A8.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A8.T1.2.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="A8.T1.3.2" class="ltx_text" style="font-size:90%;">Tasks used in the probe-based evaluation suite</span></figcaption>
<table id="A8.T1.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A8.T1.4.1.1" class="ltx_tr">
<th id="A8.T1.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t"><span id="A8.T1.4.1.1.1.1" class="ltx_text ltx_font_bold">Task Name</span></th>
<th id="A8.T1.4.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_t"><span id="A8.T1.4.1.1.2.1" class="ltx_text ltx_font_bold">Details</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A8.T1.4.2.1" class="ltx_tr">
<td id="A8.T1.4.2.1.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">amazon</td>
<td id="A8.T1.4.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">McAuley and Leskovec [<a href="#bib.bib41" title="" class="ltx_ref">2013</a>]</cite></td>
</tr>
<tr id="A8.T1.4.3.2" class="ltx_tr">
<td id="A8.T1.4.3.2.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">sciq</td>
<td id="A8.T1.4.3.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Welbl et al. [<a href="#bib.bib67" title="" class="ltx_ref">2017</a>]</cite></td>
</tr>
<tr id="A8.T1.4.4.3" class="ltx_tr">
<td id="A8.T1.4.4.3.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">truthfulqa</td>
<td id="A8.T1.4.4.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Lin et al. [<a href="#bib.bib32" title="" class="ltx_ref">2021</a>]</cite></td>
</tr>
<tr id="A8.T1.4.5.4" class="ltx_tr">
<td id="A8.T1.4.5.4.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">mc_taco</td>
<td id="A8.T1.4.5.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Zhou et al. [<a href="#bib.bib72" title="" class="ltx_ref">2019</a>]</cite></td>
</tr>
<tr id="A8.T1.4.6.5" class="ltx_tr">
<td id="A8.T1.4.6.5.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">piqa</td>
<td id="A8.T1.4.6.5.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Bisk et al. [<a href="#bib.bib4" title="" class="ltx_ref">2020</a>]</cite></td>
</tr>
<tr id="A8.T1.4.7.6" class="ltx_tr">
<td id="A8.T1.4.7.6.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">quail</td>
<td id="A8.T1.4.7.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Rogers et al. [<a href="#bib.bib53" title="" class="ltx_ref">2020</a>]</cite></td>
</tr>
<tr id="A8.T1.4.8.7" class="ltx_tr">
<td id="A8.T1.4.8.7.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">quartz</td>
<td id="A8.T1.4.8.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Tafjord et al. [<a href="#bib.bib61" title="" class="ltx_ref">2019</a>]</cite></td>
</tr>
<tr id="A8.T1.4.9.8" class="ltx_tr">
<td id="A8.T1.4.9.8.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">justice</td>
<td id="A8.T1.4.9.8.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="4"><span id="A8.T1.4.9.8.2.1" class="ltx_text"><cite class="ltx_cite ltx_citemacro_citet">Hendrycks et al. [<a href="#bib.bib19" title="" class="ltx_ref">2020</a>]</cite></span></td>
</tr>
<tr id="A8.T1.4.10.9" class="ltx_tr">
<td id="A8.T1.4.10.9.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">virtue</td>
</tr>
<tr id="A8.T1.4.11.10" class="ltx_tr">
<td id="A8.T1.4.11.10.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">utilitarianism</td>
</tr>
<tr id="A8.T1.4.12.11" class="ltx_tr">
<td id="A8.T1.4.12.11.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">deontology</td>
</tr>
<tr id="A8.T1.4.13.12" class="ltx_tr">
<td id="A8.T1.4.13.12.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">commonsense_qa</td>
<td id="A8.T1.4.13.12.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Talmor et al. [<a href="#bib.bib63" title="" class="ltx_ref">2022</a>]</cite></td>
</tr>
<tr id="A8.T1.4.14.13" class="ltx_tr">
<td id="A8.T1.4.14.13.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">openbookqa</td>
<td id="A8.T1.4.14.13.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Mihaylov et al. [<a href="#bib.bib44" title="" class="ltx_ref">2018</a>]</cite></td>
</tr>
<tr id="A8.T1.4.15.14" class="ltx_tr">
<td id="A8.T1.4.15.14.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">base64</td>
<td id="A8.T1.4.15.14.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">discrimination of base64 vs pretraining data</td>
</tr>
<tr id="A8.T1.4.16.15" class="ltx_tr">
<td id="A8.T1.4.16.15.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_isalive</td>
<td id="A8.T1.4.16.15.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="25"><span id="A8.T1.4.16.15.2.1" class="ltx_text"><cite class="ltx_cite ltx_citemacro_citet">Gurnee et al. [<a href="#bib.bib18" title="" class="ltx_ref">2023</a>]</cite></span></td>
</tr>
<tr id="A8.T1.4.17.16" class="ltx_tr">
<td id="A8.T1.4.17.16.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_sex_or_gender</td>
</tr>
<tr id="A8.T1.4.18.17" class="ltx_tr">
<td id="A8.T1.4.18.17.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_occupation_isjournalist</td>
</tr>
<tr id="A8.T1.4.19.18" class="ltx_tr">
<td id="A8.T1.4.19.18.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_occupation_isathlete</td>
</tr>
<tr id="A8.T1.4.20.19" class="ltx_tr">
<td id="A8.T1.4.20.19.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_occupation_isactor</td>
</tr>
<tr id="A8.T1.4.21.20" class="ltx_tr">
<td id="A8.T1.4.21.20.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_occupation_ispolitician</td>
</tr>
<tr id="A8.T1.4.22.21" class="ltx_tr">
<td id="A8.T1.4.22.21.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_occupation_issinger</td>
</tr>
<tr id="A8.T1.4.23.22" class="ltx_tr">
<td id="A8.T1.4.23.22.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wikidata_occupation_isresearcher</td>
</tr>
<tr id="A8.T1.4.24.23" class="ltx_tr">
<td id="A8.T1.4.24.23.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_high-school</td>
</tr>
<tr id="A8.T1.4.25.24" class="ltx_tr">
<td id="A8.T1.4.25.24.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_living-room</td>
</tr>
<tr id="A8.T1.4.26.25" class="ltx_tr">
<td id="A8.T1.4.26.25.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_social-security</td>
</tr>
<tr id="A8.T1.4.27.26" class="ltx_tr">
<td id="A8.T1.4.27.26.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_credit-card</td>
</tr>
<tr id="A8.T1.4.28.27" class="ltx_tr">
<td id="A8.T1.4.28.27.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_blood-pressure</td>
</tr>
<tr id="A8.T1.4.29.28" class="ltx_tr">
<td id="A8.T1.4.29.28.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_prime-factors</td>
</tr>
<tr id="A8.T1.4.30.29" class="ltx_tr">
<td id="A8.T1.4.30.29.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_social-media</td>
</tr>
<tr id="A8.T1.4.31.30" class="ltx_tr">
<td id="A8.T1.4.31.30.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_gene-expression</td>
</tr>
<tr id="A8.T1.4.32.31" class="ltx_tr">
<td id="A8.T1.4.32.31.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_control-group</td>
</tr>
<tr id="A8.T1.4.33.32" class="ltx_tr">
<td id="A8.T1.4.33.32.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_magnetic-field</td>
</tr>
<tr id="A8.T1.4.34.33" class="ltx_tr">
<td id="A8.T1.4.34.33.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_cell-lines</td>
</tr>
<tr id="A8.T1.4.35.34" class="ltx_tr">
<td id="A8.T1.4.35.34.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_trial-court</td>
</tr>
<tr id="A8.T1.4.36.35" class="ltx_tr">
<td id="A8.T1.4.36.35.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_second-derivative</td>
</tr>
<tr id="A8.T1.4.37.36" class="ltx_tr">
<td id="A8.T1.4.37.36.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_north-america</td>
</tr>
<tr id="A8.T1.4.38.37" class="ltx_tr">
<td id="A8.T1.4.38.37.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_human-rights</td>
</tr>
<tr id="A8.T1.4.39.38" class="ltx_tr">
<td id="A8.T1.4.39.38.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_side-effects</td>
</tr>
<tr id="A8.T1.4.40.39" class="ltx_tr">
<td id="A8.T1.4.40.39.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_public-health</td>
</tr>
<tr id="A8.T1.4.41.40" class="ltx_tr">
<td id="A8.T1.4.41.40.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_federal-government</td>
<td id="A8.T1.4.41.40.2" class="ltx_td ltx_border_r"></td>
</tr>
<tr id="A8.T1.4.42.41" class="ltx_tr">
<td id="A8.T1.4.42.41.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_third-party</td>
<td id="A8.T1.4.42.41.2" class="ltx_td ltx_border_r"></td>
</tr>
<tr id="A8.T1.4.43.42" class="ltx_tr">
<td id="A8.T1.4.43.42.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_clinical-trials</td>
<td id="A8.T1.4.43.42.2" class="ltx_td ltx_border_r"></td>
</tr>
<tr id="A8.T1.4.44.43" class="ltx_tr">
<td id="A8.T1.4.44.43.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">phrase_mental-health</td>
<td id="A8.T1.4.44.43.2" class="ltx_td ltx_border_r"></td>
</tr>
<tr id="A8.T1.4.45.44" class="ltx_tr">
<td id="A8.T1.4.45.44.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">social_iqa</td>
<td id="A8.T1.4.45.44.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Sap et al. [<a href="#bib.bib55" title="" class="ltx_ref">2019</a>]</cite></td>
</tr>
<tr id="A8.T1.4.46.45" class="ltx_tr">
<td id="A8.T1.4.46.45.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">wic</td>
<td id="A8.T1.4.46.45.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="2"><span id="A8.T1.4.46.45.2.1" class="ltx_text"><cite class="ltx_cite ltx_citemacro_citet">Wang et al. [<a href="#bib.bib66" title="" class="ltx_ref">2018</a>]</cite></span></td>
</tr>
<tr id="A8.T1.4.47.46" class="ltx_tr">
<td id="A8.T1.4.47.46.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">cola</td>
</tr>
<tr id="A8.T1.4.48.47" class="ltx_tr">
<td id="A8.T1.4.48.47.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">gpt2disc</td>
<td id="A8.T1.4.48.47.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">OpenAI [<a href="#bib.bib49" title="" class="ltx_ref">2019</a>]</cite></td>
</tr>
<tr id="A8.T1.4.49.48" class="ltx_tr">
<td id="A8.T1.4.49.48.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">ag_news_world</td>
<td id="A8.T1.4.49.48.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="4"><span id="A8.T1.4.49.48.2.1" class="ltx_text"><cite class="ltx_cite ltx_citemacro_citet"><a href="#bib.bib17" title="" class="ltx_ref">Gulli </a></cite></span></td>
</tr>
<tr id="A8.T1.4.50.49" class="ltx_tr">
<td id="A8.T1.4.50.49.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">ag_news_sports</td>
</tr>
<tr id="A8.T1.4.51.50" class="ltx_tr">
<td id="A8.T1.4.51.50.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">ag_news_business</td>
</tr>
<tr id="A8.T1.4.52.51" class="ltx_tr">
<td id="A8.T1.4.52.51.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">ag_news_scitech</td>
</tr>
<tr id="A8.T1.4.53.52" class="ltx_tr">
<td id="A8.T1.4.53.52.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_es</td>
<td id="A8.T1.4.53.52.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="8"><span id="A8.T1.4.53.52.2.1" class="ltx_text"><cite class="ltx_cite ltx_citemacro_citet">Koehn [<a href="#bib.bib27" title="" class="ltx_ref">2005</a>]</cite></span></td>
</tr>
<tr id="A8.T1.4.54.53" class="ltx_tr">
<td id="A8.T1.4.54.53.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_en</td>
</tr>
<tr id="A8.T1.4.55.54" class="ltx_tr">
<td id="A8.T1.4.55.54.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_fr</td>
</tr>
<tr id="A8.T1.4.56.55" class="ltx_tr">
<td id="A8.T1.4.56.55.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_nl</td>
</tr>
<tr id="A8.T1.4.57.56" class="ltx_tr">
<td id="A8.T1.4.57.56.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_it</td>
</tr>
<tr id="A8.T1.4.58.57" class="ltx_tr">
<td id="A8.T1.4.58.57.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_el</td>
</tr>
<tr id="A8.T1.4.59.58" class="ltx_tr">
<td id="A8.T1.4.59.58.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_de</td>
</tr>
<tr id="A8.T1.4.60.59" class="ltx_tr">
<td id="A8.T1.4.60.59.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_pt</td>
</tr>
<tr id="A8.T1.4.61.60" class="ltx_tr">
<td id="A8.T1.4.61.60.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">europarl_sv</td>
<td id="A8.T1.4.61.60.2" class="ltx_td ltx_border_r"></td>
</tr>
<tr id="A8.T1.4.62.61" class="ltx_tr">
<td id="A8.T1.4.62.61.1" class="ltx_td ltx_align_left ltx_border_b ltx_border_l ltx_border_r ltx_border_t">jigsaw</td>
<td id="A8.T1.4.62.61.2" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t"><cite class="ltx_cite ltx_citemacro_citet">Cjadams et al. [<a href="#bib.bib9" title="" class="ltx_ref">2017</a>]</cite></td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Contributions</h2>

<div id="A9.p1" class="ltx_para">
<p id="A9.p1.2" class="ltx_p"><span id="A9.p1.2.1" class="ltx_text ltx_font_bold">Leo Gao</span> implemented the autoencoder training codebase and basic infrastructure for GPT-4 experiments. Leo worked on the systems, including kernels, parallelism, numerics, data processing, etc. Leo conducted most scaling and architecture experiments: TopK and AuxK, tied initialization, number of latents, subject model size, batch size, token budget, optimal lr, <math id="A9.p1.1.m1.1" class="ltx_Math" alttext="L(C)" display="inline"><semantics id="A9.p1.1.m1.1a"><mrow id="A9.p1.1.m1.1.2" xref="A9.p1.1.m1.1.2.cmml"><mi id="A9.p1.1.m1.1.2.2" xref="A9.p1.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A9.p1.1.m1.1.2.1" xref="A9.p1.1.m1.1.2.1.cmml">​</mo><mrow id="A9.p1.1.m1.1.2.3.2" xref="A9.p1.1.m1.1.2.cmml"><mo stretchy="false" id="A9.p1.1.m1.1.2.3.2.1" xref="A9.p1.1.m1.1.2.cmml">(</mo><mi id="A9.p1.1.m1.1.1" xref="A9.p1.1.m1.1.1.cmml">C</mi><mo stretchy="false" id="A9.p1.1.m1.1.2.3.2.2" xref="A9.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A9.p1.1.m1.1b"><apply id="A9.p1.1.m1.1.2.cmml" xref="A9.p1.1.m1.1.2"><times id="A9.p1.1.m1.1.2.1.cmml" xref="A9.p1.1.m1.1.2.1"></times><ci id="A9.p1.1.m1.1.2.2.cmml" xref="A9.p1.1.m1.1.2.2">𝐿</ci><ci id="A9.p1.1.m1.1.1.cmml" xref="A9.p1.1.m1.1.1">𝐶</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p1.1.m1.1c">L(C)</annotation></semantics></math>, <math id="A9.p1.2.m2.1" class="ltx_Math" alttext="L(N)" display="inline"><semantics id="A9.p1.2.m2.1a"><mrow id="A9.p1.2.m2.1.2" xref="A9.p1.2.m2.1.2.cmml"><mi id="A9.p1.2.m2.1.2.2" xref="A9.p1.2.m2.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A9.p1.2.m2.1.2.1" xref="A9.p1.2.m2.1.2.1.cmml">​</mo><mrow id="A9.p1.2.m2.1.2.3.2" xref="A9.p1.2.m2.1.2.cmml"><mo stretchy="false" id="A9.p1.2.m2.1.2.3.2.1" xref="A9.p1.2.m2.1.2.cmml">(</mo><mi id="A9.p1.2.m2.1.1" xref="A9.p1.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="A9.p1.2.m2.1.2.3.2.2" xref="A9.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A9.p1.2.m2.1b"><apply id="A9.p1.2.m2.1.2.cmml" xref="A9.p1.2.m2.1.2"><times id="A9.p1.2.m2.1.2.1.cmml" xref="A9.p1.2.m2.1.2.1"></times><ci id="A9.p1.2.m2.1.2.2.cmml" xref="A9.p1.2.m2.1.2.2">𝐿</ci><ci id="A9.p1.2.m2.1.1.cmml" xref="A9.p1.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p1.2.m2.1c">L(N)</annotation></semantics></math>, random data, etc., and iterated on many other architecture and algorithmic choices. Leo designed and iterated on the probe based metric. Leo did investigations into downstream loss, feature recall/precision, and some early explorations into circuit sparsity.</p>
</div>
<div id="A9.p2" class="ltx_para">
<p id="A9.p2.2" class="ltx_p"><span id="A9.p2.2.1" class="ltx_text ltx_font_bold">Tom Dupré la Tour</span> studied activation shrinkage, progressive recovery, and Multi-TopK. Tom implemented and trained the Gated and ProLU baselines, and trained and analyzed different layers and locations in GPT-2 small. Tom helped refine the scaling laws for <math id="A9.p2.1.m1.2" class="ltx_Math" alttext="L(N,K)" display="inline"><semantics id="A9.p2.1.m1.2a"><mrow id="A9.p2.1.m1.2.3" xref="A9.p2.1.m1.2.3.cmml"><mi id="A9.p2.1.m1.2.3.2" xref="A9.p2.1.m1.2.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A9.p2.1.m1.2.3.1" xref="A9.p2.1.m1.2.3.1.cmml">​</mo><mrow id="A9.p2.1.m1.2.3.3.2" xref="A9.p2.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="A9.p2.1.m1.2.3.3.2.1" xref="A9.p2.1.m1.2.3.3.1.cmml">(</mo><mi id="A9.p2.1.m1.1.1" xref="A9.p2.1.m1.1.1.cmml">N</mi><mo id="A9.p2.1.m1.2.3.3.2.2" xref="A9.p2.1.m1.2.3.3.1.cmml">,</mo><mi id="A9.p2.1.m1.2.2" xref="A9.p2.1.m1.2.2.cmml">K</mi><mo stretchy="false" id="A9.p2.1.m1.2.3.3.2.3" xref="A9.p2.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A9.p2.1.m1.2b"><apply id="A9.p2.1.m1.2.3.cmml" xref="A9.p2.1.m1.2.3"><times id="A9.p2.1.m1.2.3.1.cmml" xref="A9.p2.1.m1.2.3.1"></times><ci id="A9.p2.1.m1.2.3.2.cmml" xref="A9.p2.1.m1.2.3.2">𝐿</ci><interval closure="open" id="A9.p2.1.m1.2.3.3.1.cmml" xref="A9.p2.1.m1.2.3.3.2"><ci id="A9.p2.1.m1.1.1.cmml" xref="A9.p2.1.m1.1.1">𝑁</ci><ci id="A9.p2.1.m1.2.2.cmml" xref="A9.p2.1.m1.2.2">𝐾</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p2.1.m1.2c">L(N,K)</annotation></semantics></math> and <math id="A9.p2.2.m2.1" class="ltx_Math" alttext="L_{s}(N)" display="inline"><semantics id="A9.p2.2.m2.1a"><mrow id="A9.p2.2.m2.1.2" xref="A9.p2.2.m2.1.2.cmml"><msub id="A9.p2.2.m2.1.2.2" xref="A9.p2.2.m2.1.2.2.cmml"><mi id="A9.p2.2.m2.1.2.2.2" xref="A9.p2.2.m2.1.2.2.2.cmml">L</mi><mi id="A9.p2.2.m2.1.2.2.3" xref="A9.p2.2.m2.1.2.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="A9.p2.2.m2.1.2.1" xref="A9.p2.2.m2.1.2.1.cmml">​</mo><mrow id="A9.p2.2.m2.1.2.3.2" xref="A9.p2.2.m2.1.2.cmml"><mo stretchy="false" id="A9.p2.2.m2.1.2.3.2.1" xref="A9.p2.2.m2.1.2.cmml">(</mo><mi id="A9.p2.2.m2.1.1" xref="A9.p2.2.m2.1.1.cmml">N</mi><mo stretchy="false" id="A9.p2.2.m2.1.2.3.2.2" xref="A9.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A9.p2.2.m2.1b"><apply id="A9.p2.2.m2.1.2.cmml" xref="A9.p2.2.m2.1.2"><times id="A9.p2.2.m2.1.2.1.cmml" xref="A9.p2.2.m2.1.2.1"></times><apply id="A9.p2.2.m2.1.2.2.cmml" xref="A9.p2.2.m2.1.2.2"><csymbol cd="ambiguous" id="A9.p2.2.m2.1.2.2.1.cmml" xref="A9.p2.2.m2.1.2.2">subscript</csymbol><ci id="A9.p2.2.m2.1.2.2.2.cmml" xref="A9.p2.2.m2.1.2.2.2">𝐿</ci><ci id="A9.p2.2.m2.1.2.2.3.cmml" xref="A9.p2.2.m2.1.2.2.3">𝑠</ci></apply><ci id="A9.p2.2.m2.1.1.cmml" xref="A9.p2.2.m2.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p2.2.m2.1c">L_{s}(N)</annotation></semantics></math>. Tom discovered the latent sub-spaces (refining an idea and code originally from Carroll Wainwright).</p>
</div>
<div id="A9.p3" class="ltx_para">
<p id="A9.p3.1" class="ltx_p"><span id="A9.p3.1.1" class="ltx_text ltx_font_bold">Henk Tillman</span> worked on N2G explanations and LM-based explainer scoring. Henk worked on infrastructure for scraping activations. Henk worked on finding qualitatively interesting features, including safety-related features.</p>
</div>
<div id="A9.p4" class="ltx_para">
<p id="A9.p4.1" class="ltx_p"><span id="A9.p4.1.1" class="ltx_text ltx_font_bold">Jeff Wu</span> studied ablation effects and sparsity, and ablation reconstruction. Jeff managed infrastructure for metrics, and wrote the visualizer data collation and website code. Jeff analyzed overall cost of having a fully sparse bottleneck, and analyzed recurring dense features. Cathy and Jeff worked on finding safety-relevant features using attribution. Jeff managed core researchers on the project.</p>
</div>
<div id="A9.p5" class="ltx_para">
<p id="A9.p5.1" class="ltx_p"><span id="A9.p5.1.1" class="ltx_text ltx_font_bold">Gabriel Goh</span> suggested using TopK, and contributed intuitions about TopK, AuxK, and optimization.</p>
</div>
<div id="A9.p6" class="ltx_para">
<p id="A9.p6.1" class="ltx_p"><span id="A9.p6.1.1" class="ltx_text ltx_font_bold">Rajan Troll</span> contributed intuitions about and advised on optimization, scaling, and systems.</p>
</div>
<div id="A9.p7" class="ltx_para">
<p id="A9.p7.1" class="ltx_p"><span id="A9.p7.1.1" class="ltx_text ltx_font_bold">Alec Radford</span> suggested using the irreducible loss term and contributed intuitions about and advised on the probe based metric, optimization, and scaling.</p>
</div>
<div id="A9.p8" class="ltx_para">
<p id="A9.p8.1" class="ltx_p"><span id="A9.p8.1.1" class="ltx_text ltx_font_bold">Jan Leike</span> and <span id="A9.p8.1.2" class="ltx_text ltx_font_bold">Ilya Sutskever</span> managed and led the Superalignment team.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.04092" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.04093" class="ar5iv-text-button ar5iv-severity-ok">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.04093">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.04093" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.04094" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Jul  5 19:17:37 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.04093
Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Since language models learn many concepts, autoencoders need to be very large to recover all relevant features. However, studying the properties of autoencoder scaling is difficult due to the need to balance reconstruction and sparsity objectives and the presence of dead latents. We propose using k-sparse autoencodersto directly control sparsity, simplifying tuning and improving the reconstruction-sparsity frontier. Additionally, we find modifications that result in few dead latents, even at the largest scales we tried. Using these techniques, we find clean scaling laws with respect to autoencoder size and sparsity. We also introduce several new metrics for evaluating feature quality based on the recovery of hypothesized features, the explainability of activation patterns, and the sparsity of downstream effects. These metrics all generally improve with autoencoder size. To demonstrate the scalability of our approach, we train a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens. We releasecode and autoencoders for open-source models, as well as avisualizer.111Our open source code can be found athttps://github.com/openai/sparse_autoencoderand our visualizer is hosted athttps://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html
abstract
Scaling and evaluating sparse autoencoders
Abstract
2406.04093
1IntroductionSparse autoencoders (SAEs)  have shown great promise for finding featuresand circuitsin language models. Unfortunately, they are difficult to train due to their extreme sparsity, so prior work has primarily focused on training relatively small sparse autoencoders on small language models.We develop a state-of-the-art methodology to reliably train extremely wide and sparse autoencoders with very few dead latents on the activations of any language model. We systematically study the scaling laws with respect to sparsity, autoencoder size, and language model size. To demonstrate that our methodology can scale reliably, we train a 16 million latent autoencoder on GPT-4residual stream activations.Because improving reconstruction and sparsity is not the ultimate objective of sparse autoencoders, we also explore better methods for quantifying autoencoder quality. We study quantities corresponding to: whether certain hypothesized features were recovered, whether downstream effects are sparse, and whether features can be explained with both high precision and recall.Our contributions:1.InSection 2, we describe a state-of-the-art recipe for training sparse autoencoders.2.InSection 3, we demonstrate clean scaling laws and scale to large numbers of latents.3.InSection 4, we introduce metrics of latent quality and find larger sparse autoencoders are generally better according to these metrics.We also releasecode, a full suite of GPT-2 small autoencoders,
anda feature visualizerfor GPT-2 small autoencoders and the 16 million latent GPT-4 autoencoder.Figure 1:Scaling laws for TopK autoencoders trained on GPT-4 activations.
(Left)  Optimal loss for a fixed compute budget.
(Right)  Joint scaling law of loss at convergence with fixed number of total latentsand fixed sparsity (number of active latents) . Details inSection 3.
introduction
Scaling and evaluating sparse autoencoders
1Introduction
2406.04093
2Methods2.1SetupInputs:We train autoencoders on the residual streams of both GPT-2 smalland models from
a series of increasing sized models sharing GPT-4 architecture and training setup,
including GPT-4 itself222All presented results either have qualitatively similar results on both GPT-2 small and GPT-4 models, or were only ran on one model class..
We choose a layer near the end of the network, which should contain many features without being specialized for next-token predictions (seeSection F.1for more discussion) . Specifically, we use a layerof the way into the network for GPT-4 series models, and we use layer 8 (of the way)  for GPT-2 small.
We use a context length of 64 tokens for all experiments.
We subtract the mean over thedimension and normalize to all inputs to unit norm, prior to passing to the autoencoder (or computing reconstruction errors) .Evaluation:After training, we evaluate autoencoders on sparsity L0, and reconstruction mean-squared error (MSE) . We report a normalized version of all MSE numbers, where we divide by a baseline reconstruction error of always predicting the mean activations.Hyperparameters:To simplify analysis, we do not consider learning rate warmup or decay unless otherwise noted. We sweep learning rates at small scales and extrapolate the trend of optimal learning rates for large scale. SeeAppendix Afor other optimization details.2.2Baseline: ReLU autoencodersFor an input vectorfrom the residual stream, andlatent dimensions, we use baseline ReLU autoencoders from. The encoder and decoder are defined by:(1) with,,, and. The training loss is defined by, whereis the reconstruction MSE,is an L1penalty promoting sparsity in latent activations, andis a hyperparameter that needs to be tuned.2.3TopK activation functionWe use a-sparse autoencoder, which directly controls the number of active latents by using an activation function (TopK)  that only keeps thelargest latents, zeroing the rest. The encoder is thus defined as:(2) and the decoder is unchanged. The training loss is simply.Using-sparse autoencoders has a number of benefits:•It removes the need for the L1penalty. L1is an imperfect approximation of L0, and it introduces a bias of shrinking all positive activations toward zero
(Section 5.1) .•It enables setting the L0directly, as opposed to tuning an L1coefficient, enabling simpler model comparison and rapid iteration. It can also be used in combination with arbitrary activation functions.333In our code, we also apply a ReLU to guarantee activations to be non-negative. However, the training curves are indistinguishable, as thelargest activations are almost always positive for reasonable choices of.•It empirically outperforms baseline ReLU autoencoders on the sparsity-reconstruction frontier (Figure 2a) , and this gap increases with scale (Figure 2b) .•It increases monosemanticity of random activating examples by effectively clamping small activations to zero
(Section 4.3) .(a) At a fixed number of latents () , TopK has a better reconstruction-sparsity trade off than ReLU and ProLU, and is comparable to Gated.(b) At a fixed sparsity level () , scaling laws are steeper for TopK than ReLU.555Non-TopK cannot set a precise L0, so we interpolated using a piecewise linear function in log-log space.Figure 2:Comparison between TopK and other activation functions.2.4Preventing dead latentsDead latents pose another significant difficulty in autoencoder training. In larger autoencoders, an increasingly large proportion of latents stop activating entirely at some point in training. For example,train a 34 million latent autoencoder with only 12 million alive latents, and in our ablations we find up to 90% dead latents666We follow, and consider a latent dead if it has not activated in 10 million tokenswhen no mitigations are applied (Figure 15) . This results in substantially worse MSE and makes training computationally wasteful.
We find two important ingredients for preventing dead latents: we initialize the encoder to the transpose of the decoder, and we use an auxiliary loss that models reconstruction error using the top-dead latents (seeSection A.2for more details) . Using these techniques, even in our largest (16 million latent)  autoencoder only 7% of latents are dead.
body
Scaling and evaluating sparse autoencoders
2Methods
2406.04093
3Scaling lawsFigure 3:Varying the learning rate jointly with the number of latents. Number of tokens to convergence shown above each point.Figure 4:Larger subject models in the GPT-4 family require more latents to get to the same MSE () .3.1Number of latentsDue to the broad capabilities of frontier models such as GPT-4, we hypothesize that faithfully representing model state will require large numbers of sparse features. We consider two primary approaches to choose autoencoder size and token budget:3.1.1Training to compute-MSE frontier () Firstly, following, we train autoencoders to the optimal MSE given the available compute, disregarding convergence. This method was introduced for pre-training language models. We find that MSE follows a power lawof compute, though the smallest models are off trend
(Figure 1) .However, latents are the important artifact of training (not reconstruction predictions) , whereas for language models we typically care only about token predictions. Comparing MSE across differentis thus not a fair comparison — the latents have a looser information bottleneck with larger, so lower MSE is more easily achieved. Thus, this approach is arguably unprincipled for autoencoder training.3.1.2Training to convergence () We also look at training autoencoders to convergence (within some) . This gives a bound on the best possible reconstruction achievable by our training method if we disregard compute efficiency. In practice, we would ideally train to some intermediate token budget betweenand.We find that the largest learning rate that converges scales with(Figure 4) . We also find that the optimal learning rate foris about four times smaller than the optimal learning rate for.We find that the number of tokens to convergence increases as approximatelyfor GPT-2 small andfor GPT-4 (Figure 11) . This must break at some point – if token budget continues to increase sublinearly, the number of tokens each latent receives gradient signal on would approach zero.777One slight complication is that in the infinite width limit, TopK autoencoders with our initialization scheme are actually optimal at init using our init scheme (Section A.1) , so this allows for an exponent very slightly less than; however, this happens very slowly withso is unlikely to be a major factor at realistic scales.3.1.3Irreducible lossScaling laws sometimes include an irreducible loss term, such that. We find that including an irreducible loss term substantially improves the quality of our fits for bothand.It was initially not clear to us that there should be a nonzero irreducible loss.
One possibility is that there are other kinds of structures in the activations. In the extreme case, unstructured noise in the activations is substantially harder to model and would have an exponent close to zero (Appendix G) . Existence of some unstructured noise would explain a bend in the power law.3.1.4Jointly fitting sparsity () We find that MSE follows a joint scaling law along the number of latentsand the sparsity level(Figure 1b) . Because reconstruction becomes trivial asapproaches,
this scaling law only holds for the smallregime.
Our joint scaling law fit on GPT-4 autoencoders is:(3) with,,,,, and.
We can see thatis negative,
which means that the scaling lawgets steeper asincreases.is negative too, which means that the irreducible loss decreases with.3.2Subject model sizeSince language models are likely to keep growing in size, we would also like to understand how sparse autoencoders scale as the subject models get larger.
We find that if we holdconstant, larger subject models require larger autoencoders to achieve the same MSE, and the exponent is worse (Figure 4) .(a) For a fixed number of latents () , the downstream-loss/sparsity trade-off is better for TopK autoencoders than for other activation functions.(b) For a fixed sparsity level () , a given MSE level leads to a lower downstream-loss for TopK autoencoders than for other activation functions.Figure 5:Comparison between TopK and other activation functions on downstream loss. Comparisons done for GPT-2 small, seeFigure 13for GPT-4.
body
Scaling and evaluating sparse autoencoders
3Scaling laws
2406.04093
4EvaluationWe demonstrated inSection 3that our larger autoencoders scale well in terms of MSE and sparsity (see also a comparison of activation functions inSection 5.2) . However, the end goal of autoencoders is not to improve the sparsity-reconstruction frontier (which degenerates in the limit888Improving the reconstruction-sparsity frontier is not always strictly better. An infinitely wide maximally sparse ()  autoencoder can perfectly reconstruct by assigning latents densely in, while being completely structure-less and uninteresting.) , but rather to find features useful for applications, such as mechanistic interpretability.
Therefore, we measure autoencoder quality with the following metrics:1.Downstream loss: How good is the language model loss if the residual stream latent is replaced with the autoencoder reconstruction of that latent? (Section 4.1) 2.Probe loss: Do autoencoders recover features that we believe they might have? (Section 4.2) 3.Explainability: Are there simple explanations that are both necessary and sufficient for the activation of the autoencoder latent? (Section 4.3) 4.Ablation sparsity: Does ablating individual latents have a sparse effect on downstream logits? (Section 4.5) These metrics provide evidence that autoencoders generally get better when the number of total latents increases.
The impact of the number of active latents L0is more complicated. Increasing L0makes explanations based on token patterns worse, but makes probe loss and ablation sparsity better. All of these trends also break when L0gets close to, a regime in which latents also become quite dense (seeSection E.5for detailed discussion) .4.1Downstream lossAn autoencoder with non-zero reconstruction error may not succeed at modeling the features most relevant for behavior.
To measure whether we model features relevant to language modeling, we follow prior workand consider downstream Kullback-Leibler (KL)  divergence and cross-entropy loss.999Because a perfect reconstruction would lead to a non-zero cross-entropy loss, we actually consider the difference to the perfect-autoencoder cross-entropy (“delta cross-entropy”) .In both cases, we test an autoencoder by replacing the residual stream by the reconstructed value during the forward pass, and seeing how it affects downstream predictions. We find that-sparse autoencoders improve more on downstream loss than on MSE over prior methods (Figure 5a) .
We also find that MSE has a clean power law relationship with both KL divergence, and difference of cross entropy loss (Figure 5b) , when keeping sparsity L0fixed and only varying autoencoder size. Note that while this trend is clean for our trained autoencoders, we can observe instances where it breaks such as when modulatingat test time (seeSection 5.3) .One additional issue is that raw loss numbers alone are difficult to interpret—we would like to know how good it is in an absolute sense. Prior workuse the loss of ablating activations to zero as a baseline and report the fraction of loss recovered from that baseline. However, because ablating the residual stream to zero causes very high downstream loss, this means that even very poorly explaining the behavior can result in high scores.101010For completeness, the zero-ablation fidelity metric of our 16M autoencoder is 98.2%.Instead, we believe a more natural metric is to consider the relative amount of pretraining compute needed to train a language model of comparable downstream loss. For example, when our 16 million latent autoencoder is substituted into GPT-4, we get a language modeling loss corresponding to 10% of the pretraining compute of GPT-4.(a) Probe loss(b) Logit diff sparsityFigure 6:The probe loss and logit diff metrics as a function of number of total latentsand active latents, for GPT-2 small autoencoders. More total latents (higher)  generally improves all metrics (yellow = better) . Both metrics are worse at L0, a regime in which solutions are dense (seeSection E.5) .4.2Recovering known features with 1d probesIf we expect that a specific feature (e.g sentiment, language identification)  should be discovered by a high quality autoencoder, then one metric of autoencoder quality is to check whether these features are present.
Based on this intuition, we curated a set of 61 binary classification datasets (details inTable 1) . For each task, we train a 1d logistic probe on each latent using the Newton-Raphson method to predict the task, and record the best cross entropy loss (across latents) .111111This is similar to the approach of, but always using, and regressing on autoencoder latents instead of neurons.That is:(4) whereis theth pre-activation autoencoder latent, andis a binary label.Results on GPT-2 small are shown in6(a) . We find that probe score increases and then decreases asincreases. We find that TopK generally achieves better probe scores than ReLU (Figure 23) , and both are substantially better
than when using directly residual stream channels. SeeFigure 32for results on several GPT-4 autoencoders: we observe that this metric improves throughout training, despite there being no supervised training signal; and we find that it beats a baseline using channels of the residual stream. SeeFigure 33for scores broken down by component.This metric has the advantage that it is computationally cheap. However, it also has a major limitation, which is that it leans on strong assumptions about what kinds of features are natural.4.3Finding simple explanations for featuresAnecdotally, our autoencoders find many features that have quickly recognizable patterns that suggest explanations when viewing random activations (Section E.1) . However, this can create an “illusion” of interpretability, where explanations are overly broad, and thus have good recall but poor precision.
For example,propose an automated interpretability score which disproportionately depends on recall. They find a feature activating at the end of the phrase “don’t stop” or “can’t stop”, but an explanation activating on all instances of “stop” achieves a high interpretability score. As we scale autoencoders and the features get sparser and more specific, this kind of failure becomes more severe.(a)  A feature with precision = 0.97, recall = 0.56(b)  A feature with precision = 0.06, recall = 0.05Recall Eval Sequences:\n\nThereforeshe’snotgoingtoplaypranksbuttogiveasweettime-arendezvouschoice.[Warning:RipewithWinePuns]\n\nVitalLacerda(September.[Warning:DevelopedwithEvolutionaryPuns]\n\nMailbags\n\nTrial&al6606Luke.A.Mmepwn12Thorny[B]Availablevoiceactors,hispictureofPresidentPutin.\n\nPresidentPrankster\n\nDmitryRogozin,Precision Eval Sequences:away:\n\nItching.\n\nPrickling.\n\nThesearenotallofthe%70%99%\n\n71NoPwn1ntendedi7-9503.4ottoneeWhimsicottM0Prankster/InfiltratorPrankster/InfofChimeraplayersusingone-dimensionalteams:PrankofChimeraplayersusingone-dimensionalteams:PranksterSporeSeed,ShellSmash,PoisonRecall Eval Sequences:\nMeal106h30Oats,doubleservingofWhey,6’s.\n\n04Jul65to01Aug65\n\n1/3unat\n\n2013-11-1609:47:18(25) chat:<DiamondCard0000-00260076]119pages5:[00260,calmit2013-11-1120:00:43(25) chat:It’svanityPrecision Eval Sequences:1TueMar1322:37:59EST2001i686Locale:LANG=CKiwi76added01:08-Apr28\n\nCheersCliveandhopefullyyoucansee.aspx?key=248858886|000100&CMC=&PN=&Is.aspx?key=248858886|000101&CMC=&PN=&Iskey=248858886|000102&CMC=&PN=&IsFigure 7:Qualitative examples of latents with high and low precision/recall N2G explanations.Key:Green= Ground truth feature activation,Underline= N2G predicted feature activationUnfortunately, precision is extremely expensive to evaluate when the simulations are using GPT-4 as in. As an initial exploration, we focus on an improved version of Neuron to Graph (N2G) , a substantially less expressive but much cheaper method that outputs explanations in the form of collections of n-grams with wildcards.
In the future, we would like to explore ways to make it more tractable to approximate precision for arbitrary English explanations.To construct a N2G explanation, we start with some sequences that activate the latent. For each one, we find the shortest suffix that still activates the latent.121212with at least half the original activation strengthWe then check whether any position in the n-gram can be replaced by a padding token, to insert wildcard tokens. We also check whether the explanation should be dependent on absolute position by checking whether inserting a padding token at the beginning matters.
We use a random sample of up to 16 nonzero activations to build the graph, and another 16 as true positives for computing recall.Results for GPT-2 small are found in25(a) and25(b) . Note that dense token patterns are trivial to explain, thuslatents are easy to explain on average since many latents activate extremely densely (seeSection E.5) 131313This highlights an issue with our precision/recall metrics, which care only about binarized values. We also propose a more expensive metric which uses simulated valuesSection 4.4and addresses this issue..
In general, autoencoders with more total latents and fewer active latents are easiest to model with N2G.We also obtain evidence that TopK models have fewer spurious positive activations than their ReLU counterparts. N2G explanations have significantly better recall (>1.5x)  and only slightly worse precision (>0.9x)  for TopK models with the same(resulting in better F1 scores)  and similar L0(Figure 24) .4.4Explanation reconstructionWhen our goal is for a model’s activations to be interpretable, one question we can ask is: how much performance do we sacrifice if we use only the parts of the model that we can interpret?Our downstream loss metric measures how much of the performance we’re capturing (but our features could be uninterpretable) , and our explanation based metric measures how monosemantic our features are (but they might not explain most of the model) . This suggests combining our downstream loss and explanation metrics, by using our explanations to simulate autoencoder latents, and then checking downstream loss after decoding. This metric also has the advantage that it values both recall and precision in a way that is principled, and also values recall more for latents that activate more densely.We tried this with N2G explanations. N2G produces a simulated value based on the node in the trie, but we scale this value to minimize variance explained. Specifically, we compute, whereis the simulated value andis the true value, and we estimate this quantity over a training set of tokens. Results for GPT-2 are shown inFigure 8. We find that we can explain more of GPT-2 small than just explaining bigrams, and that larger and sparser autoencoders result in better downstream loss.Figure 8:Downstream loss on GPT-2 with various residual stream ablations at layer 8. N2G explanations of autoencoder latents improves downstream loss with largerand smaller.4.5Sparsity of ablation effectsIf the underlying computations learned by a language model are sparse,
one hypothesis is that natural features are not only sparse in terms of activations,
but also in terms of downstream effects. Anecdotally, we observed that ablation effects often are interpretable (see our visualizer) .
Therefore, we developed a metric to measure the sparsity of downstream effects on the output logits.At a particular token index, we obtain the latents at the residual stream, and proceed to ablate each autoencoder latent one by one, and compare the resulting logits before and after ablation. This process leads tologit differences per ablation and affected token, whereis the size of the token vocabulary.
Because a constant difference at every logit does not affect the post-softmax probabilities,
we subtract at each token the median logit difference value. Finally, we concatenate these vectors together across some set offuture tokens (at the ablated index or later)  to obtain a vector oftotal numbers.
We then measure the sparsity of this vector via,
which corresponds to an “effective number of vocab tokens affected”. We normalize byto have a fraction between 0 and 1, with smaller values corresponding to sparser effects.We perform this for various autoencoders trained on the post-MLP residual stream at layer 8 in GPT-2 small, with. Results are shown in6(b) . Promisingly, models trained with largerhave latents with sparser effects. However, the trend reverses at, indicating that asapproaches, the autoencoder learns latents with less interpretable effects.
Note that latents are sparse in an absolute sense, having aof 10-14%
, whereas ablating residual stream channels gives 60%
(slightly better than the theoretical value offor random vectors) .
body
Scaling and evaluating sparse autoencoders
4Evaluation
2406.04093
5Understanding the TopK activation function5.1TopK prevents activation shrinkageA major drawback of the L1penalty is that it tends to shrink all activations toward zero.
Our proposed TopK activation function prevents activation shrinkage, as it entirely removes the need for an L1penalty.
To empirically measure the magnitude of activation shrinkage, we consider whether different (and potentially larger)  activations would result in better reconstruction given a fixed decoder. We first run the encoder to obtain a set of activated latents, save the sparsity mask, and then optimize only the nonzero values to minimize MSE.141414unlike the “inference-time optimization” procedure inThis refinement method has been proposed multiple times such as in-SVD, the relaxed Lasso, or ITI. We solve for the optimal activations with a positivity constraint using projected gradient descent.This refinement procedure tends to increase activations in ReLU models on average, but not in TopK models (Figure 9a) , which indicates that TopK is not impacted by activation shrinkage. The magnitude of the refinement is also smaller for TopK models than for ReLU models. In both ReLU and TopK models, the refinement procedure noticeably improves the reconstruction MSE (Figure 9b) , and the downstream next-token-prediction cross-entropy (Figure 9c) . However, this refinement only closes part of the gap between ReLU and TopK models.Figure 9:Latent activations can be refined to improve reconstruction from a frozen set of latents. For ReLU autoencoders, the refinement is biased toward positive values, consistent with compensating for the shrinkage caused by the L1penalty. For TopK autoencoders, the refinement is not biased, and also smaller in magnitude. The refinement only closes part of the gap between ReLU and TopK.5.2Comparison with other activation functionsOther recent works on sparse autoencoders have proposed different ways to address the L1activation shrinkage, and Pareto improve the L0-MSE frontier.propose to fine-tune a scaling parameter per latent, to correct for the L1activation shrinkage.
In Gated sparse autoencoders, the selection of which latents are active is separate from the estimation of the activation magnitudes. This separation allows autoencoders to better estimate the activation magnitude, and avoid the L1activation shrinkage.
Another approach is to replace the ReLU activation function with a ProLU(also known as TRec, or JumpReLU) , which sets all values below a positive threshold to zero. Because the parameteris non-differentiable, it requires a approximate gradient such as a ReLU equivalent (ProLU-ReLU)  or a straight-through estimator (ProLU-STE) .We compared these different approaches in terms of reconstruction MSE, number of active latents L0, and downstream cross-entropy loss (Figure 2and5) . We find that they significantly improve the reconstruction-sparsity Pareto frontier, with TopK having the best performance overall.5.3Progressive recoveryFigure 10:Sparsity levels can be changed at test time by replacing the activation function with either TopK()  or JumpReLU() , for a given valueor. TopK tends to overfit to the value ofused during training, but using Multi-TopK improves generalization to larger.In a progressive code, a partial transmission still allows reconstructing the signal with reasonable fidelity. For autoencoders, learning a progressive code means that ordering latents by activation magnitude gives a way to progressively recover the original vector. To study this property, we replace the autoencoder activation function (after training)  by a TopK()  activation function whereis different than during training. We then evaluate each value ofby placing it in the L0-MSE plane (Figure 10) .We find that training with TopK only gives a progressive code up to the value ofused during training. MSE keeps improving for values slightly over(a result also described in) , then gets substantially worse asincreases (note that the effect on downstream loss is more muted) . This can be interpreted as some sort of overfitting to the value.5.3.1Multi-TopKTo mitigate this issue, we sum multiple TopK losses with different values of(Multi-TopK) . For example, usingis enough to obtain a progressive code over all(note however that training with Multi-TopK does slightly worse than TopK at) .
Training with the baseline ReLU only gives a progressive code up to a value that corresponds to using all positive latents.5.3.2Fixed sparsity versus fixed thresholdAt test time, the activation function can also be replaced by a JumpReLU activation, which activates above a fixed threshold,. In contrast to TopK, JumpReLU leads to a selection of active latents where the number of active latents can vary across tokens.
Results for replacing the activation function at test-time with a JumpReLU are shown in dashed lines inFigure 10.For autoencoders trained with TopK, the test-time TopK and JumpReLU curves are superimposed only for values corresponding to an L0below the training L0, otherwise the JumpReLU activation is worse than the TopK activation. This discrepancy disappears with Multi-TopK, where both curves are nearly superimposed, which means that the model can be used with either a fixed or a dynamic number of latents per token without loss in reconstruction.
The two curves are also superimposed for autoencoders trained with ReLU.
Interestingly, it is sometimes more efficient to train a ReLU model with a low L1penalty and to use a TopK or JumpReLU at test time, than to use a higher L1penalty that would give a similar sparsity level (a result independently described in) .
body
Scaling and evaluating sparse autoencoders
5Understanding the TopK activation function
2406.04093
6Limitations and Future DirectionsWe believe many improvements can be made to our autoencoders.•TopK forces every token to use exactlylatents, which is likely suboptimal. Ideally we would constrainrather than.•The optimization can likely be greatly improved, for example with learning rate scheduling,151515Anecdotally, we also found that lowering learning rates helped with decreasing dead latents.better optimizers, and better aux losses for preventing dead latents.•Much more could be done to understand what metrics best track relevance to downstream applications, and to study those applications themselves. Applications include: finding vectors for steering behavior, doing anomaly detection, identifying circuits, and more.•We’re excited about work in the direction of combining MoEand autoencoders, which would substantially improve the asymptotic cost of autoencoder training, and enable much larger autoencoders.•A large fraction of the random activations of features we find, especially in GPT-4, are not yet adequately monosemantic. We believe that with improved techniques and greater scale161616both in number of latents and in training tokensthis is potentially surmountable.•Our probe based metric is quite noisy, which could be improved by having a greater breadth of tasks and higher quality tasks.•While we use n2g for its computational efficiency, it is only able to capture very simple patterns. We believe there is a lot of room for improvement in terms of more expressive explanation methods that are also cheap enough to simulate to estimate explanation precision.•A context length of 64 tokens is potentially too few tokens to exhibit the most interesting behaviors of GPT-4.
body
Scaling and evaluating sparse autoencoders
6Limitations and Future Directions
2406.04093
7Related workSparse coding on an over-complete dictionary was introduced by.refined the idea by proposing to learn the dictionary from the data, without supervision. This approach has been particularly influential in image processing, as seen for example in.
Later,proposed the autoencoder architecture to perform dimensionality reduction.
Combining these concepts, sparse autoencoders were developedto train autoencoders with sparsity priors, such as the L1penalty, to extract sparse features.refined this concept by introducing-sparse autoencoders, which use a TopK activation function instead of the L1penalty.evaluates autoencoders using a metric that measures recovery of features from previously discovered circuits.More recently, sparse autoencoders were applied to language models,
and multiple sparse autoencoders were trained on small open-source language models.showed that the resulting features from sparse autoencoders can find sparse circuits in language models.pointed out that sparse autoencoders are subject to activation shrinking from L1penalties, a property of L1penalties first described in.andproposed to use different activation functions to address activation shrinkage in sparse autoencoders.proposed to train sparse autoencoders on downstream KL instead of reconstruction MSE.studied scaling laws for language models which examine how loss varies with various hyperparameters.explore scaling laws related to sparsity using a bilinear fit.studied scaling laws specifically for autoencoders, defining the loss as a specific balance of reconstruction and sparsity (rather than simply reconstruction, while holding sparsity fixed) .
body
Scaling and evaluating sparse autoencoders
7Related work
2406.04093
Acknowledgments and Disclosure of FundingWe are deeply grateful to Jan Leike and Ilya Sutskever for leading the Superalignment team and creating the research environment which made this work possible. We thank David Farhi for supporting our work after their departure.We thank Cathy Yeh for explorations in finding features. We thank Carroll Wainwright for initial insights into clustering latents.We thank Steven Bills, Dan Mossing, Cathy Yeh, William Saunders, Boaz Barak, Jakub Pachocki, and Jan Leike for many discussions about autoencoders and their applications.
We thank Manas Joglekar for some improvements on an earlier version of our activation loading. We thank William Saunders for an early autoencoder visualization implementation.We thank Trenton Bricken, Dan Mossing, and Neil Chowdhury for feedback on an earlier version of this manuscript.We thank Trenton Bricken, Lawrence Chan, Hoagy Cunningham, Adam Goucher, Ryan Greenblatt, Tristan Hume, Jan Hendrik Kirchner, Jake Mendel, Neel Nanda, Noa Nabeshima, Chris Olah, Logan Riggs, Fabien Roger, Buck Shlegeris, John Schulman, Lee Sharkey, Glen Taggart, Adly Templeton, Vikrant Varma for valuable discussions.
conclusion
Scaling and evaluating sparse autoencoders
Acknowledgments and Disclosure of Funding