A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
Sparse Autoencoders;SAEs;LLMs;interpretability
Yes
No ethics review needed.
Same as above
1. This paper studies an interesting problem, that is, learned sparse features can become less understandable by "absorbing" token-aligned features.
2. understanding how width and sparsity of SAEs affects its training performance can be helpful for future SAE training.
3. the paper is written in an easy-to-understand way.
This paper explores the use of Sparse Autoencoders (SAEs) to decompose activations of Large Language Models (LLMs) into interpretable latent features. It addresses two main questions: the degree to which SAEs yield monosemantic (single-meaning) and interpretable latents, and how adjustments to the sparsity and size of SAEs impact interpretability. Through a controlled first-letter identification task with complete ground-truth labels, the authors find more nuanced insights compared to previous studies.
1. It would be valuable to explore practical examples where insights from feature absorption can enhance interpretability. For instance, analyzing the first letters of all tokens associated with specific latent activations might offer new explanations for features that previously seemed opaque. This approach could reveal subtle patterns in latent activation, aiding in the interpretation of challenging features.
2. This work presents a specific form of feature absorption, but it raises the question of whether other variations might exist. Are there additional contexts or scenarios where feature absorption manifests differently, potentially impacting interpretability in distinct ways? Identifying these cases could deepen our understanding of the phenomenon and refine the strategies needed to address it.
Yes
No ethics review needed.
I list my concerns and questions in the "weaknesses" section.
As the large language model becomes increasingly popular, it is really important to get a good understanding of these techniques. This paper investigates a meaningful problem and attempts to provides some insights and points out some research directions, which should be very useful to the community.
This paper investigates the interpretability of the extracted latent/features by Sparse Autoencoders. The authors found that the Sparse Autoencoder does extract monosemantic and interpretable features and changing the hyperparameters of the Sparse Autoencoder could not eliminate this.
There are several weaknesses for this paper.
1) The observation and conclusion are merely based on one model and one task (first-letter identification task). Thus, it is really difficult to say how general and convincing these conclusions could be. I am afraid it is not safe to draw a solid conclusion based a single setting.
2)  The authors should elaborate their method better. The current version simply lists some background and then posts a pseudo code (algorithm) there. More descriptions should benefit readers to get a better understanding.
3) Some contents cause confusing instead of help. For example, the Figure1 is really confusing and hard to interpreter and even worse, the authors do not provide sufficient descriptions of it in the main paper.
4) The overall presentation is a little bit messy and it is really hard to follow the content.
Yes
No ethics review needed.
- Did you find any rules for predicting whether the feature activation for the first letter of a given word would be absorbed? (i.e. do you have any hypothesies/ways to differentiate situations in which feature absorbtion occurs or does not occur?)
- Describes an interesting setting in which to study feature disentanglement for natural language.
- Discovers a new phenomenon, 'feature absorbtion', which could pose a problem for desired future applications of Sparse Autoencoders.
- Setting allows for a thorough analysis of the failure cases for feature absorbtion and feature splitting, and said analysis was good and informative.
- Paper has a clear statement of the problem and good structure. Good use of the 'S'/'_short' example to help build intuition.
This paper proposes a new simplified setting for studying features discovered by Sparse Autoencoders (SAEs) trained on language model activations. It supposes that SAEs should discover features corresponding to the first letter of a given token, but finds that features which are predictive of a given letter occasionally fail and are 'absorbed' into the activation of another feature. They then measure and explore this phenomenon in the context of their setting.
- It would be useful to explore the commonalities between different instances of absorbtion of SAE features, or otherwise find more ways to verify/expand upon the claim/statement that "feature absorption is likely a logical consequence of SAE sparsity loss".
Yes
No ethics review needed.
**Questions** 

1. The concept of "approximately token-aligned" latents is central to claim 3 (lines 69-71), but is not defined anywhere. How is token-alignment defined, and is it quantified or measured in these experiments?

2. When the F1 score of an SAE is computed, as in Figure 2 (lines 169-172), how are SAE activations converted into a binary classification? Is the classification "does/doesn't activate". Is a non-zero threshold used? If so, how is that threshold chosen?

3. In Figure 7b, when the mean absorption rate is \~35%, is that indicating that for roughly 35% of tokens, there is a latent absorbing that token? The vocabulary size of Gemma-2B is \~256k (https://arxiv.org/html/2408.00118v1#:~:text=Vocab%20size-,256128,-256128) and 35% of that is \~90k. If that is the case, how are 90k tokens absorbed into 65k latents?

4. In figure 13, why is the high cosine similarity in the .4-.6 range, and not closer to 1? This range of cosine similarities is suggestive of learning the "true" feature mixed with a second, orthogonal feature. 

5. It seems that the paper usually uses cosine similarity of the LR direction with the encoder direction, but Figures 4 and 13 use decoders instead of encoders. Why is that?

6. What does Algorithm 1 (Lines 116-127) return? One ablation effect per token per latent? Or are these averaged in some way?

**Other Comments**

If it is permitted for authors to make revisions before the final submission, there are several small changes that could improve the quality of the paper:

- Lines 94-107: the paper should reference (Lieberum et al., 2024) since that is the exact architecture of SAE being studied. 

- Line 100: Although ReLU is often used in SAE architectures, it would be better to write a generic activation function in this equation, especially considering that the SAEs studied in the paper use JumpReLU, not ReLU.

- Line 118 and Line 147: The description "include the SAE error term" is ambiguous in the text. It is clarified in the glossary, and by the glossary's citation to (Marks et al., 2024), but the main paper neither cites (Marks et al., 2024) here, nor links to the glossary. Is there a way to include hyperlinks to the glossary in the text itself? 

- Line 205: There is a citation to (Gao et al., 2024) for sparse probing. It is likely supposed to cite (Gurnee et al., 2023) as in the background section. (Gao et al., 2024) use k-sparse SAEs, which are distinct from k-sparse probes and the SAE architecture studied in the paper.

- Lines 291-292: "The token where the SAE feature activates is highlighted in green" should be "yellow" since the highlight is yellow.

- Line 404: The paper says "it is difficult to apply [(Karvonen et al, 2024)] to existing SAEs trained on real LLM activations." One of the main contributions of (Karvonen et al, 2024) is very applicable to this paper, namely coverage (Section 3.2 in https://arxiv.org/pdf/2408.00113). Coverage for a given set of properties (in this case, "first letter identification") is defined by, for each property, finding the SAE latent and threshold which results in the best F1 score for classifying that property, then averaging those F1 scores across the properties. This is very similar to what is shown in Figure 2, though with a different selection rule for SAE latents. The paper could make reference to the similarity of this method to coverage.

- Line 429, possible typo: "..find that by training an SAE on the decoder *or* another SEE, a technique...", the word "or" should be "of".

- (Uncertain) The citation of "(Huben et al., 2024)" should instead refer to the paper as "(Cunningham et al., 2024)". The order of the authors is different between the OpenReview page (https://openreview.net/forum?id=F76bwRSLeK) and the paper itself (https://openreview.net/pdf?id=F76bwRSLeK). In such a case it is most likely proper to follow the order of authors in the paper.

**References**

Leo Gao, Tom Dupre ́ la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024.

Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dic- tionary learning for language model interpretability with board game models. In ICML 2024 Workshop on Mechanistic Interpretability, 2024.

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsi- mas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum? id=JYs1R9IMJr.

Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=F76bwRSLeK.

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Ja ́nos Krama ́r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2, August 2024.

Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language mod- els. Computing Research Repository, arXiv:2403.19647, 2024. URL https://arxiv.org/ abs/2403.19647.
The paper explores a natural task, first letter classification, and does so well. It establishes that this information is linearly available in the model (by training an LR probe with high F1-score), and that SAEs find latents that align with the probe direction (Figure 4b and especially Figure 13), and that these latents are often good classifiers, though not as good as probes (Figure 2a).

The paper provides a novel perspective to a documented issue of feature splitting in SAEs. It provides good theoretical justification of why feature absorption is likely to happen, and why it provides challenges to interpretability.
This paper documents and measures a particular form of feature splitting in the Gemma Scope family of sparse autoencoders (SAEs), which the authors call feature absorption. The prototypical case of feature absorption studied is when a latent is a high-precision classifier for a property like "starts with the letter L", but has poor recall due to other latents "absorbing" individual tokens, so that the main latent's activations are given by the rule "starts with the letter L, except for the tokens laser, lions, [several other exceptions]".
The paper could do more to demonstrate that feature absorption happens as described, namely that: 
1) There is a latent which is a high-precision classifier for "starts with letter [x]".
2) There is a particular subset of tokens which start with this letter but which the latent fails to activate on.
3) There are separate token-aligned latents that individually classify those tokens.
4) If that subset of tokens is removed, the latent is a high-precision, high-recall classifier for "starts with the letter [x], except for that subset of tokens".

The authors demonstrate 1) and 2) well in sections 4.1 and 4.3 respectively, but never show 3) or 4) besides a single example in section 4.2. The authors could address 3) by adding an additional step to the experiment in section 4.3, in which they confirm that "the SAE latent receiving the largest negative magnitude ablation effect" is a token-classifying latent.

Lines 148-157: The metric described can fail at its stated goal to "[measure] the propensity of the model to choose the correct starting letter as opposed to other letters" because averaging logits can interact poorly with the softmax activation function. For example, if "A" is the correct letter, but the model logits are g[A]=10, g[B]=25, g[C]=g[D]=...=g[Z]=0, then the model would produce B with near-certainty. But the provided metric yields m=10-(25+0+0+...+0)/25=10-1=9, the same value as if g[A]=9, g[B]=...=g[Z]=0. This metric would be more convincing if the average were replaced by a max, or if it were a softmax over letter tokens. 

The authors only study a single family of SAEs on a single language model. Alternative SAE architectures may not demonstrate feature absorption.
@inproceedings{
anonymous2024a,
title={A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=LC2KxRwC3n},
note={under review}
}
Sparse Autoencoders (SAEs) have emerged as a promising approach to decompose the activations of Large Language Models (LLMs) into human-interpretable latents. In this paper, we pose two questions. First, to what extent do SAEs extract monosemantic and interpretable latents? Second, to what extent does varying the sparsity or the size of the SAE affect monosemanticity / interpretability? By investigating these questions in the context of a simple first-letter identification task where we have complete access to ground truth labels for all tokens in the vocabulary, we are able to provide more detail than prior investigations. Critically, we identify a problematic form of feature-splitting we call "feature absorption" where seemingly monosemantic latents fail to fire in cases where they clearly should. Our investigation suggests that varying SAE size or sparsity is insufficient to solve this issue, and that there are deeper conceptual issues in need of resolution.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Sparse Autoencoders SAEs LLMs interpretability
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8d42a1db1b7526616a11cd0a9fe75da2c8017685.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/9d1d41d0877e5e864baa96b95bac453fcd7ca973.zip
A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission