Automatically Interpreting Millions of Features in Large Language Models
interpretability;language model;sae;features;explanation
Yes
No ethics review needed.
- Missing a highly relevant work, "Explaining black box text modules in natural language with language models". How does their scoring method compare to the ones proposed in this paper?
- Can the authors explain  the negative correlation between the fuzzing score and intervention score in figure 4? If they are both useful scoring methods, why would the correlation be negative?
- Unlike the claim in the paper, figure 5 is still showing statistical alignment across layers. Can the authors provide evidence for semantic similarity? (e.g., compute explanation similarity across layers instead of the matrix statistics)

*Minor issues*
- The authors say “we introduced *five* new techniques” in the abstract, and “We addressed issues with the conventional simulation-based scoring and introduced *four* new scoring techniques” in the conclusions. The readers might get confused in terms of the number of methods actually introduced in the paper, if it’s five, please be consistent throughout the paper.
- Given the large sizes of SAEs nowadays and an increasing need for model explainability, automatically generating explanations of SAE latents efficiently is an important topic.
- The paper is well written, with ablations of design choices clearly described.
This paper introduced five automated scoring methods to score the explanations of SAE latents, and discussed the shortcomings of existing scoring techniques. The paper also finds that SAE trained on nearby layers are highly similar, and provided actionable insights for practitioners to train wider SAEs instead of narrower SAEs to be efficient when there’s a compute constraint.
- Format: There are no line numbers, and it's showing "Under review as a conference paper at ICLR **2024**" instead of **2025** at the top.
- The low correlation between different evaluation methods in Table 1 is concerning. Since the simulation method proposed in prior work is vetted and established, the new ones proposed in this work should at least have strong rank correlation (> 0.7) with it to prove that they work. Since this the scoring methods are the primary contribution in this paper, the authors should conduct more rigorous tests to ensure their validity. I would also encourage the authors to conduct experiments with ground truth explanations (e.g., SAE latents with known explanations found in prior work, or easily constructed an embedding model responding to a known concept/explanation), to make a stronger case in terms of the reliability of these new methods.
To add on, instead of correlation of the raw scores, it might make more sense to look at the "rank" correlations of different methods.
- The authors claim that the new methods are more efficient than prior scoring methods without actually quantifying the efficiency gain to support the claim. If efficiency gain is one of the highlights of these scoring methods, the authors should consider comparing runtime of different methods to support the claim.
- The author mention “Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top-k postprocessing.” in the abstract as one of the main findings, but the details cannot be found in the main paper but in the appendix. The authors should consider moving it to the main paper if this is one of the main claims.
- Reproducibility: The author mentioned a plan to open-source the project, but it's hard to evaluate the quality of their code for reproducibility purpose either since it's not provided as one of the supplementary files.
Yes
No ethics review needed.
The framework proposed in this paper is a valuable tool for the SAE community. It provides an automated pipeline to generate and evaluate natural language explanations for sparse autoencoder features. However, the authors better consider to write a more accessible version for readers who are not familiar with SAEs, especially for section 3. It is a bit difficult to follow it.
- The open-source framework is comprehensive and valuable for large-scale SAE analysis. The experiments are well designed to illustrate the effectiveness of the proposed method.
- The metrics proposed in this paper provides more dimensions to evaluate the generated explanations, which would be valuable for the SAE community.
- Some of the findings are meaningful to the SAE community. For example, larger latent SAE learn more dataset-specific latents. The relations between different sampling approaches and the generated explanations. And the high correlations between latents at adjacent layers.
The paper builds an open-source automated pipeline to generate and evaluate natural language explanations for sparse autoencoder features with LLMs. The framework has been evaluated on various dimensions, including SAE size, activation function, loss, and LLMs. Five new scoring techniques are proposed. The paper finds that SAEs trained on nearby layers' the residual stream are highly similar. And they are also more interpretable than neurons.
- The method is a bit hard to understand for readers who are not familiar with SAEs. For example, how section 3.1 is related to 3.2? Would be more illustrative if a figure of the whole pipeline is provided.
- It would be more clear to provide a simple example of explaining the latents of SAEs. And even better if an example involves the whole workflow of this framework is provided.
Yes
No ethics review needed.
-	Why is the context length of 256 chosen for activation collection? Is this value based on empirical selection?
-	Why is the activating example limited to only 32 tokens, given that short contexts may hinder the correct identification of latents with complex activation patterns?
-	What accounts for the differences in explanations between “randomly sampling” and “uniformly sampling”?
-	In Section 5, the statement that “if the explanation for latent α at layer j is very different from the explanation for the same latent at layer j + 1, this would suggest that our pipeline is inconsistent and noisy” raises a hypothesis. Can you elaborate on it?
-	This paper focuses on an important research problem of SAEs producing a large number of latent features that require automatic explanations and evaluations.
-	Several evaluation metrics have been proposed to assess the generated explanations, comparing them across different explainer models, SAEs, and layers.
-	Some findings are interesting. For instance, sampling examples that are shown to the explainer model may increase the scores of features, highlighting a problem with current auto-interpretability evaluations. Experimental results suggest a priority for training wider SAEs on a smaller subset of residual stream layers. These may provide valuable insights for future research.
This paper investigates sparse autoencoders (SAEs), which project activation representations into a sparse high-dimensional latent space for interpretability. To automatically explain the large number of latent features, this paper proposes an LLM-based framework, explaining millions of latents across multiple models, layers, and SAE architectures. Four evaluation methods are proposed, including detection, fuzzing, surprisal, embedding, measuring the extent to which an explanation enables a scorer to discriminate between activating and non-activating contexts. Additionally, an intervention scoring is proposed to interpret a feature’s counterfactual impact on model output.
-	The writing and clarity of the paper could be improved. The current version is somewhat difficult to follow and understand. Please see my detailed questions below.
-	Many observations are presented without in-depth analysis or further exploration. For instance, in Section 3.1, the impact of context length and latent space size on activation data is mentioned, but it is unclear how these results relate to the selection of 256 tokens. In Section 3.2, the claim that “showing such short contexts to the explainer model hinders the correct identification of latents with complex activation patterns” seems to contradict the use of short activating examples with 32 tokens. In Section 3.3, while qualitative results for different sampling strategies are provided, it is not clear how to optimize the sampling process. Additionally, in Section 4.1, the statement “The imperfect correlations hint at either shortcomings of the scoring metrics or the fact that these metrics can measure different qualities of explanations” lacks clarity. What specific qualities of explanations are measured by these automatic metrics?
-	The generated explanations largely rely on the prompts and explainer models. It is unclear how generalizable the results are and to what extent specific prompts and explainer models may affect the quality of the generated explanations.
-	While several automatic evaluations are compared, it is unclear whether they accurately reflect the quality of explanations. It would be beneficial to conduct human evaluations, at least on a small set, and compare the correlation with automatic evaluations.
Yes
No ethics review needed.
1. How is the correlation of the proposed evaluation metric with the original metric?
2. How is the efficiency of the proposed evaluation metric compared to the original metric?
3. Previous work (https://transformer-circuits.pub/2023/monosemantic-features) has already shown SAEs have more interepretable features than neurons. What's the purpose of validating this result?
4. Why comparing SAEs in adjacent layers? (Why is it interesing?)
5. How is comparing SAEs in adjacent layers related to your evaluation metric?
6. Are there any prior work that supports your method in evaluating the SAEs in adjacent layers?
- The scale gets improved from previous sota.
- This work lies in an interesting direction.
The paper presents an open-source automated pipeline that uses large language models to generate natural language explanations for millions of features in Sparse Autoencoders (SAEs), addressing the challenge of manual interpretation. It introduces five efficient scoring techniques, including intervention scoring, and demonstrates that SAEs are more interpretable than neurons, offering insights into semantic similarity across model layers.
- This work did not compare the new evaluation metric with the previous evaluation metric (https://openai.com/index/language-models-can-explain-neurons-in-language-models/) in a solid form. Having similar conclusions as previous approach should not serve as a solid evidence that the new metric is as good as the previous evaluation metric.
- Comparing SAEs in adjacent layers (1) lacks support of motivation and (2) is not well supported. See Questions for details.
- The presentation is poor: readers can not capture the main contribution of this paper with a normal reading flow. The contribution of this work seems to be concentrated on a new evaluation metric, but Section 5 cuts in to discuss about behaviors of the behaviors of SAEs. I would strongly recommend to reorganize the paper to one single claim, with evidence from both sides supporting it.
We build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs
@inproceedings{
anonymous2024automatically,
title={Automatically Interpreting Millions of  Features in Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5lIXRf8Lnw},
note={under review}
}
While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which may be more easily interpretable. But these SAEs can have millions of distinct latent features, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs. We test our pipeline on SAEs of varying sizes, activation functions, and losses, trained on three different open-weight LLMs. We present new techniques to score the quality of explanations that are cheaper to run than the previous state of the art. We propose guidelines for how to generate better explanations that remain valid for a broader set of activating contexts, and discuss common pitfalls with the current scoring techniques. We also introduce a novel similarity metric for SAEs, and find that SAEs trained on nearby layers of the residual stream are much more similar than ones trained on adjacent MLPs. We anticipate that the open-source framework proposed can improve future evaluations of the interpretability of SAEs and will enable more work on this front.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
interpretability language model sae features explanation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3f1995c78920e3adea6fa7365df7840bb8d0304b.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Automatically Interpreting Millions of  Features in Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission