SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders
Mechanistic interpretability;Large language models;Sparse autoencoders;Sparse dictionary learning;Unsupervised learning;Interpretable AI
Yes
N/A
No ethics review needed.
My main concern regarding this paper is its writing and presentation.
Studying intrinsic features that are interpretable in large language models is an interesting and important direction.
The  large sparse autoencoders (SAEs) and indirect object identification (IOI) look promising. This is a relatively new direction and the proposed method and findings make a good attempt in this direction. The proposed reconstruction method seems to be sound.
This work tries to scale up the feature evaluation problem for large sparse autoencoders (SAEs) in large language models, towards interpretable AI. The main contribution of the work is a new reconstruction approach for task-specific features with large training reduction of the SAEs. The overall method and experimental results look fine.
This is a paper difficult to read. It is mainly based on two prior works, 

[1] . "Interpretability in the wild: a circuit for indirect object identification in GPT-2 small", Wang et al. 2023, 
which defines the problem of indirect object identification (IOI) with a discovery of circuit; and

[2]  "Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control", Makelov et al. 
that gives a framework for supervised learning of sparse autoencoders (SAEs) for the indirect object identification (IOI) task.

I am not very familiar with the IOI and SAE literature, which seems to be a recently proposed problem in large language models. [1] and [2] were written in a relatively clean and clear way for readers to quickly understand the motivation and problem definition.

However, the paper in study is much harder to read than [1] and [2]. This may be also due to my unfamiliarity to this direction. This paper seems to make an assumption of having the readers with all the prior knowledge about IOI and SAEs by quickly throwing out the jargon words without much explanation.
Yes
No ethics review needed.
- I suggest improving the writing, especially in the method section. 
- The motivation for accelerating training of SAEs need to be better
  justified.
- How can one assess the accuracy of the cross-section discovery?
+ Interpretation of components in LLMs is a highly relevant topic.
+ SAEs is an interesting direction that recently attracted attention.
+ Automated identification of cross-sections is a step forward. The
  previous work relied on prior knowledge and thus was limited on what
  it could be applied. Using automated discovery of cross-sections,
  this limitation is alleviated.
Authors consider the question of evaluating sparse auto-encoder
models, which are studied for interpreting large scale models. They
start from a recent article and build on top of it. The previous
article already proposed a strategy for evaluating SAE models using
supervised dictionary features, extracted for a given task and points
(cross-sections) in a language model. Here, this work is extended by
first identifying the cross-section automatically and then using a
different training strategy for the SAEs. This way, authors claim that
the evaluation can be done for large-scale LLMs.
- overall, I think the problem definition and the description can be
  better written. Author rely heavily on the previous work, i.e.,
  Makelov et al., for their description. However, the current article
  on its own is not trivial to understand.
- It is unclear how the supervised dictionaries are extracted in this
  work and whether this is something new. Makelov already discusses in
  appendix A.6 the MSE estimation of the supervised
  dictionaries. Furthermore, in that model $u$ vectors are extracted
  using MSE. Here, it is unclear how $u$ vectors are
  extracted. Equation 5 and 6 only describes the weighting of the
  vectors.
- Projection-based reconstruction with residual streams is not really
  justified in my opinion. More specifically, this section makes me
  question the goal of this work. In Makelov et al.'s work, authors
  aim to evaluate SAEs, thus the cost of training SAEs is not really
  an issue. If the goal here is the same, why is the training of SAEs
  all of a sudden an issue? I am not sure if the motivation for
  reducing the training cost and the development for the residual
  stream is clear at all.
- Certain parts of the article are put in the Appendix, e.g.,
  A.5. This makes me wonder whether given the page limitations of
  conferences, a journal would be a better venue for this work.
- Discovery of the cross-sections is the main novelty in my opinion,
  however, this is also a part that is not trivial to evaluate and
  assess the accuracy. Authors should perhaps discuss this a bit
  further. Is there a way to really show that this is indeed a good
  method? 

Minor:
- the last $x_{corr}$ on line 264 is most likely $x_{clean}$.
Yes
No ethics review needed.
Abstract has some contradictions in it, it seems that the paper is about an evaluation framework, however, at the same time it seems like there are some model improvements, e.g., ‘introduce a novel reconstruction method’. 

Abstract. The validation is also unclear, the authors say that they validate the results on  novel tasks on 3 datasets, however, it is unclear what is the conclusion of these validations. Could the authors clarify this?

Could the authors clarify the novelty of the introduced approach?

Intro 2nd paragraph – the connection between the interpretability of LLMs and SAEs is not clear – could the authors clarify the connection?

Intro 3rd paragraph – it is not clear what circuit discovery has to do with evaluation of SAE, could the authors clarify this?

Intro: Could you please clarify what is meant by large frontier model?

Intro: What the authors mean by linearly represented features?

Intro: The use of manual experimentation is unclear. 

Intro: What the authors mean by sublayer activations and residual stream SAE?

Please make all the legends in all figures bigger.
The paper is dealing with a problem that Is of interest to the ICLR community.
The paper introduces an evaluation framework to evaluate sparse autoencoders called SAGE. The approach consists of attribute cross-sections, supervised feature-dictionary reconstruction and projection-based reconstruction.  The framework is evaluated on novel tasks on Pythia 70M, GPT-2 small and Gemma-2-2B.
The main weakness of this paper Is it’s presentation, it is very difficult to get through the manuscript and understand the introduced evaluation framework. One would expect to have a clear answer to the following three questions after reading the abstract and the intro: (1) what is done, (2) why it is done, and (3) how it is done. However, after reading the paper multiple times, the reviewer still struggles to find the responses to the above-mentioned questions.  For details, please see Questions box. 

The current paper presentation makes it difficult to assess the papers’ novelty, impact and significance of the results.
Yes
No ethics review needed.
It would be great if you could address weakness 1. I also think the presentation of the method and results should be made much less abstract, i.e. with some concrete examples.
The chosen task is important and highly relevant.
The paper proposes a scheme to automate SAE evaluations for large models/complex tasks and for situations where ground truth features are not known.

There are several claimed contributions:

A novel (as far as I can tell) approach to identify cross sections by leveraging clean and corrupted examples w.r.t the attribute of interest.

“A novel projection-based reconstruction using residual stream SAE”

A modification to the supervised feature dictionary approach of Makelov (2024) in which they instead “propose to compute weights to minimize the MSE loss between the reconstruction and the activation:”

Taken together, these innovations allow SAE evaluations to be “scaled,” apparently meaning w.r.t tasks with a lack of ground truth features or increased model size.
Philosophically, I don’t understand the point of this paper. If I am reading it correctly, and it’s possible I am not, the idea is to generate pseudo-ground truth features via “supervised feature dictionary learning” for cross-sections corresponding to features of interest (which are also identified automatically). Then it is claimed these features can be used to evaluate SAEs. But this seems to me to defy the whole point of SAE evaluation in the first place, which is to isolate network components which are interpretable to humans. 

The presentation of the method is also generally pretty confusing because it’s entangled with the presentation of background and related work. I also feel like it’s too abstract and needs a lot more concrete examples and figures to be grounded effectively. 

The experimental results are hard to read, both due to the small font and the chart-style presentation.
This paper introduces SAGE, a framework to scale realistic ground truth evaluations of sparse autoencoders to large, state-of-the-art language models such as Gemma-2-2B.
@inproceedings{
anonymous2024sage,
title={{SAGE}: Scalable Ground Truth Evaluations for Large Sparse Autoencoders},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sknUS8X9q0},
note={under review}
}
A key challenge in interpretability is to decompose model activations into meaningful features. Sparse autoencoders (SAEs) have emerged as a promising tool for this task. However, a central problem in evaluating the quality of SAEs is the absence of ground truth features to serve as an evaluation gold standard. Current evaluation methods for SAEs are therefore confronted with a significant trade-off: SAEs can either leverage toy models or other proxies with predefined ground truth features; or they use extensive prior knowledge of realistic task circuits. The former limits the generalizability of the evaluation results, while the latter limits the range of models and tasks that can be used for evaluations. We introduce SAGE: Scalable Autoencoder Ground-truth Evaluation, an evaluation framework for SAEs that enables obtaining high-quality feature dictionaries for diverse tasks and feature distributions without relying on prior knowledge. Specifically, we lift previous limitations by showing that ground truth evaluations on realistic tasks can be automated and scaled. First, we show that we can automatically identify the cross-sections in the model where task-specific features are active. Second, we demonstrate that we can then compute the ground truth features at these cross-sections. Third, we introduce a novel reconstruction method which significantly reduces the amount of trained SAEs needed for the evaluation. This addresses scalability limitations in prior work and significantly simplifies the practical evaluations. We validate our results by evaluating SAEs on novel tasks on Pythia70M, GPT-2 Small, and Gemma-2-2B, thus demonstrating the scalability of our method to state-of-the-art open-source frontier models. These advancements pave the way for generalizable, large-scale evaluations of SAEs in interpretability research.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mechanistic interpretability Large language models Sparse autoencoders Sparse dictionary learning Unsupervised learning Interpretable AI
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b12d3510a6f949124f02544a43354a3723fd291e.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/8f8fea29401e61849d24971e12949b30439791f8.zip
SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission