Sparse autoencoders reveal selective remapping of visual concepts during adaptation
interpretability;vision-language models;sparse autoencoder;adaptation
Yes
No ethics review needed.
* "Specifically, we suggest taking all tokens including class (CLS) and image tokens from the second last attention block output and feeding them to SAE rather than limiting it to CLS token" Can the authors explain why they chose this particular layer for their analysis? Prior work has shown that self-supervised ViTs learn different invariances (hence, concepts) at different depths [1, 2], so one can expect that the SAE results from them would be very different. Are the results derived by the authors on this particular layer generalizable to more layers at different depoths, or did the authors chose this as a representative case study? If so, it would be interesting to see some ablations on how the results change with layer depth.
* I would be interested in seeing the results of experiment 3 (adaptation methods and SAE) with fine-tuning. Have the authors considered doing an experiment with fine-tuned CLIP for comparison? What are the potential bottlenecks in running such an analysis. And what results can be expected?
* Typo L146-147: Then we compute *how* frequently these features are activated for the given input.


# References 

1. Walmer, M., Suri, S., Gupta, K., & Shrivastava, A. (2023). Teaching matters: Investigating the role of supervision in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7486-7496).
2. Shekhar, S., Bordes, F., Vincent, P., & Morcos, A. (2023). Objectives matter: Understanding the impact of self-supervised objectives on vision transformer representations. arXiv preprint arXiv:2304.13089.
* The authors do a good job at utilizing both feature based and example based summary statistics to demonstrate presence of concepts in the SAE representations, and motivate all four statistics well in Section 2.2. Unlike prior work on inferring CLIP concepts from SAEs, which tend to rely on singular feature space statistics, this makes the author's results a bit more comprehensive.
* The results from replacing CLIP features with SAE features and its impact on accuracy shows that the SAE conceptual features capture class discriminative information quite well. I think this result opens up more avenues on mechanistic interpretability research, as further exploration can be done into what makes a concept important vis-a-vis a class, what are the training dynamics of the concept learning for class discriminative performance (are there 'easier' concepts that are learned first e.g. winter before harder concepts e.g. christmas), and several other open research questions.
The authors perform sparse autoencoder (SAE) training on the features of a pre-trained vision transfer (ViT) from the CLIP image encoder.. The outputs of the 11th layer of the CLIP ViT is used to train a two layer SAE with reconstruction + sparsity objective. The SAE encoder features are then used to validate the presence of concepts in CLIP representations, and their role in classification as well as downstream tasks. The authors first utilize feature and example based statistics to sanity check that concepts are indeed present in SAEs trained on the CLIP visual representations, as well as multi-modal CLIP representations. Next, they verify that these conceptual features are relevant for class discriminative performance of CLIP zero-shot classification. Lastly, the authors show that SAE conceptual features after prompt-based adaptation show a suppression in non class-relevant features, while and increase in the activation of class-relevant features. New conceptual features are not learned during this adaptation procedure.
* The organization of the paper makes it hard for someone who is not very familiar with the field of sparse autoencoder based mechanistic interpretability of neural networks to follow the flow of ideas. For example Section 2 details the training procedure and the feature statistics calculated, but the experimental details are ommitted and then presented in Section 4. Similarly, the Intro and Section 2.1 cover a brief overview of SAEs and their use in intepretability research, but the actual related work section is not presented until Section 5. While I understand that the authors are eager to share their key results in an earlier section (especially since this is a intepretability focussed work), this organization made it more work for me as a reader to understand the paper.
* The scope of experiments performed and the subsequential observations is somewhat limited. The first experiment, showing presence of concepts in CLIP visual features, was already shown in principle by Fry et al and Daujotas et al. The result about CLIP attention maps focussing on features at different scales and regions is not surprising, but another sanity check for the SAE interpretability apporach. The second emperimental design is inspired from previous SAE results in LLMs e.g. Templeton et al, and extrapolated from a language (only) transformer setting to a multimodal setting.
Yes
No ethics review needed.
See weaknecess.
- The paper transfers the methodology of SAE from NLP to vision. The idea of identifying visual concepts from CLIP features is quite interesting.
- From visualizations, it seems that the SAE on top of CLIP sucessfully identified some meaningful semantic concertps in both image-level and patch-level.
This paper introduced Sparse AutoEncoder (SAE), which previously trained to disentagle and interprete LLM hidden states, to analysis CLIP features. The authors trained a SAE on top of CLIP features with ImageNet samples, and carried out several experiments with it demonstrating 3 points: 
- Section 3.1: certain demension in SAE feature correlates to certain visual concepts
- Section 3.2: targeted ablation of neurons yeilds degraded performance
- Section 3.3: CLIP model "re-use already known concepts" when it is fine-tuned for downstream datasets
1. **Lacks quantitative evaluation of SAE's reliability**. Although SAEs for LLMs are widely studied in NLP, to my best knowledge, the usage of SAE for CLIP is only reported in several blog posts that have not been peer-reviewed. Since its effectiveness and reliability are not sufficiently justified in a convincing way, directly using it to draw conclusions is quite risky.

    - For example, how can we ensure that the training settings listed in Section 4.1 are properly set? Are 49,152 hidden dimensions enough for the vision domain? Does this SAE achieve monosemanticity? 

    - Examples in the case study are certainly not sufficient for a rigorous evaluation. I would suggest conducting a large-scale evaluation using vision datasets with fine-grained attribute annotations to answer the above questions.


1. **Potential distribution shift between base and fine-tuned CLIP**. As stated in Line 325, the authors used the SAE trained on top of pretrained CLIP to interpret the fine-tuned CLIP. As MaPLe fine-tunes both image and text encoders, there are no guarantees that the feature distribution will remain the same after fine-tuning, and all the conclusions in Section 3.3 might be inaccurate as a result.


1. **Regarding "re-mapping" or "re-using" visual concepts**. As highlighted in the paper title and section titles, the authors wanted to demonstrate the connection of visual concepts between base and fine-tuned CLIP. However, I am a bit confused regarding the terminology used, as neither "re-mapping" nor "re-using" is properly defined in the paper. The authors are encouraged to provide a clear definition of these terms and highlight how the results (e.g., Fig. 6c) demonstrate this point.


1. **Significance of Contribution**. Overall, I find the new insights provided in this paper are not as substantial as I expected when I read only the title. Training SAE on CLIP features is not novel. It has been previously reported in several blog posts, yet this paper does not provide a more formal and quantitative evaluation. The results in Section 3.1 and 3.2 are somewhat expected and are not particularly surprising. 

    I would suggest that the authors conduct large-scale quantitative validation to prove the effectiveness of CLIP's SAE, e.g., ablation of hyperparameters, training settings, and types of CLIP models (convolutional encoders are not covered in the presented study). Additionally, clearly stating the relation and differences between SAE and existing interpretability methods in the vision domain would help readers understand the contributions.
Yes
n/a
No ethics review needed.
- Please see "weaknesses" above for a few questions. 
- How would the results change if the authors use a subset (or a totally different set) of "SAE feature statisics"? Why did the authors chose the ones they use in the paper (eg, sparsity, label entropy, mean activation, etc). Was any ablation study performed to select these features? Were they selected in previous work?
- How would the proposed approach perform on other language-vision models besides CLIP? Would the selected hyperparameters need to be different in this case?
- Better understanding foundantion models (and what their features encode) is an extremely important problem---specially as these models get more powerful.
- The proposed approach is simple  and leverages the well-understood sparse autoencoder approach to interpret the features.
In this paper, the authors propose a new way to try to interpret features learned from a vision-language (in particular, a CLIP model). The proposed approach train SAE on all image tokens. The author then show that the trained SAE can be used to discover different concepts from the CLIP vision transformer. The authors then investigate how the learned "interpretable features" relates the output of the model.
- The paper has some very strong claims that has not been properly demonstrated. Eg, L249 says "CLIP understands sophisticated concepts from input images not only responding to basic patterns such as color or shape ". Showing a a few (potentially cherrypicked) qualitative examples is not enough to show/demonstrate/prove the results on the paper. It would make results stronger if  the authors could provide more quantitative metrics to measure the sophistication of detected concepts. This would make the results on the paper more convincing/interesting.
- Some sessions of the paper are not very clear. For example, the authors leverage MaPLe model (eg "adaptation methods analysis") and nowhere in the paper they explain how this model works. It would make the paper more readable if the authors could provide a brief overview of this model on the manuscript.
- A lot of design choices seem to be made ad-hoc. For example, why chose the second-to-last layer? Why use the SAE feature statistics used? Why 49,152 dimensions? Could the authors provide some justification for these choices? How were those choices made? Was any ablation study performed to choose those values?
Yes
No ethics review needed.
- Section 3.3 (ADAPTATION METHODS AND SAE) is difficult to follow. Can you provide a more concise and coherent explanation of the logic behind the analysis, supported by a smaller set of essential figures?
- What does “suddenly capture new concepts” mean in this context? In human cognition, new concepts are usually built upon existing knowledge. Given this, how does this study distinguish between the reuse of old concepts and the acquisition of new concepts?
- The paper offers a detailed and fine-grained exploration of CLIP by processing the entire token sequence with SAEs, allowing for a deeper understanding of the model's visual feature representations.
- It tackles the adaptation dynamics of foundation models like CLIP, which is a relatively under-explored topic.
This paper utilizes sparse autoencoders (SAE) to interpret visual concepts learned by the CLIP vision model and investigates how these concepts are affected by adaptation techniques such as MaPLe. The key finding is that during adaptation, the model primarily reuses existing concepts rather than learning entirely new ones.
- The clarity and organization of the analyses are insufficient. The main body of the paper frequently refers to figures in the appendix, many of which are poorly annotated, making it difficult for readers to follow the core arguments. The authors should revise the manuscript to consolidate the most important findings in the main body and present them in a clearer and more structured format.
- The paper lacks sufficient discussion on the broader implications and significance of its findings. Are any results surprising or do they challenge conventional views within the field? The authors should also elaborate on the practical value of their discoveries.
@inproceedings{
anonymous2024sparse,
title={Sparse autoencoders reveal selective remapping of visual concepts during adaptation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=imT03YXlG2},
note={under review}
}
Compared to conventional machine learning models, foundation models excel at adaptation to a variety of downstream tasks with limited examples. Adapting foundation models for specific purposes has become a standard, yet it is an open question which mechanisms are in place during adaptation. Here we propose a variant of sparse auto-encoders to discover 49,000 candidate concepts relevant to the ImageNet reference task. We explore how these concepts influence to the model output and extend it to investigate how recent state-of-the-art adaptation techniques like MaPLe change the association of model inputs to these concepts. While activations of concepts slightly change between adapted and non-adapted models, we find that the majority of gains on common adaptation tasks can be explained with the existing concepts within the foundation model. This work provides a concrete framework to train and use SAEs for vision transformers and provides insights into the existing adaptation techniques.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
interpretability vision-language models sparse autoencoder adaptation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f454159f94270b0e2e0c591391e91e9bb07e1b72.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Sparse autoencoders reveal selective remapping of visual concepts during adaptation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission