Evaluating Synthetic Activations composed of SAE Latents in GPT-2
Mechanistic Interpretability;SAEs;Activations;SAE Latents
Wrong formatting template.
Submission Desk Rejected by Program Chairs
We investigate how synthetic activations composed of SAE latents compare to real model-generated activations in GPT-2, revealing insights into the importance of SAE latents relationships and activation plateau characteristics.
@misc{
petrova2024evaluating,
title={Evaluating Synthetic Activations composed of {SAE} Latents in {GPT}-2},
author={Nora Petrova and Giorgi Giglemiani and Chatrik Singh Mangat and Stefan Heimersheim and Jett Janiak},
year={2024},
url={https://openreview.net/forum?id=U0y32WKeOd}
}
Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Nora_Petrova1 ~Giorgi_Giglemiani1 ~Chatrik_Singh_Mangat1 ~Stefan_Heimersheim1 ~Jett_Janiak1
Nora Petrova Giorgi Giglemiani Chatrik Singh Mangat Stefan Heimersheim Jett Janiak
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mechanistic Interpretability SAEs Activations SAE Latents
I certify that there is no acknowledgement section in this submission for double blind review.
petrova|evaluating_synthetic_activations_composed_of_sae_latents_in_gpt2
/pdf/6856cb7a877fc44bcb357136e4358aeb26b7c7ef.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Evaluating Synthetic Activations composed of SAE Latents in GPT-2
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission