Interpreting the Second-Order Effects of Neurons in CLIP
CLIP;neurons;interpretability
__“What is the complexity of the proposed approach compared to other interpretability-focused methods?”__

Current interpretability approaches range from simple decomposition-based approaches (e.g. modeling the direct effect of components - TextSpan [1]/LogitLens [2]) to training-based approaches that aim to learn an interpretable decomposition of the representation (e.g. Sparse Autoencoders [3]). Our method does not rely on learning and instead uses classical sparse decomposition techniques, which makes it less compute-intensive. On the other hand, it looks at more specific computational paths than the direct effects or the indirect effect, and therefore it can provide in some cases more useful interpretation that can be used for improving downstream performance (see Table 3). As shown in A.8, the compute recruitments of this method are not significant in comparison to the training-based approaches.  

[2] Nostalgebraist, interpreting GPT: the logit lens, AI Alignment Forum 2022

[3] Cunningham et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models, ICLR 2024


__“Can the authors provide the texts' weights corresponding to each neuron's sparse decomposition? A full list of some neurons would also be helpful.”__

We provide an example for all the 128 text representations that correspond to neuron 4, layer 9 in the model and their corresponding weight coefficients (see Figure 5 for corresponding images with a highest second-order effect):

| Text         | Value | Text       | Value | Text     | Value  | Text   | Value |
|--------------|-------------|--------------|-------------|--------------|-------------|--------------|-------------|
| snowy | 0.28    | fuchs | -0.07   | discus | -0.05   | plato | 0.03    |
| frost      | 0.19    | timberland | 0.07    | treasury | -0.05   | transistor       | 0.03    |
| closings | 0.16    | thunderstorm     | -0.07   | narnia | 0.05    | viper | -0.03   |
| advent   | 0.15    | nav | -0.07   | tasmanian        | 0.05    | sligo | 0.03    |
| southwark        | -0.14   | auditorium       | -0.07   | burnley | 0.05    | pendleton        | -0.03   |
| grassy | -0.13   | chinatown        | -0.07   | elk | 0.05    | purdue | 0.03    |
| subcontractors   | -0.12   | atherosclerosis  | 0.07    | mosque | 0.05    | loss | -0.03   |
| wipers  | 0.11    | soybeans         | 0.07    | chores | 0.05    | bild | 0.03    |
| smugmug | 0.11    | ridges | -0.06   | couplings        | -0.05   | composites       | -0.03   |
| molten | 0.11    | naxos | -0.06   | kemp | -0.05   | packard | -0.03   |
| bosses | -0.11   | kazakhstan       | 0.06    | chinook | 0.05    | slr | -0.03   |
| angola | -0.11   | macedonian       | -0.06   | charleston       | -0.05   | steamboat        | 0.03    |
| reflective       | -0.11   | autoconf         | -0.06   | chocolate        | 0.05    | lesotho | 0.02    |
| ozark | -0.11   | dispenser        | -0.06   | woodstock        | -0.05   | genie | -0.02   |
| grouping         | 0.1     | pupils | -0.06   | pcos | 0.05    | griffith         | -0.02   |
| oakville         | 0.1     | firenze | 0.06    | dhl | -0.04   | garrison         | -0.02   |
| mikasa | -0.1    | ledger | -0.06   | manning | -0.04   | telluride        | -0.02   |
| exhibitionism    | -0.1    | beckham | 0.06    | amtrak | 0.04    | regency | 0.02    |
| horns | -0.1    | sasha | 0.06    | swimsuit         | -0.04   | xtc | 0.02    |
| anyone | -0.09   | pistons | 0.06    | peril | 0.04    | platelet         | -0.02   |
| orbit | -0.09   | kauai | 0.06    | hopper | 0.04    | coldplay         | -0.02   |
| testimonial      | -0.09   | banded | 0.06    | chairs | 0.04    | gump | -0.02   |
| clemente         | -0.09   | susceptible      | 0.06    | forte | 0.04    | shakira | -0.02   |
| orbit        | -0.09   | kauai        | 0.06    | hopper       | 0.04    | coldplay     | -0.02   |
| testimonial  | -0.09   | banded       | 0.06    | chairs       | 0.04    | gump         | -0.02   |
| clemente     | -0.09   | susceptible  | 0.06    | forte        | 0.04    | shakira      | -0.02   |
| vail   | 0.09    | tolkien      | -0.06   | bonnet | 0.04    | atpase | 0.02    |
| mimic | -0.09   | tortoise     | -0.06   | prominence   | 0.04    | weir | 0.02    |
| clay | -0.08   | braille      | -0.06   | drosophila   | 0.04    | lama | -0.02   |
| sprite       | -0.08   | counters     | 0.05    | neurosurgery | -0.04   | greyhound    | -0.02   |
| supervised   | 0.08    | september    | -0.05   | brackets     | 0.04    | clerks       | 0.02    |
| hardwood     | -0.08   | martial      | 0.05    | srv | 0.03    | social       | 0.01    |
| oj | 0.08    | flaws        | -0.05   | visor        | -0.03   | monet        | 0.01    |
| celsius      | 0.08    | airbrush     | -0.05   | commodore    | 0.03    | restaurant   | 0.01    |
| sidewalk     | 0.07    | sfo | -0.05   | maintained   | -0.03   | peridot      | 0.01    |
cont.
We thank the reviewer for the valuable comments and questions.

__“Can the authors provide a discussion on TextSPAN, highlighting the differences in the decomposition and analysis of the direct effects of the neurons?”__

[1] explores the direct effect of different components in CLIP. As shown in [1], the direct effects of the MLPs are very low – mean-ablating the direct effects of all the neurons in all the layers results in a negligible drop in performance (please see Table 1 in [1]). Moreover, the main observation is that most of the direct effect comes from the late attention layers. Therefore, in our analysis, we look at the information flow from neurons through such attention layers, directly to the output. A different way of looking at it is that in our analysis we decompose the contributions of attention layers into further flows from previous layers through them, and focus on the flows from individual neurons.   

__“The focus on a single dataset, i.e., ImageNet, is a bit restrictive in terms of analyzing the behavior of the proposed approach”__

Indeed, we mostly focus on ImageNet classification as our downstream evaluation metric, to analyze how much our ablations change the final representations, similarly to [1]. Nevertheless, as shown in Figure 10, performing similar analyses on other datasets (e.g., ImageNet-R) shows similar trends. 

__“What happens when trying to generate adversarial examples in a more complicated dataset?”__

Our approach for adversarial attacks does not rely on the images of the datasets. We only use the class names from the datasets to showcase that we can generate an image that will be classified as one class while having objects from the other class. We specifically used classes that look very different (random CIFAR class pairs - e.g., horse and automobile) to produce those examples. We provide additional examples for other classes in Figure 15. 

__“Did the authors reproduce the results for all the methods in Table 4?”__

Yes, we reproduced all the baselines and re-evaluated the different approaches. The difference in numbers from TextSpan is due to a different choice of a model – we focus on the model that was pre-trained by OpenAI on their data, while TextSpan focuses on a model that was trained by OpenCLIP on LAION.  

__“Qualitatively, what are the differences when choosing a different sparse set size m?”__

We found that different m’s provide relatively similar results when looking at the top coefficients. Nevertheless, larger m’s allow better reconstruction of the second-order of a neuron. That suggests that neurons are polysemantic and correspond to multiple texts, but many of them have only a few significant and coherent text directions for which they are responsible.
 
__“What is the motivation behind the consideration of a binary classification task instead of a classical setting? What classifier is used and how is it trained?”__

We aim to produce an adversarial attack on CLIP. Therefore, we use CLIP as the zero-shot classifier, by comparing the output image representation to the text representations of the two class descriptions (e.g., “An image of a dog”/”An image of a cat”). 

Typically, adversarial attacks are images that cause a classifier to label them as one class while they are perceived by humans as another class. This is the reason for choosing pairs of classes.  As there is no training or any other usage of the dataset except for the class names, we decided to choose class pairs that are very different and CLIP can usually easily distinguish between them (e.g., ship and truck). We show additional examples from other pairs of classes (not only from CIFAR10) in Figure 15. 

__“The authors mention that the generated adversarial images lie on the manifold of generated images differently from non-semantic adversarial attacks. Can the authors elaborate on this claim?”__

Classical (non-semantic) adversarial attacks modify images directly by adding pixel noise to the images, which causes the image to be classified incorrectly. Differently from this approach, our images are generated from a generative model that is trained to model the natural image distribution (for example – a diffusion model). We do not tamper with the generative model or the image pixels to make it adversarial.
We thank the reviewer for their question.

__“What is the relation between exploring what examples the neurons activate on vs. exploring their contribution in the output space, what are the advantages/limitations of one over the other, does one show complementary aspects that the other cannot observe?”__

There are non-trivial relations between the highly activated images of a neuron (its “input space”) and its output space. Here, for example, we rely on the assumption that if a neuron’s second effect contributes toward some direction (e.g. the “dog” text representation), it probably means that the neuron is activated on images that have the concept in them. While it is mostly the case and allows us to attack the model, we found some instances where a neuron has relatively high activation patterns on images of a concept, but since the neuron is followed by a GeLU (which can produce negative values for positive inputs), its second-order contribution direction is negative. Nevertheless, in most cases, there is a clear positive correspondence between the concepts in highly activated images and the neurons’ output space. Moreover, in most cases, there is a clear correspondence between the activation patterns and the output space (e.g., a dog segment will be highly activated for a neuron that has a high contribution towards the “dog” text representation). This property allows us to mine for the neurons with a high contribution toward a text representation and use their activation maps for zero-shot segmentation (see Figure 7 and Table 4).
We thank the reviewer for the valuable comments and questions.

__“Sparse coding to find textual descriptions of neurons may be very computationally expensive”__

Empirically, we didn’t find the sparse coding technique being too expensive. We used an off-the-shelf sklearn library (CPU-based solution) and did not try to optimize this part. Applying this sparse coding technique to all the neurons we analyzed (one layer in the model) with a dictionary of ~30000 entries took a few hours (please see Appendix A.8 for more details). 

__“Not considering nonlinearities in second-order effects (Eqn 5)”__

Indeed, for simplicity, we do not mention the nonlinearities (that come from the layer normalizations) in the main text. Nevertheless, in all of our experiments, we include this part in the computation of second-order effects. We included the extended derivations that we use to compute the second-order effects with layer normalization in Appendix A.6. 

__“Are there any gradient-based approaches to finding textual descriptions of each neuron's second-order effect?”__

To the best of our knowledge, there are no gradient-based approaches for finding textual descriptions for neurons, mostly due to the difficulty of back-propagating into a discrete space. 

__“Were any additional text-to-image models evaluated other than DeepFloyd IF, if so was there similar success to images generated with DeepFloyd IF?”__

We specifically used DeepFloyd IF as our model, because it does not use CLIP model in its conditioning. Nevertheless, the same approach can be applied to other generative models as well. Moreover, we suspect that with better models, that follow the prompt better, our adversarial results will improve, and we plan to analyze it in the future.
We thank the reviewer for the valuable questions.
 
__“What happens if you apply the same method to the text encoder in CLIP?”__

Although the same technique can be applied to the text encoder, we did not do that. More specifically, the text encoder is also a transformer, so we can compute the second-order effects similarly, and then apply sparse decomposition into a dictionary of text/image representations. We focused here on the vision encoder for exploring the polysemantic behavior of vision neurons with text and to demonstrate that the second order is useful for finding adversarial images. We plan to explore the text encoder in future work.

__“Have the authors tried it on other variants of CLIP?”__

We evaluated our model on different model sizes and found similar trends (please see Figures 9 and 12 in the supplementary material). 

__“Do the findings of this paper apply to other domains like medical imaging (MedCLIP)?”__

We did not apply it to medical imaging but we think that the same findings can be applied there as well, as the architecture of the model is similar.
Yes
No ethics review needed.
Please see the Weaknesses section.
This paper draws inspiration from recent approaches that aim to examine and evaluate the functionality of each neuron in a given architecture. Automated interpretability constitutes an important challenge for modern architectures and this work aims to approach this in a different way via the contribution of neurons to the output representation and the information flow through the MSA blocks.
This work presents a novel approach for examining potential second-order effects of neurons of CLIP representations and how these can be used in the context of zero-shot segmentation and generation of adversarial images. 

To this end, the authors focus on the contribution of each individual neuron to the output in terms of second-order effects via the computation of the additive contribution of each neuron through the MSAs and projection to the input space. 

The experimental evaluation focuses on the empirical analysis of the obtained effects and how these insights can be used in the context of generating "semantic" adversarial examples and using said effects for zero-shot segmentation.
The connection of the proposed approach to highly relevant work is a bit lacking. Can the authors provide a discussion on [1], highlighting the differences in the decomposition and analysis of the direct effects of the neurons?

I find the focus on a single dataset, i.e., ImageNet, to be a bit restrictive in terms of analysing the behavior of the proposed approach. Indeed, most approaches in this line of work considered additional datasets, e.g., Waterbirds, CUB and Places365. The same applies for the adversarial examples setting, where the authors only consider CIFAR-10. What happens when trying to generate adversarial examples in a more complicated dataset, e.g., CUB or ImageNet?

Did the authors reproduce the results for all the method in Table 4? Since they are different than the ones reported [1], I would expect that to be the case. 

Can the authors provide the weights of the texts corresponding to the sparse decomposition of each neuron? A full list for some neurons would also be helpful. 
Qualitatively, what are the differences when choosing a different sparse set size m. 

What is the motivation behind the consideration a binary classification task instead of a classical setting? What classifier is used and how is it trained?

The authors mention that the generated adversarial images lie on the manifold of generated images differently from non-semantic adversarial attacks. Can the authors elaborate on this claim?

What is the complexity of the proposed approach compared to other interpretability focused methods?

[1] Gandelsman et al.,  Interpreting CLIP’s image representation via text-based decomposition, ICLR 2024
Yes
No ethics review needed.
1. What happens if you apply the same method to the text encoder in CLIP?
2. Have the authors tried it on other variants of CLIP like MaskCLIP?
3. Do the findings of this paper apply to other domains like medical imaging (MedCLIP)?
1. The technical contributions are sound  and interesting.
2. The paper is well written. 
3. The paper included thorough evaluations.
The paper presents an interpretability study focused on understanding the second-order effects of neurons in CLIP. The authors propose a novel "second-order lens" to analyze neuron contributions that flow through attention heads to the model output.
Generally good paper so please see questions.
Yes
No ethics review needed.
- Are there any gradient based approaches to finding textual descriptions of each neuron's second order effect?
- Were any additional text to image models evaluated other than DeepFloyd IF, if so was there similar success to images generated with DeepFloyd IF?
- Extensive empirical validation of second order effects (e.g. second order effect neuron sparseness)
- Intuitive and interesting applications of second order effect control in the semantic adversarial example generation
- Increased understanding of internal attention model mechanism through semantic adversarial examples
- Improved segmentation results over TextSpan
This paper introduces an approach to generalize the first order effects of a neuron in attention networks into second order effects through successive layers. This methodology helps uncover the sparseness of the neuron second order effect as well as the decomposition of each neurons second order effect into a single direction vector.
- Sparse coding to find textual descriptions of neurons may be very computationally expensive
- Not considering nonlinearities in second order effects (Eqn 5)
Yes
No ethics review needed.
1. In the related work section, the paper lists several other neurons interpretability works that explore what concepts the neuron activated on, and saying that in contrast their work focuses on the neuron's contribution to the output space. My question is: what is the relation between the two approaches (exploring what examples the neurons activate on vs. exploring their contribution in the output space), what are the advantages / limitations of one over the other, does one show complementary aspects that the other cannot  observe?
1. The paper is well motivated, as exploring the functions of neurons in CLIP model facilitates better understanding of its representation learning mechanism, and guide further improvements
2. The paper proposes to study the "second order" effect of neurons, which is demonstrated to show a clearer signal of their contribution comparing to first order and indirect effects
3. The paper shows an interesting finding that each neuron's contribution to the joint text-image output space is approximately a single direction vector, and implies a ploysemy property.
4. The paper further exploits the finding of 3 for generating adversarial images and application of semantic segmentation, showing that by taking advantage of polysemy neuron's output representations, it's possible to fool the CLIP classifier by using words that's spuriously correlated with a wrong class; and that by aggregating the activation maps of neurons that highly correlated with a query concept, its semantic segmentation mask can be acquired.
This paper proposes to interpret the functional of individual neurons in CLIP by investigating their "second order" effects at the joint text-image output space. Specifically, a neuron's contribution to the output feature space is defined as the summing of the effects flowing from the neuron through the later attention heads, to the output. Experimental results show several observations: (1) the effects of an individual neuron are highly selective. (2) Each neuron's contribution in the output feature space can be approximated by a single direction, which can be decomposed into representations of a sparse set of texts. (3) By exploiting the polysemantic property of neuron, it is possible to generate adversarial images that fool the CLIP classifier. (4) The semantic-awareness of neurons can be utilized for semantic segmentation.
Please see questions
We present the second-order lens to interpret neurons in CLIP with text, and use it for segmentation, properties discovery in images, and mass production of semantic adversarial attacks.
@inproceedings{
anonymous2024interpreting,
title={Interpreting the Second-Order Effects of Neurons in {CLIP}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GPDcvoFGOL},
note={under review}
}
We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that a automated interpretation of neurons can be used for model deception and for introducing new model capabilities
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
CLIP neurons interpretability
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2dd20644c38591d4a0fc47b9e61c1c925b583ad9.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Interpreting the Second-Order Effects of Neurons in CLIP
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission