A Principled Evaluation Framework for Neuron Explanations
interpretability;mechanistic interpretability;explainable AI;trustworthy machine learning
Yes
None.
No ethics review needed.
- Equation (1). Why consider layer $l$ after specifying target neuron $k$ ?
    - Equation (1). Is $t_k$ a textual description of neuron function or a one-hot vector based on concepts?
    - Equation (1). What is the process for generating explanation $t$ ?
    - Section 4.1. In general machine learning, it is common to comprehensively consider recall, precision, and F1 score. Could you explain the connection between individually assessing precision or recall and the more prevalent approach of evaluating all three metrics together?
- This paper provides a detailed summary of existing work and unifies them under a mathematical framework.
    - The experimental results of this paper reveal shortcomings in the majority of existing evaluation metrics.
The existing methods often use completely different metrics to evaluate how good their descriptions are, but it is not clear how they compare to each other. This paper unifies many existing explanation evaluation methods under one mathematical framework. This paper proposes two simple sanity checks on the evaluation metrics and shows that many commonly used metrics fail these tests and do not change their score after massive changes to the concept labels. Finally, this paper proposes guidelines that future evaluations should follow and identifies good evaluation metrics such as correlation.
- The clarity of this paper could be improved. It is necessary in Equation (1) to present the form of function $\mathcal{G}$ and samples from the probing dataset $\mathcal{D}$ to help readers understand the process of neuron explanation.
    - More information is necessary to explain how the results in the table in Figure 2 were computed. The author did not specify which images in Figure 2 represent the ground truth, which were used for generating prediction results.
    - It is essential to provide additional details on how the results in the table in Figure 2 are computed. The author does not clarify which images in Figure 2 are used to generate prediction results and how these concepts are determined. For example, Equation (10) states $M(a_k,c_t)={B(a_k)}_i$, and at line 331, ${B(a_k)}_{i}=1$, so why does the precision for "Animal" equal 0.5 instead of 0?
Yes
No ethics review needed.
- Have you considered applying the proposed sanity checks to evaluation metrics in other domains, such as NLP or multimodal systems? This could demonstrate the broader applicability of your framework.

- Would an analysis that incorporates feature visualization methods alter or reinforce your conclusions about the reliability of different metrics?

- Extend your analysis to include neurons from intermediate layers and various model architectures to assess the framework's robustness across different scenarios.

- Incorporate qualitative analyses or practical case studies or concept-based dataset ?

**Overall, this paper is a notable contribution to the evaluation of neuron explanations. However, there are some major limitations and a major rewrite would further strengthen its overall impact on the field.**
This paper is well-written and adress an important problem to the field of explainability:

1. **Unified Framework**: The paper effectively consolidates various existing evaluation metrics under one theoretical umbrella, making it easier to compare and contrast them. This is a valuable step toward standardizing neuron explanation evaluations.
2. **Introduction of Sanity Checks**: The proposed "Missing Labels" and "Extra Labels" tests are innovative and help identify metrics that may produce unreliable results. These tests are practical tools for researchers aiming to choose or develop more robust evaluation metrics.
The paper introduces a unified mathematical framework for evaluating neuron explanations and proposes two new sanity checks to assess the reliability of these evaluation metrics. The work highlights the shortcomings of many current metrics and recommends correlation (with uniform sampling) as the most reliable metric. Through theoretical analysis and comprehensive experiments, the paper provides insights into the biases of common metrics and emphasizes the importance of unbiased sampling.
Nevertheless, this paper has issues, some more significant than others. I will categorize these into **major problems (M)** and **minor problems (m)**. It is important to emphasize that all these issues are addressable and do not undermine the overall quality of the paper.

**M.1.** Related Work Insufficiency: the related work section is lacking for an ICLR-level submission, citing only 25 references. The paper overlooks key literature on concept extraction, concept importance estimation, and human-centered evaluations of neurons and concepts—some of which directly relate to the challenges discussed in the paper.

**M.3.** Limited Discussion on Metric Limitations: while correlation is presented as the most reliable metric, the paper does not thoroughly explore its potential limitations or scenarios where it may fall short! A more balanced discussion that addresses these limitations would enrich the analysis.

**M.5** Focus on Final Layer Neurons: the experiments are predominantly focused on final layer neurons, which might not represent the complexity found in intermediate layers. Including analyses of hidden layers would provide a more comprehensive evaluation of the framework's applicability across the entire network.

**Minor Problems (m)**:

**m1.** Impact of Hyperparameter α: the paper does not discuss the effect of the α parameter on the results. Is there an optimal α that depends on the specific metric used? A brief analysis would be helpful.

**m.2** Lengthy Recap of Basic Metric Definitions: the paper devotes two pages to revisiting standard classification metrics such as precision, F1-score, AUC, and cosine similarity. For an ICLR audience, this level of detail is unnecessary and could be condensed or omitted.

**m.3.** Redundant Explanations of Failure Cases: what the paper describes as "Failure cases of recall" and "Failure cases of precision" are related to standard type I and type II errors in statistics. Entire paragraphs dedicated to these concepts are redundant and could be streamlined.

**m.4** Model-wise Table Splits Without Aggregate Analysis: the tables present results model-wise but do not offer aggregate data. Including aggregated results could provide a clearer overall picture of the metric performance.
Yes
No ethics review needed.
Please refer to the weakness part.
1. The authors focus on an important topic of the evaluation of explanation methods.
2. The authors compare many different evaluation metrics in experiments.
This paper focuses on the evaluation of explanation methods for neurons in DNNs. The authors first formalize the evaluation task in a unified framework. Then, they identify the theoretical flaws of existing evaluation metrics, and further propose two sanity checks for evaluation methods. In experiments, they find that most evaluation methods cannot pass the sanity check, while the correlation metric and cosine similarity exhibited good performance.
1. This paper is only limited to explanation methods based on simple concepts. Specifically, the explanation in this paper only refers to whether a neuron represents a specific concept, where the concept is simplified as the class label or superclass labels. However, the explanation for a neuron is much more complex and not limited to representing a single concept, such as the receptive filed of a neuron or saliency map. The proposed evaluation principles cannot be scaled to other explanation methods.
2. The proposed method is ad-hoc. Why do you only adjust the concept label for testing? Why not try to adjust neural activations? The theoretical guarantee for the completeness of the proposed method is needed.
3. The mathematical formulation of the explanation $t_k$ is unclear. Does it represent a scalar, a vector, or a text description? 
4. All the metrics and sanity checks seem to be limited to the explanation for a neuron. How do you compute these evaluation metrics and sanity check if the explanation is based on linear combination of neurons? Many papers have claimed a concept is usually jointly represented by the combination of multiple neurons, rather than a single neuron.
5. The completeness of sanity checks in the paper is not guaranteed. In other words, even if an evaluation metric passes two sanity checks, it still cannot grauantee this metric to be a faithful metric. The limitation of the evaluation method needs to be formulated in mathematics.
6. All experiments in the paper are conducted using class labels or superclass labels as the concept. If the concepts are more fine-grained attributes like stripe or wings, is the proposed method powerful enough to still identify the difference between different evaluation metrics?
Yes
No ethics review needed.
See weaknesses.
1. The study tackles an important research question in XAI by aiming to compare and contrast existing evaluation metrics for neuron explanation methods and understand the evaluation pipeline with increased clarity.

2. The definitions and explanations related to the mathematical framework are well articulated.

3. The introduction of two sanity checks based on the mathematical framework seems sensible. These checks provide practical tools for assessing the robustness of the evaluation metrics.

4. The findings effectively highlight the shortcomings of existing evaluation methods and underscore crucial considerations for designing robust evaluation metrics. This enhances our understanding of how to measure the effectiveness of neuron explanations accurately.
The paper addresses a critical issue in neuron explanation methods: the works of neuron explanation methods often use completely different metrics to evaluate how good their explanations are. It proposes a unified mathematical framework to unify existing evaluation metrices. 

Furthermore, the paper introduces two sanity checks for evaluation metrics: the Missing Labels test and the Extra Labels test. These tests are designed to assess the robustness of commonly used metrics. The findings reveal that many of these metrics fail to pass at least one of these fundamental tests. A key outcome of the research is the identification of Correlation as the most effective overall evaluation metric.
1.	Ambiguity in Parameter Selection: The paper does not provide a clear rationale for the choice of threshold hyperparameters in equations (5) and (6), nor for the value of \alpha in the second experimental setting, or the ratios 0.5/2 in the first and second sanity checks. It remains unclear how these values were determined and whether different values might impact the results.

2.	While I understand that the paper focuses on evaluating Function 1, considering the title “A Principled Evaluation Framework for Neuron Explanations,” it would be beneficial to discuss the relationship between evaluating Function 1 and Function 2 more thoroughly. It would be advantageous to demonstrate whether the current evaluation framework could be adapted to evaluate Function 2 with some modifications. If this is not feasible, I would suggest that the authors narrow their claims prior to Section 2.3 to better reflect the specific focus of the framework.

3.	In Setup 4 of Table 7, all methods fail to achieve high accuracy. Could the authors explain why this is the case or why this setup is particularly more challenging? In this scenario, even the best method achieved less than 80% accuracy. Does this indicate that the current best evaluation methods are still not perfect?
Yes
No ethics review needed.
All the weaknesses listed above are indirect questions for the authors. Therefore, please consider them as points to discuss. The score should be considered a placeholder. I will significantly revise it if the authors provide further clarification or supporting evidence or adapt the narrative and claims of the paper to the raised concerns.

All the following questions did not affect my evaluation but I would appreciate a clarification:
- What is the rationale for assigning citations to the metrics in Table 1? Some of the metrics have been used in multiple papers, but only one of them is reported (e.g. WPMI)
- There are some simplifications in the setup and definitions of the tested metrics and the way these metrics have been used in the original papers (e.g., correlation score or IoU). Is this correct or are they the same? If the answer is negative, is there a mathematical proof ensuring that the results for the simplified and unified versions hold for the settings tested by the original papers? If so, please include this information in the appendix.
- It is not clear why some metrics, like the Cosine cubed, are included. They have been used only as an optimization metric for CBMs and not strictly for comparing and evaluating neuron explanations (even if the target is related). If the authors prefer to include this kind of metrics, there are many other similar optimization metrics in the CBM research area that should be included.
- The motivation for considering correlation a better choice than cosine similarity seems quite weak as reported in the paper (just one sentence vs better results overall!). Can authors provide an example of “significant errors when explaining neurons whose average activation is different from zero”?
- Evaluating the quality of explanations and addressing the fragmentation of the metrics is arguably the most important problem in the field of explainable artificial intelligence. The field needs more papers in this direction.

- The development of more sanity checks for explanations is important and would be a good contribution


- The pure results in the tables (without the analysis and claims - see weaknesses and questions), are significant and important to the field and can raise awareness about the weaknesses of metrics used for evaluating explanations.
The paper studies the problem of evaluating neuron explanations. Specifically, it proposes a unification of some of the metrics used for neuron explanations and two sanity checks that a good metric needs to pass. The sanity checks work by replacing the class labels associated with the data points in the probing dataset. While the first check removes half of the labels for a given class, the second doubles their count. The idea is to use such labels as a concept, keep the explanations associated with the neurons the same, and check the sensitivity of the metrics to the change. To compute the ground truth explanations (which will be kept fixed), the authors consider the output neurons of vision models where the ground truth is the label to be predicted and the class that optimizes the overlap between firing rate and class samples for a hidden layer. The authors claim that good metrics should exhibit a large change in the scores and, based on such claims, identify and claim the best metric among the tested ones as the one where the gap is the largest.
- The most significant weakness relates to the claims and analysis of the results. **There is a mismatch between what the authors claim,** (assess which metric is the best one to evaluate explanations) **and what the results and the sanity check support** (asses which metric is the most sensitive to the concept label noise in a probing dataset -  i.e., the (non-) robustness of metrics in the presence of concept label noise). In a few words, claiming that the best metric to evaluate explanations is one that exhibits the largest sensitivity to label noise is misleading and not conceptually precise. I would suggest maintaining all the experiments and changing the narrative and the analysis to better reflect the results of the experiments (for example moving the focus from "finding the best metric for evaluating neuron explanations" to "benchmarking metrics in the presence of label noise"). 


This first point requires a detailed description, provided below. The authors are encouraged to provide further clarification or supporting evidence or adapt the narrative and claims of the paper to the raised concern:

Consider the setup of the output neurons considered by the authors. In this case, the tests:
- do not change the input (so the concepts are still included or not included in the input), 
- do not change the explanation (which is truthful as claimed by the authors to be the ground truth), and
- do not change the neuron’s behavior. 

This setup means that the explanation is still the right one because it reflects what the neuron actually learned and its behavior. Therefore, in an ideal (and unrealistic) scenario, a perfect metric should still assign a high score with the explanation associated with the neuron both before and after the tests based on the semantics of the input and the behavior of the neuron and ignoring the label associated with the input. 

Conversely, *what the tests are doing is **introducing label noise** into the probing dataset.*
The problem of label noise is well-studied and well-known in binary classification setups. Ideally, a perfect metric should be immune to label noise, as stated before. However, such a metric is unrealistic, and at the practical level, f1 or AUC are the best and standard de facto metric. Analyzing the results in the table, the best metrics identified by the authors are more sensitive to label noise than these standard metrics for binary classification problems. It is not clear why a metric should be more sensitive to label noise than these latter metrics. This is a good sign that the claim of the authors about the criteria to establish the best metric could be misleading and needs to be reconsidered or further assessed. Again, there is nothing wrong with the experiments or the sanity checks; they could be a wonderful resource against adversarial attacks or to check robustness against label noise. **However, I argue these tests should not be used to identify the best metric to evaluate the quality of explanations.**

- **There is a mismatch between the generality of the claims (best metrics for neural explanations) and the specific settings analyzed**. Given the scope and the expected impact of the paper, it is essential to be rigorous and to state explicitly at the beginning of the paper (alongside the claims) what are the assumptions and settings where this behavior has been observed and holds true. Currently, these must be inferred implicitly from the paper. In the current state, the experiments focus only on (i) the vision domain (ii), models trained on ImageNet, (iii) settings where the concepts correspond to the classes on which the model was trained on, and (iv) concepts are identified by their binary presence. Several of these assumptions are not standards (e.g., (iii) (iv)). The authors should provide more evidence of the generality of their claims in different domains (e.g., nlp), models (trained on different datasets and not trained on the concepts), and setups (e.g., non-binary concepts) claims and explicitly state these assumptions.

- The organization of the paper could be slightly improved. For example, the metrics evaluated in this paper are presented only as equations in the main text. There is no textual description of them. Since they are the main focus of the paper, they deserve more attention.  It is unclear what these metrics measure, why they were proposed, and what their strengths and weaknesses are. Some details, such as the setup where they were proposed (e.g., domain, granularity) and ranges, are included in tables and the appendix, but these are not sufficient to fully understand the metrics. To save space, certain sections that are not the main focus of the paper do not need detailed descriptions in the main text (e.g., function 2 in Section 2.3 and associated discussions, or even the generation part of Section 2.2).

- It is unclear what rationale and process were used to select the metrics analyzed. As with the previous points, since the claims involve identifying the best metric to use in future work, it is important to adopt a clear rationale and adhere to it.  Some metrics included in the analysis have been used only in one paper (or at least it seems so by looking at the table references), so popularity does not seem to be the criterion. Meanwhile, metrics used in other papers are missing from the analysis. Adopting a rigorous strategy (e.g. popularity, publication venues, date of publication, compatibility with the proposed unification, etc.) and explicitly stating the procedure used to select the metrics would strengthen the paper.
Yes
No ethics review needed.
1. Are there any technical or conceptual challenges in constructing the underlying uniform framework? Does it reveal any non-trivial or surprising relationships, such as the unexpected categorization of certain metrics within this framework?

2. See other comments in the weaknesses section

3. It would be helpful if the authors could discuss the limitations of their unifying framework, the sanity checks performed, or aspects of the experiments section. Could the authors elaborate on some of the constraints or limitations of their work?
1. A unifying framework for various metrics appears to be beneficial for researchers in this field.

2. The "sanity checks" conducted could provide valuable insights for those studying the topic.

3. The paper is thoughtfully and clearly written.

4. The paper combines both a "formal" aspect that unifies the definitions and practical experiments.
This paper unifies various metrics used for evaluating mechanistic interpretability tasks into a single unified framework. It then introduces a series of "sanity checks" to assess the reliability of these metrics, demonstrating how some existing metrics fail these tests. Through experiments where the "ground truth" of a concept is known, the authors conclude that the "correlation" metric with uniform sampling yields the most accurate results.
1. The primary contribution of this paper is in unifying various metric frameworks within mechanistic interpretability. However, this approach appears relatively "straightforward" and doesn’t seem to involve any technically complex process (the authors may clarify if they disagree). In traditional XAI, for instance, numerous papers propose unifying different notations, yet these often entail either technical or mathematical complexities ("demonstrating that X fits within this category is technically challenging because...") or yield unexpected insights ("it's surprising that X can also be captured under our framework because..."). This framework does not seem to meet those criteria.

2. I'm not sure why the "sanity checks" are regarded as a reliable evaluation framework for the metrics. Why should we trust the assurances provided by these "sanity checks" but not the guarantees offered by the metrics themselves?

3. The choice of the "correlation" metric appears somewhat rigid. Are there any potential issues with using this metric, even though it performed best in the experiments section?
We unify methods for evaluating neuron explanations under one framework and show many popular methods fail basic sanity checks.
@inproceedings{
anonymous2024a,
title={A Principled Evaluation Framework for Neuron Explanations},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=todLTYB1I7},
note={under review}
}
Understanding the function of individual units in a neural network is an important building block for mechanistic interpretability. This is often done by generating a simple text explanation of the behavior of individual neurons or units. However, for these explanations to be useful, we must understand how reliable and truthful they are. In this work we unify many existing explanation evaluation methods under one mathematical framework. This allows us to compare and contrast existing evaluation metrics and understand the evaluation pipeline with increased clarity. We propose two simple sanity checks on the evaluation metrics and show that many commonly used metrics fail these tests and do not change their score after massive changes to the concept labels. Based on our experimental and theoretical results, we propose guidelines that future evaluations should follow and identify good evaluation metrics such as correlation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
interpretability mechanistic interpretability explainable AI trustworthy machine learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fa9e95d48944f4ad26f7815c5d8e0e3a6cb6eda0.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A Principled Evaluation Framework for Neuron Explanations
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission