Benchmarking XAI Explanations with Human-Aligned Evaluations
Explainable artificial intelligence (XAI);Dataset;Benchmark
Yes
No ethics review needed.
1. On page 18, in Appendix A1.1, it’s stated that "the task we focus on is indoor scene labeling," but this was not mentioned in the main paper. Could you please clarify this?

2. The authors mention a low correlation between widely used XAI methods and human assessments. Does this imply that existing metrics do not accurately align with human evaluations? If so, how do the authors conclude that human and metric-based assessments cover complementary aspects? What complementary information is provided by the existing metric-based approaches that human-subject-based evaluation does not capture?

Also, See the weaknesses mentioned above.
1. The paper addresses an important problem of benchmarking XAI methods in a comprehensive and efficient way, aligning evaluations with human assessments, which I believe is an important gap in the literature. 

2. The proposed benchmark allows for the comparison of XAI methods across different modalities, facilitating evaluations of both explanation-based and saliency-based methods.

3. The paper introduce a data-driven metric to mimic human assessments in evaluating the interpretability of explanations.

4. The authors created six carefully designed questions and a comprehensive set of criteria to assess various desired properties while collecting human labels.

5. Different XAI methods are evaluated against human assessments to benchmark the quality of their explanations. Additionally, human scores are compared with different XAI metrics. 

6. The authors have indicated their willingness to open-source their code, annotations, and models.
The paper introduces PASTA, a perceptual assessment system designed to benchmark explainable AI (XAI) techniques in a human-centric manner. The authors first integrate annotated images from the COCO, Pascal Parts, Cats Dogs Cars, and Monumai datasets to create a benchmark dataset for XAI, which is also used to train classifier models for generating explanations. For the final assessment of XAI methods, the authors curate a PASTA evaluation benchmark comprising 100 images. They apply 46 distinct combinations of 21 different XAI techniques on these 100 images, creating a dataset of 4,600 instances. Each instance includes an image, its ground truth label, a classifier’s predicted label, and the explanation generated by a specific XAI technique.

Additionally, to compare various XAI methods, including saliency-based and explanation-based approaches, the authors develop an automated evaluation metric that models human preferences based on a database of human evaluations. This metric enables robust evaluations across modalities. The experiments, conducted on 21 different XAI techniques and datasets, demonstrate that saliency-based explanation techniques such as LIME and SHAP align more closely with human intuition than explanation-based methods.
1. How many human subjects were involved in the benchmark creation? Is it five? This information is not explicitly stated in the main paper. Since the benchmark aims to align with human evaluations, it would be valuable to provide details about the annotators, including their ethnicity, gender, age, and other relevant demographics. Ideally, involving annotators from diverse backgrounds—such as varying ages, genders, and ethnicities—would help reduce potential biases in the benchmark.

2. The evaluation benchmark includes only 100 images, which may not be statistically sufficient for conclusive insights into model behavior across different XAI techniques.

3. There appears to be a possibility that some images could overlap between training and test sets. While the authors demonstrated generalization by separating these sets explicitly in Section 4.4, it is not clear to me  why they did not consistently apply such strict separation across all experiments. 

4. The method utilizes existing object grounding techniques, such as Grounding DINO, to generate bounding boxes for the “Cats Dogs Cars” dataset, which might lead to suboptimal performance. It is also unclear if the authors conducted any ablation studies on different grounding or object detection methods before selecting Grounding DINO. Was there a particular reason for choosing this specific method? 

5. Many essential information related to the proposed approach are not included in the main paper. For example, the complete list of perturbations need to be included in the main paper, not on the appendix.
Yes
No ethics review needed.
Please check the limitations and kindly answer the correlation of the design of XAI methods and a deep learning architecture for the evaluation of the XAI methods. Another question that needs to be highlighted are the use of CLIP in the pipeline as it will be dependent on the embedded representations on the CLIP space.
1. The paper tries to answer an important question about the evaluation of the XAI methods. 
2. The overall evaluation protocol and the research questions designed based on the fidelity, complexity, objectivity and robustness are presented in a right manner.
This paper introduces a framework for human centric method of the explainable AI techniques (XAI). This paper performs evaluation on four datasets: COCO, Pascal Parts, Cats Dogs Cars and Monum AI.
1. First and foremost to perform any sort of evaluation the authors should consider the methodology used and the architecture used specially in gradient based methods. eg: GradCAM does not work well with transformers, rather it is a methodology that works significantly better on CNNs due to the architectural composition. Chefer et. al. Transformer interpretability beyond attention visualization.
2. One of the major limitation of this work is use of another network suck as CLIP as explanations evaluator, this limits the model to the limitations of the CLIP. eg: CLIP embedding space limits it for the fine grained understanding in the joint embedding space.
3. The paper is more suitable for a user-based study conference rather than ICLR.
Yes
No ethics review needed.
(1) Please justify the difference between the proposed study and previous user studies on XAI.

(2) How can the PASTA model scale to broader domains, and help model development?

(3) Please consider reorganizing the paper, and including more in-depth analyses of the main paper

(4) It would be reasonable to perform a more comprehensive study on the automatic rating part.
(1) This paper tackles an important challenge in XAI: the alignment between model explanations and human evaluations.

(2) It carries out human study with a broad range of explanation methods and multiple datasets, which can facilitate the development of future XAI methods.

(3) With an automatic model for estimating human ratings, the study can be extended to help improve the trustworthiness of deep networks.

(4) Extensive experiments are provided in the supplementary materials, providing the opportunity for more in-depth analyses.
This paper conducts a user study on the comparative effectiveness of different explainable AI (XAI) methods, and proposes an automatic XAI evaluation model. In particular, it asks human participants to rate different aspects of the explanations given carefully designed questions, and train a model that takes in the explanations (either text or saliency maps) to predict the ratings. Experimental results reveal the advantages of visual explanations over concept ones, and show the feasibility of predicting human ratings based on model explanations.
(1) The paper emphasizes the limitations of existing studies in scaling to broader domains, due to difficulties of data collection and the subjectivity of human evaluation. Nevertheless, these problems are also well addressed in the proposed method. Specifically, the paper centers around a user study for rating model explanations, which also requires significant amounts of manual labor and does not alleviate annotator biases. While a quantitative evaluation model is proposed, with only the results in Table 3 (comparison to the baseline without explanations and inter-subject agreement), it is unclear how it can help evaluate XAI approaches in different domains. I would suggest involving the PASTA model for training other models, and exploring its effectiveness in enhancing the trustworthiness of deep networks.

(2) Despite the wide coverage of the user study, the analyses in the main paper are relatively shallow and fall short of providing important insights. In particular, while the paper highlights comparing over 10 classifiers and XAI methods, its conclusions focus on only two categories of XAI methods and three backbones. Table 1 indicates close-to-zero correlations between human ratings and the XAI methods, but without detailed explanations. It is reasonable to move certain results from the supplementary materials and perform more in-depth analyses following previous studies (e.g., [ref1, ref2]).

(3) The proposed PASTA model essentially feeds the visual or textual explanations to CLIP encoders to derive the human ratings. Such a design can have several limitations: First, CLIP is heavily tuned toward semantics and can have its own biases. It would be good to test with various designs, e.g., with recent approaches of projecting multi-modal data to a pretrained LLM space (e.g., [ref3]). Second, the model seems to only utilize explanations (or explanations on top of images) as inputs. While this is okay with simple visual stimuli that have a dominant object for classification, it may not generalize to other applications that demand complex reasoning with a rich set of visual elements (e.g., visual reasoning). It can be useful to consider at least including the ground truth answers as inputs. Third, relating to Table 3, I would expect comparisons with various methods, as the problem itself is just a regression. 

References:

[ref1] Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations. IEEE TPAMI, 2024.

[ref2] What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. NeurIPS, 2022.

[ref3] Visual Instruction Tuning. NeurIPS, 2023.
Yes
No ethics review needed.
Please see the points under Weaknesses above.
1. The paper conducts a comprehensive investigation of XAI methods, providing a convincing and promising set of evaluation criteria and protocols that offer a strong foundation for future research.

2. Experiments demonstrate that PASTA achieves human-level or better performance under the proposed new protocol.
This paper proposes backbone for XAI, introducing a human evaluation protocol. The authors construct a benchmark dataset consisting of triplets of images, explanations, and labels, allowing for quantitative evaluation of XAI methods. Additionally, the paper consolidates different evaluation criteria and presents a question-based protocol for data annotation.
1. The process of generating computed explanations and annotating concepts is unclear. The PASTA-metric’s applicability to unseen datasets or new XAI methods appears limited, as the generalization to other contexts or modalities is not fully validated. The four datasets used may not represent the full range of XAI applications, potentially limiting the framework’s relevance for domains beyond standard object recognition. From Table 5 in the supplementary material, it seems that concepts are defined by class names in the datasets. Would these pre-defined concepts limit the model’s generalization? If concepts are simply classes, it may keep the model as a “black-box”. So how to define the explainability?

2. Including more figures and equations to illustrate the metrics would be beneficial. For instance, how are scores for faithfulness calculated? This additional detail would enhance clarity.

3. Participant Background: The study included 15 participants—are they experts in this field? Is this number sufficient for robust annotation?

4. The main sections are somewhat unclear, and parts seem to serve as an overview of the supplementary material, which makes the paper challenging to follow. I recommend the authors provide a clearer narrative or at least one example before referring to supplementary sections.

5. There are minor typos, such as “tree losses” in Line 1563, which should be “three losses,” and issues with brackets in Fig. 8.

6. I also recommend comparing PASTA with traditional protocols or providing a visualization to demonstrate the effectiveness of the backbones.
@inproceedings{
anonymous2024benchmarking,
title={Benchmarking {XAI} Explanations with Human-Aligned Evaluations},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6KZ80APcxf},
note={under review}
}
In this paper, we introduce PASTA (Perceptual Assessment System for explanaTion of Artificial intelligence), a novel framework for a human-centric evaluation of XAI techniques in computer vision.
Our first key contribution is a human evaluation of XAI explanations on four diverse datasets—COCO, Pascal Parts, Cats Dogs Cars, and MonumAI—which constitutes the first large-scale benchmark dataset for XAI, with annotations at both the image and concept levels. This dataset allows for robust evaluation and comparison across various XAI methods. Our second major contribution is a data-based metric for assessing the interpretability of explanations. It mimics human preferences, based on a database of human evaluations of explanations in the PASTA-dataset. With its dataset and metric, the PASTA framework provides consistent and reliable comparisons between XAI techniques, in a way that is scalable but still aligned with human evaluations. Additionally, our benchmark allows for comparisons between explanations across different modalities, an aspect previously unaddressed. Our findings indicate that humans tend to prefer saliency maps over other explanation types. Moreover, we provide evidence that human assessments show a low correlation with existing XAI metrics that are numerically simulated by probing the model.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Explainable artificial intelligence (XAI) Dataset Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/424f94dbc2401cd890595901b8bf79363e540615.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Benchmarking XAI Explanations with Human-Aligned Evaluations
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission