AIM: Adversarial Information Masking for Evaluating EEG-DL Interpretations
Explainable AI;Post-hoc explanation;EEG;Feature attribution method;Saliency map;in-distribution imputation
Yes
No ethics review needed.
What are the footnotes in Table 2 ?

It might be fair to cite https://www.nature.com/articles/s42256-023-00620-w .
It is a reasonable application study about faithfulness evaluation for a particular field - which is an acceptable type of invention. A good set of experiments in three domains, also for multiple networks. They measure also consistency in the sense of rank correlations.
The authors present approaches on how to mask EEG input data in spatial, frequency and temporal domains. The aim of this masking is faithfulness evaluation of attribution maps. The present novel ideas for masking in the temporal domain, also with likely a novelty for the frequency domain. Besides conventional approximate in-distribution masking they also evaluate masking by copying from adversarially crafted samples. The present results for several networks and several attribution methods. They investigate the question whether the sign of attribution maps carries information and investigate an unexpected result in the frequency domain.
In particular for the temporal domain, but also for the frequency domain, trying out different non-adversarial masking methods would make the evaluation more interesting. For the temporal domain that would be different stochastic processes.
Yes
No ethics review needed.
What follows from the evaluation results?
What are the consequences for the practitioner?
The paper addresses a relevant problem, namely evaluation of post-hoc explanations. The paper is original in the sense that it proposes two imputation techniques specifically tailored for multivariate EEG data. The proposals are based on the ROAD and AR frameworks as the  and carefully integrate the spatial, spectral and temporal dimension of multivariate EEG data. The overall originally and quality of the proposed approach is rather limited and specific to models trained for EEG analysis. It is unclear how to generalise the approach beyond this specific application domain.
The paper is well written and easy to understand. The experimental evaluation is ok, but could be more detailed and deep. Currently it is not clear how follows, e.g., from the results in Table 2 or 3.
Overall, the contribution is rather incremental and will probably be of interest / significance only to a limited (EEG) community.
The paper addresses the problem of evaluation of post-hoc explanations in the context of models trained on EEG decoding tasks. 
In particular, it focuses on the evaluation of faithfulness of explanations and proposes a novel framework involving multi-domain adversarial information masking (AIM) based on Multi-Domain Adversarial Robustness (mdAR), which overcomes some of the limitations of standard faithfulness evaluation approaches. The framework is validated on multiple model architectures and EEG datasets.
The contributions of the paper are very specific and may be of interest to a limited community, mainly only researchers training and explaining NN for EEG analysis. There has been a lot of research on faithfulness evaluation of explanations. The proposed method represents an incremental contribution to this field. The experimental evaluation is not 100% convincing. It is unclear to me what follows from the evaluation results. Shall we only use some of the methods which perform well in Table 2 for the analysis of EEG-based explanations? Are the results consistent with other evaluation approaches? What are the consequences of the evaluation for the practioner?. 
Currently, the paper reads to me as proposing yet another faithfulness evolution metric, here specifically for EEG analysis tasks. The overall originality of the contribution and relevance for the ICLR research community is rather limited. Therefore I recommend "reject".
Yes
No ethics review needed.
- Improve clarity and style of writing
- Be more concise in your contributions 
- Evaluate your framework with respect to other frameworks
- Give more context regarding the EEG experiments and show more example attributions
-	Addresses a real gap in the EEG/DL field
-	Seems to generate useful findings
-	Three EEG domains are considered
The manuscript aims to contribute to the evaluation of interpretations of deep learning models applied to EEG. It does so by identifying issues with traditional adversarial robustness evaluation for EEG and proposing alternative information masking methods to evaluate the faithfulness of feature attribution methods. Their framework is able to differentiate between attribution methods on spatial, temporal, and spectral domains and thereby seem to generate useful findings for the field. Such progress is valuable as explainability of neural networks in EEG data is not well-studied.
1.Writing: Unfortunately, the level of the writing of the manuscript is poor. Especially the first half of the paper, outlining the motivations, prior literature, and outline of the paper, is difficult to follow. The paper could really benefit from another thorough round of editing as the many grammatical errors lead to semantic ambiguity. A few examples that I am unable to understand:
a. L218-219: ‘ability to exert data distribution’
b. L220-221:‘computationally exhaustive while remain biased or uncontrollable’.
c. L271-273: "(…), whose value are concluded to reflect certain series trend.”
Also the experiments are hard to follow and it is difficult to assess the contributions of this work. 

2. Evaluation: Is it possible to perform some form of cross-framework comparison? It is difficult to understand the advantage of the proposed framework over existing ones. For example, why did the authors choose not to analyze where and why frameworks agree or disagree? Would synthetic data enable a comparison between frameworks? Understanding both the advantages and disadvantages of this new framework would be very valuable.

Minor:
-	It might be useful to briefly describe what the the proposed metrics (AOC, ABC) conceptionally mean and how they differ.
-	Recent work on EEG-DL concerns the use of large amounts of resting state data for clinical predictions (e.g. [1-3]). Do the authors believe the proposed framework could be relevant for such work as well? Understandably, task-based explanations are easier to verify and interpret. However, the datasets used by the authors are small, while it may be argued that deep learning models may be particularly interesting in case of larger datasets, which to my knowledge tend to be resting-state.

References:
1.	https://openreview.net/forum?id=QzTpTRVtrP
2.	https://arxiv.org/abs/2305.10351
3.	https://arxiv.org/abs/2409.07480
A robust framework for quantifying the faithfulness of post-hoc explanations in EEG deep learning models, employing adversarial information masking for effective in-distribution imputation.
@inproceedings{
anonymous2024aim,
title={{AIM}: Adversarial Information Masking for Evaluating {EEG}-{DL} Interpretations},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=B5i88Tj1nk},
note={under review}
}
The analysis of post-hoc explanations for deep learning models has revealed significant benefits in enhancing transparency and advancing explainable artificial intelligence (XAI). Despite this progress, a critical gap remains in the reliable evaluation of the faithfulness of explanations, particularly for data that encompasses multi-domain information. To address the challenges of distribution shifts and information leakage present in existing methods, we propose a novel framework involving multi-domain adversarial information masking (AIM) based on Multi-Domain Adversarial Robustness (mdAR), enabling in-distribution imputation and facilitating a reliable assessment of faithfulness for feature attribution methods. Extensive experiments validate our approach across various model architectures and multi-channel electroencephalogram (EEG) datasets. The code and sample data for this work are available at https://anonymous.4open.science/r/EEG-explanation-faithfulness-5C05.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Explainable AI Post-hoc explanation EEG Feature attribution method Saliency map in-distribution imputation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4b08822f11fb0dc8feaff2f89e657f037a17f727.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
AIM: Adversarial Information Masking for Evaluating EEG-DL Interpretations
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission