Factor Graph-based Interpretable Neural Networks
interpretable neural network;factor graph;perturbation;explanation rectification;graph learning
Yes
No ethics review needed.
- Defining the logic rule set seems expensive. Would it be difficult to construct a factor graph by connecting concepts and categories with a bipartite complete graph and estimating the weights $w_i$?
- How do you distinguish between coexistence and exclusion in the factor graph?
- It would be better to describe the specific estimation algorithm of $w_i$ in the Appendix.
- Line 336-337: What is "attributional training"?
- The proposed method enables both the detection of logical errors and the correction of these errors within a single framework.
- Compared to other concept-based interpretable neural networks, the proposed method achieves higher comprehensiveness in explanations, regardless of whether the perturbations are known or unknown.
The paper proposed a method using factor graphs to correct errors in concept-based explanations caused by perturbations. 
Specifically, it constructs a factor graph using predefined logical rules between concepts or between concepts and categories. 
This graph helps identify logical errors in the output from concept-based interpretable neural networks by evaluating the likelihood of these errors. 
Additionally, by leveraging the factor graph, it is possible to correct these logical errors in the explanations. 
Experimental comparisons on three datasets demonstrate that the proposed method outperforms existing concept-based approaches.
- While the proposed method assumes that explanations change due to perturbations without affecting predictions, this seems unrealistic. Particularly in interpretable neural networks with a concept bottleneck structure, as assumed in this study, changes in the concepts outputted by the neural network would naturally lead to changes in predictions, which undermines this assumption.
- The proposed method requires predefined logic rules between concepts and categories. If these rules are already known, wouldn’t it be possible to detect inconsistencies between concepts and predictions without the need for the factor graph? The advantage of using a factor graph is unclear.
- As noted in the minor comments below, there is room for improvement in the writing.

Minor comments:
- The explanation of Figure 3 is insufficient.
- There is no reference to Figure 4.
Yes
No ethics review needed.
- Can the authors provide a comparison against one of the methods injecting knowledge (prior or learnt)?
- Scalability: The two limitations that have been reported (domain knowledge changes, correct prediction categories) are non-negligible. How could this method be extended to face these limitations?
- **Good presentation and structure** The paper is well structured and easy to read. The mathematical definition of the proposed method is clear and well-defined. 
- **Nice experimental campaign**: Extensive experiments on three datasets (CUB, MIMIC-III EWS, and Synthetic-MNIST) demonstrate the superior performance of AGAIN over the compared baselines.
The paper introduces AGAIN (fActor GrAph-based Interpretable Neural Network), which generates comprehensible explanations for neural network decisions even under unknown perturbations. Unlike traditional adversarial training methods, AGAIN incorporates logical rules through a factor graph to identify and rectify explanatory logical errors during inference. The model is validated on three datasets: CUB, MIMIC-III EWS, and Synthetic-MNIST, demonstrating superior performance compared to existing baselines
## Major Issues

- **Novelty, Related work and Compared methods**: the main issue with the current paper is that it only considers methods injecting knowledge into the models by means of factor graphs. However, the integration of knowledge into model predictions has been fully studied under different point of views: probabilistic (e.g., DeepProblog [1] and all its variants), logic constraints (see the survey of Giunchiglia et al. [2]). Also, it is not the first method defending against adversarial attack with knowledge and without employing adversarial training. Some of these methods have been already employed to defend against adversarial attacks, such as [3-4]. [5] is a survey entirely dedicated to enhancing interpretability and adversarial robustness with prior knowledge. [6] has shown in the context of concept-based models that it can learn logic rules at training time and use them at inference time to defend against adversarial attacks. This is also reflected in the experimental campaign that is extensive but does not consider any methods injecting prior knowledge to defend against adversarial attacks. The only compared methods are CBMs or adversarial-trained versions of the same models. 


- **Paper positioning and Preliminaries**: the method provides explanations and a defence mechanism that is based on concept predictions; thus, it applies only to concept-based models. Most of the compared methods also belongs to this niche. While this is fully acceptable, explicit references to concept-based models only appears in Section 3-4. Therefore, I think it should state earlier that this is the focus of the paper, as most of the related work mentioned in the paper does not focus on concept-based explanations. Furthermore, there is no explicit mentions to concept-based models in the preliminaries. The “Interpretable Neural Network” paragraph should include citations to this literature and explain concept-based models.

## Minor Issues
- P.2 “[…] even if the adversarial samples are available, retraining is only effective for at most one perturbation type” I think this statement is quite strong, and although it may have been proved for some attacks in Tan & Tian, 2023, I don’t think it is a general assumption that we can make. I think this sentence should be soften to “retraining is effective only for few perturbation types at the same time”. 
- P.2 “[…] to ensure the expectation of the potential function is in its maximum value”. It is not clear at this point what is the potential function. Consider rephrasing this sentence.
- P.2 “The explanations that are further regenerated align with the exogenous logic.” Not clear in this case what are the exogenous factors. 
- P.2-3: The term "defenses against comprehensibility" seems a bit misleading. It implies that the goal is to prevent explanations from being understandable, which is not the case. Instead, the focus is on defending the comprehensibility of explanations against perturbations. “defenses of comprehensibility” colud be a more appropriate definition.


[1] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, Luc De Raedt: DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS 2018: 3753-3763

[2] Giunchiglia, E., Stoian, M. C., & Lukasiewicz, T. (7 2022). Deep Learning with Logical Constraints. In L. D. Raedt (Ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 (pp. 5478–5485). doi:10.24963/ijcai.2022/767

[3] Yin, M., Li, S., Cai, Z., Song, C., Asif, M. S., Roy-Chowdhury, A. K., & Krishnamurthy, S. V. (2021). Exploiting multi-object relationships for detecting adversarial attacks in complex scenes. In proceedings of the IEEE/CVF international conference on computer vision (pp. 7858-7867).

[4] Melacci, Stefano, et al. "Domain knowledge alleviates adversarial attacks in multi-label classifiers." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.12 (2021): 9944-9959.

[5] Mumuni, Fuseini, and Alhassan Mumuni. "Improving deep learning with prior knowledge and cognitive models: A survey on enhancing interpretability, adversarial robustness and zero-shot learning." Cognitive Systems Research (2023): 101188.

[6] Ciravegna, Gabriele, et al. "Logic explained networks." Artificial Intelligence 314 (2023): 103822.
Yes
No ethics review needed.
How is the reasoning conducted on G? How is the probability estimation implemented in Equation 1? Please provide more details.
1. The paper proposes a novel and interesting idea that uses the logical correctness of explanation to detect and defense against noises and adversaries.
2. The three stage framework is reasonable to me. 
3. The examples used in the paper are intuitive.
This paper proposes AGAIN (fActor GrAph-based Interpretable Neural Network), a new approach to maintain comprehensible neural network explanations when inputs are affected by unknown perturbations. AGAIN builds factor graphs from explanations, and integrates logical rules through the graphs. The system detects adversaries by identifying violations of real-world logic, and uses an interactive intervention strategy to fix incorrect explanations. Tested on three datasets (CUB, MIMIC-III EWS, and Synthetic-MNIST), AGAIN outperformed existing methods in generating comprehensible explanations under both known and unknown perturbations.
1. The concept bottleneck model is not easy to scale, as it requires manual annotations of concepts.
2. The implementation details of Section 4.2 is not very clear (the introduction is too conceptual). For example, is $\mathcal{G}$ a graph or a model? At the beginning I thought $\mathcal{G}$ is a graph, but in Line 221 it "reasons about a conditional probability".
3. The datasets used in this work are not very strong. I doubt if the work is applicable to real-world situations. At least, the datasets used cannot reflect adversarial scenarios in practice.
Yes
No ethics review needed.
1. The authors employed black-box perturbations to test robustness. Adding input noise could provide more convincing evidence.
2. I am concerned about the potential impact of large factor graphs on computational efficiency. I would like to see an analysis on this problem.
3. Currently, the algorithm explores all intervention options to select the optimal one. Is it possible to employ a simplified strategy, such as a heuristic or greedy algorithm, to reduce computational load?
1. The paper presents an innovative method, AGAIN, that combines factor graphs with concept-level explanations to improve model interpretability under unknown perturbations.
2. The authors evaluate AGAIN across multiple datasets and baseline models, providing a broad view of its effectiveness.
3. The paper is clear and well-structured.
This paper proposes AGAIN, a neural network model that generates comprehensible explanations under unknown perturbations by integrating logical rules directly during inference, rather than relying on adversarial training. Using a factor graph to identify and rectify logical inconsistencies, AGAIN demonstrates superior interpretability and robustness compared to existing methods.
1. The factor graph requires predefined logical rules, which could be challenging to construct or generalize across different domains.
2. The algorithm’s process of exploring all possible intervention options to find the optimal solution could create computational overhead.
Yes
No ethics review needed.
See Weaknesses
The paper offers a well-motivated and clear presentation of a unique approach. The use of a factor graph to handle perturbations is innovative and highly relevant to the research community. Furthermore, the authors have provided sufficient theoretical support and empirical evidence to justify the effectiveness of their approach.
In this paper, the authors propose a novel method called AGAIN to generate comprehensible explanations under unknown perturbations by employing a factor graph approach. The method presents a significant contribution to the field, addressing an important gap with a new perspective. The paper is well-structured, and the authors have provided thorough theoretical justifications and experimental analyses. These analyses effectively demonstrate the superiority of the proposed method over existing ones.
1. Clarification on Figure 3:
   In Figure 3, does the final output include hat{c}_{re} and prediction? Does the hat{c}_{re} mean the interpretation? Additionally, the figure lacks clarity on how hc is generated, which is crucial for understanding the method. Adding details on hc generation would make the figure more comprehensive.
 
2. Comparison Metrics:
   The authors compare their method with all baselines using the LSM metric. However, they only compare their method with CBM in terms of accuracy (P-ACC and E-ACC). It would be beneficial to extend the accuracy comparison to include all baselines to provide a more complete evaluation of the method’s performance.
 
3. Figure 7 Interpretation and Readability:
   In Figure 7, for each example, the authors provide two bar charts. Does the left bar chart represent the initial interpretation, and the right bar chart represent the interpretation combined with the factor graph? Clarification on this aspect would enhance the understanding of the figure. Additionally, some symbols and text in Figure 7 are overlapping.
 
4. Inconsistency in Line 463 and Table 5:
   The authors mention in line 463 that their method is compared with two baselines on the Synthetic-MNIST dataset. However, Table 5 lists four baselines. Furthermore, "ProbCBM" in Table 5 should be corrected to "ProCBM" for consistency. It is recommended that the authors proofread the paper to eliminate such inconsistencies and typographical errors.
@inproceedings{
anonymous2024factor,
title={Factor Graph-based Interpretable Neural Networks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=10DtLPsdro},
note={under review}
}
Comprehensible neural network explanations are foundations for a better understanding of decisions, especially when the input data are infused with malicious perturbations. Existing solutions generally mitigate the impact of perturbations through adversarial training, yet they fail to generate comprehensible explanations under unknown perturbations. To address this challenge, we propose AGAIN, a fActor GrAph-based Interpretable neural Network, which is capable of generating comprehensible explanations under unknown perturbations. Instead of retraining like previous solutions, the proposed AGAIN directly integrates logical rules by which logical errors in explanations are identified and rectified during inference. Specifically, we construct the factor graph to express logical rules between explanations and categories. By treating logical rules as exogenous knowledge, AGAIN can identify incomprehensible explanations that violate real-world logic. Furthermore, we propose an interactive intervention switch strategy rectifying explanations based on the logical guidance from the factor graph without learning perturbations, which overcomes the inherent limitation of adversarial training-based methods in defending only against known perturbations. Additionally, we theoretically demonstrate the effectiveness of employing factor graph by proving that the comprehensibility of explanations is strongly correlated with factor graph. Extensive experiments are conducted on three datasets and experimental results illustrate the superior performance of AGAIN compared to state-of-the-art baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
interpretable neural network factor graph perturbation explanation rectification graph learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e27cefe47e843cdc9ae47f185ccbf3238aac30a7.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Factor Graph-based Interpretable Neural Networks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission