Exploring the Causal Mechanisms: Towards Robust and Explainable Algorithm Selection
Algorithm Selection;Automated Machine Learning;Robustness;Explainability
Yes
No ethics review needed.
The questions are sufficiently described in the weaknesses section.
* This paper proposes a novel way to treat the algorithm selection problem from causal perspective, and awares us of the bias caused by distribution shift in algorithm selection. It also builds an adequate causal framework to treat the problem under Pearl’s causal framework.

* Experiment results endorse the superior performance of this method in terms of accuracy, robustness, etc. comparing with previous methods in algorithm selection.
This work introduces causality to explore underlying mechanisms in algorithm selection problem. Based on Pearl’s causal framework, it proposes a structural equation model (SEM) based on a causal DAG among problem features and algorithm features. Neural network-based method is proposed to fit the model, through minimizing a mixture of reconstruction, sparsity, acyclicity and 	selection loss. As both demonstrated in texts and experiments, this method is featured by its robustness under distribution shift and explainability towards understanding the mechanism between problem and algorithm features, and it outperforms other methods in most instances especially when constructing dense causal graphs.
Some parts of the paper are hard to catch up. e.g.

* **Causal Learning Structure** In section 2.2, the paper considers incorporating graph information of DAG through designing the first layer of NN as an adjacency matrix. It is argued that it leads to a consistent model. However, there seems to be no theory or references to prove the consistency.

* **Loss function** The loss function is designed to be a weighted sum of four different losses. As these four losses measure completely different aspects, it is suggested to discuss on the weight so as to make them comparable. Besides, if the graph is pre-specified, then what is the use of sparsity and acyclic losses? If it is to be discovered, there should be an illustration on how to construct the DAG to ensure there is only directed flow from problem features to algorithm features.

* **Do-calculus** The notation of do-calculus in section 3.2 needs to be clarified. e.g. in $do(\textbf{PF}=\textbf{PF}+\delta_{\textbf{PF}})$, it should be clarified which PF stands for variable and which stands for specific values.

Overall, the paper is well-motivated by incorporating causal frameworks and methods (Do-calculus, SEM, Causal learning) to deal with algorithm selection. However, it seems more efforts ought to be spent on fulfilling details of this method and explaining its rationality.
Yes
No ethics review needed.
1. The authors claim to find the optimal algorithm. However, the candidate algorithms are only divided into selected (S=1) and not selected (S=0). I want to know whether there is only one selected algorithm. If not, which one is the optimal?

2. In the section of "Demonstration of Explainability", only the feature index is demonstrated in Figure 4. The semantic meaning of them is not known. It limits the explainability of the shown demonstration.
The proposed method achieve robustness to distribution shift, which is an important quality to the application of methods. Empirical results show the effectiveness of CausalAS in different scenarios of distribution shift. The improvement margin is significant. Based on the intermediate product (i.e. causal graph), the CausalAS method provides two kinds of explanations, which is critical to the transparency of the method. The above two properties together contribute the trustworthy application of the method.
This paper investigate the problem of selecting optimal algorithm for particular problem instance. Originally, the algorithm feature is predicted based on the problem feature. The correlation-based machine learning methods can be applied to solve this. However, this kind of methods are vulnerable to data bias and distribution shift. To address these issues and improve the transparency, this paper introduce causal structure learning to explore the underlying mechansim of algorithm selection. The experimental results show that the proposed CausalAS method achieve robustness to distribution shift, and provide explainability through causal graph and counterfactual explanation.
1.I think the novelty of the proposed method is limited. The proposed method is similar to that in [1]. The design of loss function and the given assumptions are similar to the counterpart in [1]. And the problem formulation seems to be directly transformed from recommendation (i.e. problem corresponds to user, algorithm corresponds to item). And the authors did not give citation to this important reference. 


[1] Yue He, Zimu Wang, Peng Cui, Hao Zou, Yafeng Zhang, Qiang Cui, Yong Jiang. CausPref: Causal Preference Learning for Out-of-Distribution Recommendation.
Yes
No ethics review needed.
See Weakness.
1. Robust and explainable algorithm selection is important and of interest.
2. Incorporating causality into this area is apealing.
This paper presents a new method towards robust and explainable algorithmic selection by using of causality. However, both of the clarity and novelty is lack.
1. Main issue 1. Clarity. The overall presentation is poor, as authors cannot highlight their contribution and distinguish their work from existing one. For example, counterfactual explanation is well-motivated, and their causal versions are also very popular in previous work. However, authors claim that "we measure the minimal intervention from the perspectives of explanation complexity and explanation strength,", which is wield, as the framework of minimial intervention for CE has already been established in NeurIPS 2021. 
 - (a) What is the physical meaning of AF and PF? Can you provide more detailed clarification? 
-  (b) What is the core task of this paper? 
-  (c) A more clear version of paper is required. Definition of algorithmic selection, the objective of the task, with surrounding definitions and running examples.

2. Lack of novelty. 
- (a) Why the causality is required? I am not convinced by your illustration. Is it introduced for just dealing with distributional shift? If so, please provide formal characterization on shift. If not, please justify this point in detail.
- (b) Searching DAG in continuous spaces is always a popular approach, and the counterfactual explanation is well-studied. Please justify your contribution. 

3. Minor issues.
 You cannot define conditional distribution as P(AF | PF), as PF it self serves as random vectors. Please use more strict and formal illustration in theoretical analysis.
We introduce causality to describe the underlying mechanisms of the algorithm selection task, and design a robust and explainable algorithm selection model.
@inproceedings{
anonymous2024exploring,
title={Exploring the Causal Mechanisms: Towards Robust and Explainable Algorithm Selection},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7Fh57rIpXT},
note={under review}
}
Algorithm selection aims to identify the optimal performing algorithm before execution. Existing techniques typically focus on the observed correlations between algorithm performance and meta-features. However, little research has explored the underlying mechanisms of algorithm selection, specifically what characteristics an algorithm must possess to effectively tackle problems with certain feature values. This gap not only limits the explainability but also makes existing models vulnerable to data bias and distribution shift. This paper introduces causality to describe this mechanism, proposing a novel modeling paradigm that aligns more closely with the fundamental logic of algorithm selection. By leveraging causal relationships to characterize the algorithm feature distribution conditioned on problem features, our approach enhances robustness against marginal distribution changes and allows for finer-grained predictions through the reconstruction of optimal algorithm features, with the final decision relying on differences between reconstructed and rejected algorithm features. Furthermore, we demonstrate that, the learned causal graph and the proposed counterfactual calculations offer our approach with both model-level and instance-level explainability. Extensive experiments on the ASlib benchmark validate the advantages of the proposed model in terms of robustness and explainability. The code will make publicly available after the review process.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Algorithm Selection Automated Machine Learning Robustness Explainability
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b4e88848cc0b26943a4681e2fd4761680ba97881.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/8197d85c4dee28cf3b84b4280fb2fffd53c3a352.pdf
Exploring the Causal Mechanisms: Towards Robust and Explainable Algorithm Selection
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission