A Meta-Learning Approach to Bayesian Causal Discovery
neural processes;bayesian causal discovery;transformers
Yes
No ethics review needed.
1. Is it nontrivial to consider the setting where latent confounders between variables exist? 

2. Is the proposed method computationally efficient when a function class is chosen as neural networks? 

3. How sensitive the proposed framework is regarding to the sparcity of the graph? 

4. Are there any interesting example where the Bayesian approach beats the Markov-equivalence-based method?
1. This paper provides a well-structured and self-contained summary of prior works in Bayesian causal discovery, allowing readers to clearly follow the evolution of Bayesian approaches for learning causal graphs. As a reviewer, this summary enables me to track the advancements in the field and understand the recent progress in addressing challenges like scalability and uncertainty in causal inference.

2. The paper clearly outlines its unique contributions in Table 1, highlighting significant advancements over existing methods. The proposed model addresses critical aspects such as acyclicity, permutation invariance, and edge dependencies—all of which are critical for accurate causal graph learning. These improvements over prior approaches directly enhance the reliability and precision of inferred causal structures, making the contributions both clear and impactful.

3. The proposed BCNP is particularly attractive due to its innovative use of Neural Processes to directly sample the posterior distribution $ P(G | X) $. This approach enables BCNP to achieve scalability and uncertainty-aware inference by learning across tasks (= datasets), allowing the model to efficiently adapt to new datasets while maintaining a robust representation of causal uncertainty. This combination of scalability, direct posterior sampling, and adaptability positions BCNP as a strong advancement over existing Bayesian causal discovery methods.
This paper proposes the Bayesian Causal Neural Process (BCNP), a Neural Process-based framework for learning a posterior distribution over causal graphs given a dataset $P(G \mid X)$ . The proposed method is scalable and captures distinctions between graphs within the same Markov equivalence class. Since BCNP learns causal structures across multiple (synthetic) datasets, it is considered a meta-learning approach.
1. One of the main limitations of this paper is the assumption that there are no latent confounders between variables. This assumption may limit the applicability of the method to real-world datasets where unobserved confounders are common. 

2. I think this method is heavily relying on the assumptions on the choice of modeling class of functions F, the types of graphs and the noise model. Unlike fully nonparametric approaches like the PC algorithm, which do not impose strict functional or noise assumptions, BCNP’s effectiveness depends on these choices being well-suited to the data.

3. The proposed framework may be sensitive to hyperparameter settings and architecture choices, such as the hyperparameters for the encoder-decoder network, or the types of prior distributions. Fine-tuning these parameters is often data-dependent, which could reduce the model's robustness and generalization capabiltiy.
Yes
No ethics review needed.
- For DiBS and BayesDAG, the authors mention that they “keep all other hyperparameters to their preset values.” What if these hyperparameters were tuned? Would the results differ significantly? For instance, DiBS and BayesDAG perform well on the Syntren dataset for certain metrics, so tuning may further widen this performance gap.
- In the caption, it says, “Each dataset contains D nodes and N samples.” Does this imply that all datasets must be of the same size? What if the sample sizes vary, such as 100, 200, or 500? Should N be set to 100, or can larger datasets (e.g., 200 and 500 samples) be split into multiple parts (e.g., 2 or 5 parts) to train the model? In that case, should there be weighting adjustments (1/2, 1/5), also training perhaps only for data-related parameters and not for those related to the graph prior?
- If there is no training involved, does your model function like a standard Bayesian causal discovery method with a uniform prior?
- Each variable might have distinct functional priors, such as some being linear, others non-linear, discrete, or having different means. Does the encoder “implicitly” reorder these variables (perhaps using attention mechanisms) to align them by their functional type? I’m trying to conceptually grasp how the encoder handles such diverse functional characteristics under the hood.
- What if permutation equivariance among nodes was not enforced? Would this potentially improve performance, especially if nodes are ordered consistently between training and test data?
- In Line 382, could you clarify what is meant by “correlations between edges” when dealing with just two variables, X and Y, with at most a single edge?

Minor comments
- It would be better to have some annotations in Figure 2 (given that the figure takes lots of space)
- Not quite sure whether it is sufficient to say the dependence in permutation and lower triangular binary matrices is modeled because they share the same representation R0.
- 047, 137 citep for Wenzel et al. 
- 115 citing PC, GES might be good (given you have some space for 119)
- 141 Bayesian metal → meta
- 305 Is the grammar correct? (The matrix … and thus … )
- 340 space before Mena
- 389 In section section 4.2
- 389 its 
- 414 space after comma
- 480 enable
- 485 period
- BayesDAG is not highlighted in Table 4 for Linear data/SHD
- Figure 2 shouldn’t o be the input to Theta? or onto the edge between R^L1 to Theta?
- Sorry for suggesting my preference here but can we align Theta and Phi (parameters, vertically) and align Qs and As with the same horizontal line as Theta and Phi, respectively, and put Gs at far right.

(I can change to weak accept (or accept) depending on the response and others' reviews.)
- As summarized in Table 1, this approach implements several key desiderata for a Bayesian meta-learning model more effectively than existing alternatives.
- The paper is well-written, allowing readers to easily follow the motivation and desiderata of Bayesian meta-learning models.
This paper introduces a Bayesian meta-learning model for causal discovery, named the Bayesian Causal Neural Process. This "meta-learning" algorithm learns a model from pairs of datasets and their associated causal graphs. Specifically, the model incorporates a neural process to address model uncertainty and employs an encoder-decoder architecture carefully designed to achieve desirable properties for causal discovery in a meta-learning context. These properties include permutation invariance to sample order, permutation equivariance to nodes, and acyclicity, achieved by decomposing the adjacency matrix into a lower triangular matrix and a permutation matrix—some of which build on existing approaches. The authors demonstrate the empirical superiority of this model over other explicit Bayesian and Bayesian meta-learning models
- While I appreciate the overall architecture of the model, it is challenging to identify which components provide truly novel technical contributions. For instance, the results on pages 4 and 5 appear to largely rely on existing findings, and, if I'm not mistaken, page 6 includes results from Annadani et al. (2024). The contributions of this paper are not clearly or explicitly articulated.
- The Bayesian prior is learned through pairs of datasets and directed acyclic graphs (DAGs). From my perspective, learning P(X,G) is likely very challenging. Indeed, Appendix B.1 notes that, for a case involving two variables, the model was trained on 200,000 datasets. I understand that Section 4.1 is intended to evaluate the posterior, but in Appendix B.2, 500,000 datasets are used for training. This information would be better suited for the main text, and details on computational resources should be transparently reported there as well (they are not provided even in the appendix).
- I find it difficult to be fully convinced by the meta-learning approach to causal discovery. While I understand the importance of establishing a strong Bayesian prior, can we realistically expect access to hundreds of thousands of dataset-graph pairs?
Yes
No ethics review needed.
1) The ideas that the paper uses to impose acyclicity and permutation invariance seem applicable in general. Can existing meta-learning approaches be modified to get the same? While this is not the focus of the paper. It would make it stronger to highlight it if that's the case.

2) The experimental validation for the case when the true posterior is known, is done only for the two-variable case. Is it possible to scale this up? 

3) How can the posterior over the graphs be used for inference on that of the functional parameters? 

4) How is the performance on Gene Regulatory Network simulated data like SERGIO? I believe it would be useful to add that in given its importance. 

5) The paper says that the metrics average over the posterior. I believe this is a problem with any metric. Are there alternatives that the authors propose?
The paper proposes a technique that can directly sample from the posterior over graphs. The main contribution of the paper is in the architecture of the encoder-decoder network that captures necessary properties like the permutation invariance by cross-attention, and ayclic DAGs and edge dependencies by sampling the decomposition of a DAG into permutation and lower-triangular matrices. The comparison of the proposed model is done with existing meta-learning approaches and Bayesian approaches using multiple metrics. The proposed model is shown to be competitive on synthetic and simulated datasets. The paper is written clearly and flows well.
In Bayesian causal discovery, the goal is to approximate the posterior over graphs. Since computing the posterior over graphs requires computing posterior over functional parameters, previous approaches have tried to approximate the posterior over function parameters.  This paper uses neural processes to approximate the posterior over graphs directly by learning a map from the dataset to the set of distributions. The neural process is an encoder-decoder network that claims to sample, permutation-invariant, acyclic graphs that capture edge dependencies.
My main concern is that of this being an incremental contribution. The framework is not new and the only contributions I see are a change in how permutation invariance and acyclicity is incorporated. Please let me know if I am missing something else. I have a detailed list of questions below whose answers might help strengthen the paper.
Yes
No ethics review needed.
1. As my main concerns, I would appreciate the authors to elaborate on the abovementioned weaknesses.
2. Given that the proposed method does not utilize interventional data, using CPDAG E-SHD (as used in previous papers such as in BayesDAG) might be more reflective of the performance of the models compared with E-SHD.
3. Have the authors tried any experiments on low data regime where the number of samples is close to the number of nodes? This is particularly important since Bayesian Causal Discovery is most desirable in low data regime where $n \approx d$. Do you have an intuition about how the proposed model performs in these particular settings?
4. In several experiments, other baselines such as BayesDAG and DiBS achieve a better E-SHD compared to the proposed method. Could you elaborate on why this happens?
* The paper is well-written and clear and includes a range of well-designed experiments.
The paper introduces the Bayesian Causal Neural Process (BCNP), a novel approach for Bayesian causal discovery that leverages meta-learning to address challenges in traditional methods. BCNP learns a mapping from datasets to posterior distributions over causal graphs, thereby bypassing the need to explicitly infer the posterior over causal mechanisms and allowing for efficient sampling from high-dimensional spaces. By encoding key properties like permutation equivariance and edge dependencies, BCNP accurately approximates the true posterior and generates acyclic graph samples. The paper demonstrates BCNP's effectiveness through experiments on synthetic and semi-synthetic datasets, showcasing its superior performance compared to existing Bayesian and meta-learning models for causal discovery.
For me, there are two major concerns regarding the novelty and contributions of the proposed method to the community:

1. The paper presents the direct mapping from data to a posterior over graphs ($P(G|D)$) as a main advantage, bypassing the need to model causal mechanisms. However, learning a distribution over causal mechanisms is crucial for many applications, such as causal inference and treatment effect estimation [1, 2]. Could the authors elaborate on potential applications of the proposed approach beyond the discovery of the causal graph?
2. The proposed method builds on several modules from existing work. For instance, the representation of the posterior distribution as permutation and upper triangular matrices is also similar to [3]. Although the method outperforms baselines in graph-based metrics in causal discovery, the specific novelty remains unclear, particularly given the first concern.

[1] Emezue, C. C., et al. (2023). Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation. arXiv, 2307.04988. https://arxiv.org/abs/2307.04988v3

[2] Schölkopf, B., et al. (2021). Towards Causal Representation Learning. arXiv, 2102.11107. https://arxiv.org/abs/2102.11107v1

[3] Cundy, C., et al. (2021). BCD Nets: Scalable Variational Approaches for Bayesian Causal Discovery. arXiv, 2112.02761. https://arxiv.org/abs/2112.02761v1
We propose a method that allows for sampling from an approximate posterior over causal structures using neural processes.
@inproceedings{
anonymous2024a,
title={A Meta-Learning Approach to Bayesian Causal Discovery},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=eeJz7eDWKO},
note={under review}
}
Discovering a unique causal structure is difficult due to both inherent identifiability issues, and the consequences of finite data.
As such, uncertainty over causal structures, such as those obtained from a Bayesian posterior, are often necessary for downstream tasks.
Finding an accurate approximation to this posterior is challenging, due to the large number of possible causal graphs, as well as the difficulty in the subproblem of finding posteriors over the functional relationships of the causal edges.
Recent works have used Bayesian meta learning to view the problem of posterior estimation as a supervised learning task.
Yet, these methods are limited as they cannot reliably sample from the posterior over causal structures and fail to encode key properties of the posterior, such as correlation between edges and permutation equivariance with respect to nodes.
To address these limitations, we propose a Bayesian meta learning model that allows for sampling causal structures from the posterior and encodes these key properties.
We compare our meta-Bayesian causal discovery against existing Bayesian causal discovery methods, demonstrating the advantages of directly learning a posterior over causal structure.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
neural processes bayesian causal discovery transformers
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/32519215af7aa69b8b5284f894de3b1871413f4a.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/314825d909c85dc914c22315bead1a25f68e8a51.zip
A Meta-Learning Approach to Bayesian Causal Discovery
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission