Standardizing Structural Causal Models
Causality;Causal Discovery;Structural Causal Model;Standardized Structural Causal Model;Variance Artifact;Covariance Artifact;Simulation;Benchmark
Yes
No ethics review needed.
Evaluation on real-world datasets - Although the whole paper is about standardizing synthetic datasets for benchmarking, it would be interesting to see how the structure learning algorithms perform on standardized /internally-standardized versions of real-world datasets.
Causal structure learning methods require synthetic datasets to benchmark against. This work addresses artifacts that vanilla generating methods have. This makes it an interesting and relevant direction of research. The main contribution is the proposed method that is simple and seems to address these artifacts. The experimental evaluation is thorough and the theorems are relevant to the issues with standardized SCMs. The paper is also written well.
Sampling datasets from an SCM results in artifacts that can be exploited to infer causal ordering. This paper proposes a different method to standardize SCMs for generating benchmarkable datasets. Earlier work proposes post-hoc standardization that is vulnerable to deterministic relations downstream in long chains (correlation between adjacent nodes gets closer to 1). This work proposes generating new variables with standardized parents that bypasses this problem, called internal standardization. The paper contains results on linear Gaussian SCMs with forest causal graphs that show that standardizing SCMs can make them partially identifiable in some settings whereas internally standardized SCMs are nonidentifiable. Experimental evaluation includes comparing current causal structure learning algorithms performance on unstandardized, standardized,  and internally standardized data.
Few concerns follow: 
1) Most of the theory is for the linear, Gaussian case where while analysis is tractable, most practical situations are non-Gaussian. Therefore, while the partial identifiability result is still of importance, I am not that convinced with the impact of the nonidentifiability result. 
2) Even in the experimental evaluation, the nonlinear systems are evaluated on samples from a Gaussian process. Is it possible to introduce nonlinearity in a different way? 
3) For a proposal that is relatively incremental, I would urge the authors to also focus on the nonlinear case to strengthen the paper.
Yes
No ethics review needed.
1. Didnot define the weights formally before using them in Lemma 1.
2. In theorem 3, "all except one edge". Can you elaborate on this claim a little more?
3. It is also not entirely clear on how to extend iSCMs to handle interventions.
4. Are iSCMs a valid way of modeling real world datasets? Can you comment on this a little more?
1. The paper is very well written, explaining the standardization process clearly. It is interesting to note that the iSCMs are also an SCM.

2. The partial identifiability result of Theorem 3, arguing that for standardized SCMs, given the markov equivalence graph, one can recover the almost entire graph, while iSCMs are robust to such identification (Theorem 4) is an important result. This provides a grounding that iSCMs can be used for benchmarking. 

3. The empirical validation of the results that the iSCMs are not easily beatable compared to SCMs.
In this paper, the authors address an important issue in benchmarking causal discovery algorithms: how do we build robust synthetic datasets. The main challenge is that many popular algorithms make use of the patterns (artifacts) among the available synthetic datasets, such as the correlations increase along the causal ordering to learn the causal graph, even when it should be impossible to do so. In this work, the authors describe a simple standardization process of generating synthetic data that makes them less identifiable from prior knowledge, such as correlations, and can be useful for benchmarking and beyond. The standardization process, resulting in causal models called iSCMS, involves normalizing each variable (to zero mean and unit variance) and then continuing this process iteratively over the topological ordering.
1. The authors approach works for sparse graphs, such as directed acylic forests. It is not entirely clear if the process of normalization (i.e., topological ordering) is the reason why it cannot be extended to graphs beyond forests.

2. It seems that the process of generation each sample is computationally expensive, depends linearly on the depth of the topological ordering. Also it is not clear if it extends for nonlinear causal dependencies.

3. Generalizability Beyond linear models is not clear.
Yes
No ethics review needed.
I understand that retaining ground truth is challenging, but as noted in the weaknesses, there are cases where ground truth can be obtained. Are there additional instances where $R^2$-sortability values are reported for such cases?
1. The paper proposes a simple yet effective approach to avoid  $R^2$ -sortability, operating under the assumption that the increase of  $R^2$  along the causal order is an artifact.
2. The paper is well-organized, providing both theoretical justifications and experimental results to support its claims.
The authors propose internally-standardized SCMs (iSCMs), which standardize each variable during data generation to counteract the increase in variance or correlation along the causal order. Unlike post-hoc standardization, iSCMs avoid Var- and $R^2$-sortability and demonstrate robustness in large systems, as shown in the simulation study.
My main concern is the paper’s treatment of increasing correlation along the causal order as an “artifact.” In reality, this pattern could exist in real-world data. Standardizing variables in the data generation process may, in fact, introduce more “artificiality,” as fewer studies have shown this pattern in real data. As reported in [1], the R²-sortability for a real dataset in Section 4.2 was 0.82. Thus, while iSCMs may serve as a useful supplementary benchmark, they may not be the definitive or sole benchmark for causal modeling.

[1] Reisach, Alexander, et al. "A scale-invariant sorting criterion to find a causal order in additive noise models." Advances in Neural Information Processing Systems 36 (2024).
Yes
No ethics review needed.
1. Line 147: "existing identifiability results only concern the unstandardized distributions $p(x)$ of SCMs." I am not sure if I agree with this claim, as most causal discovery methods either do not rely on any parametric assumptions (e.g., PC), or rely on assumptions that do not depend on variance (e.g., LiNGAM).

2. The chain structure in Figure 1 is a special case where, for each $x_i$, $i=1, \cdots, 10$, all variables $x_j$ with $j < i$ are ancestors of $x_i$. This structure leads to linear growth in variances, which in turn affects correlations. Would this be resolved if the underlying model had a sparser structure or a smaller tree depth?

3. Line 459: "in standardized SCMs, the parent-child relationships become more deterministic for larger graphs, which makes independence testing less reliable." I am not sure if I agree with this claim either, as the PC algorithm is mainly based on *conditional* independence tests, and it recursively removes edges from the fully connected graph only when the variables are conditionally independent. This means that marginal dependencies (deterministic relations) should not have a huge impact on the the algorithm.
1. The observation about the variance and pairwise correlation of variables in SCMs is very interesting and has not been widely studied. This idea could be used for further evaluation of the performance of structure learning algorithms, especially when no theoretical identifiability results are provided.
2. The organization of the paper is clear and easy to follow.
The authors observe that while SCMs are widely assumed to be the underlying causal model in causal structure learning algorithms, variance and pairwise correlations among the observed variables in an SCM may provide extra information about the underlying model. This information may not be removed by simply standardizing the observed variables and may be used by existing structure learning algorithms. Therefore, they propose internally-standardized SCMs, where the standardization is done within the recursive data-generating process. The authors provide partial identification of the proposed model and also test the performance of existing algorithms on their proposed model.
1. Lack of empirical evidence for the advantage of the proposed model. The authors mention that SCM may suffer from the problem of increasing correlation, whereas iSCM does not. However, there is no clear evidence that this is actually the case in real-life scenarios. If iSCM offers no advantage over SCM in real-life scenarios (i.e., increasing correlation may actually occur in real data), then this may not be a meaningful benchmark, as iSCM is a submodel of SCM and has more model assumptions than SCM.

2. Limited novelty of the proposed theoretical result. The authors claim to provide the first identifiability result for standardized SCMs. However, since standardized SCMs can be considered a special case of SCM, existing identifiability results for SCMs that do not rely on variances (such as LiNGAM) should also apply here (see Q1 below). Additionally, Theorem 3 states that if the underlying model (1) is a linear SCM, (2) has equal noise variance, (3) has a tree structure, and (4) has all edge weights with absolute values greater than 1, then these conditions are sufficient to identify the causal model from standardized data given the MEC. This is a highly restrictive setting, particularly conditions (3) and (4), and there may be other identification methods under the same setting that do not rely on pairwise correlation properties.
Yes
No ethics review needed.
1. Theorem 2 does show that there is no convergence of CEV towards 1 when we increase the depth, still, it displays a dependence on the density: this is completely overlooked by the authors, both in the theory and the experiments. Is there a specific reason? As described in the weaknesses section, I would like to see experiments accounting for different density levels. In practice, I expect to see that nodes with a lot of parents have a larger CEV, such that when directing edges, it is more likely that that node with a higher CEV is a child rather than a parent: under this conjecture, R2 is still a signal about the correct topological order which could be exploited (although weaker than the one provided by SCMs). SF graphs with preferential attachment might be a good testing ground to see this effect, as they produce an unbalance in the number of edges between different nodes. 
2. I believe that there is a big point where iSCMs would be beneficial and that is a bit overlooked. One important use of synthetic data is for training amortized inference models such as AVICI. It is reasonable to think that these models exploit shortcuts as R2 or var-sortability. Yet, this is not really studied in the paper: all the experiments of Figure 5 are performed on SCMs that are non-R2 sortable in the first place, so we don’t really understand whether AVICI trained on standardized data benefits of R2 sortability or not.
The paper proposes a practical solution to remove varsortability and R2-sortability in samples from simulated structural causal models. The available empirical evidence suggests that their approach succeeds in removing these shortcuts, and the provided theory (Theorem 2) supports these observations (at least in the linear case). In general, the paper is well presented and has the potential of a valuable contribution to the causal discovery community.
This paper addresses the well-known problem of the presence of shortcuts in structural causal models commonly simulated for the benchmarking of causal discovery algorithms. The authors propose iSCM, a variation of standardized structural causal models where standardization occurs at each step of the generation along the topological order, instead of standardizing the dataset post hoc, when all variables are already sampled. They provide some empirical evidence and theory showing that this removes the dependence of the Cause-Explained-Variance (one source of shortcuts in SCMs, standardized or not) from the depth of the causal graph. Additionally, they carry out experiments to compare several algorithms’ performance over SCMs, Standardized SCMs, and iSCM.
1. This paper stresses that iSCMs are preferable to SCM for benchmarking, or generally that SCMs are ill-suited for the purpose (L80-82, L49-52, L187-188). I believe this is not supported by the available empirical evidence, both in the paper and in the literature: methods like e.g. LiNGAM, RESIT, CAM, and SCORE (and all other methods based on score matching that are state of the art) have been shown by extensive benchmarking to be neutral about rescaling (see Montagna et al., 2023). The bottom line is that whether SCMs are ill-suited depends on the algorithms at hand: in general, such claims should be supported by more extensive empirical evidence, as I discuss in the next point.
2. The empirical section is limited. One reason is the aforementioned lack of well-known / SOTA methodologies that might be unaffected by standardization. Moreover, the authors do not consider the effect of increasing edge density. Given that the authors’ aim is to propose new SCMs for benchmarking, and the common practice of testing at different density and sparsity levels, I believe these experiments should be added to the paper. To control density on ER graphs in a way that reasonably scales with the number of nodes, I suggest fixing the probability of edge rather than the average degree (as the latter does not scale with graph size). Finally, the experiments of Fig. 5 are carried out on datasets that are not R2-sortable in the first place. The only clear evidence that iSCM removes R2-sortability is provided in Figure 4, but this is still limited, at least for the lack of testing on different density levels, which, as I argued before, is important. 
3. The claim that the existing identifiability results only concern the *unstandardized* distributions of SCMs (L147) is not true, or at least I don’t see that: let aside the case where the assumption enabling identifiability is that of equal variance of the noise terms for each variable - which are of minor importance, as they concern the very specific case of LiNGAM models with equal variance of noises: LiNGAM, ANM, and PNL identifiability results are agnostic about the variance. So any identifiability result about SCMs generating according to the aforementioned model classes, would hold both for standardized SCMs and for iSCMs.

I am willing to increase my score given improved experiments and removal/explanation of somewhat controversial claims as those in the first and third points of the weaknesses section.
Model for benchmarking causal structure learning methods that does not induce variance and covariance artifacts in datasets.
@inproceedings{
anonymous2024standardizing,
title={Standardizing Structural Causal Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=aXuWowhIYt},
note={under review}
}
Synthetic datasets generated by structural causal models (SCMs) are commonly used for benchmarking causal structure learning algorithms. However, the variances and pairwise correlations in SCM data tend to increase along the causal ordering. Several popular algorithms exploit these artifacts, possibly leading to conclusions that do not generalize to real-world settings. Existing metrics like $\operatorname{Var}$-sortability and $\operatorname{R^2}$-sortability quantify these patterns, but they do not provide tools to remedy them. To address this, we propose internally-standardized structural causal models (iSCMs), a modification of SCMs that introduces a standardization operation at each variable during the generative process. By construction, iSCMs are not $\operatorname{Var}$-sortable, and as we show experimentally, not $\operatorname{R^2}$-sortable either for commonly-used graph families. Moreover, contrary to the post-hoc standardization of data generated by standard SCMs, we prove that linear iSCMs are less identifiable from prior knowledge on the weights and do not collapse to deterministic relationships in large systems, which may make iSCMs a useful model in causal inference beyond the benchmarking problem studied here.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Causality Causal Discovery Structural Causal Model Standardized Structural Causal Model Variance Artifact Covariance Artifact Simulation Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5448a9ab16110e6b097a97c4c88c09541f007358.pdf
causal reasoning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/688121ed0c7781010215bb0237d033a88339c15e.zip
Standardizing Structural Causal Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission