Learning Graph Invariance by Harnessing Spuriosity
Out-of-Distribution Generalization;Invariant Learning;Graph Neural Networks
Yes
No ethics review needed.
See the Weaknesses
1.  The structure of the paper is clear and easy to follow.
2.  The paper conducts comprehensive experiments to demonstrate the performance of proposed method.
This paper proposes LIRS, a learning framework designed to Learn graph invariance, which consists of: a) The biased infomax principle, and b) The class-conditioned crossentropy loss, which elicit effective spuriosity learning and invariant learning respectively, aiming to learn more invariant features. Different from most existing approaches that directly learn the invariant features, LIRS takes an indirect approach by first learning the spurious features and then removing them from the ERM-learned features.
1.  The novelty seems limited. The core idea of LIRS is three parts, i.e., graph Invariance learning by removing spurious features, the biased infomax principle, and the class-conditioned cross-entropy loss. However, the first part is inspired by [1], the second part seems to be a combination of SCL[2] and DGI[3], the third part is the generalized cross-entropy (GCE) method[4]. The technical contribution is a little weak. It is recommended that the author provide a clearer description of the contribution.
2.  The proposed LIRS requires training the GNN encoder twice, which is more cost than the end-to-end methods. It is recommended that the authors provide an overall time cost.
3.  LIRS takes an indirect approach by first learning the spurious features by self-supervised and then removing them from the ERM-learned features. OOD-GCL[5] proposes a contrastive learning module with invariance regularization so that spurious correlations between latent factors in graph representation can be eliminated and OOD predictions in downstream tasks can be achieved. It is suggested that the authors analyze and compare OOD-GCL that directly learns invariant representations by self-supervised.

[1] Empowering graph invariance learning with deep spurious infomax. ICML, 2024.

[2] Supervised contrastive learning. NeurIPS, 2020.

[3] Deep graph infomax. ICLR, 2019.

[4] Generalized cross entropy loss for training deep neural networks with noisy labels. NeurIPS, 2018.

[5] Disentangled Graph Self-supervised Learning for Out-of-Distribution Generalization. ICML 2024.
Yes
No ethics review needed.
Is there an analysis of how the multi-step process in this indirect learning approach affects the overall results? Additionally, during empirical training, what observations did you make? If the final performance does not improve, how would you analyze the sources of performance issues?
1. The paper is well-written and easy to follow.
2. The strategy of decoupling invariant and spurious features is common in the field, but learning the spurious features first before removing them offers a new perspective. This approach provides a novel way to tackle the challenges of OOD generalization in graph representation learning.
This paper presents LIRS (Learning graph Invariance by Removing Spurious features), a novel framework for improving Out-of-Distribution (OOD) generalization in graph representation learning. It critiques existing methods like IRM and VRex for capturing only limited invariant features. Instead of directly learning invariant features, LIRS first identifies and removes spurious features from those learned via Empirical Risk Minimization (ERM). The framework utilizes the biased infomax principle for learning spurious features and a class-conditioned cross-entropy loss for isolating invariant features. This work offers a robust approach to enhancing OOD generalization in graph learning.
Complexity: The process of first identifying spurious features and then removing them may introduce additional complexity in the learning pipeline. Discussing the computational efficiency and potential challenges in this two-stage process would be useful for understanding the trade-offs involved.
Yes
No ethics review needed.
1.  About $L_{inv}$, I think it's implemented using cross entropy, $\hat{s}^{i}$ is the prediction of the environmental label ${s}^{(i)}$ by using $\hat{h_G}$ , and look forward to by minimizing the cross entropy to make prediction closer to real label. Intuitively, i don't think this is helpful for learning invariant learning, because if the learned representation is the invariant feature, I don't think $\hat{h_G}$ should be able to predict the correct  ${s}^{(i)}$, since it should only be related to labels. Mathematically speaking, the authors claim in Proposition 7 in the Appendix that minimizing $L_{inv}$ is equivalent to maximizing the conditional entropy $H(s^{(i)}|\hat{h_G^i})$, 
which I doubt is valid. Known $L_ {inv} $ is cross entropy, we have a $H(s^{(i)}|\hat{h_G^i})$ = $H(s^{(i)},\hat{h_G^i})$-$H(\hat{h_G^i})$ .Minimizing the cross-entropy loss ($H(s^{(i)},\hat{h_G^i})$) fails to maximize the conditional entropy.
2. LIRS requires multi-step training and involves multiple models (GSAT,SVM, etc.). Is it less efficient than existing graph-invariant learning models? 
3. What is the difference between you and the existing method (EQuAD)? Is LIRS an incremental improvement version of it?
1. I agree that learning invariant features indirectly is intuitively likely to be easier and achieve better performance than a direct approach.

2. The authors provide some experimental analysis to support its motivation, as well as some theoretical analysis to justify the two optimization objectives.

3. Good experimental performance.
This paper presents a graph invariant learning method based on spurious feature learning, termed LIRS. The authors argue that directly learning invariant features is challenging, prompting them to first learn spurious features in order to decouple them from graph representation to get the invariant parts, thus this approach necessitates training the encoder for two times, separately. They introduce a learning objective called bias infomax for the spurious feature encoder, and a loss function $L_{inv}$ designed to encourage the encoder to learn invariant features. The authors validate the effectiveness of both loss functions through theoretical analysis. 


However, I find the theoretical feasibility of $L_{inv}$  to be somewhat questionable (see the question part). Additionally, the proposed method appears to be more complex than standard graph invariant learning methods, yet there is no discussion of its efficiency (e.g., time complexity or reported training times), which raises concerns for me. Therefore, I rated it a 5, anticipating that the authors will address my issues, which may prompt me to reconsider my score.
1. The author lacks a clear description of the differences between this method and the existing methods. As far as I know, for example, EQuAD, which the author has cited many times, I checked the article and found that the model pipeline of LIRS is very close to it, and the differences seem to be only whether utilize bias infomax and the definition of $s$ in $L_{inv}$.

2. Missing details about the model, such as how to optimize bias infomax? As far as I know mutual information is hard to calculate, how does GSAT get the subgraph? Why choose this approach?
We investigate the advantages of learning graph invariant features indirectly over existing OOD methods, and propose a learning framework that adopts this indirect learning paradigm for graph OOD generalization.
@inproceedings{
anonymous2024learning,
title={Learning Graph Invariance by Harnessing Spuriosity},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=UsVJlgD1F7},
note={under review}
}
Recently, graph invariant learning has become the \textit{de facto} approach to tackle the Out-of-Distribution (OOD) generalization failure in graph representation learning.
They generically follow the framework of invariant risk minimization to capture the invariance of graph data from different environments.
Despite some success, it remains unclear to what extent existing approaches have captured invariant features for OOD generalization on graphs.
In this work, we find that representative OOD methods such as IRM and VRex, and their variants on graph invariant learning may have captured a limited set of invariant features. 
To tackle this challenge, we propose LIRS, a novel learning framework designed to Learn graph Invariance by Removing Spurious features.
Different from most existing approaches that \textit{directly} learn the invariant features, 
LIRS takes an \textit{indirect} approach by first learning the spurious features and then removing them from the ERM-learned features, which contains both spurious and invariant features. We demonstrate that learning the invariant graph features in an \textit{indirect} way can learn a more comprehensive set of invariant features. Moreover, our proposed method outperforms the second-best method by as much as 25.50% across all competitive baseline methods, highlighting its effectiveness in learning graph invariant features.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Out-of-Distribution Generalization Invariant Learning Graph Neural Networks
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/de0fb23c149ba5732534c0c68debd1e992f57b33.pdf
learning on graphs and other geometries & topologies
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Learning Graph Invariance by Harnessing Spuriosity
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission