Cross-Domain Graph Data Scaling: A Showcase with Diffusion Models
Graph Foundation Model;Graph Data Augmentation;Diffusion Models
Yes
No ethics review needed.
Please see the weakness part.
- The paper demonstrates an interesting cross-domain transfer ability, where the pre-trained model can be adapted to various downstream tasks at different levels (node, edge, and graph).
- The presentation is clear, with well-explained methods and experimental results, making it easy to follow the proposed approach and findings.
The paper proposes a graph structure augmentation framework built on discrete diffusion models. They first pre-train a structure-only discrete diffusion model on graphs from multiple domains. During downstream adaptation, the model generates synthetic graph through guided generation, using a task-specific head to align these generated structures with the target task. This framework aims to enhance performance across various tasks, including node classification, link prediction and graph prediction. The experiment also shows that the downstream performance increases when the amount of pre-training data increases.
- I have some concerns about the comparison of UniAug with baseline pre-training methods. The authors mention that they used the same pre-training dataset for UniAug and baselines. They also calculated node degrees as inputs and replaced node features with node degrees for downstream testing. However, these baseline models were not originally designed to be trained and tested in this way. As shown in Table 13, when evaluated in their original semi-supervised or self-supervised settings, these baselines achieve results similar to UniAug. This is different from the larger improvement suggested in Table 2. A more accurate comparison would keep the baselines in their original configurations, and it would also help to see similar comparisons for link prediction and node classification tasks.
- Additionally, a discussion of pre-training time for UniAug compared to the baselines would improve the analysis. This can offer a better understanding of UniAug’s efficiency compared to other methods.
- A more detailed description of the sizes of Small, Full, and Extra pre-training datasets should be provided. From the description, the Extra dataset includes 1,000 more subgraphs from the GitHub Star dataset than the Full dataset. This addition results in large improvements for link prediction tasks but shows little effect on graph classification tasks. Thus I am curious whether the improvements come from the increased amount of data or from the diversity added by the new subgraphs. A more detailed breakdown of the dataset sizes and an analysis of data diversity versus quantity would help clarify the reasons behind these improvements.
Yes
n/a
No ethics review needed.
See Weaknesses.
+ The research topic of this paper—How to effectively leverage the increasing scale of data across domains for graph learning, is highly significant and meaningful, representing a critical issue that urgently needs to be addressed in the current field.
+ This paper collects thousands of graphs from varied domains with diverse patterns to explore the potential of data scaling for graph learning.
+ This paper provides a solution based on the diffusion model from a structural augmentation perspective. This augmentation paradigm strategically circumvents feature heterogeneity and fully utilizes downstream inductive biases in a plug-and-play manner.
This paper explores a meaningful issue: How to effectively leverage the increasing scale of data across domains for graph learning? To achieve effective data scaling, this paper proposes a universal graph augmentation framework, UniAug. This framework pre-trains a discrete diffusion model on massive graphs across domains to learn the structural patterns, and conducts structure augmentation with the help of the pre-trained diffusion model. Experiments shows that this framework can leverage the data scaling laws and achieve performance improvements across various downstream tasks.
+ The authors should clearly illustrate the scale and domain variety of the pre-training data collection in the paper and make it public for the community to conduct further research on the issue. Meanwhile, providing code examples is also essential.
+ The authors should consider whether excessively introducing additional datasets for pre-training is appropriate and whether it might produce negative effects.
+ Related work about data scaling on graphs should be include in the paper [1, 2, 3].
1. [1] Liu J, Mao H, Chen Z, et al. Neural scaling laws on graphs[J]. arXiv preprint arXiv:2402.02054, 2024.
2. [2] Ma Q, Mao H, Liu J, et al. Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?[J]. arXiv preprint arXiv:2408.11243, 2024.
3. [3] Wang Z, Li Y, Ding B, et al. Exploring Neural Scaling Law and Data Pruning Methods For Node Classification on Large-scale Graphs[C]//Proceedings of the ACM on Web Conference 2024. 2024: 780-791.
Yes
No ethics review needed.
1. The authors present experimental results with varying pretraining scales in Figure 3. Although there is a general performance improvement between the SMALL, FULL, and EXTRA scales, there are instances where the model pretrained on smaller graph datasets outperforms those pretrained on larger datasets, such as on the Enzymes and Erdos datasets. Could the authors provide a detailed explanation for these observations?

2. Can the proposed method be applied to more practical scenarios, such as few-shot or zero-shot learning?

3. The proposed method leverages a diffusion model to generate additional structural information. While the model demonstrates empirically desirable performance, I am curious whether there is a theoretical understanding that supports its efficacy.
1. The paper is well-written and easy to follow, presenting a simple yet effective approach that uses diffusion models to augment structural data for cross-domain graphs.

2. Addressing the challenge of designing models for cross-graph learning is highly valuable and relevant.

3. The experiments are comprehensive, covering approximately 30 graphs across more than five domains.
The paper proposes a model called UniAug, based on a graph diffusion model. The approach consists of two steps: first, pretraining on diverse graphs from different domains, and second, using the model to generate synthetic structures for downstream tasks. Experimental results on approximately 30 cross-domain graphs demonstrate the empirical effectiveness of the method.
1. The authors claim to propose a graph foundation model for cross-domain graphs. However, the method appears to be more of a universal graph structural augmentor that utilizes graph diffusion models for generating additional structures, rather than a foundation model capable of inference across various graph types, like [1,2,3]. Additionally, although the authors demonstrate that pretrained diffusion models can generate synthetic structures to enhance performance on various downstream tasks, the concept of using graph diffusion models to create structures is not entirely new [4,5], which may limit the novelty of the approach.

2. The experimental results are not entirely convincing. The authors use node degrees as features for self-supervised baselines, which can significantly impair model performance when replacing original node features with node degrees, whereas using original node features for the proposed method. Although the authors elaborate on this issue in Appendix D, the experimental descriptions still confuse me, and I cannot find strong evidence that the proposed UniAug outperforms existing SSL methods under comparable settings (i.e., using the same feature setup).

3. The applicability of the method may be limited in certain scenarios. When applying it to downstream tasks, such as node and graph classification, the authors use node labels to guide the generation of graph structures. However, the method may struggle with graphs that lack sufficient label information. It would be helpful to understand whether the approach can be effectively applied to downstream graphs with limited or no label data.

References:

[1] One for all: Towards training one graph model for all classification tasks. ICLR, 2024.

[2] GraphAny: A Foundation Model for Node Classification on Any Graph. Arxiv, 2024.

[3] Zerog: Investigating cross-dataset zero-shot transferability in graphs. KDD, 2024.

[4] SimDiff: Simple Denoising Probabilistic Latent Diffusion Model for Data Augmentation on Multi-modal Knowledge Graph. KDD, 2024.

[5] Data-Centric Learning from Unlabeled Graphs with Diffusion Model. NeurIPS, 2023.
We introduce a universal graph structure augmentor for cross-domain data scaling.
@inproceedings{
anonymous2024crossdomain,
title={Cross-Domain Graph Data Scaling: A Showcase with Diffusion Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ILSZZNlbqw},
note={under review}
}
Models for natural language and images benefit from data scaling behavior: the more data fed into the model, the better they perform. This 'better with more' phenomenon enables the effectiveness of large-scale pre-training on vast amounts of data. However, current graph pre-training methods struggle to scale up data due to heterogeneity across graphs. To achieve effective data scaling, we aim to develop a general model that is able to capture diverse data patterns of graphs and can be utilized to adaptively help the downstream tasks. To this end, we propose UniAug, a universal graph structure augmentor built on a diffusion model. We first pre-train a discrete diffusion model on thousands of graphs across domains to learn the graph structural patterns. In the downstream phase, we provide adaptive enhancement by conducting graph structure augmentation with the help of the pre-trained diffusion model via guided generation. By leveraging the pre-trained diffusion model for structure augmentation, we consistently achieve performance improvements across various downstream tasks in a plug-and-play manner. To the best of our knowledge, this study represents the first demonstration of a data-scaling graph structure augmentor on graphs across domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Graph Foundation Model Graph Data Augmentation Diffusion Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/eb1e6725494797edd0239bf07bf729d12d0f0434.pdf
learning on graphs and other geometries & topologies
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/c2df06a46c5d0b8288f4f057c443b0ff4b88e10a.zip
Cross-Domain Graph Data Scaling: A Showcase with Diffusion Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission