Demystifying GNN Distillation by Replacing the GNN
GNN;Graph Neural Network
Yes
No ethics review needed.
See weakness section.
1. Interesting observation: The paper demystifies GNN distillation by highlighting their role as regularizers. 

2. The proposed approach of label histogram and pseudo labeling is easy/simple and effective

3. The authors do highlight limitations of their work. ( homophilic prior vs heterophily).

4. code is provided

5. Ablation studies are also provided.
The authors first investigate " Why are distillation methods so successful? ".

To determine whether GNNs ability comes from increased expressivity or a inductive bias, they analyze their performance against MLPs with varying training set sizes. The authors in their work show that GNNs primarily act as regularizers in distillation, rather than increasing model expressivity. Further, they show that GNNs benefit from unlabeled data through the message passing mechanism.

The paper proposes  method for node classification that replaces GNNs with regularization techniques: consistency loss and iterative pseudo-labeling. This approach leverages unlabeled data and captures structural information using label histograms.

The authors perform empirical evaluation on diverse datasets. They show their method performs empirically better than existing methods. Further, ablation is conducted for diff components to understand its importance.

Overall a good paper.
1. Would recommend authors to add homophily ratio for each dataset so that one can understand how the results vary with diff homophily rates.

2. Why is only GraphSage taken as base GNN in Table3? Why GNNs like GAT are not considered?

3. What is the impact of gamma in eq.3? impact of weightage to itself and neighbor?
 
4. Does the degree of a node have some role to play? Since label histogram and consistency loss use neighborhood data. How is the performance on nodes of different degrees? Are there any studies? How much is the impact of degree/ availability of neighborhood data etc. on performance. Are there any studies on noisy neighbors?
Yes
No ethics review needed.
Most of the questions and suggestions are already mentioned in the weaknesses section. I would like to mention some minor points here.

1. The authors should use "GCN" explicitly instead of "GNN" in their claims, since they only use GCN architecture in the analysis and don't extend their analysis to general GNN models.

2. How does this method compute the label histogram of unobserved test nodes in the inductive setting? Will it access the graph during inference?

3. Inference time of this method should be discussed in the paper. Comparison of training costs would also be helpful.

4. I think Label histogram as a structural embedding is more important than the other two techniques, while little theoretical or empirical insight of it is discussed in the paper. More intuitions and studies on this technique would be helpful.
1. The insight that distillation loss could be equivalent to smoothing regularization loss is interesting.

2. The writing is clear and straightforward. One can easily follow the paper.

3. Label histogram as a structural embedding seems much more effective and reasonable than the deepwalk embedding used in NOSMOG.

4. Notable improvements in performance are observed with the proposed techniques in some datasets.
This paper studies the GNN-to-MLP distillation problem. This paper introduces a framework including three techniques: smoothing-regularization loss, label-histogram as a structural embedding, and iterative pseudo label smoothing. Considering the motivations, the proposed techniques are reasonable and effective. These techniques are based on the observations of the performance gap between GNNs and MLPs with varying label rates. The authors also claim that GNN distillation loss could be equivalent to smoothing regularization loss under certain conditions. The experiments are conducted on several datasets and the results show that the proposed techniques are effective. The ablation study also shows the effectiveness of each technique.
1. (Soundness) Several claims are not well supported or not appropriate. For example, in lines 132-133, the authors claim that "MLPs overfit" with observations that MLPs perform poorly in few-labels setting. As far as I know, overfitting describes the performance gap between training and test sets, which is not the case here. The authors could provide the train-test performance gaps of GNNs and MLPs to support this claim. Another example is the claim in lines 139-140 that "distillation overcomes it by increasing the size of the training set with GNN pseudo-labels." This claim is not well supported, since for distillation, not only more pseudo-labels are introduced, but also the information inside these pseudo-labels is distinctive to what MLP itself could provide. The result of an additional ablative experiment in which GNN pseudo-labels are replaced with MLP's own pseudo-labels (in a self-training way) may help to validate this claim in a more convincing way.

2. (Novelty) Regularizing the MLP using graph structure is not a new idea. Some related works [1,2] are missing in this topic. Furthermore, the smoothing-regularization loss seems equivalent to the contrastive loss of [4] under the homophily assumption. The authors should make a distinction between these works and their work.

3. (Novelty) The iterative pseudo-labeling is also a common practice [5] in topics of GNN few-labels learning. Furthermore, this technique lacks sufficient intuition since the topic of this paper is distillation, not few-labels learning. The authors should provide more insights on why this technique is effective in distillation, especially in a normal setting where labels are sufficient.

4. (Experiment) As far as I know, the main focused benchmark in GNN-to-MLP distillation is the inductive setting. However, the authors put this important inductive setting in the appendix while the main experiments are conducted in the transductive setting. Furthermore, the proposed method is even surpassed by the weak baseline GLNN in some datasets in this setting, which makes it hard to believe the effectiveness of the proposed techniques.

5. (Experiment) Only GraphSAGE is used as the GNN model. The authors should provide more GNN teachers such as GCN, GAT, APPNP, etc. to make the results more convincing.

6. (Contribution) Most of the insights and techniques heavily rely on the assumption of homophily, which limits the contribution of the paper.






[1] Wu, T., Zhao, Z., Wang, J., Bai, X., Wang, L., Wong, N., & Yang, Y. (2023). Edge-free but structure-aware: Prototype-guided knowledge distillation from gnns to mlps. arXiv preprint arXiv:2303.13763.

[2] Zhang, H., Wang, S., Ioannidis, V. N., Adeshina, S., Zhang, J., Qin, X., ... & Yu, P. S. (2023). Orthoreg: Improving graph-regularized mlps via orthogonality regularization. arXiv preprint arXiv:2302.00109.

[3] Shin, Y. M., & Shin, W. Y. (2023). Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs. arXiv preprint arXiv:2311.17781.

[4] Hu, Y., You, H., Wang, Z., Wang, Z., Zhou, E., & Gao, Y. (2021). Graph-mlp: Node classification without message passing in graph. arXiv preprint arXiv:2106.04051.

[5] Li, Q., Han, Z., & Wu, X. M. (2018, April). Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).
Yes
Yes, Discrimination / bias / fairness concerns
1. During the iteration process, where does the node  with pseudo-labels come from? Will they be used for testing?


2. In Section 4.3.1, is the consistency loss highly correlated with the datasets? Using such a loss on the high homophily dataset used in the experiment seems to easily improve performance. Can you compare [homophily prediction encouraged by GNN during distillation] with [ homophily prediction encouraged by your suggested loss during MLP training]?


3. Can there be more analysis in the experiment, for example, on which specific nodes did your proposed method perform well? Is the performance improvement node brought by your method consistent with the performance improvement node brought by the GNN2MLP distillation process?

4. Line 343, "As our goal is to understand GNN distillation, we want to optimize the MLP model such that during inference we do not utilize neighboring nodes. Thus, we apply prediction smoothing only to the pseudo-labels within the iterative training algorithm." What is the causal relationship here?

Overall, I think the perspective of this paper is reasonable and interesting. But the manuscript is worth further revisions, and if the author can address my concerns, I am willing to increase my score.  :  )
1. To understanding the process of GNN2MLP is quite important. The previous methods empirically confirmed the distillation process is  effective, but this work for the first time summarizes them into a unified hypothesis and verifies it through experiments.

2. The MLP trained by the author achieved performance comparable to GNN on different dataset sizes
This paper discusses the distillation process from GNN to MLP. Rather than suggesting a new technique, the authors aim to demystify GNN distillation methods. They propose two hypotheses:(1) GNNs serve as consistency regularizers, and (2) GNNs leverage unlabeled data effectively. Their experimentals indicate that the distillation process of GNN2MLP can be replaced by other manually designed methods (without the need for GNN)
1. All hypotheses have been experimentally proven, not theoretically. And the author only reported accuracy and lacked further analysis of the experimental results

2. During the iteration process, it seems that the nodes with pseudo-labels are constantly being replaced into the training set, and these nodes appear to come from outside the training set, which may lead to unfairness in the evaluation.
Yes
No ethics review needed.
Please address my concerns about weakness.
1. The research perspective of the paper is interesting. It demonstrates that GNNs in distillation methods act as regularizers, which is supported by experiments comparing GNN distillation with an alternative approach of training MLPs with direct regularization.
2. The replacement of implicit GNN regularization with explicit terms can achieve comparable results to GNN distillation without training GNNs.
3. The writing is clear and good. The descriptions and explanations of individual methods are easy to follow.
This paper focuses on demystifying the role of Graph Neural Networks (GNNs) in graph distillation methods for graph node classification. It is found that GNNs primarily function as regularizers during distillation. Through theoretical and empirical investigations, the authors replace GNN distillation with explicit regularization strategies and propose label histogram as an alternative to positional embedding. The paper also validates its claims through comprehensive experiments on multiple datasets and comparisons with existing state-of-the-art methods.
1. The analysis is mainly focused on homophily graphs, and the performance on heterophily graphs is not well explored. A proper addition to this part of the results would strengthen my confidence on this paper. Another important issue is that I feel that “neighborhood smoothing” is an overly strong assumption, which makes the proposed method less applicable.
2. Proposition 1 and its proof seem to be based only on a linear setting. However, the actual MLP contains nonlinear activation, and I'm curious if the GNN distillation loss and consistency loss are still equivalent?
3. I have noticed that the datasets used in this paper are actually small, and it is very necessary to validate the effectiveness of the method on larger-scale ogb-arxiv and ogb-products datasets.
4. Lack of comparison with important baselines mentioned in related work, e.g. VQGraph and FF-G2M, etc.
5. The technical contribution of the article as a whole is limited, because the three proposed methods (1) label consistency; (2) iterative pseudo-labeling; and (3) label histograms share similar ideas with many past works. Combining them is indeed effective but an incremental contribution.
@inproceedings{
anonymous2024demystifying,
title={Demystifying {GNN} Distillation by Replacing the {GNN}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VfYShlQbj7},
note={under review}
}
It has recently emerged that Multilayer Perceptrons (MLPs) can achieve excellent performance on graph node classification, but only if they distill a previously-trained Graph Neural Network (GNN). This finding is confusing; if MLPs are expressive enough to perform node classification, what is the role of the GNNs? This paper aims to answer this question. Rather than suggesting a new technique, we aim to demystify GNN distillation methods. Through our analysis, we identify the key properties of GNNs that enable them to serve as effective regularizers, thereby overcoming limited training data. We validate our analysis by demonstrating an MLP training process that successfully leverages GNN-like properties without actually training a GNN.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
GNN Graph Neural Network
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2be7d0303a77e7f7c6eb94c2c641bfec5fcd0907.pdf
learning on graphs and other geometries & topologies
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/d3554557106adf506f22f2d0857a8a10c2e48dad.zip
Demystifying GNN Distillation by Replacing the GNN
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission