KD-HGRL: Knowledge Distillation for Multi-Task Heterogeneous Graph Representation Learning
Knowledge Distillation;Graph Neural Network;Embedding Transfer;Heterogeneous Graph;Contrastive Learning;Self-Supervised Learning.
Yes
No ethics review needed.
1. What is the input node feature for the student model? What is the $Z_j$ in equation 11?

2. What's the meaning of the second term in equation 13? Can you further explain on this term? 

3. Why do you incorporate the representation learned by teacher model in the final prediction in equation 14? If so, what's the point of distilling the knowledge from teacher model to student model? Why don't you just train the teacher model for the downstream task? Could you provide the intuition and the reason for that? In addition, in the summarization of contribution, the authors mention that the knowledge distillation mechanism balances performance and computational efficiency. If teacher model is used for final prediction, then how do you ensure the computational efficiency?
1. This paper proposed to capture topological and semantic information of nodes.
2. The experimental results show that the proposed method outperform all baseline methods across multiple datasets in two tasks.
This paper presents a novel knowledge distillation framework called KD-HGRL for heterogeneous graph representation learning, consisting of a teacher model and a student model. The teacher model uses self-supervised contrastive learning with the semantic view and the topological view, while the student model learn the knowledge distillated by the teacher model.
1. There are many typos in the paper.
- Please add the whitespace between the text and the corresponding citation.
- What is the $L_{supervised}$ in equation 17? Should it be $L_{bpr}$?
- In equation 10, the objective function is the summation over N different nodes. However, in figure 1, the objective function is the summation over C different node types. The authors should correct the mismatch between these two objective functions.
- In equation 16,  $Z_v^{s}td$ should be $Z_v^{std}$, $Z_v^{t}ch$ should be $Z_v^{tch}$, $Z_v^{s}$ should be $Z_v^{std}$ and $Z_v^{t}$ should be $Z_v^{tch}$.

2. The motivation of this paper is not appealing. Many existing methods have shown the great performance in the limited label scenarios. Besides, the pretrained models aims to address this issue. What's the advantage of the proposed method over the existing methods?

3. In the experiment, it's better to highlight the method with the best performance.

4. In the introduction, the authors mention that of main issue of current supervised learning method is that their performance is highly dependent on having access to large amounts of labeled data, which is the key motivation of the proposed method. However, in both node classification and link prediction task in the experiment, the training datasets is 60% (80%), which cannot reflect the performance of the proposed method in the limited label scenario. It's recommended to conduct the experiments in a limited label scenario, e.g. 5% or 1% labeled nodes in node classification problem.

5. What's the time complexity of the proposed method compared with the baseline methods?
Yes
No ethics review needed.
See weaknesses.
- Good presentation and readability
- Promising experimental results on node-level tasks
The paper proposes a framework for self-supervised learning on heterogeneous graphs. The self-supervised learning is based on node-level contrastive learning. The paper also proposes to utilize knowledge distillation to transfer the knowledge from the large pre-trained graph model. The framework is tested on node-level tasks, namely node classification and link prediction. Experimental results are promising for the proposed framework in comparison with the baseline models.
- Unclear experimental design: the key in knowledge distillation is to transfer knowledge from large GNNs to smaller GNNs. In the experiments, the authors do not provide the size comparison between the teacher and student models. When comparing with the baseline models, it is also essential to show that smaller GNN models with transferred knowledge can outperform the other baseline GNN models with the same parameter scale.
- Only node-level tasks are evaluated: general graph structural knowledge is expected to benefit both graph-level and node-level tasks. Both the framework and the experiments seem to focus only on node-level knowledge.
- Lack of novelty: graph contrastive learning is widely adopted as a self-supervised task and there exist many similar prior works (e.g., [1]). The proposed framework does not introduce a fundamentally innovative self-supervised graph learning task and is mainly a combination of several previous graph self-supervised techniques. The contributions are more from the engineering perspective rather than the research perspective.

[1] Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. 2021. Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (KDD '21).
Yes
No ethics review needed.
1- What is the primary contribution of this work that distinguishes it from existing literature?

2- What methodology is employed to determine the hyperparameter settings?
Experiments on real-world benchmarks show that KD-HGRL outperforms current state-of-the-art methods.
The KD-HGRL employs knowledge distillation for multi-task heterogeneous graph representation learning. The teacher model generates node embeddings via self-supervised learning by incorporating both semantic and topological views. Then, knowledge is transferred from the teacher to the lighter weight student model. The framework improves performance on downstream tasks like node classification and link prediction.
The discussion surrounding the novelties of the proposed method is insufficient, and the distinctions between this work and existing methodologies remain unclear. While the concept of learning node embeddings from dual views is acknowledged in the literature, it lacks originality in this context. Additionally, the utilization of teacher-student models and knowledge distillation to transfer knowledge from a pretrained teacher model to a student model is not a new contribution. This technique has been well-established in graph representation learning for several years (e.g., [R1]) and has been applied recently to heterogeneous graphs as demonstrated in works such as [R2, R3, R4]. A more thorough exploration of the unique contributions of the proposed method in relation to these established approaches would enhance the clarity and impact of the submission.

[R1]  J. Ma and Q. Mei, Graph Representation Learning via Multi-task Knowledge Distillation, 2019
[R2]  C. Wang et al., Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding, WWW 2022
[R3] J. Fu et al., Heterogeneous graph knowledge distillation neural network incorporating multiple relations and cross-semantic interactions, Information Sciences, 2023
[R4] J. Liu et al., HIRE: Distilling High-order Relational Knowledge From Heterogeneous Graph Neural Networks, 2022

2- It is not clear how the authors select the value of hyperparameters \alpha, \gamma, \lambda_1, \lambda_2
Yes
No ethics review needed.
Please refer to the question section.
1. The proposed method achieves best performance compared with the baselines reported in the experiments, showing its effectiveness. 

2. Overall, the paper clearly presents the proposed method and the conducted experiments.
The paper proposed a two-stage graph learning framework for heterogeneous graphs. In the first stage, the teacher model is pre-trained in SSL and the representations from two views will be learned for contrastive learning. Then the knowledge learned by the teacher model will be transferred to the student model with knowledge distillation for downstream tasks.
1. Since the proposed method includes the pre-training (teacher) stage, some self-supervised learning-based methods proposed for heterogeneous graphs should be compared in the experiments, like MVSE[1] and HGMAE[2]

2. It seems the performance improvements are mainly brought by extra training stage, views and distillation process, the authors should provide complexity analysis in the paper to show if the sacrificed efficiency is worthwhile. 

3. The effectiveness of some designs in the paper is not very clear, like the incorporation of two views in the pre-training and the class centroid representations. The authors should conduct a more comprehensive ablation study to demonstrate the effectiveness of the components in the framework. 

4. It is not clear how the hyper-parameters ($\alpha$, $\lambda$) are set in this work and their impacts on the final performance. 
 
5.	In line 292, the authors mention that a homogeneous graph is generated for the target graph, however, it is still not clear how the homogeneous graph is built. Does the graph build on a specific relation (meta-path) and why?

[1] Zhao et al., Multi-view self-supervised heterogeneous graph embedding, In PKDD 2021.

[2] Tian et al., Heterogeneous Graph Masked Autoencoders, In AAAI 2023.
Yes
No ethics review needed.
1. Existing methods on heterogeneous graphs already handle the node classification task well, what is the need to use knowledge distillation on heterogeneous graphs.
2. The paper claims that lightweight can be achieved through knowledge distillation, but no experimental verification is provided.
3. How to ensure that comparative learning under both views in the teacher phase does not weaken the complementary information of each of the semantic and topological views.
4. The experimental setup of the article does not cover all datasets for example lack of ablation experiments on Freebase and AMiner datasets.
1. Developed a novel knowledge distillation framework (KD-HGRL) tailored for heterogeneous graph representation learning, focusing on tasks like node classification and link prediction.
2. Integrates both local and global information to improve task-specific performance on node classification and link prediction.
3. Combines embeddings from a node and its class-specific neighbors, leveraging both self-supervised and supervised learning, to enhance the student's performance across heterogeneous graph tasks.
This paper proposes a Knowledge Distillation Framework for Heterogeneous Graph Representation Learning (KD-HGRL) for node classification and link prediction tasks. The framework contains two phases: teacher and student. The teacher phase uses self-supervised comparative learning to learn graph node embeddings from both semantic and topological perspectives. The student phase passes these representations to a lightweight combined GCN and MLP model, which enhances the performance of the student model through knowledge distillation to supervise the loss to accomplish a specific task. The approach improves the performance of the student model in node classification and link prediction tasks.
1. The need to employ knowledge distillation on reisomorphic maps is not presented clearly.
2. Although the student model implements a lightweight design, the computational overhead of the teacher model is not analyzed.
3. The authors claim that the proposed method can be better applied to large datasets, but there are no relevant experiments to prove it.
4. Some of the diagrams in the paper are too scribbled to affect the reader's perception, such as Figures 2, 3.
5. Some of the writing in the text is problematic, such as line 412.
KDHGNN enhances representation learning in heterogeneous graphs through knowledge distillation and self-supervised contrastive learning for improved node classification and link prediction
@inproceedings{
anonymous2024kdhgrl,
title={{KD}-{HGRL}: Knowledge Distillation for Multi-Task Heterogeneous Graph Representation Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=OMN1UtDbRf},
note={under review}
}
Heterogeneous graphs capture complex relationships in real-world systems like social networks and knowledge graphs. Heterogeneous Graph Neural Networks (HGNNs) have been proposed for processing these graphs. Still, they often require extensive labeled data, especially for training on large networks, which is impractical or even impossible to obtain. The key question is whether we can train heterogeneous graphs using self-supervised learning and then transfer this knowledge to downstream tasks like node classification and link prediction. To address this, we propose a novel framework, KD-HGRL, which leverages \textbf{K}nowledge \textbf{D}istillation for multi-task \textbf{H}eterogenous \textbf{G}raph \textbf{R}epresentation \textbf{L}earning. Our approach consists of two main phases: teacher and student. The teacher phase generates rich node embeddings from two views—the semantic view and the topological view. A self-supervised learning strategy trains the teacher model without needing labeled data. The student model uses a light graph neural network with a multi-layer perception to predict the task output. A novel knowledge distillation strategy transfers the wealthy, learned knowledge, which reflects the graph's topological structure and semantic information, from the pre-trained teacher to the student model. This allows the student model to understand the downstream tasks graph better. The results from several experiments performed on real-world benchmarks demonstrate the superiority of the proposed method over state-of-the-art approaches.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Knowledge Distillation Graph Neural Network Embedding Transfer Heterogeneous Graph Contrastive Learning Self-Supervised Learning.
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bcf059b537ea7fee9abe3ff6a1ced8f395b0c914.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/8b1a392f970e8b5ddb8fab4fa7f2b77a83e42103.zip
KD-HGRL: Knowledge Distillation for Multi-Task Heterogeneous Graph Representation Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission