Multi-modal Prompt Learning Empowers Graph Neural Networks with Semantic Knowledge
Graph Foundation Model;Multi-modal Prompt Learning;Graph Neural Network;Language Models;Contrastive Learning
Yes
No ethics review needed.
See Weaknesses.
1.  The proposed Morpher can generalize GNN to unseen classes.
2.  The authors provide codes for reproducibility.
3.  This paper is easy to follow.
This paper proposes Morpher, which leverages the text modality to align downstream tasks and data with any pre-trained GNN given only a few semantically labeled samples. The key idea is to embed the graphs directly in the same space as the LLM by multi-modal prompt learning. Experiments demonstrate the superior performance of Morpher in few-shot, multi-task-level, and cross-domain settings.
1. The authors only show that AIO [76] is unable to learn good representations of the downstream data. In my opinion, some state-of-the-art graph prompt methods such as PRODIGY [Ref1] have addressed this issue.
2. From Appendix D.3 in [Ref1], PRODIGY can also apply to the zero-shot classification. Therefore, the comparison of PRODIGY and Morpher is necessary.
3. Some references such as [Ref2, Ref3] are missing.



[Ref1] PRODIGY: Enabling In-context Learning Over Graphs. NeurIPS 2023.

[Ref2] Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning. ICLR 2024.

[Ref3] Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Bias. TPAMI 2024.
Yes
No ethics review needed.
Please refer to the above weaknesses.
S1. The manuscript is well-organized and clearly written, with structured and detailed mathematical derivations. For instance, in the "Improved Graph Prompt Design" section, the paper rigorously derives the formulation for balancing cross-connections between input nodes and prompt nodes. 
S2. The proposed sparse cross-connection mechanism in Morpher prevents prompt features from overwhelming graph features, offering a balanced integration for improved feature alignment.
S3. Comprehensive experiments across diverse datasets, including Cora, CiteSeer, and synthetic zero-shot datasets, effectively demonstrate Morpher's adaptability and robustness.
S4. Results show that Morpher achieves superior performance over baselines in few-shot and zero-shot tasks, validating its cross-domain generalization capabilities.
This paper introduces a new paradigm for enhancing Graph Neural Networks (GNNs) through multi-modal prompt learning, which leverages text data to align downstream tasks with pre-trained GNNs using only a few semantically labeled samples. The approach embeds graphs in the same semantic space as large language models (LLMs) by jointly learning graphs and text prompts. Building on theoretical insights, the authors improve existing graph prompt methods and propose a multi-modal framework that fully utilizes knowledge from pre-trained models without requiring fine-tuning of either the GNN or LLM. This setup significantly reduces the number of learnable parameters. Extensive experiments on real-world datasets demonstrate the paradigm's effectiveness across few-shot, multi-task, and cross-domain settings. Additionally, the framework includes a zero-shot classification prototype that enables GNNs to generalize to previously unseen classes.
W1. This paper proposes a sparse cross-connection mechanism for Morpher to prevent prompt features from overloading graph features, but could this also limit information exchange between graph and text modalities, weakening alignment quality? For complex tasks requiring richer feature fusion, this limited interaction may be insufficient. 

W2. Morpher’s reliance on graph-text fusion assumes complementary information from both modalities, but in tasks with weaker associations, might this integration introduce noise and reduce performance?

W3. The paper does not analyze how differences in data distribution between pre-training and target datasets influence transfer performance, which is crucial for domain adaptation. Quantifying these differences (e.g., via JS or KL divergence) could clarify Morpher’s sensitivity to dataset similarity.

W4. The domain transfer experiment only tests on limited datasets (MUTAG and PubMed), which share similar characteristics in molecular biology. It is unclear if the findings generalize to more diverse domains like social networks.

W5. In the "Zero-Shot Classification Prototype" section, the zero-shot experiment relies on synthetic datasets (ZERO-Cora, ZERO-CiteSeer), which are generated by modifying features and labels and may fail to capture the complexity of natural data distributions. I wonder about Morpher’s generalization performance on real-world datasets.

W6. The author mentions "Figure 5" in the text, though no such figure exists; it seems they meant to refer to "Table 4" instead.

W7. Some relevant works are missing, e.g., Killing Two Birds with One Stone: Cross-modal Reinforced Prompting for Graph and Language Tasks, KDD 2024.
Natural Language Is All a Graph Needs. arxiv. 2023
Yes
No ethics review needed.
- In Section 3, it is stated that the number of cross-connections is controlled to $n_e$ by thresholding. How are the best edges selected? Is this selection random, or is there a specific criterion?
- The objective function used for training Morpher does not incorporate labels or the task but focuses solely on aligning the prompting parameters for the GNN and LLM. This raises concerns about potential misalignment with pre-trained task-specific information, as merely aligning representations without consideration of the task could lead to forgetting previously learned knowledge. What is the rationale behind this objective function, and how does it mitigate this issue?
- Given that the objective function does not involve labels and the GNN and LLM have frozen parameters, how is the model evaluated in a few-shot setting with labels? Does this imply that the base GNN and LLM are pre-trained, with prompting applied afterward without using labels? If so, the contributions to domain transfer and zero-shot learning may be limited, as the fundamental learning from one domain to another occurs prior to prompting. The proposed method may primarily align representations of the LLM and GNN, which has already been explored in few-shot settings within a single domain.
- The proposed method does not involve training the base GNN or LLM, which reduces the number of parameters needed for training and enhances complexity and versatility.
- The authors reasonably justify the shortcomings of one current state-of-the-art model and propose modifications to mitigate its issues, empirically demonstrating the improvements achieved by these modifications.
- The main part of the prompting method, Morpher, is explained clearly with formulations, making it easy to understand.
- The authors provide sufficient and clear visualizations for their proposed methods.
This paper proposes a new framework for integrating pre-trained GNNs with LLMs, leveraging the reasoning capabilities of LLMs alongside the structural representation advantages of GNNs. Specifically, the authors prompt both the LLM and the GNN with trainable parameters while keeping the parameters of both base modules frozen. This prompting, along with a cross-model projector, assists in aligning the embeddings produced by the GNN and the LLM for an input instance, such as a graph. Additionally, they introduce a variant of a current GNN prompting method that addresses its shortcomings through an attention mechanism.
- The authors do not compare their work with state-of-the-art approaches that combine LLMs and GNNs, such as [1]
- While the authors claim to provide a theoretical analysis that "in many cases, state-of-the-art graph prompt is unable to learn good representations," this analysis is limited to Lemma 3.1. This lemma presents a trivial conclusion that does not significantly contribute to the development of the method or elaborate on the effectiveness of current prompting methods. It may be beneficial to remove this lemma and its proof, as it has been widely discussed in the literature on GNNs, and cite related works [2], which would help streamline the paper.
- In the Background section, few-shot prompt learning is only addressed in the context of methods that learn trainable parameters. It would be valuable to also discuss methods that do not train parameters, particularly regarding the intrinsic in-context learning of these methods.
- Some definitions and formulations appear extraneous. For instance, Equation 2 addresses concepts that are reiterated in Equation 6 and relate to matching dimensionality. The term $\tilde{h}$ does not appear in subsequent equations, and similarly, the introduction of $e^t$ in Equation 5 is later replaced by $e^t_{norm,i}$ without further context. Streamlining these sections could reduce unnecessary complexity.
- The authors do not discuss the inference process of their method at test time. It would be helpful to clarify whether the same model must be used at inference time and whether the LLM or the GNN makes the final prediction.
- The claim that this method is the first to address zero-shot classification for GNNs to unseen classes may overlook existing approaches. Referencing works such as [1] and [3] and comparing with them would provide necessary context.
- While the authors mention generating textual features from numerical features of the original datasets, it appears that for node classification datasets, only the labels of the nodes are converted to text, as indicated in Appendix B.2, rather than the node features themselves. Moreover, converting high-dimensional feature vectors to text features is not trivial and could exceed the acceptable input length for LLMs. This oversight may limit the model's generalizability to node/edge-level tasks due to the neglect of node features.


[1] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, & Muhan Zhang (2024). One For All: Towards Training One Graph Model For All Classification Tasks. In The Twelfth International Conference on Learning Representations.

[2] Keyulu Xu, Weihua Hu, Jure Leskovec, & Stefanie Jegelka (2019). How Powerful are Graph Neural Networks?. In International Conference on Learning Representations.

[3] Chen, J., Mi, R., Wang, H., Wu, H., Mo, J., Guo, J., Lai, Z., Zhang, L., & Leung, V. (2024). A Review of Few-Shot and Zero-Shot Learning for Node Classification in Social Networks. IEEE Transactions on Computational Social Systems, 1-15.
Yes
No ethics review needed.
1. How does different text labels affect the model performance? Does randomizing the class text label lead to comparable performance in few-shot setting? How fine-grained text can the model take? Will adding explanations to the text label help improve the performance?
2. Why does the test accuracy of zero-shot class decrease as the training epoch increases?
3. What’s the distribution of graph and text representation? Does there still exist domain shift after the alignment? Can you visualize the distributions of graph and text representations before and after alignment, using t-SNE or UMAP?
1. Morpher is able to adapt to various downstream tasks with limited labels. It is also able to do zero-shot classification on unseen classes.
2. They introduce an improved method of graph prompting without overwhelming or limiting the input graph's features.
3. The use of a contrastive loss function to align the graph and text embeddings helps bridge the gap between graph structure and semantic text features, enhancing the model's flexibility across modalities.
This paper introduces a multimodal prompt learning method called Morpher that uses both graph and text prompts to align graph representations with LLM’s semantic embeddings. This method is able to do zero-shot classification for unseen graph classes. In their paradigm, the pre-trained GNN and the LLM are kept frozen, making it much more efficient than finetuning. They also introduce an improved graph prompting method that cross-connections between the prompt graph and the input graph share the same density with the input graph. By using a projection layer and training with contrastive loss, they align the graph and text prompt embeddings in the same space, allowing for effective adaptation to various tasks with limited downstream data.
1. The effectiveness of Morpher relies on the availability and quality of semantic labels in the text associated with graph data. When these labels are noisy, incomplete, or inconsistent, the alignment might be weakened and the performance of downstream tasks might be reduced.
2. Freezing GNN and LLM reduces the number of trainable parameters, but it also limits adaptation flexibility. Certain tasks might require fine-tuning the GNN or LLM to capture task-specific information.
3. The independently pre-trained GNN and LLM may cause huge representation gaps between the two modalities, causing difficulty in the alignment process.
4. Applying the graph prompting might be computationally expensive on large and complex graphs.
@inproceedings{
anonymous2024multimodal,
title={Multi-modal Prompt Learning Empowers Graph Neural Networks with Semantic Knowledge},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ax4ZOytBV2},
note={under review}
}
While great success has been achieved in building generalizable language models, three fundamental issues hinder GNN-based graph foundation models: the scarcity of labeled data, different levels of downstream tasks, and the conceptual gaps between domains. In depth, though the labels of real graphs are associated with semantic information, most graph learning frameworks ignore it by turning semantic labels into numerical labels. In this work, to address these issues, we present a new paradigm that leverages the text modality to align downstream tasks and data with any pre-trained GNN given only a few semantically labeled samples. Our paradigm embeds the graphs directly in the same space as the LLM by learning both graph prompts and text prompts simultaneously. To accomplish this, we improve state-of-the-art graph prompt method based on our theoretical findings. Then, we propose the first multi-modal prompt learning approach for exploiting the knowledge in pre-trained models. Notably, in our paradigm, the pre-trained GNN and the LLM are kept frozen, so the number of learnable parameters is much smaller than fine-tuning any pre-trained model. Through extensive experiments on real-world datasets, we demonstrate the superior performance of our paradigm in few-shot, multi-task-level, and cross-domain settings. Moreover, we build the first zero-shot classification prototype that can generalize GNNs to unseen classes. The code is provided in the supplementary materials.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Graph Foundation Model Multi-modal Prompt Learning Graph Neural Network Language Models Contrastive Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9993823a67dd20b806352621c8eff50b607045c0.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/0bdfc8563e647dad1485890f413a365ff3778f3f.zip
Multi-modal Prompt Learning Empowers Graph Neural Networks with Semantic Knowledge
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission