Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders
Chuang Liu Yuyao Wang Yibing Zhan Xueqi Ma Dapeng Tao Jia Wu Wenbin Hu


Partial Renormalization of Quasiparticle Interactions
    Kun Chen
    April 29, 2024
=====================================================




   Graph masked autoencoders (GMAE) have emerged as a significant advancement in self-supervised pre-training for graph-structured data. Previous GMAE models primarily utilize a straightforward random masking strategy for nodes or edges during training. However, this strategy fails to consider the varying significance of different nodes within the graph structure. In this paper, we investigate the potential of leveraging the graph's structural composition as a fundamental and unique prior in the masked pre-training process.  To this end, we introduce a novel structure-guided masking strategy (i.e., StructMAE), designed to refine the existing GMAE models. StructMAE involves two steps: 1) Structure-based Scoring: Each node is evaluated and assigned a score reflecting its structural significance.  Two distinct types of scoring manners are proposed: predefined and learnable scoring. 2) Structure-guided Masking: With the obtained assessment scores, we develop an easy-to-hard masking strategy that gradually increases the structural awareness of the self-supervised reconstruction task. Specifically, the strategy begins with random masking and progresses to masking structure-informative nodes based on the assessment scores. This design gradually and effectively guides the model in learning graph structural information. Furthermore, extensive experiments consistently demonstrate that our StructMAE method outperforms existing state-of-the-art GMAE models in both unsupervised and transfer learning tasks. Codes are available at <https://github.com/LiuChuang0059/StructMAE>.





















§ INTRODUCTION






In domains, such as academic, social, and biological networks, graph-structured data often lacks labels. This problem is especially acute in biochemical graphs due to the expense of wet-lab experiments. To address this challenge, many techniques have been developed to fully exploit the existing massive amounts of unlabeled data, aiming to enhance graph model training. Among these approaches, self-supervised graph pre-training (SSGP) methods are prominent due to their effectiveness, attracting significant interest in academic and industrial realms <cit.>. 

Currently, SSGP methods are categorized into two primary streams:  1) Contrastive methods, such as GraphCL <cit.> and SimGRACE <cit.>, which utilize contrastive learning principles to reveal the intrinsic structure and interconnections within graph data. 2) Generative methods, including GraphMAE <cit.> and MaskGAE <cit.>. These methods focus on learning node representations through a reconstruction objective. Moreover, generative methods have proved to be simpler and more effective than contrastive approaches that require carefully designed augmentation and sampling strategies. The efficacy of generative methods is further underscored by the enormous successes of models such as BERT and ChatGPT in Natural Language Processing (NLP) <cit.> and MAE in Computer Vision (CV) <cit.>. These successes highlight the significant potential of generative approaches across various domains. Accordingly, this paper explores the capabilities of generative methods, specifically graph masked autoencoders (GMAE), in graph learning tasks and recognizes their potential as evidenced in other fields.






GMAE fundamentally involves randomly masking a proportion of input data (i.e., nodes or edges) and leveraging the reconstruction of the removed contents to guide the representation learning. Despite GMAE's promising results, its random masking approach, which assigns equal probability to all nodes in a graph - a universally adopted strategy, presents a suboptimal strategy. Specifically, the masked nodes are sometimes overly simplistic to predict (i.e., the atom C in Figure <ref> (a)) with only neighborhood information. In such cases, the model's pre-training phase may not be sufficiently informative, thereby hindering the acquisition of valuable knowledge. However, if we mask a large number of key informative nodes (i.e., SO3 in Figure <ref> (b)), the model may fail to perceive the graph’s overall structural information. In summary, the indiscriminate nature of random masking, which fails to distinguish nodes of varying informational values, potentially leads to low data efficiency and compromises the quality of the learned graph representations.  Therefore, this raises the question: is there a more effective  masking strategy than random sampling for enhancing GMAE's pre-training process? 














This paper answers the aforementioned open question by introducing StructMAE, which features a novel structure-guided masking strategy designed to enhance GMAE's pre-training process. The key insight of our method is to inject prior graph structure knowledge into the masking process to guide model learning. Specifically, StructMAE comprises two principal components: 1) Structure-based Scoring: We recognize that the node reconstruction complexity is inherently linked to its structural significance within the graph.  Therefore, we derive a scoring method to assess the node significance, distinguishing between informative and less-informative nodes based on structural considerations. In addition, two scoring method variants are proposed: predefined and learnable, which are discussed in detail in Section <ref>. 2) Structure-guided Masking: With node significance scores established, we propose an easy-to-hard masking strategy that gradually increases the difficulty of the self-supervised reconstruction task. This approach commences with the masking of less-informative nodes, progressively shifting towards masking more informative nodes as the model's learning progresses. This strategic progression in masking difficulty is designed to enable the model to gradually and effectively assimilate the graph's structural information.


To evaluate the effectiveness of the StructMAE model, we conduct comprehensive experiments on a range of widely-used datasets, notably the large-scale Open Graph Benchmark (OGB) <cit.>,  covering two graph learning tasks: unsupervised and transfer learning. The experimental results consistently demonstrate that StructMAE's performance surpasses that of existing state-of-the-art models in contrastive and generative pre-training domains. This exceptional performance demonstrates our structure-guided masking approach's advantages over conventional random masking methods. The principal contributions are summarized as follows:


  
  * We introduce StructMAE, a novel node masking strategy tailored for GMAE. This strategy utilizes the structural information inherent in graphs to gradually and effectively direct the masking process, significantly enhancing the model's capability for representation learning.

  

  
  * We evaluate StructMAE through extensive experiments, comparing its performance with generative and contrastive baselines across two graph tasks on various real-world graph datasets, including the OGB dataset. The experimental results consistently validate StructMAE's effectiveness.













§ RELATED WORK





  
Self-supervised Graph Pre-training. 
Inspired by the success of pre-trained language models like BERT <cit.>, T5 <cit.>, and ChatGPT <cit.>, numerous efforts have been directed towards SSGP. Based on model architectures and objective designs, SSGP is naturally divided into contrastive and generative methods. First, contrastive self-supervised learning has dominated graph representation learning in the past two years. Its success is largely due to elaborate data augmentation designs, negative sampling, and contrastive loss. For instance, DGI <cit.> and InfoGraph <cit.>, based on mutual information, leverage corruptions to construct negative pairs. Similarly, models such as SimGRACE <cit.> and GraphCL <cit.> utilize in-batch negatives.  Differing from previous methods that execute augmentation on graphs, COSTA <cit.> implements augmentation within the embedding space, aiding in the mitigation of sampling bias. Moreover, POT <cit.>  advances graph contrastive learning (GCL) training through the employment of a node compactness metric, assessing adherence to the GCL principle. Second, generative self-supervised learning focuses on recovering missing parts of the input data. For example, GAE <cit.> is a conventional method that reconstructs the adjacency matrix. Multiple GAE variants utilize graph reconstruction to pre-train Graph Neural Networks (GNNs), including  ARVGA <cit.> and SIGVAE <cit.>. Recently, a paradigm shift towards GMAE has shown promising results in various tasks <cit.>. A detailed introduction will be provided in the following section.








  
Graph Masked Autoencoders. 
GMAE mainly involves reconstructing the contents (e.g., nodes and edges) that are randomly masked from the input using autoencoder architecture. A notable example of GMAE is GraphMAE <cit.>, which reconstructs randomly masked input node features with several innovative designs, including re-mask decoding and scaled cosine error. In addition, MaskGAE <cit.>, S2GAE <cit.>, and GiGaMAE <cit.> jointly reconstruct the masked edges and node degrees. SimSGT <cit.>, and GCMAE <cit.> combine contrastive learning with GMAE, whereas RARE <cit.> employs self-distillation to enhance GMAE's performance. Unlike the above mentioned GMAE methods, which primarily use message-passing GNNs as backbone models, GMAE-GT <cit.> utilizes a graph transformer <cit.> as its encoder backbone. There are also several GMAE applications, including heterogeneous graph representation learning <cit.>, protein surface prediction <cit.>, and action recognition <cit.>. However, all the aforementioned methods employ a random masking method when masking graph contents and thus overlook the importance of GMAE mask strategies, potentially inhibiting the model's capabilities. The work most closely related to this study is MoAma <cit.>, which similarly focuses on designing superior mask strategies. However, this approach uses motifs which rely on domain knowledge and manual motif pre-definition, limiting its generalizability across various domains.















§ PRELIMINARIES





  
Notations.  A graph 𝒢 =(𝒱, ℰ)  can be represented by an adjacency matrix 𝐀∈{0, 1}^ n × n and a node feature matrix 𝐗∈ℝ^ n × d, where 𝒱 denotes the node sets, ℰ denotes the edge sets,  n is the number of nodes, d is the dimension of the node features, and  𝐀[i, j]=1 if there exists an edge between nodes v_i and v_j, otherwise, 𝐀[i, j]=0. 




  
Graph Masked Autoencoders.  GMAE operates as a self-supervised pre-training framework, which focuses on recovering masked node features or edges based on the representations of unmasked nodes. To illustrate this method, we focus on the reconstruction of node features as an example. GMAE comprises two essential components: an encoder (f_E(·)) and a decoder (f_D(·)). The encoder maps each unmasked node v ∈𝒱_unmask to a d-dimensional vector 𝐡_v ∈ℝ^d, with 𝒱_unmask representing the set of unmasked nodes, while the decoder reconstructs the masked node features from these vectors. The entire process can be formally represented as:

    𝐇_unmask=f_E(𝐀, 𝐗_unmask);   𝐗^'=f_D(𝐀, 𝐇_unmask),

where 𝐗_unmask and 𝐇_unmask denote the features and embeddings of unmasked nodes, respectively, and  𝐗^' represents the reconstructed features of all the nodes. Then, GMAE optimizes the model by minimizing the discrepancy between the reconstructed representation of masked nodes, 𝐗^'_mask⊂𝐗^',  and their original features, 𝐗_mask⊂𝐗.




§ METHODOLOGY






This section presents the model architecture of StructMAE in detail. First, we provide a detailed introduction to the masking module in GMAE (<ref>). Subsequently, a comprehensive exploration (<ref>) of the masking strategy is provided. Then, the details of the proposed StrucMAE are elucidated  (<ref>). Finally, the overall StructMAE architecture, encompassing training and inference details, is expounded  (<ref>).






 §.§ Introducing the GMAE Masking Module
 


Prior to processing graph data within the GMAE encoder, a subset of nodes 𝒱_mask⊂𝒱 is sampled for masking. According to the methodology described in <cit.>, the features of these sampled nodes are replaced with a learnable vector x_[M]∈ℝ^d. Accordingly, for a node v_i within the masked node subset 𝒱_mask, its feature 𝐱_i in the altered feature matrix 𝐗 is defined as follows:

    𝐱_i= 𝐱_[M]    v_i ∈𝒱_mask
    𝐱_i     v_i ∉𝒱_mask.

Regarding the approach for selecting nodes to mask, most existing works <cit.> utilize a random masking strategy. This method assigns an equal masking probability of masking to each node within a graph. A more detailed exploration of the random masking strategy will be presented in the following section (<ref>).









 §.§ Reconsidering the Random Masking Strategy


In GMAE context, the masking strategy is crucial, as it significantly influences the type of information that the model learns. In previous studies <cit.>,
nodes within a graph are masked randomly, each with an equal probability. However, this strategy overlooks the varying structural information of different nodes, which has been shown to be crucial in graph learning tasks. Therefore, we conduct a preliminary experiment to explore whether the incorporation of structure priors could augment pre-training efficacy. 


In the experiment, models are pre-trained on the MNIST dataset and evaluated in unsupervised settings. Nodes forming numerical values are identified as those possessing rich structural information. Subsequently, the masking probability for these structurally informative nodes is manually increased. The results, presented on the left side of Figure <ref>, reveal the impact of this modified masking strategy on unsupervised accuracy. The results indicate that: 1) A marginal increase in the masking probability for nodes with rich structural information enhances the model's pre-training learning. Furthermore, up to a probability threshold of 0.2, a corresponding gradual increase in the model's accuracy is observed. 2) Conversely, excessively raising the masking probability of these structurally significant nodes detrimentally affects the model's training. These findings corroborate our initial discussion and highlight the importance of proposing an effective method to integrate structural information into the masking process.










 §.§ The Proposed Masking Strategy


Inspired by the preceding discussion, we introduce an innovative structure-guided masking approach for GMAE, named StructMAE (as depicted in Figure <ref> (b)). StructMAE involves integrating the graph's structural knowledge into the masking process, thereby directing the model's learning trajectory more effectively. StructMAE is composed of two primary elements:  Structure-based Scoring (SBS) and Structure-guided Masking (SGM).  









  §.§.§ Structure-based Scoring

The SBS evaluates the significance of each node based on its structural role within the graph. This evaluation facilitates the identification of nodes that are pivotal for the model to learn, allowing for a more targeted masking approach. To determine the importance of nodes, we introduce two distinct methodologies: the pre-defined and learnable methods.



  
Pre-defined Structure-based Scoring. The predefined method involves using a set of predetermined criteria, based on the known structural information, to evaluate node importance. Specifically, the computation of importance score 𝐒∈ℝ^n is achieved using the PageRank algorithm <cit.>, a well-established technique for evaluating node significance based on graph structures. Thus, the importance score of node v_i is defined as:

    s_i= 1-e/n + e ∑_j ∈𝒩_is_j/L_j,

where e denotes the damping factor, L_i represents the degree of node v_i, and 𝒩_i denotes the set of neighboring nodes of node v_i. In addition to PageRank, other prevalent metrics for assessing node importance comprise degree, closeness centrality, and betweenness centrality. Each of these methods is based on different underlying principles, offering diverse perspectives on a node’s role and influence within a graph.  Although these methods constitute more intricate ways of evaluating node importance, our empirical findings suggest that PageRank serves as a straightforward and effective measure. A detailed discussion and comparison of these methods is presented in the following section.



  
Learnable Structure-based Scoring.  In contrast to the predefined method, the learnable approach dynamically assesses node significance based on the evolving state of the graph during the learning process. Specifically, it integrates the formulation of the assessment metric with masked graph modeling, enabling end-to-end learning of this metric. To accomplish this, we employ a lightweight scoring network, denoted as f_S(·), which assesses the importance of each node v_i. The scoring network's design is similar to a GNN-style layer, and it effectively captures graph structural information. Thus, the importance score s_i ∈𝐒 for node v_i is calculated as follows:

    s_i=Sigmoid( f_S(𝐱_i, 𝐀)),    i=1, …, n.

A higher score s_i indicates greater importance of the corresponding node v_i. Following the scoring, node features are sorted in descending order based on these scores. The ordered node features and their respective scores are represented as  {𝐱_i^'} and {s_i^'}, respectively, where i=1, …, n. To facilitate learning the scoring network f_S(·), the predicted scores are multiplied by the node features to serve as modulating factors. This operation is formally expressed as:

    𝐗̂={𝐱_i^''|x_i^''=x_i^' * s_i^'},    i=1, …, n,

where 𝐗̂ denotes the set of node features after scoring. This mechanism ensures that the scoring network is continually updated and refined throughout the model's training process.

In this section, we explore the two distinct SBS methods proposed for StructMAE. First, the pre-defined SBS method offers simplicity and is particularly effective when the structural characteristics are well-understood and can be explicitly defined beforehand. Conversely,  the learnable SBS is more adaptable and can cater to complex and variable graph structures, making it suitable for scenarios where the node significance cannot be easily predetermined.



  §.§.§ Structure-guided Masking

The SGM component utilizes the scores generated by the SBS to guide its masking decisions. It selectively and progressively masks nodes in an easy-to-hard manner, thereby enhancing the model's capacity to effectively learn and represent graph structures.  Specifically, in the initial training stage, a subset of easy nodes with lower scores is masked, making it easier for the model to predict them using basic neighboring information. As training progresses, the masking strategy evolves to encompass more challenging nodes. This enables the model to capture intricate structural information and, consequently, enhances its learning capabilities.

This masking strategy relies on the importance scoring matrix 𝐒. It is hypothesized that nodes with higher 𝐒 scores are more informative and significant. Consequently, we gradually increase the masking probabilities for these high-scoring nodes during masked graph modeling. To implement this, we rank the nodes based on their scores 𝐒={s_1, s_2, ⋯, s_n} and identify the top K indices that constitute the set 𝒴. The masking probability for each node is then determined as:

    γ_i=ϵ+ β    v_i ∈𝒴
     0     v_i ∉𝒴,

where ϵ denotes random noise drawn from a uniform distribution (ϵ∼ U(0,1)) and β represents the increased probability assigned to nodes with higher scores. In this instance, nodes in the set 𝒴 are considered more informative, and consequently, the model is anticipated to prioritize these nodes. The number of masked nodes, denoted by K, is dynamically adjusted throughout the training process. Initially set at zero, K progressively increases with epoch according to the following formula:

    K(t)=p · n ·√(t/T),

where K(t) represents the K value at epoch t, n denotes the total number of nodes in the graph, p is the predefined mask ratio, and T represents the total number of training epochs. This approach enables the model to progressively concentrate on more challenging nodes, thereby enhancing its acquisition of complex structural information.










 §.§ Overall StructMAE Architecture





  
Training Process. The StructMAE training process begins with an input graph from which a specified proportion of nodes is chosen based on our selective masking strategy. The selected nodes are subsequently masked using a mask-token. The graph, now with partially masked features, is subsequently fed into the encoder, which generates encoded representations of the nodes. After that, the decoder module is responsible for predicting and reconstructing the features of masked nodes.  For optimization, we adopt the scaled cosine error as utilized in GraphMAE <cit.>.



  
Inference and Downstream Tasks. StructMAE is designed to cater to two distinct downstream applications: unsupervised and transfer learning.  In unsupervised learning, the encoder processes the input graph without masking during the inference stage.  The node embeddings generated by the encoder are then utilized for graph classification tasks with linear classifiers or support vector machines. In the transfer learning context, the pre-trained models are fine-tuned on different datasets. This fine-tuning enables the model to adjust to new data domains, leveraging the foundational knowledge gained during its initial training on the source dataset. Each of these downstream tasks emphasizes the versatility and applicability of StructMAE in diverse graph learning scenarios.
































































§ EXPERIMENT








 §.§ Unsupervised Representation Learning




  
Objective. To assess the efficacy of the pre-trained model in its feature extraction capability, we subject it to a series of unsupervised tasks. Achieving success in these tasks will highlight the model's proficiency in learning high-quality and informative representations, which are crucial for various downstream graph analytics tasks.



  
Settings. Datasets. We employ seven real-world datasets, including MUTAG, IMDB-B, IMDB-M, PROTEINS, COLLAB, REDDIT-B, and NCI1, involving diverse domains and sizes. Baseline Models. To demonstrate the effectiveness of our proposed method, we compare StructMAE with the following 10 baseline models: 1) Two supervised models: GIN gin and DiffPool diffpool; 2) Six contrastive models: Infograph infoGraph:, GraphCL graphCL, JOAO joao, GCC gcc, InfoGCL infogcl, and SimGRACE <cit.>; 3) Two  generative models: GraphMAE <cit.> and S2GAE <cit.>. We report the results from previous papers according to graph classification research norms. Implementation Details. In the evaluation protocol, we initially generate graph embeddings using the encoder and readout function. Then, the encoded graph-level representations are fed into a downstream LIBSVM <cit.> classifier for label prediction, consistent with other baseline models. The performance is assessed by measuring the mean accuracy obtained from a 10-fold cross-validation, and this evaluation is repeated five times to ensure robustness.  







  
Results. The results are detailed in Table <ref>, where StructMAE-P denotes StructMAE with predefined SBS, and StructMAE-L is with learnable SBS. Analyzing these results enables deriving several observations:  1) State-of-the-art Performance:  StructMAE-L outperforms existing self-supervised baselines on four out of seven datasets. Furthermore, it attains state-of-the-art performance considering the average rank across these datasets. Meanwhile, StructMAE-P  maintains competitive performance against other self-supervised methods. These results emphatically demonstrates the efficacy of the proposed StructMAE approach.  Please note that,  StructMAE focuses solely on masking nodes, whereas S2GAE extends its masking strategy to include edges. The differing approach of S2GAE underscores a potential avenue for further development in StructMAE. 2) Comparison with Supervised Methods: Remarkably, the StructMAE-L, though self-supervised, attain comparable or superior performance on certain datasets, including PROTEINS, IMDB-B, and COLLAB. This finding indicates that the representations learned by StructMAE are high-quality and informative, aligning with supervised learning benchmarks. 3) Comparison with GraphMAE: In comparison to GraphMAE, which employs a random node masking strategy, StructMAE consistently outperforms it, providing further evidence to support our hypothesis that incorporating structural knowledge into the masking process can significantly enhance the model's learning capabilities. 4) Predefined vs. Learnable SBS: A notable trend is that the StructMAE-L's performance generally surpasses that of StructMAE-P, particularly on complex datasets such as COLLAB and REDDIT-B. This trend indicates the greater adaptability and effectiveness of the learnable SBS in handling complex and variable graph structures, as opposed to the predefined method. These results collectively validate the core principles behind StructMAE and its components, highlighting its potential as a powerful tool for unsupervised representation learning in graph-structured data.




 §.§ Transfer Learning




  
Objective. The primary goal of the transfer learning task is to evaluate the transferability of the pre-training scheme utilized in StructMAE. This involves pre-training the model on a specific dataset and fine-tuning it with different datasets.



  
Settings. Datasets. During the initial pre-training phase, StructMAE is trained on a dataset comprising two million unlabeled molecules obtained from the ZINC15 <cit.> dataset. Subsequently, the model is fine-tuned on eight classification benchmark datasets featured in the MoleculeNet dataset <cit.>. In our evaluation, we adopt a scaffold-split approach for splitting the datasets, as outlined in <cit.>. Baseline Models. To demonstrate the effectiveness of our proposed method, we compare StructMAE with the following 10 baseline models: 1) Three unsupervised models: Infomax, AttrMasking and ContextPred <cit.>; 2) Four contrastive models: GraphCL graphCL, JOAO joao,   GraphLOG <cit.>, and RGCL <cit.>; 3) Three generative models: GraphMAE <cit.>, GraphMAE2 <cit.>, and Mole-BERT <cit.>. We report the results of baseline models from previous papers according to research norms. Implementation Details. Experiments are conducted 10 times, and the mean and standard deviation of the ROC-AUC scores are reported. According to the default settings used in prior research <cit.>, a 5-layer GIN model <cit.> is employed as the encoder and a single-layer GIN as the decoder in our StructMAE framework. 







  
Results. The detailed results in Table <ref> offer insightful observations into StructMAE's performance within the transfer learning context.
1) State-of-the-Art Performance Across Datasets:  StructMAE-P and StructMAE-L demonstrate state-of-the-art performance across an ensemble of eight datasets. Specifically, StructMAE-P and StructMAE-L achieved 1.4% and 1.5% improvements in average performance metrics, respectively. Additionally, each method individually achieves top-tier performance on several datasets. These results validate the StructMAE's ability to effectively generalize learned representations across diverse datasets.
2) Superiority of StructMAE-L: The results reveal the superior performance of StructMAE-L over StructMAE-P on the average metrics. This trend further highlights the enhanced adaptability and effectiveness of the learnable SBS method, similar to observations made previously in unsupervised learning.







  


 §.§ Ablation Study







A detailed study is conducted to evaluate the impact of different components within StructMAE. It is important to note that, except for the components under analysis, all other aspects of the model remain consistent with the comprehensive StructMAE. The findings, as outlined below, provide valuable insights into the significance of each component: 1) Efficacy of Pre-defined Metrics: As shown in Table <ref>, the PageRank metric consistently demonstrates superior performance compared to other pre-defined metrics. However, other metrics also demonstrate commendable performances, suggesting their potential applications in specific scenarios. 2)  Scoring Function Comparison:   As displayed in Table <ref>, we observe that the performance of GNN scoring function outperforms the Multilayer Perceptron (MLP). This outcome emphasizes the importance of incorporating structural information into the scoring process. Furthermore, the combined use of GNN and MLP consistently yields superior performance compared to using either one independently. 3) Masking Strategy Efficiency:  The results are detailed in Table <ref>, where Top, Middle, and Bottom denote the masking of the top, middle, and bottom p*n nodes, respectively, based on the masking probability. Analysis of the results indicates that the easy-to-hard masking strategy consistently surpasses the performance of other methods. This finding supports our perspective on the effectiveness of gradually increasing the learning challenge, validating the strategic design of our training approach. 4) Role of Random Noise: Notably, a significant drop in performance is observed in the absence of random noise, as indicated in Table <ref>. This indicates that the inclusion of randomness in the model enables access to a broader range of information and simultaneously strengthens the learning process robustness. 










§ CONCLUSION
 
This study proposes the StructMAE model, a novel structure-guided masking strategy that incorporates prior structural knowledge into the masking process, thereby enhancing the pre-training model's learning efficiency. The StructMAE framework consists of two pivotal steps: SBS and SGM. Extensive experiments, encompassing two distinct graph learning tasks, demonstrate that StructMAE significantly outperforms existing self-supervised pre-training methods. These results highlight our approach's effectiveness in leveraging structural information for improved model performance. Despite its competitive performance, StructMAE still has room for improvement. For instance, 1) devising more effective scoring methods to fully exploit the structural information, 2) extending the structure-guided masking strategy to encompass edge masking, and 3) expanding structure-guided masking to a broader spectrum of tasks (e.g., node classification).





§ ACKNOWLEDGMENTS

The work of Wenbin Hu was supported by the National Key Research and Development Program of China (2023YFC2705700). This work was supported in part by the Natural Science Foundation of China (No. 82174230), Artificial Intelligence Innovation Project of Wuhan Science and Technology Bureau (No. 2022010702040070), Natural Science Foundation of Shenzhen City (No. JCYJ20230807090211021).




named_short











§ PARAMETER ANALYSIS

In this section, we investigate the effects of varying the extra masking probability assigned to nodes deemed structurally significant in the graph. The detailed results are presented in Figure <ref>. Analyzing the results, we aim to elucidate how different levels of emphasis on structural information during the masking process affect the model's training performance. Our findings reveal a notable trend: the model typically achieves peak accuracy when the additional masking probability is set to either 0.25 or 0.50. This observation implies the existence of an optimal range for integrating structural information into the masking process. When properly balanced, this integration enhances the model's training performance, enabling it to focus more effectively on key nodes within the graph. However, deviating beyond this optimal range can lead to detrimental effects. Excessive emphasis on structurally significant nodes, indicated by higher extra masking probabilities, appears to impede the model's learning process. This could result from the model becoming overly biased towards certain nodes, potentially overlooking other valuable information distributed across the graph.




§ VISUALIZATION
 

For a deeper exploration of the structure information embedded in the pre-trained representations generated by StructMAE (trained on the ZINC15 dataset), we assess the similarity (i.e., cosine similarity) between a given query molecule and other molecules, displaying the top-most similar molecules. As shown in Figure <ref>, the molecule ranked highest by StructMAE (Top@1) demonstrates a significant similarity to the query molecule, mirroring both the types of nodes and the graph structure. In contrast, the Top@1 molecule identified by GraphMAE <cit.> shares only atom types with the query molecule and is significantly deficient in replicating essential structural features (e.g., the subgraphs highlighted in grey or  blue colors). Therefore, this illustrates that our structure-guided masking can assist the model in more accurately capturing the graph structural information.



























§ IMPLEMENTATION DETAILS








 §.§ Details for Experiments on Unsupervised Representation Learning



  
Environment.
In unsupervised representation learning, we implement StructMAE with Python (3.8), Pytorch (2.0.0), scikit-learn (1.3.2) and Pytorch Geometric (2.4.0). All experiments are conducted on a Linux server with 8 NVIDIA A100 GPUs. 



  
Training Details. The Adam <cit.> optimizer and batch normalization are employed for both StructMAE-P and StructMAE-L. The mask ratio is searched within {0.25, 0.5, 0.75} for most datasets, and the additional masking probability (denoted as β in the main manuscript) is searched within {0.25, 0.5, 0.75, 0.85}. The hidden dimension is set as 512 for most datasets, except 32 for MUTAG and 256 for COLLAB. Particularly for StructMAE-L, the ratio of warm-up epochs is set as 0.0 for most datasets, except 0.2 for PROTEINS. Besides, the balance parameter between GNN and MLP is searched within {0.0, 0.1, 1.0, 10.0, 100.0}. In terms of the GNN function used in the scoring process, a one-layer GIN <cit.> is employed for most datasets. However, for MUTAG, we use a GCN <cit.> to better suit its specific structure and properties. For a comprehensive understanding of our model configurations, detailed hyperparameters for StructMAE-P and StructMAE-L are presented in Table <ref> and Table <ref>, respectively. 











  
Evaluation.
During the evaluation phase, we focus on the generation and utilization of graph embeddings for the classification task.  202 Generation of Graph Embeddings: We employ the encoder and readout function of our model to create graph embeddings. 
 203 Classification using LIBSVM: The generated embeddings are subsequently fed into a LIBSVM <cit.> classifier. This approach is in line with the practices adopted by other baseline models  <cit.> in the field.  204 Performance Assessment: To assess the performance of our model, we utilize mean accuracy as the primary metric. This accuracy is derived from a 10-fold cross-validation process, ensuring a comprehensive evaluation. To enhance the robustness and reliability of our results, this 10-fold cross-validation is repeated five times.
 205 Baseline Configuration: Each of the baseline models included in our comparison is configured based on the recommended settings provided in their official implementations.
















 §.§ Details for Experiments on Transfer Learning
 


  
Environment.
In transfer learning, we implement StructMAE with Python (3.8), Pytorch (1.13.1), scikit-learn (1.3.2), rdkit (2022.03.2) and Pytorch Geometric (2.0.3). All experiments are conducted on a Linux server with 8 NVIDIA A100 GPUs. 



  
Training Details.

In the transfer learning task, the experimental setup for StructMAE closely follows the configurations used in GraphMAE <cit.>. We have made specific adjustments to the mask ratio and introduced new hyperparameters to tailor the pre-training and finetuning phases to our research objectives. 1) Pre-Training Phase:  For StructMAE-P, we set the mask ratio at 0.5. Additionally, the extra probability, denoted as β, is configured to 0.25.   In the case of StructMAE-L (Learnable Version), the mask ratio and β are both set to 0.5. Furthermore, the balance parameter, represented as α, is fixed at 1.0. 2) Finetuning Phase:  During the finetuning phase, we employ the Adam optimizer to refine the model's performance further. Key hyperparameters, including the learning rate, batch size, and dropout ratio, are tuned to optimize the model for each specific dataset. 
The learning rate is varied among {0.001, 0.005}; the batch size are selected from {32, 64, 128};  the dropout ratio is adjusted between {0.4, 0.5, 0.6}. To provide a comprehensive view of our finetuning phase, detailed hyperparameter configurations are presented in Table <ref>.




  
Evaluation. In the finetuning phase, we adopt a scaffold-split approach for splitting the datasets and report the mean and standard deviation of ROC-AUC scores in 10 experiments following the guidelines presented in <cit.>.













§ INTRODUCTION OF DATASETS AND BASELINES








 §.§ Unsupervised Representation Learning





  
Datasets. In unsupervised representation learning, we use a total of seven real-world datasets, including MUTAG, IMDB-B, IMDB-M, PROTEINS, COLLAB, REDDIT-B, and NCI1, which vary in content domains and dataset sizes. The datasets used can be downloaded from PyTorch Geometric (PyG) <cit.> [<https://github.com/pyg-team/pytorch_geometric>], and the dataset statistics are summarized in Table <ref>. 



  
Baselines. To demonstrate the effectiveness of our proposed method in unsupervised representation learning, we compare StructMAE with the following 10 baseline models:  
  
  1) Two Supervised Models:


  
  * GIN <cit.> is a graph neural network architecture designed for learning on graph-structured data. It employs a message passing scheme that incorporates neighborhood aggregation and graph isomorphism testing.
  
  * DiffPool <cit.> is a graph neural network architecture that addresses the flat nature of current GNN methods by introducing a differentiable graph pooling module, enabling hierarchical representations of graphs. 


 2) Six Contrastive Models:
 
 
  
  * InfoGraph <cit.> is a method for learning graph-level representations by maximizing mutual information between graph-level and substructure representations. 
  
  * GraphCL <cit.> is a framework for learning
unsupervised representations of graph data with various graph augmentations.
  
  * JOAO <cit.> is a unified bi-level optimization framework that could automatically selects augmentations, addressing the limitations of manual augmentation selection in GraphCL.
  
  * GCC <cit.> is a self-supervised pre-training framework for graph neural networks, designed to capture universal network topological properties across multiple datasets.
  
  * InfoGCL <cit.> is an information-aware graph contrastive learning framework following the Information Bottleneck principle to minimize information loss during graph representation learning.
  
  * SimGRACE <cit.> is a novel framework for graph contrastive learning using the original graph and a perturbed version as inputs to two correlated encoders.


3) Two Generative models:

 
  
  * GraphMAE <cit.> is a masked graph autoencoder designed to address issues negatively impacting the development of graph autoencoder (GAE) <cit.> in generative self-supervised learning on graphs. 
  
  * S2GAE <cit.> jointly reconstruct the masked edges and node degrees.
  
  
  
  

For most baselines, we refer to their implementations provided by the original paper. If unavailable, we utilize the implementation provided by PyG <cit.>. The links of codes are included in Table <ref>.

















 §.§ Transfer Learning



  
Datasets. In transfer learning, our model is initially pre-trained in two million unlabeled molecules sampled from the ZINC15  <cit.> and then finetuned in eight classification benchmark datasets contained in MoleculeNet <cit.>, including BBBP, Tox21, ToxCast, SIDER, ClinTox, MUV, HIV, and BACE. The datasets used can be downloaded from Pretrain-GNNs <cit.> [<https://github.com/snap-stanford/pretrain-gnns>], and the dataset statistics are summarized in Table <ref>. 

 

ZINC is a publicly available dataset developed for virtual screening, ligand discovery, pharamcophore screens, and benchmarking. ZINC15 is a new version published in 2015, containing over 120 million purchasable “drug-like” compounds represented as graphs. The task related to this dataset involves conducting regression analysis. 

 

MoleculeNet is a benchmark dataset specifically designed for evaluating machine learning methods on molecular properties. This comprehensive collection currently encompasses over 700,000 compounds tested across various properties. The tasks associated with this dataset can be evaluated through ROC-AUC, AUC-PRC, RMSE, and MAE scores.



  
Baselines. To demonstrate the effectiveness of our proposed method in transfer learning, we compare StructMAE with the following 10 baseline models: 

  1) Three Unsupervised models:


  
  * Infomax, AttrMasking and ContextPred are three unsupervised methods from Pretrain-GNNs <cit.>. It proposes various self-supervised methods for pre-training GNNs to address distributional differences and scarcity of task-specific labels, achieving great performance for molecular property prediction and protein function prediction.


 2) Four Contrastive models:
 
 
  
  * GraphCL <cit.> is a framework for learning
unsupervised representations of graph data with various graph augmentations.
  
  * JOAO <cit.> is a unified bi-level optimization framework that could automatically selects augmentations, addressing the limitations of manual augmentation selection in GraphCL.
  
  * GraphLOG <cit.> is a suite for studying the logical generalization capabilities of GNNs. 
  
  * RGCL <cit.> is a framework integrating invariant rationale discovery and rationale-aware pre-training.


3) Three Generative models:

 
  
  * GraphMAE <cit.> is a masked graph autoencoder designed to address issues negatively impacting the development of GAE <cit.> in generative self-supervised learning on graphs. 
  
  * GraphMAE2 <cit.> extends GraphMAE by employing strategies such as multi-view decoding and node sampling. 
  
  * Mole-BERT <cit.> represents a specialized pre-training framework designed for GNNs with a focus on molecular applications. At the heart of Mole-BERT is the innovative use of a variant of VQ-VAE, which is adeptly employed for the context-aware encoding of atom attributes.


  
  
  
  
  

For most baselines, we refer to their implementations provided by the original paper. If unavailable, we utilize the implementation provided by PyG <cit.>. The links of codes are included in Table <ref>.
















 








