Question: Represent the scientific article to find a short TLDR

Question: Graph Neural Networks (GNNs) have received tremendous attention recently due to their power in handling graph data for different downstream tasks across different application domains. The key of GNN is its graph convolutional filters, and recently various kinds of filters are designed. However, there still lacks in-depth analysis on Whether there exists a best filter that can perform best on all graph data; Which graph properties will influence the optimal choice of graph filter; How to design appropriate filter adaptive to the graph data. In this paper, we focus on addressing the above three questions. We first propose a novel assessment tool to evaluate the effectiveness of graph convolutional filters for a given graph. Using the assessment tool, we find out that there is no single filter as a `silver bullet' that perform the best on all possible graphs. In addition, different graph structure properties will influence the optimal graph convolutional filter's design choice. Based on these findings, we develop Adaptive Filter Graph Neural Network (AFGNN), a simple but powerful model that can adaptively learn task-specific filter. For a given graph, it leverages graph filter assessment as regularization and learns to combine from a set of base filters. Experiments on both synthetic and real-world benchmark datasets demonstrate that our proposed model can indeed learn an appropriate filter and perform well on graph tasks. Graph Neural Networks (GNNs) are a family of powerful tools for representation learning on graph data, which has been drawing more and more attention over the past several years. GNNs can obtain informative node representations for a graph of arbitrary size and attributes, and has shown great effectiveness in graph-related down-stream applications, such as node classification , graph classification (b), graph matching , recommendation systems , and knowledge graphs . As GNNs have superior performance in graph-related tasks, the question as to what makes GNNs so powerful is naturally raised. Note that GNNs adopt the concept of the convolution operation into graph domain. To obtain a representation of a specific node in a GNN, the node aggregates representations of its neighbors with a convolutional filter. For a task related to graph topology, the convolutional filter can help GNN nodes to get better task-specific representations . Therefore, it is the filter that makes GNNs powerful, and thus the key to designing robust and accurate GNNs is to design proper graph convolutional filters. Recently, many GNN architectures are proposed with their own graph filter designs. However, none of them have properly answered the following fundamental questions of GNNs: Is there a best filter that works for all graphs? If not, what are the properties of graph structure that will influence the performance of graph convolutional filters? Can we design an algorithm to adaptively find the appropriate filter for a given graph? In this paper, we focus on addressing the above three questions for semi-supervised node classification task. Inspired by studies in Linear Discriminant Analysis (LDA), we propose a Graph Filter Discriminant (GFD) Score metric to measure the power of a graph convolutional filter in discriminating node representations of different classes on a specific graph. We have analyzed all the existing GNNs' filters with this assessment method to answer the three aforementioned questions. We found that no single filter design can achieve optimal on all possible graphs. In other words, for different graph data, we should adopt different graph convolutional filters to achieve optimal performance. We then experimentally and theoretically analyze how graph structure properties influence the optimal choice of graph convolutional filters. Based on all of our findings, we propose the Adaptive Filter Graph Neural Network (AF-GNN), which can adaptively learn a proper model for the given graph. We use the Graph Filter Discriminant Score (GFD) as a an extra loss term to guide the network to learn a good data-specific filter, which is a linear combination of a set of base filters. We show that the proposed Adaptive Filter can better capture graph topology and separate features on both real-world datasets and synthetic datasets. We highlight our main contributions as follows: • We propose an assessment tool: Graph Filter Discriminant Score, to analyze the effectiveness of graph convolutional filters. Using this tool, we find that no best filter can work for all graphs, the optimal choice of a graph convolutional filter depends on the graph data. • We propose Adaptive Filter Graph Neural Network that can adaptively learn a proper filter for a specific graph using the GFD Score as guidance. • We show that the proposed model can find better filters and achieve better performance compared to existing GNNs, on both real-word and newly created benchmark datasets. Semi-Supervised Node Classification. Let Y be the class assignment vector for all the nodes in V. C indicates the total number of classes, and Y v ∈ {1, · · ·, C} indicates the class that node v belongs to. The goal of semi-supervised node classification is to learn a mapping function f: V → {1, · · ·, C} using the labeled nodes, and predict the class labels for the unlabeled nodes, i.e., Y v = f (v), by leveraging both node features X and graph structure A. Graph Data Generator. Intuitively, semi-supervised node classification requires both node features (X) and the graph structure (A) to be correlated to the intrinsic node labels (Y) to some extent. To systematically analyze the performance of different GNN filters, we test their performance under different graph data with different properties, i.e., graphs with different X, A, Y. Intuitively, both graph topology and node features have to be correlated with the node labels, if including both can enhance the performance of node classification task. To better understand the roles played by each component, we assume the graphical model to generate a graph data is as described in Fig. 1(a). To better disclose the relationship between different graph filters and properties of different graph data, we further make assumptions on how X and A are generated when Y is given, as it is difficult to obtain those properties from real-world data. Therefore, we study simulated data to support a thorough analysis. We now describe the generation of Y, X|Y, and A|Y respectively. Generating Y: Each node is randomly assigned with a class label with probability proportional to its class size. We assume each class c is associated with n c nodes. Generating X|Y: We assume that node features are sampled from a distribution determined by their corresponding labels. For example, we can sample node features of class c from a multivariate Gaussian distribution with the parameters conditioned on class c:. For another example, we can sample node features of class c from a circular distribution with radius r c and noise noise c conditioned on c. Generating A|Y: We follow the most classic class-aware graph generator, i.e. stochastic block model , to generate graph structure conditioned on class labels. SBM has several simple assumptions that edges are generated via Bernoulli distributions independently and the parameter of the Bernoulli distribution is determined by the classes of the corresponding pair of nodes v i and v j, i.e., A ij |Y i, Y j ∼ Ber(p YiYj), where p YiYj is a parameter determined by the two corresponding classes. In a simple two-class case, p = p 11 = p 22 denotes the probability that the linked pair belongs to the same class, while q = p 12 = p 21 denotes the probability that the linked pair belongs to different classes. We call p+q 2 the "density of graph", which controls the overall connectivity of a graph, and we call |p − q| the "density gap", which controls how closely the graph generated by SBM correlates with labels. We assume p ≥ q in all the following sections. Degree Corrected SBM , which is a variation of SBM, adds another parameter γ to control the "power-law coefficient" of degree distribution among nodes. Figure 1 Graph Convolutional Filters. By examining various GNN designs, we find that most of the GNN operators can fit into a unified framework, i.e., for the l-th layer: which describes the three-step process that involves: a graph convolutional operation (can also be regarded as feature propagation or feature smoothing) denoted as F(G)H (l−1), a linear transformation denoted by multiplying W, and a non-linear transformation denoted by σ(·). Clearly, the graph convolutional operation F(G)H (l−1) is the key step that helps GNNs to improve performance. Thus, to design a good GNN, a powerful graph convolutional filter F(G) is crucial. We analyze the effectiveness of graph filters for existing GNNs in the following. The work of GCN first adopts the convolutional operation on graphs and use the filter F(G) =D −1/2ÃD−1/2. Here,Ã = A + I is the self-augmented adjacency matrix, andD = diag(d 1, ...,d n) is the corresponding degree matrix, whered i = n j=1Ã ij. Some studies (a;) use a filter F(G) = (D −1/2ÃD−1/2) k that is similar in form to GCN's filter, but with a pre-defined exponent k greater than one. This would help a node to obtain information from its further neighbors without redundant computation cost. Several other studies propose to use sampling to speed up GNN training (b; ; a) ), which can be considered as a sparser version of GCN's filter. Another set of GNNs consider using a learnable graph convolutional filter. For example, and both propose to use F(G) = A+ I where is a learnable parameter to augment self-loop skip connection. Graph Attention Networks proposes to assign attention weight to different nodes in a neighborhood, which can be considered as a flexible learnable graph convolutional filter. Their graph filters applied on a feature matrix X can be considered as: ∀i, j, where N i is the neighborhood of node i, α is a learnable weight vector, and || indicates concatenation. In this section, we introduce a novel assessment tool for analyzing graph convolutional filters. We first review the Fisher score, which is widely used in Linear Discriminant Analysis to quantify the linear separability of two sets of features. With the Fisher score, we propose the Graph Filter Discriminant Score metric to evaluate the graph convolutional filter on how well it can separate nodes in different classes. Fisher Score. When coming to non-graph data, the Fisher Score is used to assess the linear separability between two classes. Given two classes of features X (i) and X (j), the Fisher Score is defined as the ratio of the variance between the classes (inter-class distance) to the variance within the classes (inner-class distance) under the best linear projection w of the original feature: where µ (i) and µ (j) denotes the mean vector of X (i) and X (j) respectively, Σ (i) and Σ (j) denotes the variance of X (i) and X (j) respectively, and w denotes the linear projection vector which we can understand as a rotation of the coordinate system, and the max w operation is to find the best direction in which these two class of nodes are most separable. As the numerator of J indicates interclass distance and the denominator of J indicates inner-class distance a larger value of J indicates higher separability. Note that for given features, we can directly get the closed form solution of the optimal w, with which Fisher Score could be deformed as: The detailed proof is provided in appendix A.2. Graph Filter Discriminant Score. As mentioned before, the key component that empowers GNNs is the graph convolutional filter F(G). Intuitively, an effective filter should make the representations of nodes in different classes more separable. Therefore, we propose to use Fisher Scores of the node representations before and after applying the graph convolutional filter in order to evaluate this filter. For each pair of classes (i, j), we define their Fisher Difference as, which is the difference of their Fisher Score of representations after applying the filter F(G) and their Fisher Score of initial representations. We then define the GFD Score for the filter F(G) with respect to feature matrix X as follows: where n c is the number of nodes in class c. Note that the GFD Score is a weighted sum of the Fisher Difference for each pair of classes. Intuitively, the larger the GFD score, the more effective is this corresponding filter to increase the separability of node features. The Fisher Score can be extended to evaluate non-linearly separable data in addition to linearly separable data. We claim the rationale of such measure by showing that the graph convolution can actually help non-linearly separable data to be linearly separable if the graph filter is chosen properly for a given graph. As shown in Figure 2 (a)∼(d), if we use a proper filter, the convolutional operation can transform three circular distributions, which are non-linearly separable, into three linearly separable clusters. Moreover, as shown in Figure 2 (e)∼(h), even if the original features of different classes are sampled from the same distribution, the proper graph convolutional filter can help to linearly separate the data. This phenomenon shows that if the graph structure (A) is correlated with the task (Y), a proper filter alone is powerful enough to empower GNNs with non-linearity, without any non-linear activation. This phenomenon is also supported by the promising of SGC (a), which removes all the non-linear activations in the GCN architecture. Therefore, we claim that the proposed GFD is a reasonable metric to evaluate a graph filter's effectiveness, and a good graph filter for a given graph should have a higher GFD score on that graph. With the help of the assessment tool, we now examine existing filters and try to answer the two fundamental questions: Is there a best filter that works for all graphs? If not, what are the properties of graph data that will influence the performance of graph convolutional filters? The GFD Score we introduced in the above section can be applied to any filter on any given graph. From Table 3, we can see that most of the current GNNs fall into the following filter family: {(Â) k }, where the baseÂ is a normalized adjacency matrix, and k is the order of the filter. Note that there are some other variants of GNN filters that do not fall into this family, for example, GAT, but the analysis is similar. Without loss of generality, we focus on analyzing this filter family. The two main components of this filter family are the normalization strategy (Â) and the order to use (k). For (h) F isher = 17.8018 Figure 2: Each row corresponds to a graph. The i-th column corresponds to the feature distribution after applying filter (D −1/2ÃD−1/2) i−1. Both graphs include three classes of same size and has structure generated by SBM (p = 0.6, q = 0.03). The first graph's feature follows a circular distribution with radius = 1, 0.9, 0.8 and Gaussian noise = 0.02 for each class. The second graph's feature follows a circular distribution with radius = 1 and Gaussian noise = 0.02 for all classes. simplicity, we study the roles of these two components separately, using our assessing tool to show whether there exists an optimal choice of filter for different graph data. If an optimal choice does not exist, we determine the factors that will influence our choice of component. Through the analysis, we choose SBM and DCSBM introduced previously to generate the structures of synthetic graphs, and choose multivariate Gaussian distributions to generate features of synthetic graphs. We focusing on the structure properties that influence the optimal choice of filter. We enumerate the hyper-parameters to generate graphs with different structure properties, including the power law coefficient (γ) that controls the power law degree distribution of the graph, label ratio (n1 n2) that indicates how balance are the classes of this graph, density (p+q 2) that indicates the overall connectivity of the graph, and density gap (|p − q|) that indicates structural separability of the graph. As these properties are significant for real-world graphs, our generated synthetic graphs can cover a large range of possible graph properties, and are representative for analyzing different filters. Analyzing Filter's Normalization Strategy. We consider three normalization strategies, including row normalization D −1 A, column normalization AD −1, and symmetric normalization D −1/2 AD −1/2. We calculate GFD scores of these three graph filters for graphs generated with different parameters. As shown in Figure 3, no single normalization strategy is optimal for all graphs. Here we give an empirical explanation to this phenomenon. Note that, with the same order, each filter has the same receptive field, and different normalization strategies affect only on how to assign weights to the neighboring nodes. The row normalization strategy simply takes the mean of features of the node's neighbors. Clearly, this would help to keep every node's new representations in the same range. On the contrary, column normalization and symmetric normalization, might keep a larger representation for higher-degree nodes. Using a column-normalized adjacency matrix as the base of the graph convolutional filter is similar to the PageRank algorithm. While a node propagates its features to neighbors, this normalization strategy takes its degree into account. Thus, column normalization can be helpful when the when node degree plays an important role for classification. Symmetric normalization combines the properties from both the row normalization and the column normalization. Even in the case where row normalization and column normalization do not perform well, symmetric normalization still leads to promising performance. We now examine which graph properties influence our choice of the optimal normalization strategy, which may vary per graph. We find that power law coefficient γ is an important factor that influences the choice of normalization. As shown in Figure 3, when power-law coefficient γ decreases (graph's extent of power-law grows), row normalization tends to have better performance. This is because row normalization helps to keep node representations in the same range, so that large representations of high degree nodes can be avoided. Therefore, it prevents nodes with similar degrees getting closer to each other and messing the classification tasks where node degrees are not important. We also find that the label ratio (n1 n2) matters. As shown in Figure 3, when the size of each class becomes more imbalanced, column normalization tends to work better. This is because column normalization better leverages degree property during representation smoothing, as nodes in largesize classes tend to have larger representation since they are more likely to have higher degree. This can help nodes within different classes become more separable. We then analyze what would be the best order for filters. With a highorder filter, a node can obtain information from its further neighbors, and thus the amount of information it receives during the feature propagation increases. But do we always need more information under any circumstances? The answer is no. Still, we find that, for different graphs, the order that in the best performance would be different 1. Since there is no best filter order for all the cases, we explore the factors that can influence the choice of order. We find that the density of graph and density gap between two classes have a big impact. As shown in Figure 4, when the density or density gap increases, the filter with higher order tends to be a better choice. We provide an intuitive explanation for this phenomenon as follows. Note that the feature propagation scheme is based on the assumption that nodes in the same class have a closer connection. On one hand, when the density increases, the connections between nodes are closer. Therefore, high-order filters can help gather richer information and thus reduce the variance of the obtained new node representations, so it helps nodes in the same class get smoother representations. On the other hand, when the density gap decreases, for a node, the size of neighbors within the same class becomes similar to the size of neighbors within different classes. Thus conducting high-order graph convolution operations will mix the representations of all nodes regardless of classes, which will make node classification more difficult. Based on previous analysis, we now answer the last question: Can we design an algorithm to adaptively find the appropriate filter for a given graph? We develop a simple but powerful model, the Adaptive Filter Graph Neural Network (AFGNN). For a given graph, AFGNNs can learn to combine an effective filter from a set of filter bases, guided by GDF Scores. Adaptive Filter Graph Neural Network (AFGNN). For simplicity, we only consider finding the optimal filter for one family of graph convolutional filters: where k is the maximum order. Note that, we also include the identity matrix, which serves as a skip-connection, to maintain the original feature representation. Based on our previous analysis, for graphs that are not closely correlated to tasks (i.e., small density gap in SBM), the identity matrix will outperform all the other convolutional filters. We denote the above 3k + 1 filters as, the l-th layer of AFGNN is defined as a learnable linear combination of these filter bases: where ψ (l) is the learnable vector to combine base filters and α (l) is its softmax-normalized version. Comparing to GNNs with fixed filters such as GCN and SGC, our proposed AFGNN can adaptively learn a filter based on any given graph. As we have shown that no single fixed filter can perform optimally for all graphs, we conclude that an adaptive filter has more capacity to learn better representations. Comparing to other GNNs with learnable filters such as GAT, AFGNN is computationally cheaper and achieves similar or better performance on most existing benchmarks and our synthetic datasets (as shown in the experiment section). We leave expanding the base filter family and adding more complex filters such as GAT into our filter bases as future work. Training Loss. To train this AFGNN model, we can simply optimize the whole model via any downstream tasks, i.e., node classification. However, as most of the semi-supervised node classification datasets only contain limited training data, the enlarged filter space will make the model prone to over-fitting. Thus, we decide to add the GFD Score as an loss term into the training loss to guide the optimization of filter weights, i.e., ψ (l) and to prevent overfitting: where L CE is the cross-entropy loss of the node classification, and L GF D is defined as the cumulative negation of GFD Score for the learned adaptive filter F AF GN N (G) (l) at each layer with respect to its input representation H (l−1). During the training, we minimize L to learn the proper model. With a different choice of the weight λ for GFD loss, we can categorize our model into: AFGNN 0: With λ = 0, the model is only trained by L CE, which might be prone to over-fitting when data is not sufficient. AFGNN 1: With λ = 1, the model is trained by both L CE and L GF D simultaneously. AFGNN ∞: This case is not exactly λ = ∞, and the training process is different from the other two cases. We implement the training iteratively: we optimize the combination of base filters by training only with GFD loss L GF D, then we optimize the linear transofrmation parameter W l s with classification loss L CE. Note that the input feature H = X is invariant, we can pre-train the optimal filter for first layer and fix it. Dataset We first evaluate AFGNN on three widely used benchmark datasets: Cora, Citeseer, and Pubmed . As these datasets are not sensitive enough to differentiate the models, we need more powerful datasets that can evaluate the pros and cons of each model. Based on our findings in section 3.2, we generate two synthetic benchmarks called SmallGap and SmallRatio. SmallGap corresponds to the case in which the density gap of the graph is close to 1. This indicates that the graph structure does not correlate much to the task, thus I would be the best filter in this case. SmallRatio corresponds to the case in which the label ratio is small, i.e. the size of one class is clearly smaller than the other, and column normalization AD −1 is the best normalization 2. Baselines and Settings. We compare against 5 baselines, including GCN, GIN, SGC, GFNN, and GAT. To make fair comparisons, for all the baseline GNNs, we set the number of layers (or orders) to be 2, and tune the parameters including learning rate, weight decay, and number of epochs 3. For all the benchmark datasets, we follow the data split convention 2. For the synthetic dataset, we conduct 5-fold cross-validation, randomly split the nodes into 5 groups of the same size, take one group as the training set, one as the validation set and the remaining three as the test set. Each time we pick Classification Performance. As is shown in Table 1, our proposed AFGNN ∞ model can consistently achieve competitive test accuracy. On Pubmed, SmallGap, and SmallRatio, AFGNN ∞ can achieve the best among all the baseline models. On Cora and Citeseer, though GAT outperforms our proposed model a little bit, however, as shown in Table 6,7, GAT takes a longer time to train and converge, and has more memory cost as well. Also, when the given graph is simple, GAT would suffer unavoidable overfitting problem. We further compare our AFGNN 0, AFGNN 1, AFGNN ∞ to examine the role of GFD loss. The AFGNN 0 performs quite poorly on all the datasets, implying that the larger search space of the filter without GFD loss is prone to over-fitting, while AFGNN 1 and AFGNN ∞ perform much better. Also, AGFNN ∞ has superior performance compared to AFGNN 1, which indicates the GFD Score is indeed a very powerful assessment tool. Graph Filter Discriminant Analysis. We are also interested to see whether the proposed method can indeed learn the best combination of filters from the base filter family. To do so, we calculate the GFD Score of the first-layer filter learned by AFGNN 0, AFGNN 1, AFGNN ∞ and the seven base filters on the test set for each dataset. For the AFGNN models, the filter is trained with the training set for each dataset. Table 2 4 and Figure 5 show the , we can see that our proposed method can indeed learn a combined filter on all the datasets. Specifically, in all the benchmark datasets, the best base filter is (D −1Ã) 2, and our proposed adaptive filter not only picks out the best base filter but also learns a better combination. For the two synthetic datasets, where I and (ÃD −1) 2 are the best filters, our algorithm can also learn to pick them out. We thereby conclude that the proposed GFD loss can help find an appropriate filter for a given dataset. Understanding the graph convolutional filters in GNNs is very important, as it can help to determine whether a GNN will work on a given graph, and can provide important guidance for GNN design. In our paper, we focus on the semi-supervised node classification task. We first propose the Graph Filter Discriminant Score as an assessment tool for graph convolutional filter evaluation, and then apply this GFD Score to analyze a family of existing filters as a case study. Using this tool, we learn that no single fixed filter can produce optimal on all graphs. We then develop a simple but powerful GNN model: Adapative Filter Graph Neural Network, which can learn to combine a family of filters and obtain a task-specific powerful filter. We also propose to add the negative GFD Score as an extra component to the objective function, it can act as a guidance for the model to learn a more effective filter. Experiments show that our approach outperforms many existing GNNs on both benchmark and synthetic graphs. Graph Convolutional Filters F(G) = Q, where Q is parametric attention function of X and A Table 3 summarized the graph filters for existing GNNs. Proof According to the in linear discriminant analysis, the maximum separation occurs when w ∝ (Note that, when we want to apply this fisher linear discriminant score in our problem, the linear transformation part in our classifier (and also the linear transformation part in GNN) will help to find the best w. Thus, we can directly plug the optimum solution w * = c into this formula, here c is a scalar. Then, we'll have: Thus we completed the proof. A.3.1 EXAMPLES OF "NO BEST NORMALIZATION STRATEGY FOR ALL" Figure 6 provides two examples to show there is no best normalization strategy for all graphs. For both examples, we fix the order of filter to be 2. The first row shows a case in which row normalization is better than the other two. The corresponding graph contains 2 classes of nodes with size 500. The graph structure is generated by DCSBM with p = 0.3, q = 0.05, power law coefficient γ = −0.9. The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (0.2,0.2) and respectively. In this example, we can clearly see that with other two normalization strategy, some high-degree hubs show up in the upper right corner from both class, which is harmful for classification. We generate this example to illustrate the benefit of row normalization because row normalization would be very helpful for a graph with power law degree distribution, which contains some nodes with unusually large degree (those nodes are called hubs), since it can help avoid those hubs obtaining larger representations and thus be mis-classified. The second row shows a case in which column normalization is better than the other two. The corresponding graph contains 2 classes of nodes with size 900 and 100 respectively. The graph structure is generated by SBM with p = 0.3, q = 0.2. The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (-0.2,-0.2) and (0.2,0.2) respectively. We generate this example to illustrate the benefit of column normalization because under this case, we should consider taking more degree information into consideration. Therefore, column normalization would be more helpful. Figure 7 provides two examples to show there is no best order for all graphs. For both examples, we fix the normalization strategy to be row normalization, and varies order to be 2, 4, 6. The first row shows a case in which small order is better than the large ones. The corresponding graph contains 2 classes of nodes with same size 500. The graph structure is generated by SBM with p = 0.215, q = 0.2. The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (0.5,0.5) and respectively. The second row shows a case in which large order is better than the smaller ones. The corresponding graph contains 2 classes of nodes with same size 500. The graph structure is generated by SBM with p = 0.75, q = 0.6. The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (0.5,0.5) and respectively. For the curves indicating how powerlaw coefficient influence the choice of normalization in Figure 3, we generate the corresponding graphs structure by DCSBM with fixed p = 0.3, q = 0.2 and varies the powerlaw coefficient from -0.3 to 0. The graph contains two classes of nodes, and is of size 400 and 600 for each class respectively. The feature for each class satisfies multivariate normal distribution with identity co-variance matrix, and with mean and (0.2,0.2). For the curves indicating how label ratio influence the choice of normalization in Figure 3, we generate the corresponding graphs structure by SBM with fixed p = 0.3, q = 0.1 and varies the label ratio. The graph contains a total number of 1000 nodes in two classes. The feature for each class satisfies multivariate normal distribution with identity co-variance matrix, and with mean and (0.5,0.5). For the curves indicating how density influence the choice of normalization in Figure 4, we generate the corresponding graphs structure by SBM with fixed density gap p/q = 1.5 and varies the density by varying q. The graph contains two classes of node of size 500. The feature for each class satisfies multivariate normal distribution with identity co-variance matrix, and with mean and (0.5,0.5). For the curves indicating how density gap influence the choice of normalization in Figure 4, we generate the corresponding graphs structure by SBM with fixed density p + q = 0.6 and varies the density gap. The graph contains two classes of node of size 500. The feature for each class satisfies multivariate normal distribution with identity co-variance matrix, and with mean (-0.2,-0.2) and (0.2,0.2). The following flowchart (Figure 8) describes the process of how a one-layer AFGNN tackle node classification task. We reduced the dimension of feature by t-SNE . We annotate the filter and the GFD Score in title of each subfigure. Note that, identity also corresponds to the initial feature. The figure is the feature representation obtained after conduct graph convolution operation once with the corresponding filter. We use three benchmark dataset: Cora, Citeseer and Pubmed for the node classification task. Their statictics are in table4. Beside number of nodes, edges, classes, the dimension of feature, and the data split strategy, we also show the class ratio variance, which can indicates if this dataset is imbalance or not, density gap, which indicates the dependency of structure and labels, and density, which indicates the overall connectivity of a graph. We provide the degree distribution in Figure 10, and we can clearly find that these benchmark datasets has power law degree distribution. Nodes 2708 3327 19717 Edges 5429 4732 44338 Classes 7 6 3 Feature 1433 3703 500 Train 140 120 60 Validation 500 500 500 Test 1000 1000 We tune the number of epochs based on convergence performance. For learning rate and weight decay, we follows the parameter setting provides by the corresponding public implementations unless we find better parameters. The tuned parameters can be found in our code resource. We report the accuracy of node classification task for baseline models on Cora, Citeseer, and Pubmed provided by corresponding literature. Since GIN is not originally evaluated on node classification task, we do not have the reported number here. The is in Table 5. A.9 TIME AND MEMORY COST COMPARISON Both our AFGNN model and GAT model have a learnable filter. We provide time and memory complexity comparison on benchmark datasets here to compare these two models. As shown in Table 6, GAT's time cost is at least three times of AFGNN's time cost on both Cora and Citeseer dataset. As shown in Table 8: Performance on OAG SmallRatio Dataset because it requires too much memory cost and is not able to run on GPU. Therefore, AFGNN needs less time and memory cost than GAT. We generate a real-world dataset with imbalanced classes to justify hard cases may exist in realworld datasets. We download a large scale academic graph called Open Academic Graph (OAG), and choose two fields that have a large disparity in the number of papers: "History of ideas", which consists of 1041 papers; "Public history", which consists of 150 papers. Obviously this two classes are imbalanced, and fall in the large label ratio gap problem. We run supplementary experiment on the generated OAG graph, the experiment setting remains the same as experiment settings for synthetic graphs. Table 8 shows the experiment . To evaluate the models, we compare their F1 score for each class, the weighted average F1 score (micro F1), and the average F1 score (macro F1). Our AFGNN ∞ model shows superior performance on this dataset.

Propose an assessment framework to analyze and learn graph convolutional filter

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: The advance of node pooling operations in Graph Neural Networks (GNNs) has lagged behind the feverish design of new message-passing techniques, and pooling remains an important and challenging endeavor for the design of deep architectures. In this paper, we propose a pooling operation for GNNs that leverages a differentiable unsupervised loss based on the minCut optimization objective. For each node, our method learns a soft cluster assignment vector that depends on the node features, the target inference task (e.g., a graph classification loss), and, thanks to the minCut objective, also on the connectivity structure of the graph. Graph pooling is obtained by applying the matrix of assignment vectors to the adjacency matrix and the node features. We validate the effectiveness of the proposed pooling method on a variety of supervised and unsupervised tasks. A fundamental component in deep convolutional neural networks is the pooling operation, which replaces the output of convolutions with local summaries of nearby points and is usually implemented by maximum or average operations . State-of-the-art architectures alternate convolutions, which extrapolate local patterns irrespective of the specific location on the input signal, and pooling, which lets the ensuing convolutions capture aggregated patterns. Pooling allows to learn abstract representations in deeper layers of the network by discarding information that is superfluous for the task, and keeps model complexity under control by limiting the growth of intermediate features. Graph Neural Networks (GNNs) extend the convolution operation from regular domains, such as images or time series, to data with arbitrary topologies and unordered structures described by graphs . The development of pooling strategies for GNNs, however, has lagged behind the design of newer and more effective message-passing (MP) operations , such as graph convolutions, mainly due to the difficulty of defining an aggregated version of the original graph that supports the pooled signal. A naïve pooling strategy in GNNs is to average all nodes features , but it has limited flexibility since it does not extract local summaries of the graph structure, and no further MP operations can be applied afterwards. An alternative approach consists in pre-computing coarsened versions of the original graph and then fit the data to these deterministic structures . While this aggregation accounts for the connectivity of the graph, it ignores task-specific objectives as well as the node features. In this paper, we propose a differentiable pooling operation implemented as a neural network layer, which can be seamlessly combined with other MP layers (see Fig. 1). The parameters in the pooling layer are learned by combining the task-specific loss with an unsupervised regularization term, which optimizes a continuous relaxation of the normalized minCUT objective. The minCUT identifies dense graph components, where the nodes features become locally homogeneous after the message-passing. By gradually aggregating these components, the GNN learns to distil global properties from the graph. The proposed minCUT pooling operator (minCUTpool) yields partitions that 1) cluster together nodes which have similar features and are strongly connected on the graph, and 2) take into account the objective of the downstream task. minCUT Pooling Message-passing Figure 1: A deep GNN architecture where message-passing is followed by minCUT pooling. Given a graph G = {V, E}, |V| = N, and the associated adjacency matrix A ∈ R N ×N, the K-way normalized minCUT (simply referred to as minCUT) aims at partitioning V in K disjoint subsets by removing the minimum volume of edges. The problem is equivalent to maximizing where the numerator counts the edge volume within each cluster, and the denominator counts the edges between the nodes in a cluster and the rest of the graph . Let C ∈ R N ×K be a cluster assignment matrix, so that C i,j = 1 if node i belongs to cluster j, and 0 otherwise. The minCUT problem can be expressed as where D = diag(A1 N) is the degree matrix . Since problem is NP-hard, it is usually recast in a relaxed formulation that can be solved in polynomial time and guarantees a near-optimal solution : While the optimization problem is still non-convex, there exists an optimal solution Q * = U K O, where U K ∈ R N ×K contains the eigenvectors of A corresponding to the K largest eigenvalues, and O ∈ R K×K is an orthogonal transformation . Since the elements of Q * are real values rather than binary cluster indicators, the spectral clustering (SC) approach can be used to find discrete cluster assignments. In SC, the rows of Q * are treated as node representations embedded in the eigenspace of the Laplacian, and are clustered together with standard algorithms such as k-means . One of the main limitations of SC lies in the computation of the spectrum of A, which has a memory complexity of O(N 2) and a computational complexity of O(N 3). This prevents its applicability to large datasets. To deal with such scalability issues, the constrained optimization in can be solved by gradient descent algorithms that refine the solution by iterating operations whose individual complexity is O(N 2), or even O(N) . Those algorithms search the solution on the manifold induced by the orthogonality constraint on the columns of Q, by performing gradient updates along the geodesics . Alternative approaches rely on the QR factorization to constrain the space of feasible solutions , and alleviate the cost O(N 3) of the factorization by ensuring that orthogonality holds only on one minibatch at a time . Other works based on neural networks include an autoencoder trained to map the ith row of the Laplacian to the ith components of the first K eigenvectors, to avoid the spectral decomposition . use a soft orthogonality constraint to learn spectral embeddings as a volumetric reparametrization of a precomputed Laplacian eigenbase.; propose differentiable loss functions to partition generic data and process out-of-sample data at inference time. generate balanced node partitions with a GNN, but adopt an optimization that does not encourage cluster assignments to be orthogonal. Many approaches have been proposed to process graphs with neural networks, including recurrent architectures or convolutional operations inspired by filters used in graph signal processing (; . Since our focus is on graph pooling, we base our GNN implementation on a simple MP operation, which combines the features of each node with its 1st-order neighbors. To account for the initial node features, it is possible to introduce self-loops by adding a (scaled) identity matrix to the diagonal of A . Since our pooling will modify the structure of the adjacency matrix, we prefer a MP implementation that leaves the original A unaltered and accounts for the initial node features by means of skip connections. N ×N be the symmetrically normalized adjacency matrix and X ∈ R N ×F the matrix containing the node features. The output of the MP layer is where Θ M P = {W m, W s} are the trainable weights relative to the mixing and skip component of the layer, respectively. The minCUT pooling strategy computes a cluster assignment matrix S ∈ R N ×K by means of a multi-layer perceptron, which maps each node feature x i into the ith row of S: where Θ P ool = {W 1 ∈ R F ×H, W 2 ∈ R H×K} are trainable parameters. The softmax function guarantees that s i,j ∈ and enforces the constraints S1 K = 1 N inherited from the optimization problem in. The parameters Θ M P and Θ P ool are jointly optimized by minimizing the usual task-specific loss, as well as an unsupervised loss L u, which is composed of two terms where · F indicates the Frobenius norm. The cut loss term, L c, evaluates the minCUT given by the cluster assignment S, and is bounded by −1 ≤ L c ≤ 0. Minimizing L c encourages strongly connected nodes to be clustered together, since the inner product s i, s j increases whenã i,j is large. L c has a single maximum, reached when the numerator T r(This occurs if, for each pair of connected nodes (i.e.,ã i,j > 0), the cluster assignments are orthogonal (i.e., s i, s j = 0). L c reaches its minimum, −1, when T r(S TÃ S) = T r(S TD S). This occurs when in a graph with K disconnected components the cluster assignments are equal for all the nodes in the same component and orthogonal to the cluster assignments of nodes in different components. However, L c is a non-convex function and its minimization can lead to local minima or degenerate solutions. For example, given a connected graph, a trivial optimal solution is the one that assigns all nodes to the same cluster. As a consequence of the continuous relaxation, another degenerate minimum occurs when the cluster assignments are all uniform, that is, all nodes are equally assigned to all clusters. This problem is exacerbated by prior message-passing operations, which make the node features more uniform. The orthogonality loss term, L o, penalizes the degenerate minima of L c by encouraging the cluster assignments to be orthogonal and the clusters to be of similar size. Since the two matrices in L o have unitary norm it is easy to see that 0 ≤ L o ≤ 2. Therefore, L o does not dominate over L c and the two terms can be safely summed directly (see Fig. 4 for an example). I K can be interpreted as a (rescaled) clustering matrix I K =Ŝ TŜ, whereŜ assigns exactly N/K points to each cluster. The value of the Frobenius norm between clustering matrices is not dominated by the performance on the largest clusters and, thus, can be used to optimize intra-cluster variance. Contrarily to SC methods that search for feasible solutions only within the space of orthogonal matrices, L o only introduces a soft constraint that could be violated during the learning procedure. Since L c is non-convex, the violation compromises the theoretical guarantee of convergence to the optimum of. However, we note that: 1. the cluster assignments S are well initialized: after the MP operation, the features of the connected vertices become similar and, since the MLP is a smooth function , it yields similar cluster assignments for those vertices; 2. in the GNN architecture, the minCUT objective is a regularization term and, therefore, a solution which is sub-optimal for could instead be adequate for the specific objective of the downstream task; 3. optimizing the task-specific loss helps the GNN to avoid the degenerate minima of L c. The coarsened version of the adjacency matrix and the graph signal are computed as where the entry x pool i,j in X pool ∈ R K×F is the weighted average value of feature j among the elements in cluster i. A pool ∈ R K×K is a symmetric matrix, whose entries a are the total number of edges between the nodes in the cluster i, while a pool i,j is the number of edges between cluster i and j. Since A pool corresponds to the numerator of L c in, the trace maximization yields clusters with many internal connections and weakly connected to each other. Hence, A pool will be a diagonal-dominant matrix, which describes a graph with self-loops much stronger than any other connection. Because self-loops hamper the propagation across adjacent nodes in the MP operations following the pooling layer, we compute the new adjacency matrixÃ pool by zeroing the diagonal and by applying the degree normalization where diag(·) returns the matrix diagonal. The proposed method is straightforward to implement: the cluster assignments, the loss, graph coarsening, and feature pooling are all computed with standard linear algebra operations. There are several differences between minCUTpool and classic SC methods. SC partitions the graph based on the Laplacian, but does not account for the node features. Instead, the cluster assignments s i found by minCUTpool depend on x i, which works well if connected nodes have similar features. This is a reasonable assumption in GNNs since, even in disassortative graphs (i.e., networks where dissimilar nodes are likely to be connected ), the features tend to become similar due to the MP operations. Another difference is that SC handles a single graph and is not conceived for tasks with multiple graphs to be partitioned independently. Instead, thanks to the independence of the model parameters from the number of nodes N and from the graph spectrum, minCUTpool can generalize to outof-sample data. This feature is fundamental in problems such as graph classification, where each sample is a graph with a different structure, and allows to train the model on small graphs and process larger ones at inference time. Finally, minCUTpool directly uses the soft cluster assignments rather than performing k-means afterwards. Trainable pooling methods. Similarly to our method, these approaches learn how to generate coarsened version of the graph through differentiable functions, which take as input the nodes features X and are parametrized by weights optimized on the task at hand. Diffpool is a pooling module that includes two parallel MP layers: one to compute the new node features X (t+1) and another to generate the cluster assignments S. Diffpool implements an unsupervised loss that consists of two terms. First, the link prediction term A − SS T F minimizes the Frobenius norm of the difference between the adjacency and the Gram matrix of the cluster assignments, encouraging nearby nodes to be clustered together. The second term 1 N N i=1 H(S i) minimizes the entropy of the cluster assignments to make them alike to one-hot vectors. Like minCUTpool, Diffpool clusters the vertices of annotated graphs, but yields completely different partitions, since it computes differently the clustering assignments, the coarsened adjacency matrix and, most importantly, the unsupervised loss. In Diffpool, such a loss shows pathological behaviors that are discussed later in the experiments. The approach dubbed Top-K pooling , learns a projection vector that is applied to each node feature to obtain a score. The nodes with the K highest scores are retained, the others are dropped. Since the top-K selection is not differentiable, the scores are also used as a gate/attention for the node features, letting the projection vector to be trained with backpropagation. Top-K is memory efficient as it avoids generating cluster assignments. To prevent A from becoming disconnected after nodes removal, Top-K drops the rows and the columns from A 2 and uses it as the new adjacency matrix. However, computing A 2 costs O(N 2) and it is inefficient to implement with sparse operations. Topological pooling methods. These methods pre-compute a pyramid of coarsened graphs, only taking into account the topology (A), but not the node features (X). During training, the node features are pooled with standard procedures and are fit into these deterministic graph structures. These methods are less flexible, but provide a stronger bias that can prevent degenerate solutions (e.g., coarsened graphs collapsing in a single node). The approach proposed by , which has been adopted also in other GNN architectures , exploits GRACLUS , a hierarchical algorithm based on SC. At each pooling level l, GRACLUS indetifies the pairs of maximally similar nodes i l and j l to be clustered together into a new vertex k (l+1). At inference phase, max-pooling is used to determine which node in the pair is kept. Fake vertices are added so that the number of nodes can be halved each time, but this injects noisy information in the graph. Node decimation is a method originally proposed in graph signal processing literature , which as been adapted also for GNNs . The nodes are partitioned in two sets, according to the signs of the Laplacian eigenvector associated to the largest eigenvalue. One of the two sets is dropped, reducing the number of nodes each time approximately by half. Kron reduction is used to compute a pyramid of coarsened Laplacians from the remaining nodes. A procedure proposed in diffuses a signal from designated nodes on the graph and stores the observed sequence of diffused components. The ing stream of information is interpreted as a time signal, where standard CNN pooling is applied. We also mention a pooling operation for coarsening binary unweighted graphs by aggregating maximal cliques . Nodes assigned to the same clique are summarized by max or average pooling and become a new node in the coarsened graph. We consider both supervised and unsupervised tasks, and compare minCUTpool with other GNN pooling strategies. The Appendix provides further details on the experiments and a schematic depiction of the architectures used in each task. In addition, the Appendix reports two additional experiments: i) graph reconstruction by means of an Auto Encoder with bottleneck, implemented with pooling and un-pooling layers, ii) an architecture with pooling for graph regression. To study the effectiveness of the proposed loss, we perform different node clustering tasks with a simple GNN composed of a single MP layer followed by a pooling layer. The GNN is trained by minimizing L u only, so that its effect is evaluated without the "interference" of a supervised loss. Clustering on synthetic networks We consider two simple graphs: the first is a network with 6 communities and the second is a regular grid. The adjacency matrix A is binary and the features X are the 2-D node coordinates. Fig. 2 depicts the node partitions generated by SC (a, d), Diffpool (b, e), and minCUTpool (c, f). Cluster indexes for Diffpool and minCUTpool are obtained by taking the argmax of S row-wise. Compared to SC, Diffpool and minCUTpool leverage the information contained in X. minCUTpool generates very accurate and balanced partitions, demonstrating that the cluster assignment matrix S is well formed. On the other hand, Diffpool assigns some nodes to the wrong community in the first example, and produces an imbalanced partition of the grid. Image segmentation Given an image, we build a Region Adjacency Graph (Trémeau &) using as nodes the regions generated by an oversegmentation procedure . The SC technique used in this example is the recursive normalized cut , which recursively clusters the nodes until convergence. For Diffpool and minCUTpool, we include node features consisting of the average and total color in each oversegmented region. We set the number of desired clusters to K = 4. The in Fig. 3 show that minCUTpool yields a more precise segmentation. On the other hand, SC and Diffpool aggregate wrong regions and, in addition, SC finds too many segments. Clustering on citation networks We cluster the nodes of three popular citation networks: Cora, Citeseer, and Pubmed. The nodes are documents represented by sparse bag-of-words feature vectors stored in X and the binary undirected edges in A indicate citation links between documents. Each node i is labeled with the document class y i. Once the training is over, to test the quality of the partitions generated by each method we check the agreement between the cluster assignments and the true class labels. Tab. 1 reports the Completeness Score CS(ỹ, y) = 1 −, where H(·) is the entropy. The GNN architecture configured with minCUTpool achieves a higher NMI score than SC, which does not account for the node features X when generating the partitions. Our pooling operation outperforms also Diffpool, since the minimization of the unsupervised loss in Diffpool yields degenerate solutions. The pathological behavior is shown in Fig. 4, which depicts the evolution of the NMI scores as the unsupervised losses in Diffpool and minCUTpool are minimized in training. In this task, the i-th datum is a graph with N i nodes represented by a pair {A i, X i} and must be associated to the correct label y i. We test the models on different graph classification datasets. For featureless graphs, we used the node degree information and the clustering coefficient as surrogate node features. We evaluate model performance with a 10-fold train/test split, using 10% of the training set in each fold as validation for early stopping. We adopt a fixed network architecture, MP-pool-MP-pool-MP-GlobalAvgPool-softmax, where MP is the message-passing operation in with 32 hidden units. The pooling module is implemented either by Graclus, Decimation pooling, Top-K, SAGPool , Diffpool, or the proposed minCUTpool. Each pooling method is configured to drop half of the nodes in a graph (K = N/2 in Top-K, Diffpool, and minCUTpool). As baselines, we consider the popular Weisfeiler-Lehman (WL) graph kernel , a network with only MP layers (Flat), and a fully connected network (Dense). Tab. 2 reports the classification , highlighting those that are significantly better (p-value < 0.05 w.r.t. the method with the highest mean accuracy). The comparison with Flat helps to understand if a pooling operation is useful or not. The of Dense, instead, help to quantify how much additional information is brought by the graph structure, with respect to the node features alone. It can be seen that minCUTpool obtains always equal or better with respect to every other GNN architecture. On the other hand, some pooling procedures do not always improve the performance compared to the Flat baseline, making them not advisable to use in some cases. The WL kernel generally performs worse than the GNNs, except for the Mutagenicity dataset. This is probably because Mutagenicity has smaller graphs than the other datasets, and the adopted GNN architecture is overparametrized for this task. Interestingly, in some dataset such as Proteins and COLLAB it is possible to obtain fairly good classification accuracy with the Dense architecture, meaning that the graph structure only adds limited information. TopK DiffPool minCUT Graclus Decim. Graclus and Decimation are understandably the fastest methods, since the coarsened graphs are precomputed. Among the differentiable pooling methods, minCUTpool is faster than Diffpool, which uses a slower MP layer rather than a MLP to compute cluster assignments, and than Top-K, which computes the square of A at every forward pass. We proposed a pooling layer for GNNs that coarsens a graph by taking into account both the the connectivity structure and the node features. The layer optimizes a regularization term based on the minCUT objective, which is minimized in conjunction with the task-specific loss to produce node partitions that are optimal for the task at hand. We tested the effectiveness of our pooling strategy on unsupervised node clustering tasks, by optimizing only the unsupervised clustering loss, as well as supervised graph classification tasks on several popular benchmark datasets. Results show that minCUTpool performs significantly better than existing pooling strategies for GNNs. To compare the amount of information retained by the pooling layers in the coarsened graphs, we train an autoencoder (AE) to reconstruct a input graph signal X from its pooled version. The AE architecture is MP-MP-pool-unpool-MP-MP-MP, and is trained by minimizing the mean squared error between the original and the reconstructed graph signal, X − X rec 2. All the pooling operations are configured to retain 25% of the original nodes. In Diffpool and minCUTpool, the unpool step is simply implemented by transposing the original pooling operations Top-K does not generate a cluster assignment matrix, but returns a binary mask m = {0, 1} N that indicates the nodes to drop or to retain. Therefore, an upsamplig matrix U is built by dropping the columns of the identity matrix I N that correspond to a 0 in m, U = [I N]:,m==1. The unpooling operation is performed by replacing S with U in, and the ing upscaled graph is a version of the original graph with zeroes in correspondence of the dropped nodes. 6 and 7 report the original graph signal X (the node features are the 2-D coordinates of the nodes) and the reconstruction X rec obtained by using the different pooling methods, for a ring graph and a regular grid graph. The reconstruction produced by Diffpool is worse for the ring graph, but is almost perfect for the grid graph, while minCUTpool yields good in both cases. On the other hand, Top-K clearly fails in generating a coarsened representation that maintains enough information from the original graph. This experiment highlights a major issue in Top-K pooling, which retains the nodes associated to the highest K values of a score vector s, computed by projecting the node features onto a trainable vector p: s = Xp. Nodes that are connected on the graph usually share similar features, and their similarity further increases after the MP operations, which combine the features of neighboring nodes. Retaining the nodes associated to the top K scores in s corresponds to keeping those nodes that are alike and highly connected, as it can be seen in Fig. 6-7. Therefore, Top-K discards entire portions of the graphs, which might contain important information. This explains why Top-K fails to recover the original graph signal when used as bottleneck for the AE, and yields the worse performance among all GNN methods in the graph classification task. The QM9 chemical database is a collection of ≈135k small organic molecules, associated to continuous labels describing several geometric, energetic, electronic, and thermodynamic properties 1. Each molecule in the dataset is represented as a graph {A i, X i}, where atoms are associated to nodes, and edges represent chemical bonds. The atomic number of each atom (one-hot encoded; C, N, F, O) is taken as node feature and the type of bond (one-hot encoded; single, double, triple, aromatic) can be used as edge attribute. In this experiment, we ignore the edge attributes in order to use all pooling algorithms without modifications. The purpose of this experiment is to compare the trainable pooling methods also on a graph regression task, but it must be intended as a proof of concept. In fact, the graphs in this dataset are extremely small (the average number of nodes is 8) and, therefore, a pooling operation is arguably not necessary. We consider a GNN with architecture MP-pool-MP-GlobalAvgPool-Dense, where pool is implemented by Top-K, Diffpool, or minCUTpool. The network is trained to predict a given chemical property from the input molecular graphs. Performance is evaluated with a 10-fold cross-validation, using 10% of the training set for validation in each split. The GNNs are trained for 50 epochs, using Adam with learning rate 5e-4, batch size 32, and ReLU activations. We use the mean squared error (MSE) as supervised loss. The MSE obtained on the prediction of each property for different pooling methods is reported in Tab. 3. As expected, the flat baseline with no pooling operation (MP-MP-GlobalAvgPoolDense) yields a lower error in most cases. Contrarily to the graph classification and the AE task, Top-K achieves better than Diffpool in average. Once again, minCUTpool significantly outperforms the other methods on each regression task and, in one case, also the flat baseline. Table 3: MSE on the graph regression task. The best with a statistical significance of p < 0.05 are highlighted: the best overall are in bold, the best among pooling methods are underlined. For the WL kernel, we used the implementation provided in the GraKeL library 2. The pooling strategy based on Graclus, is taken from the ChebyNets repository 3. Diffpool and minCUTpool are configured with 16 hidden neurons with linear activations in the MLP and MP layer, respectively used to compute the cluster assignment matrix S. The MP layer used to compute the propagated node features X uses an ELU activation in both architectures. The learning rate for Adam is 5e-4, and the models are trained for 10000 iterations. The details of the citation networks dataset are reported in Tab. 4. We train the GNN architectures with Adam, an L 2 penalty loss with weight 1e-4, and 16 hidden units (H) both in the MLP of minCUTpool and in the internal MP of Diffpool. Mutagenicity, Proteins, DD, COLLAB, and Reddit-2k are datasets representing real-world graphs and are taken from the repository of benchmark datasets for graph kernels 4. Bench-easy and Bench-hard 5 are datasets where the node features X and the adjacency matrix A are completely uninformative if considered alone. Hence, algorithms that account only for the node features or the graph structure will fail to classify the graphs. Since Bench-easy and Bench-hard come with a train/validation/test split, the 10-fold split is not necessary to evaluate the performance. The statistics of all the datasets are reported in Tab. 5. Fig. 8 reports the schematic representation of the minCUTpool layer; Fig. 9 the GNN architecture used in the clustering and segmentation tasks; Fig. 10 the GNN architecture used in the graph classification task; Fig. 12 the GNN architecture used in the graph regression task; Fig. 11 the graph autoencoder used in the graph signal reconstruction task.

Represent the short summary for finding the corresponding scientific article (long).

A new pooling layer for GNNs that learns how to pool nodes, according to their features, the graph connectivity, and the dowstream task objective.

Represent the TLDR for finding the corresponding scientific article!

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Knowledge graphs are structured representations of real world facts. However, they typically contain only a small subset of all possible facts. Link prediction is the task of inferring missing facts based on existing ones. We propose TuckER, a relatively simple yet powerful linear model based on Tucker decomposition of the binary tensor representation of knowledge graph triples. By using this particular decomposition, parameters are shared between relations, enabling multi-task learning. TuckER outperforms previous state-of-the-art models across several standard link prediction datasets. Vast amounts of information available in the world can be represented succinctly as entities and relations between them. Knowledge graphs are large, graph-structured databases which store facts in triple form (e s, r, e o), with e s and e o representing subject and object entities and r a relation. However, far from all available information is stored in existing knowledge graphs, which creates the need for algorithms that automatically infer missing facts. Knowledge graphs can be represented by a third-order binary tensor, where each element corresponds to a triple, 1 indicating a true fact and 0 indicating the unknown (either a false or a missing fact). The task of link prediction is to infer which of the 0 entries in the tensor are indeed false, and which are missing but actually true. A large number of approaches to link prediction so far have been linear, based on various methods of factorizing the third-order binary tensor BID12 BID22 BID19 BID7. Recently, state-of-the-art have been achieved using non-linear convolutional models BID3 BID0. Despite achieving very good per- formance, the fundamental problem with deep, non-linear models is that they are non-transparent and poorly understood, as opposed to more mathematically principled and widely studied tensor decomposition models. In this paper, we introduce TuckER (E stands for entities, R for relations), a simple linear model for link prediction in knowledge graphs, based on Tucker decomposition BID21 of the binary tensor of triples. Tucker decomposition factorizes a tensor into a core tensor multiplied by a matrix along each mode. In our case, rows of the matrices contain entity and relation embeddings, while entries of the core tensor determine the level of interaction between them. Due to having the core tensor, unlike simpler models, such as RESCAL, DistMult and ComplEx, where parameters for each relation are often learned separately, TuckER makes use of multi-task learning between different relations BID24. Subject and object entity embedding matrices are assumed equivalent, i.e. we make no distinction between the embeddings of an entity depending on whether it appears as a subject or as an object in a particular triple. Our experiments show that TuckER achieves state-of-the-art across all standard link prediction datasets. Several linear models for link prediction have previously been proposed. An early linear model, RESCAL BID12, optimizes a scoring function containing a bilinear product between subject and object entity vectors and a full rank matrix for each relation. RESCAL is prone to overfitting due to its large number of parameters, which increases quadratically in the embedding dimension with the number of relations in a knowledge graph. DistMult BID22 ) is a special case of RESCAL with a diagonal matrix per relation, which reduces overfitting. However, DistMult cannot model asymmetric relations. ComplEx BID19 extends DistMult to the complex domain. Subject and object entity embeddings for the same entity are complex conjugates, which enables ComplEx to model asymmetric relations. SimplE BID7 is a model based on Canonical Polyadic (CP) decomposition BID5.Scoring functions of all models described above and TuckER are summarized in Table 1. Table 1. Scoring functions of state-of-the-art link prediction models, the dimensionality of their relation parameters, and significant terms of their space complexity. de and dr are the dimensionalities of entity and relation embeddings, while ne and nr denote the number of entities and relations respectively. eo ∈ C de is the complex conjugate of eo, he s, te s ∈ R de are the head and tail entity embedding of entity es, and w r −1 ∈ R dr is the embedding of relation r −1 which is the inverse of relation r. · denotes the dot product and ×n denotes the tensor product along the n-th mode, f is a non-linear function, and W ∈ R de×de×dr is the core tensor of a Tucker decomposition. Relation Parameters Space Complexity BID22 e s, w r, e o w r ∈ R de O(n e d e + n r d e) ComplEx BID19 Re(e s, w r, e o) DISPLAYFORM0 DISPLAYFORM1 Let E denote the set of all entities and R the set of all relations present in a knowledge graph. A triple is represented as (e s, r, e o), with e s, e o ∈ E denoting subject and object entities respectively and r ∈ R the relation between them. In link prediction, we are given a subset of all true triples and the aim is to learn a scoring function φ that assigns a score s = φ(e s, r, e o) ∈ R which indicates whether a triple is true, with the ultimate goal of being able to correctly score all missing triples. The scoring function is either a specific form of tensor factorization in the case of linear models or a more complex (deep) neural network architecture for nonlinear models. Typically, a positive score for a particular triple indicates a true fact predicted by the model, while a negative score indicates a false one. Tucker decomposition, named after Ledyard R. Tucker BID20, decomposes a tensor into a set of matrices and a smaller core tensor. In a three-mode case, given the original tensor X ∈ R I×J×K, Tucker decomposition outputs a tensor Z ∈ R P ×Q×R and three matrices A ∈ R I×P, B ∈ R J×Q, C ∈ R K×R: DISPLAYFORM0 with × n indicating the tensor product along the n-th mode. Elements of the core tensor Z show the level of interaction between the different components. Typically, P, Q, R are smaller than I, J, K respectively, so Z can be thought of as a compressed version of X BID9 ). We propose a model that uses Tucker decomposition for link prediction on the binary tensor representation of a knowledge graph, with entity embedding matrix E that is equivalent for subject and object entities, i.e. E = A = C ∈ R ne×de and relation embedding matrix R = B ∈ R nr×dr, where n e and n r represent the number of entities and relations and d e and d r the dimensionality of entity and relation embedding vectors respectively. We define the scoring function for TuckER as: DISPLAYFORM0 where e s, e o ∈ R de are the rows of E representing the subject and object entity embedding vectors, w r ∈ R dr the rows of R representing the relation embedding vector and W ∈ R de×dr×de is the core tensor. We apply logistic sigmoid to each score φ(e s, r, e o) to obtain the predicted probability p of a triple being true. Visualization of the TuckER model architecture can be seen in FIG1. The number of parameters of TuckER increases linearly with respect to entity and relation embedding dimensionality d e and d r, as the number of entities and relations increases, since the number of parameters of W depends only on the entity and relation embedding dimensionality and not on the number of entities or relations. By having the core tensor W, unlike simpler models such as DistMult, ComplEx and SimplE, TuckER does not encode all the learned knowledge into the embeddings; some is stored in the core tensor and shared between all entities and relations through multi-task learning BID24. Following the training procedure introduced by BID3, we use 1-N scoring, i.e. we simultaneously score a pair (e s, r) with all entities e o ∈ E, in contrast to 1-1 scoring, where individual triples (e s, r, e o) are trained one at a time. We assume that a knowledge graph is only locally complete by including only the non-existing triples (e s, r, ·) and (·, r, e o) of the observed pairs (e s, r) and (r, e o) as negative samples and all observed triples as positive samples. We train our model to minimize the Bernoulli negative loglikelihood loss function. A component of the loss for one subject entity and all the object entities is defined as: DISPLAYFORM0 where p ∈ R ne is the vector of predicted probabilities and y ∈ R ne is the binary label vector. We evaluate TuckER using standard link prediction datasets. FB15k BID1 ) is a subset of Freebase, a large database of real world facts. FB15k-237 BID18 was created from FB15k by removing the inverse of many relations that are present in the training set from validation and test sets. WN18 BID1 ) is a subset of WordNet, containing lexical relations between words. WN18RR BID3 ) is a subset of WN18, created by removing the inverse relations. We implement TuckER in PyTorch BID13 and make our code available on Github 1. We choose all hyper-parameters by random search based on validation set performance. For FB15k and FB15k-237, we set entity and relation embedding dimensionality to d e = d r = 200. For WN18 and WN18RR, which both contain a significantly smaller number of relations relative to the number of entities as well as a small number of relations compared to FB15k and FB15k-237, we set d e = 200 and d r = 30. We use batch normalization BID6 and dropout BID16 to speed up training. We choose the learning rate from {0.01, 0.005, 0.003, 0.001, 0.0005} and learning rate decay from {1, 0.995, 0.99}. We find the following combinations of learning rate and learning rate decay to give the best : (0.003, 0.99) for FB15k, (0.0005, 1.0) for FB15k-237, (0.005, 0.995) for WN18 and (0.01, 1.0) for WN18RR. We train the model using Adam BID8 with the batch size 128.1 https://github.com/ibalazevic/TuckERWe evaluate each triple from the test set as in BID1: for a given triple, we generate 2n e test triples by keeping the subject entity e s and relation r fixed and replacing the object entity e o with all possible entities E and vice versa. We then rank the scores obtained. We use the filtered setting, i.e. we remove all true triples apart from the currently observed test triple. For evaluation, we use the evaluation metrics used across the link prediction literature: mean reciprocal rank (MRR) and hits@k, k ∈ {1, 3, 10}. Mean reciprocal rank is the average of the inverse of a mean rank assigned to the true triple over all n e generated triples. Hits@k measures the percentage of times the true triple is ranked in the top k of the n e generated triples. Link prediction on all datasets are shown in Tables 2 and 3. Overall, TuckER outperforms previous state-ofthe-art models on all metrics across all datasets (apart from hits@10 on WN18). Results achieved by TuckER are not only better than those of other linear models, such as DistMult, ComplEx and SimplE, but also better than those of many more complex deep neural network and reinforcement learning architectures, e.g. MINERVA, ConvE and HypER, demonstrating the expressive power of linear models. Even though at entity embedding dimensionality d e = 200 and relation embedding dimensionality d r = 30 on WN18RR TuckER has fewer parameters (∼9.4 million) than ComplEx and SimplE (∼16.4 million), it consistently obtains better than any of those models. We believe this is achieved by exploiting knowledge sharing between relations through the core tensor. We find that lower dropout values (0.1, 0.2) are required for datasets with a higher number of training triples per relation and thus less risk of overfitting (WN18 and WN18RR) and higher dropout values (0.3, 0.4, 0.5) are required for FB15k and FB15k-237. We further note that TuckER improves the of all other linear models by a larger margin on datasets with a large number of relations (e.g. +14% improvement on FB15k over ComplEx, +8% improvement over SimplE on the toughest hits@1 metric), which supports our belief that TuckER makes use of the parameters shared between similar relations to improve predictions by multi-task learning. The presence of the core tensor which allows for knowledge sharing between relations suggests that TuckER should need a lower number of parameters for obtaining good than ComplEx or SimplE. To test this, we re-implement ComplEx and SimplE with 1-N scoring, batch normalization and dropout for fair comparison, perform random search to choose best hyper-parameters and train all three models on FB15k-237 with embedding sizes d e = d r ∈ Table 2. Link prediction on WN18RR and FB15k-237. We report for ComplEx-N3 BID10 at de = 115 for WN18RR and de = 400 for FB15k-237 to ensure comparability with TuckER in terms of the overall number of parameters (original paper reports at de = 2000). The RotatE BID17 are reported without their self-adversarial negative sampling (see Appendix H in the original paper) for fair comparison, given that it improves the by ∼ 4% and it is not specific to that model only. FB15k-237Linear MRR Hits@10 Hits@3 Hits@1 MRR Hits@10 Hits@3 Hits@1DistMult BID22 no BID14 no DISPLAYFORM0 DISPLAYFORM1.151 MINERVA BID2 no BID3 FIG2 shows the obtained MRR on the test set for each model. It is important to note that at embedding dimensionalities 20, 50 and 100, TuckER has fewer parameters than ComplEx and SimplE (e.g. ComplEx and SimplE have ∼3 million and TuckER has ∼2.5 million parameters for embedding dimensionality 100). DISPLAYFORM2 We can see that the difference between the MRRs of ComplEx, SimplE and TuckER is approximately constant for embedding sizes 100 and 200. However, for lower embedding sizes, the difference between MRRs increases e.g. by 4.2% for embedding size 20 for ComplEx and by 9.9% for embedding size 20 for SimplE. At embedding size 20 (∼300k parameters), the performance of TuckER is almost as good as the performance of ComplEx and SimplE at embedding size 200 (∼6 million parameters), which supports our initial assumption. In this work, we introduce TuckER, a relatively simple yet highly flexible linear model for link prediction in knowledge graphs based on the Tucker decomposition of a binary tensor of training set triples, which achieves state-of-the-art on several standard link prediction datasets. TuckER's number of parameters grows linearly with respect to embedding dimension as the number of entities or relations in a knowledge graph increases, which makes it easily scalable to large knowledge graphs. Future work might include exploring how to incorporate knowledge on individual relation properties into the existing model.

Represent the TLDR for finding the corresponding scientific article

We propose TuckER, a relatively simple but powerful linear model for link prediction in knowledge graphs, based on Tucker decomposition of the binary tensor representation of knowledge graph triples.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!