Question: Represent the article for finding a single-sentence TLDR

Question: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance. Learning with graph structured data, such as molecules, social, biological, and financial networks, requires effective representation of their graph structure BID14. Recently, there has been a surge of interest in Graph Neural Network (GNN) approaches for representation learning of graphs BID23 BID13 BID21 BID34 BID37. GNNs broadly follow a recursive neighborhood aggregation (or message passing) scheme, where each node aggregates feature vectors of its neighbors to compute its new feature vector BID37 BID12. After k iterations of aggregation, a node is represented by its transformed feature vector, which captures the structural information within the node's k-hop neighborhood. The representation of an entire graph can then be obtained through pooling BID39, for example, by summing the representation vectors of all nodes in the graph. Many GNN variants with different neighborhood aggregation and graph-level pooling schemes have been proposed BID31 BID3 BID6 BID8 BID13 BID19 BID21 BID23 BID34 BID28 BID37 BID29 BID35 BID39. Empirically, these GNNs have achieved state-of-the-art performance in many tasks such as node classification, link prediction, and graph classification. However, the design of new GNNs is mostly based on empirical intuition, heuristics, and experimental trial-anderror. There is little theoretical understanding of the properties and limitations of GNNs, and formal analysis of GNNs' representational capacity is limited. Here, we present a theoretical framework for analyzing the representational power of GNNs. We formally characterize how expressive different GNN variants are in learning to represent and distinguish between different graph structures. Our framework is inspired by the close connection between GNNs and the Weisfeiler-Lehman (WL) graph isomorphism test BID36, a powerful test known to distinguish a broad class of graphs BID2. Similar to GNNs, the WL test iteratively updates a given node's feature vector by aggregating feature vectors of its network neighbors. What makes the WL test so powerful is its injective aggregation update that maps different node neighborhoods to different feature vectors. Our key insight is that a GNN can have as large discriminative power as the WL test if the GNN's aggregation scheme is highly expressive and can model injective functions. To mathematically formalize the above insight, our framework first represents the set of feature vectors of a given node's neighbors as a multiset, i.e., a set with possibly repeating elements. Then, the neighbor aggregation in GNNs can be thought of as an aggregation function over the multiset. Hence, to have strong representational power, a GNN must be able to aggregate different multisets into different representations. We rigorously study several variants of multiset functions and theoretically characterize their discriminative power, i.e., how well different aggregation functions can distinguish different multisets. The more discriminative the multiset function is, the more powerful the representational power of the underlying GNN.Our main are summarized as follows: 1) We show that GNNs are at most as powerful as the WL test in distinguishing graph structures.2) We establish conditions on the neighbor aggregation and graph readout functions under which the ing GNN is as powerful as the WL test.3) We identify graph structures that cannot be distinguished by popular GNN variants, such as GCN BID21 and GraphSAGE (a), and we precisely characterize the kinds of graph structures such GNN-based models can capture. We develop a simple neural architecture, Graph Isomorphism Network (GIN), and show that its discriminative/representational power is equal to the power of the WL test. We validate our theory via experiments on graph classification datasets, where the expressive power of GNNs is crucial to capture graph structures. In particular, we compare the performance of GNNs with various aggregation functions. Our confirm that the most powerful GNN by our theory, i.e., Graph Isomorphism Network (GIN), also empirically has high representational power as it almost perfectly fits the training data, whereas the less powerful GNN variants often severely underfit the training data. In addition, the representationally more powerful GNNs outperform the others by test set accuracy and achieve state-of-the-art performance on many graph classification benchmarks. We begin by summarizing some of the most common GNN models and, along the way, introduce our notation. Let G = (V, E) denote a graph with node feature vectors X v for v ∈ V. There are two tasks of interest: Node classification, where each node v ∈ V has an associated label y v and the goal is to learn a representation vector h v of v such that v's label can be predicted as y v = f (h v); Graph classification, where, given a set of graphs {G 1, ..., G N} ⊆ G and their labels {y 1, ..., y N} ⊆ Y, we aim to learn a representation vector h G that helps predict the label of an entire graph, DISPLAYFORM0 Graph Neural Networks. GNNs use the graph structure and node features X v to learn a representation vector of a node, h v, or the entire graph, h G. Modern GNNs follow a neighborhood aggregation strategy, where we iteratively update the representation of a node by aggregating representations of its neighbors. After k iterations of aggregation, a node's representation captures the structural information within its k-hop network neighborhood. Formally, the k-th layer of a GNN is GNNs is crucial. A number of architectures for AGGREGATE have been proposed. In the pooling variant of GraphSAGE BID13, AGGREGATE has been formulated as DISPLAYFORM1 DISPLAYFORM2 where W is a learnable matrix, and MAX represents an element-wise max-pooling. The COMBINE step could be a concatenation followed by a linear mapping W · h DISPLAYFORM3 as in GraphSAGE. In Graph Convolutional Networks (GCN) BID21, the element-wise mean pooling is used instead, and the AGGREGATE and COMBINE steps are integrated as follows: DISPLAYFORM4 Many other GNNs can be represented similarly to Eq. 2.1 BID37 BID12.For node classification, the node representation h DISPLAYFORM5 of the final iteration is used for prediction. For graph classification, the READOUT function aggregates node features from the final iteration to obtain the entire graph's representation h G: DISPLAYFORM6 (2.4)READOUT can be a simple permutation invariant function such as summation or a more sophisticated graph-level pooling function BID39.Weisfeiler-Lehman test. The graph isomorphism problem asks whether two graphs are topologically identical. This is a challenging problem: no polynomial-time algorithm is known for it yet BID10 BID11 BID1. Apart from some corner cases BID4, the Weisfeiler-Lehman (WL) test of graph isomorphism BID36 is an effective and computationally efficient test that distinguishes a broad class of graphs BID2. Its 1-dimensional form, "naïve vertex refinement", is analogous to neighbor aggregation in GNNs. The WL test iteratively aggregates the labels of nodes and their neighborhoods, and hashes the aggregated labels into unique new labels. The algorithm decides that two graphs are non-isomorphic if at some iteration the labels of the nodes between the two graphs differ. Based on the WL test, BID32 proposed the WL subtree kernel that measures the similarity between graphs. The kernel uses the counts of node labels at different iterations of the WL test as the feature vector of a graph. Intuitively, a node's label at the k-th iteration of WL test represents a subtree structure of height k rooted at the node (Figure 1). Thus, the graph features considered by the WL subtree kernel are essentially counts of different rooted subtrees in the graph. We start with an overview of our framework for analyzing the expressive power of GNNs. Figure 1 illustrates our idea. A GNN recursively updates each node's feature vector to capture the network structure and features of other nodes around it, i.e., its rooted subtree structures (Figure 1). Throughout the paper, we assume node input features are from a countable universe. For finite graphs, node feature vectors at deeper layers of any fixed model are also from a countable universe. For notational simplicity, we can assign each feature vector a unique label in {a, b, c . . .}. Then, feature vectors of a set of neighboring nodes form a multiset (Figure 1): the same element can appear multiple times since different nodes can have identical feature vectors. Definition 1 (Multiset). A multiset is a generalized concept of a set that allows multiple instances for its elements. More formally, a multiset is a 2-tuple X = (S, m) where S is the underlying set of X that is formed from its distinct elements, and m: S → N ≥1 gives the multiplicity of the elements. To study the representational power of a GNN, we analyze when a GNN maps two nodes to the same location in the embedding space. Intuitively, a maximally powerful GNN maps two nodes to the same location only if they have identical subtree structures with identical features on the corresponding nodes. Since subtree structures are defined recursively via node neighborhoods (Figure 1), we can reduce our analysis to the question whether a GNN maps two neighborhoods (i.e., two multisets) to the same embedding or representation. A maximally powerful GNN would never map two different neighborhoods, i.e., multisets of feature vectors, to the same representation. This means its aggregation scheme must be injective. Thus, we abstract a GNN's aggregation scheme as a class of functions over multisets that their neural networks can represent, and analyze whether they are able to represent injective multiset functions. Next, we use this reasoning to develop a maximally powerful GNN. In Section 5, we study popular GNN variants and see that their aggregation schemes are inherently not injective and thus less powerful, but that they can capture other interesting properties of graphs. First, we characterize the maximum representational capacity of a general class of GNN-based models. Ideally, a maximally powerful GNN could distinguish different graph structures by mapping them to different representations in the embedding space. This ability to map any two different graphs to different embeddings, however, implies solving the challenging graph isomorphism problem. That is, we want isomorphic graphs to be mapped to the same representation and non-isomorphic ones to different representations. In our analysis, we characterize the representational capacity of GNNs via a slightly weaker criterion: a powerful heuristic called Weisfeiler-Lehman (WL) graph isomorphism test, that is known to work well in general, with a few exceptions, e.g., regular graphs BID4 BID7 BID9. Lemma 2. Let G 1 and G 2 be any two non-isomorphic graphs. If a graph neural network A: G → R d maps G 1 and G 2 to different embeddings, the Weisfeiler-Lehman graph isomorphism test also decides G 1 and G 2 are not isomorphic. Proofs of all Lemmas and Theorems can be found in the Appendix. Hence, any aggregation-based GNN is at most as powerful as the WL test in distinguishing different graphs. A natural follow-up question is whether there exist GNNs that are, in principle, as powerful as the WL test? Our answer, in Theorem 3, is yes: if the neighbor aggregation and graph-level readout functions are injective, then the ing GNN is as powerful as the WL test. DISPLAYFORM0 With a sufficient number of GNN layers, A maps any graphs G 1 and G 2 that the Weisfeiler-Lehman test of isomorphism decides as non-isomorphic, to different embeddings if the following conditions hold: a) A aggregates and updates node features iteratively with DISPLAYFORM1 where the functions f, which operates on multisets, and φ are injective.b) A's graph-level readout, which operates on the multiset of node features h DISPLAYFORM2, is injective. We prove Theorem 3 in the appendix. For countable sets, injectiveness well characterizes whether a function preserves the distinctness of inputs. Uncountable sets, where node features are continuous, need some further considerations. In addition, it would be interesting to characterize how close together the learned features lie in a function's image. We leave these questions for future work, and focus on the case where input node features are from a countable set (that can be a subset of an uncountable set such as R n).Lemma 4. Assume the input feature space X is countable. Let g (k) be the function parameterized by a GNN's k-th layer for k = 1,..., L, where g is defined on multisets X ⊂ X of bounded size. The range of g (k), i.e., the space of node hidden features h DISPLAYFORM3 Here, it is also worth discussing an important benefit of GNNs beyond distinguishing different graphs, that is, capturing similarity of graph structures. Note that node feature vectors in the WL test are essentially one-hot encodings and thus cannot capture the similarity between subtrees. In contrast, a GNN satisfying the criteria in Theorem 3 generalizes the WL test by learning to embed the subtrees to low-dimensional space. This enables GNNs to not only discriminate different structures, but also to learn to map similar graph structures to similar embeddings and capture dependencies between graph structures. Capturing structural similarity of the node labels is shown to be helpful for generalization particularly when the co-occurrence of subtrees is sparse across different graphs or there are noisy edges and node features BID38. Having developed conditions for a maximally powerful GNN, we next develop a simple architecture, Graph Isomorphism Network (GIN), that provably satisfies the conditions in Theorem 3. This model generalizes the WL test and hence achieves maximum discriminative power among GNNs. To model injective multiset functions for the neighbor aggregation, we develop a theory of "deep multisets", i.e., parameterizing universal multiset functions with neural networks. Our next lemma states that sum aggregators can represent injective, in fact, universal functions over multisets. Lemma 5. Assume X is countable. There exists a function f: X → R n so that h(X) = x∈X f (x) is unique for each multiset X ⊂ X of bounded size. Moreover, any multiset function g can be decomposed as g (X) = φ x∈X f (x) for some function φ. We prove Lemma 5 in the appendix. The proof extends the setting in BID40 ) from sets to multisets. An important distinction between deep multisets and sets is that certain popular injective set functions, such as the mean aggregator, are not injective multiset functions. With the mechanism for modeling universal multiset functions in Lemma 5 as a building block, we can conceive aggregation schemes that can represent universal functions over a node and the multiset of its neighbors, and thus will satisfy the injectiveness condition (a) in Theorem 3. Our next corollary provides a simple and concrete formulation among many such aggregation schemes. Corollary 6. Assume X is countable. There exists a function f: X → R n so that for infinitely many choices of, including all irrational numbers, h(c, X) = (1 +) · f (c) + x∈X f (x) is unique for each pair (c, X), where c ∈ X and X ⊂ X is a multiset of bounded size. Moreover, any function g over such pairs can be decomposed as g (c, X) = ϕ (1 +) · f (c) + x∈X f (x) for some function ϕ.We can use multi-layer perceptrons (MLPs) to model and learn f and ϕ in Corollary 6, thanks to the universal approximation theorem BID16 BID15. In practice, we model f (k+1) • ϕ (k) with one MLP, because MLPs can represent the composition of functions. In the first iteration, we do not need MLPs before summation if input features are one-hot encodings as their summation alone is injective. We can make a learnable parameter or a fixed scalar. Then, GIN updates node representations as DISPLAYFORM0 Generally, there may exist many other powerful GNNs. GIN is one such example among many maximally powerful GNNs, while being simple. Node embeddings learned by GIN can be directly used for tasks like node classification and link prediction. For graph classification tasks we propose the following "readout" function that, given embeddings of individual nodes, produces the embedding of the entire graph. An important aspect of the graph-level readout is that node representations, corresponding to subtree structures, get more refined and global as the number of iterations increases. A sufficient number of iterations is key to achieving good discriminative power. Yet, features from earlier iterations may sometimes generalize better. To consider all structural information, we use information from all depths/iterations of the model. We achieve this by an architecture similar to Jumping Knowledge DISPLAYFORM0 Figure 2: Ranking by expressive power for sum, mean and max aggregators over a multiset. Left panel shows the input multiset, i.e., the network neighborhood to be aggregated. The next three panels illustrate the aspects of the multiset a given aggregator is able to capture: sum captures the full multiset, mean captures the proportion/distribution of elements of a given type, and the max aggregator ignores multiplicities (reduces the multiset to a simple set).vs. Networks BID37, where we replace Eq. 2.4 with graph representations concatenated across all iterations/layers of GIN: DISPLAYFORM1 By Theorem 3 and Corollary 6, if GIN replaces READOUT in Eq. 4.2 with summing all node features from the same iterations (we do not need an extra MLP before summation for the same reason as in Eq. 4.1), it provably generalizes the WL test and the WL subtree kernel. Next, we study GNNs that do not satisfy the conditions in Theorem 3, including GCN BID21 and GraphSAGE (a). We conduct ablation studies on two aspects of the aggregator in Eq. 4.1: 1-layer perceptrons instead of MLPs and mean or max-pooling instead of the sum. We will see that these GNN variants get confused by surprisingly simple graphs and are less powerful than the WL test. Nonetheless, models with mean aggregators like GCN perform well for node classification tasks. To better understand this, we precisely characterize what different GNN variants can and cannot capture about a graph and discuss the implications for learning with graphs. The function f in Lemma 5 helps map distinct multisets to unique embeddings. It can be parameterized by an MLP by the universal approximation theorem BID15. Nonetheless, many existing GNNs instead use a 1-layer perceptron σ • W BID8 BID21 ), a linear mapping followed by a non-linear activation function such as a ReLU. Such 1-layer mappings are examples of Generalized Linear Models BID25. Therefore, we are interested in understanding whether 1-layer perceptrons are enough for graph learning. Lemma 7 suggests that there are indeed network neighborhoods (multisets) that models with 1-layer perceptrons can never distinguish. Lemma 7. There exist finite multisets X 1 = X 2 so that for any linear mapping W, DISPLAYFORM0 The main idea of the proof for Lemma 7 is that 1-layer perceptrons can behave much like linear mappings, so the GNN layers degenerate into simply summing over neighborhood features. Our proof builds on the fact that the bias term is lacking in the linear mapping. With the bias term and sufficiently large output dimensionality, 1-layer perceptrons might be able to distinguish different multisets. Nonetheless, unlike models using MLPs, the 1-layer perceptron (even with the bias term) is not a universal approximator of multiset functions. Consequently, even if GNNs with 1-layer perceptrons can embed different graphs to different locations to some degree, such embeddings may not adequately capture structural similarity, and can be difficult for simple classifiers, e.g., linear classifiers, to fit. In Section 7, we will empirically see that GNNs with 1-layer perceptrons, when applied to graph classification, sometimes severely underfit training data and often perform worse than GNNs with MLPs in terms of test accuracy. What happens if we replace the sum in h (X) = x∈X f (x) with mean or max-pooling as in GCN and GraphSAGE? Mean and max-pooling aggregators are still well-defined multiset functions because they are permutation invariant. But, they are not injective. Figure 2 ranks the three aggregators by their representational power, and FIG0 illustrates pairs of structures that the mean and max-pooling aggregators fail to distinguish. Here, node colors denote different node features, and we assume the GNNs aggregate neighbors first before combining them with the central node labeled as v and v.In FIG0, every node has the same feature a and f (a) is the same across all nodes (for any function f). When performing neighborhood aggregation, the mean or maximum over f (a) remains f (a) and, by induction, we always obtain the same node representation everywhere. Thus, in this case mean and max-pooling aggregators fail to capture any structural information. In contrast, the sum aggregator distinguishes the structures because 2 · f (a) and 3 · f (a) give different values. The same argument can be applied to any unlabeled graph. If node degrees instead of a constant value is used as node input features, in principle, mean can recover sum, but max-pooling cannot. FIG0 suggests that mean and max have trouble distinguishing graphs with nodes that have repeating features. Let h color (r for red, g for green) denote node features transformed by f. FIG0 shows that maximum over the neighborhood of the blue nodes v and v yields max (h g, h r) and max (h g, h r, h r), which collapse to the same representation (even though the corresponding graph structures are different). Thus, max-pooling fails to distinguish them. In contrast, the sum aggregator still works because 1 2 (h g + h r) and 1 3 (h g + h r + h r) are in general not equivalent. Similarly, in FIG0, both mean and max fail as DISPLAYFORM0 To characterize the class of multisets that the mean aggregator can distinguish, consider the example X 1 = (S, m) and X 2 = (S, k · m), where X 1 and X 2 have the same set of distinct elements, but X 2 contains k copies of each element of X 1. Any mean aggregator maps X 1 and X 2 to the same embedding, because it simply takes averages over individual element features. Thus, the mean captures the distribution (proportions) of elements in a multiset, but not the exact multiset. Corollary 8. Assume X is countable. There exists a function f: X → R n so that for h(X) = 1 |X| x∈X f (x), h(X 1) = h(X 2) if and only if multisets X 1 and X 2 have the same distribution. That is, assuming |X 2 | ≥ |X 1 |, we have X 1 = (S, m) and X 2 = (S, k · m) for some k ∈ N ≥1.The mean aggregator may perform well if, for the task, the statistical and distributional information in the graph is more important than the exact structure. Moreover, when node features are diverse and rarely repeat, the mean aggregator is as powerful as the sum aggregator. This may explain why, despite the limitations identified in Section 5.2, GNNs with mean aggregators are effective for node classification tasks, such as classifying article subjects and community detection, where node features are rich and the distribution of the neighborhood features provides a strong signal for the task. The examples in FIG0 illustrate that max-pooling considers multiple nodes with the same feature as only one node (i.e., treats a multiset as a set). Max-pooling captures neither the exact structure nor the distribution. However, it may be suitable for tasks where it is important to identify representative elements or the "skeleton", rather than to distinguish the exact structure or distribution. BID27 empirically show that the max-pooling aggregator learns to identify the skeleton of a 3D point cloud and that it is robust to noise and outliers. For completeness, the next corollary shows that the max-pooling aggregator captures the underlying set of a multiset. Corollary 9. Assume X is countable. Then there exists a function f: X → R ∞ so that for h(X) = max x∈X f (x), h(X 1) = h(X 2) if and only if X 1 and X 2 have the same underlying set. There are other non-standard neighbor aggregation schemes that we do not cover, e.g., weighted average via attention BID34 and LSTM pooling BID13 BID24. We emphasize that our theoretical framework is general enough to characterize the representaional power of any aggregation-based GNNs. In the future, it would be interesting to apply our framework to analyze and understand other aggregation schemes. Despite the empirical success of GNNs, there has been relatively little work that mathematically studies their properties. An exception to this is the work of BID30 who shows that the perhaps earliest GNN model BID31 can approximate measurable functions in probability. BID22 show that their proposed architecture lies in the RKHS of graph kernels, but do not study explicitly which graphs it can distinguish. Each of these works focuses on a specific architecture and do not easily generalize to multple architectures. In contrast, our above provide a general framework for analyzing and characterizing the expressive power of a broad class of GNNs. Recently, many GNN-based architectures have been proposed, including sum aggregation and MLP encoding BID3 BID31 BID8, and most without theoretical derivation. In contrast to many prior GNN architectures, our Graph Isomorphism Network (GIN) is theoretically motivated, simple yet powerful. We evaluate and compare the training and test performance of GIN and less powerful GNN variants. Training set performance allows us to compare different GNN models based on their representational power and test set performance quantifies generalization ability. Datasets. We use 9 graph classification benchmarks: 4 bioinformatics datasets (MUTAG, PTC, NCI1, PROTEINS) and 5 social network datasets (COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY and REDDIT-MULTI5K) BID38. Importantly, our goal here is not to allow the models to rely on the input node features but mainly learn from the network structure. Thus, in the bioinformatic graphs, the nodes have categorical input features but in the social networks, they have no features. For social networks we create node features as follows: for the REDDIT datasets, we set all node feature vectors to be the same (thus, features here are uninformative); for the other social graphs, we use one-hot encodings of node degrees. Dataset statistics are summarized in Table 1, and more details of the data can be found in Appendix I. We evaluate GINs (Eqs. 4.1 and 4.2) and the less powerful GNN variants. Under the GIN framework, we consider two variants: a GIN that learns in Eq. 4.1 by gradient descent, which we call GIN-, and a simpler (slightly less powerful) 2 GIN, where in Eq. 4.1 is fixed to 0, which we call GIN-0. As we will see, GIN-0 shows strong empirical performance: not only does GIN-0 fit training data equally well as GIN-, it also demonstrates good generalization, slightly but consistently outperforming GIN-in terms of test accuracy. For the less powerful GNN variants, we consider architectures that replace the sum in the GIN-0 aggregation with mean or max-pooling 3, or replace MLPs with 1-layer perceptrons, i.e., a linear mapping followed by ReLU. In FIG1 and Table 1, a model is named by the aggregator/perceptron it uses. Here mean-1-layer and max-1-layer correspond to GCN and GraphSAGE, respectively, up to minor architecture modifications. We apply the same graph-level readout (READOUT in Eq. 4.2) for GINs and all the GNN variants, specifically, sum readout on bioinformatics datasets and mean readout on social datasets due to better test performance. Following BID38 BID26, we perform 10-fold crossvalidation with LIB-SVM BID5. We report the average and standard deviation of validation accuracies across the 10 folds within the cross-validation. For all configurations, 5 GNN layers (including the input layer) are applied, and all MLPs have 2 layers. Batch normalization BID17 is applied on every hidden layer. We use the Adam optimizer BID20 with initial learning rate 0.01 and decay the learning rate by 0.5 every 50 epochs. The hyper-parameters we tune for each dataset are: the number of hidden units ∈ {16, 32} for bioinformatics graphs and 64 for social graphs; the batch size ∈ {32, 128}; the dropout ratio ∈ {0, 0.5} after the dense layer BID33; the number of epochs, i.e., a single epoch with the best cross-validation accuracy averaged over the 10 folds was selected. Note that due to the small dataset sizes, an alternative setting, where hyper-parameter selection is done using a validation set, is extremely unstable, e.g., for MUTAG, the validation set only contains 18 data points. We also report the training accuracy of different GNNs, where all the hyper-parameters were fixed across the datasets: 5 GNN layers (including the input layer), hidden units of size 64, minibatch of size 128, and 0.5 dropout ratio. For comparison, the training accuracy of the WL subtree kernel is reported, where we set the number of iterations to 4, which is comparable to the 5 GNN layers. Baselines. We compare the GNNs above with a number of state-of-the-art baselines for graph classification: the WL subtree kernel BID32, where C-SVM BID5 ) was used as a classifier; the hyper-parameters we tune are C of the SVM and the number of WL iterations ∈ {1, 2, . . ., 6}; state-of-the-art deep learning architectures, i.e., Diffusionconvolutional neural networks (DCNN) BID0, PATCHY-SAN BID26 and Deep Graph CNN (DGCNN); Anonymous Walk Embeddings (AWL) BID18. For the deep learning methods and AWL, we report the accuracies reported in the original papers. Training set performance. We validate our theoretical analysis of the representational power of GNNs by comparing their training accuracies. Models with higher representational power should have higher training set accuracy. FIG1 shows training curves of GINs and less powerful GNN variants with the same hyper-parameter settings. First, both the theoretically most powerful GNN, i.e. GIN-and GIN-0, are able to almost perfectly fit all the training sets. In our experiments, explicit learning of in GIN-yields no gain in fitting training data compared to fixing to 0 as in GIN-0. In comparison, the GNN variants using mean/max pooling or 1-layer perceptrons severely underfit on many datasets. In particular, the training accuracy patterns align with our ranking by the models' Datasets IMDB-B IMDB-M RDT-B RDT-M5K COLLAB MUTAG PROTEINS PTC NCI1 Datasets # graphs 1000 1500 2000 5000 5000 188 1113 344 4110 # classes 2 3 2 5 3 2 2 2 74.5 ± 5.9 51.5 ± 3.6 87.9 ± 2.5 54.7 ± 2.9 73.9 ± 1.9 87.9 ± 9.8 --- SUM-MLP (GIN-0) 75.1 ± 5.1 52.3 ± 2.8 92.4 ± 2.5 57.5 ± 1.5 80.2 ± 1.9 89.4 ± 5.6 76.2 ± 2.8 64.6 ± 7.0 82.7 ± 1.7 SUM-MLP (GIN-) 74.3 ± 5.1 52.1 ± 3.6 92.2 ± 2.3 57.0 ± 1.7 80.1 ± 1.9 89.0 ± 6.0 75.9 ± 3.8 63.7 ± 8.2 82.7 ± 1.6 SUM-1-LAYER 74.1 ± 5.0 52.2 ± 2.4 90.0 ± 2.7 55.1 ± 1.6 80.6 ± 1.9 90.0 ± 8.8 76.2 ± 2.6 63.1 ± 5.7 82.0 ± 1.5 MEAN-MLP 73.7 ± 3.7 52.3 ± 3.1 50.0 ± 0.0 20.0 ± 0.0 79.2 ± 2.3 83.5 ± 6.3 75.5 ± 3.4 66.6 ± 6.9 80.9 ± 1.8 MEAN-1-LAYER (GCN) 74.0 ± 3.4 51.9 ± 3.8 50.0 ± 0.0 20.0 ± 0.0 79.0 ± 1.8 85.6 ± 5.8 76.0 ± 3.2 64.2 ± 4.3 80.2 ± 2.0 MAX-MLP 73.2 ± 5.8 51.1 ± 3.6 ---84.0 ± 6.1 76.0 ± 3.2 64.6 ± 10.2 77.8 ± 1.3 MAX-1-LAYER (GraphSAGE) 72.3 ± 5.3 50.9 ± 2.2 ---85.1 ± 7.6 75.9 ± 3.2 63.9 ± 7.7 77.7 ± 1.5 Table 1: Test set classification accuracies (%). The best-performing GNNs are highlighted with boldface. On datasets where GINs' accuracy is not strictly the highest among GNN variants, we see that GINs are still comparable to the best GNN because a paired t-test at significance level 10% does not distinguish GINs from the best; thus, GINs are also highlighted with boldface. If a baseline performs significantly better than all GNNs, we highlight it with boldface and asterisk.representational power: GNN variants with MLPs tend to have higher training accuracies than those with 1-layer perceptrons, and GNNs with sum aggregators tend to fit the training sets better than those with mean and max-pooling aggregators. On our datasets, training accuracies of the GNNs never exceed those of the WL subtree kernel. This is expected because GNNs generally have lower discriminative power than the WL test. For example, on IMDBBINARY, none of the models can perfectly fit the training set, and the GNNs achieve at most the same training accuracy as the WL kernel. This pattern aligns with our that the WL test provides an upper bound for the representational capacity of the aggregation-based GNNs. However, the WL kernel is not able to learn how to combine node features, which might be quite informative for a given prediction task as we will see next. Test set performance. Next, we compare test accuracies. Although our theoretical do not directly speak about the generalization ability of GNNs, it is reasonable to expect that GNNs with strong expressive power can accurately capture graph structures of interest and thus generalize well. Table 1 compares test accuracies of GINs (Sum-MLP), other GNN variants, as well as the state-of-the-art baselines. First, GINs, especially GIN-0, outperform (or achieve comparable performance as) the less powerful GNN variants on all the 9 datasets, achieving state-of-the-art performance. GINs shine on the social network datasets, which contain a relatively large number of training graphs. For the Reddit datasets, all nodes share the same scalar as node feature. Here, GINs and sum-aggregation GNNs accurately capture the graph structure and significantly outperform other models. Mean-aggregation GNNs, however, fail to capture any structures of the unlabeled graphs (as predicted in Section 5.2) and do not perform better than random guessing. Even if node degrees are provided as input features, mean-based GNNs perform much worse than sum-based GNNs (the accuracy of the GNN with mean-MLP aggregation is 71.2±4.6% on REDDIT-BINARY and 41.3±2.1% on REDDIT-MULTI5K).Comparing GINs (GIN-0 and GIN-), we observe that GIN-0 slightly but consistently outperforms GIN-. Since both models fit training data equally well, the better generalization of GIN-0 may be explained by its simplicity compared to GIN-. In this paper, we developed theoretical foundations for reasoning about the expressive power of GNNs, and proved tight bounds on the representational capacity of popular GNN variants. We also designed a provably maximally powerful GNN under the neighborhood aggregation framework. An interesting direction for future work is to go beyond neighborhood aggregation (or message passing) in order to pursue possibly even more powerful architectures for learning with graphs. To complete the picture, it would also be interesting to understand and improve the generalization properties of GNNs as well as better understand their optimization landscape. A PROOF FOR LEMMA 2Proof. Suppose after k iterations, a graph neural network A has A(G 1) = A(G 2) but the WL test cannot decide G 1 and G 2 are non-isomorphic. It follows that from iteration 0 to k in the WL test, G 1 and G 2 always have the same collection of node labels. In particular, because G 1 and G 2 have the same WL node labels for iteration i and i + 1 for any i = 0,..., k − 1, G 1 and G 2 have the same collection, i.e. multiset, of WL node labels l DISPLAYFORM0 as well as the same collection of node neighborhoods l DISPLAYFORM1. Otherwise, the WL test would have obtained different collections of node labels at iteration i + 1 for G 1 and G 2 as different multisets get unique new labels. The WL test always relabels different multisets of neighboring nodes into different new labels. We show that on the same graph G = G 1 or G 2, if WL node labels l DISPLAYFORM2 u for any iteration i. This apparently holds for i = 0 because WL and GNN starts with the same node features. Suppose this holds for iteration j, if for any u, v, l DISPLAYFORM3, then it must be the case that DISPLAYFORM4 By our assumption on iteration j, we must have DISPLAYFORM5 In the aggregation process of the GNN, the same AGGREGATE and COMBINE are applied. The same input, i.e. neighborhood features, generates the same output. Thus, h DISPLAYFORM6. By induction, if WL node labels l DISPLAYFORM7 u, we always have GNN node features h DISPLAYFORM8 u for any iteration i. This creates a valid mapping φ such that h DISPLAYFORM9 v ) for any v ∈ G. It follows from G 1 and G 2 have the same multiset of WL neighborhood labels that G 1 and G 2 also have the same collection of GNN neighborhood features DISPLAYFORM10 are the same. In particular, we have the same collection of GNN node features DISPLAYFORM11 for G 1 and G 2. Because the graph level readout function is permutation invariant with respect to the collection of node features, A(G 1) = A(G 2). Hence we have reached a contradiction. Proof. Let A be a graph neural network where the condition holds. Let G 1, G 2 be any graphs which the WL test decides as non-isomorphic at iteration K. Because the graph-level readout function is injective, i.e., it maps distinct multiset of node features into unique embeddings, it sufficies to show that A's neighborhood aggregation process, with sufficient iterations, embeds G 1 and G 2 into different multisets of node features. Let us assume A updates node representations as DISPLAYFORM0 with injective funtions f and φ. The WL test applies a predetermined injective hash function g to update the WL node labels l Proof. We first prove that there exists a mapping f so that x∈X f (x) is unique for each multiset X of bounded size. Because X is countable, there exists a mapping Z: X → N from x ∈ X to natural numbers. Because the cardinality of multisets X is bounded, there exists a number N ∈ N so that |X| < N for all X. Then an example of such f is f (x) = N −Z(x). This f can be viewed as a more compressed form of an one-hot vector or N -digit presentation. Thus, h(X) = x∈X f (x) is an injective function of multisets. φ x∈X f (x) is permutation invariant so it is a well-defined multiset function. For any multiset function g, we can construct such φ by letting φ x∈X f (x) = g(X). Note that such φ is well-defined because h(X) = x∈X f (x) is injective. Proof. Following the proof of Lemma 5, we consider f (x) = N −Z(x), where N and Z: X → N are the same as defined in Appendix D. Let h(c, X) ≡ (1 +) · f (c) + x∈X f (x). Our goal is show that for any (c, X) = (c, X) with c, c ∈ X and X, X ⊂ X, h(c, X) = h(c, X) holds, if is an irrational number. We prove by contradiction. For any (c, X), suppose there exists (c, X) such that (c, X) = (c, X) but h(c, X) = h(c, X) holds. Let us consider the following two cases: c = c but X = X, and c = c. For the first case, h(c, X) = h(c, X) implies x∈X f (x) = x∈X f (x). It follows from Lemma 5 that the equality will not hold, because with f (x) = N −Z(x), X = X implies x∈X f (x) = x∈X f (x). Thus, we reach a contradiction. For the second case, we can similarly rewrite h(c, X) = h(c, X) as Because is an irrational number and f (c) − f (c) is a non-zero rational number, L.H.S. of Eq. E.1 is irrational. On the other hand, R.H.S. of Eq. E.1, the sum of a finite number of rational numbers, is rational. Hence the equality in Eq. E.1 cannot hold, and we have reached a contradiction. For any function g over the pairs (c, X), we can construct such ϕ for the desired decomposition by letting ϕ (1 +) · f (c) + x∈X f (x) = g(c, X). Note that such ϕ is well-defined because h(c, X) = (1 +) · f (c) + x∈X f (x) is injective. F PROOF FOR LEMMA 7Proof. Let us consider the example X 1 = {1, 1, 1, 1, 1} and X 2 = {2, 3}, i.e. two different multisets of positive numbers that sum up to the same value. We will be using the homogeneity of ReLU.Let W be an arbitrary linear transform that maps x ∈ X 1, X 2 into R n. It is clear that, at the same coordinates, W x are either positive or negative for all x because all x in X 1 and X 2 are positive. It follows that ReLU(W x) are either positive or 0 at the same coordinate for all x in X 1, X 2. For the coordinates where ReLU(W x) are 0, we have x∈X1 ReLU (W x) = x∈X2 ReLU (W x). For the coordinates where W x are positive, linearity still holds. It follows from linearity that x∈X ReLU (W x) = ReLU W x∈X x where X could be X 1 or X 2. Because x∈X1 x = x∈X2 x, we have the following as desired.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We develop theoretical foundations for the expressive power of GNNs and design a provably most powerful GNN.

represent the short summary for finding the corresponding scientific article (long) Examples:

"A study on the benefit of sharing representation in Multi-Task Reinforcement Learning." == "We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them ing in a more effective feature extraction compared to learning a single task. Intuitively, the ing set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance. Multi-Task Learning (MTL) ambitiously aims to learn multiple tasks jointly instead of learning them separately, leveraging the assumption that the considered tasks have common properties which can be exploited by Machine Learning (ML) models to generalize the learning of each of them. For instance, the features extracted in the hidden layers of a neural network trained on multiple tasks have the advantage of being a general representation of structures common to each other. This translates into an effective way of learning multiple tasks at the same time, but it can also improve the learning of each individual task compared to learning them separately . Furthermore, the learned representation can be used to perform Transfer Learning (TL), i.e. using it as a preliminary knowledge to learn a new similar task ing in a more effective and faster learning than learning the new task from scratch . The same benefits of extraction and exploitation of common features among the tasks achieved in MTL, can be obtained in Multi-Task Reinforcement Learning (MTRL) when training a single agent on multiple Reinforcement Learning (RL) problems with common structures . In particular, in MTRL an agent can be trained on multiple tasks in the same domain, e.g. riding a bicycle or cycling while going towards a goal, or on different but similar domains, e.g. balancing a pendulum or balancing a double pendulum 1. Considering recent advances in Deep Reinforcement Learning (DRL) and the ing increase in the complexity of experimental benchmarks, the use of Deep Learning (DL) models, e.g. deep neural networks, has become a popular and effective way to extract common features among tasks in MTRL algorithms (; ;). However, despite the high representational capacity of DL models, the extraction of good features remains challenging. For instance, the performance of the learning process can degrade when unrelated tasks are used together ; another detrimental issue may occur when the training of a single model is not balanced properly among multiple tasks . Recent developments in MTRL achieve significant in feature extraction by means of algorithms specifically developed to address these issues. While some of these works rely on a single deep neural network to model the multi-task agent (; ; ;), others use multiple deep neural networks, e.g. one for each task and another for the multi-task agent (; ; ;). Intuitively, achieving good in MTRL with a single deep neural network is more desirable than using many of them, since the training time is likely much less and the whole architecture is easier to implement. In this paper we study the benefits of shared representations among tasks. We theoretically motivate the intuitive effectiveness of our method, deriving theoretical guarantees that exploit the theoretical framework provided by , in which the authors present upper bounds on the quality of learning in MTL when extracting features for multiple tasks in a single shared representation. The significancy of this is that the cost of learning the shared representation decreases with a factor O(1 / √ T), where T is the number of tasks for many function approximator hypothesis classes. The main contribution of this work is twofold. Policy-Iteration (API) 2 in the MTRL setting, and we extend the approximation error bounds in to the case of multiple tasks with different dimensionalities. Then, we show how to combine these ing in, to the best of our knowledge, the first proposed extension of the finite-time bounds of AVI/API to MTRL. Despite being an extension of previous works, we derive these to justify our approach showing how the error propagation in AVI/API can theoretically benefit from learning multiple tasks jointly. 2. We leverage these proposing a neural network architecture, for which these bounds hold with minor assumptions, that allow us to learn multiple tasks with a single regressor extracting a common representation. We show an empirical evidence of the consequence of our bounds by means of a variant of Fitted Q-Iteration (FQI) , based on our shared network and for which our bounds apply, that we call Multi Fitted Q-Iteration (MFQI). Then, we perform an empirical evaluation in challenging RL problems proposing multitask variants of the Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) algorithms. These algorithms are practical implementations of the more general AVI/API framework, designed to solve complex problems. In this case, the bounds apply to these algorithms only with some assumptions, e.g. stationary sampling distribution. The outcome of the empirical analysis joins the theoretical , showing significant performance improvements compared to the singletask version of the algorithms in various RL problems, including several MuJoCo domains. Let B(X) be the space of bounded measurable functions w.r.t. the σ-algebra σ X, and similarly B(X, L) be the same bounded by L < ∞. A Markov Decision Process (MDP) is defined as a 5-tuple M =< S, A, P, R, γ >, where S is the state space, A is the action space, P: S × A → S is the transition distribution where P(s |s, a) is the probability of reaching state s when performing action a in state s, R: S × A × S → R is the reward function, and γ ∈ is the discount factor. A deterministic policy π maps, for each state, the action to perform: π: S → A. Given a policy π, the value of an action a in a state s represents the expected discounted cumulative reward obtained by performing a in s and following π thereafter:, where r i+1 is the reward obtained after the i-th transition. The expected discounted cumulative reward is maximized by following the optimal policy π * which is the one that determines the optimal action values, i.e., the ones that satisfy the Bellman optimality equation : > where t ∈ {1, . . ., T} and T is the number of MDPs. For each MDP M (t), a deterministic policy π t: In this setting, the goal is to maximize the sum of the expected cumulative discounted reward of each task. In our theoretical analysis of the MTRL problem, the complexity of representation plays a central role. As done in , we consider the Gaussian complexity, a variant of the well-known Rademacher complexity, to measure the complexity of the representation. Given a setX ∈ X T n of n input samples for each task t ∈ {1, . . ., T}, and a class H composed of k ∈ {1, . . ., K} functions, the Gaussian complexity of a random set H(X) = {(h k (X ti)): h ∈ H} ⊆ R KT n is defined as follows: where γ tki are independent standard normal variables. We also need to define the following quantity, taken from: let γ be a vector of m random standard normal variables, and f ∈ F: Equation 2 can be viewed as a Gaussian average of Lipschitz quotients, and appears in the bounds provided in this work. Finally, we define L(F) as the upper bound of the Lipschitz constant of all the functions f in the function class F. The following theoretical study starts from the derivation of theoretical guarantees for MTRL in the AVI framework, extending the of in the MTRL scenario. Then, to bound the approximation error term in the AVI bound, we extend the described in to MTRL. As we discuss, the ing bounds described in this section clearly show the benefit of sharing representation in MTRL. To the best of our knowledge, this is the first general for MTRL; previous works have focused on finite MDPs or linear models . problem: where we use f = (f 1, . . ., f T), w = (w 1, . . ., w T), and define the minimizers of Equation asŵ, h, andf. We assume that the loss function: R × R → is 1-Lipschitz in the first argument for every value of the second argument. While this assumption may seem restrictive, the obtained can be easily scaled to the general case. To use the principal of this section, for a generic loss function, it is possible to use (·) = (·) / max, where max is the maximum value of. The expected loss over the tasks, given w, h and f is the task-averaged risk: The minimum task-averaged risk, given the set of tasks µ and the hypothesis classes W, H and F is ε * avg, and the corresponding minimizers are w *, h * and f *. We start by considering the bound for the AVI framework which applies for the single-task scenario. Theorem 1. (Theorem 3.4 of) Let K be a positive integer, and ν, we have: where with E(ε 0, . . ., ε K−1 ; r) = K−1 k=0 α 2r k ε k, the two coefficients c VI1,ρ,ν, c VI2,ρ,ν, the distributions ρ and ν, and the series α k are defined as in. In the multi-task scenario, let the average approximation error across tasks be: where Q t,k+1 =f t,k •ĥ k •ŵ t,k, and T * t is the optimal Bellman operator of task t. In the following, we extend the AVI bound of Theorem 1 to the multi-task scenario, by computing the average loss across tasks and pushing inside the average using Jensen's inequality. Theorem 2. Let K be a positive integer, and we have: Remarks Theorem 2 retains most of the properties of Theorem 3.4 of , except that the regression error in the bound is now task-averaged. Interestingly, the second term of the sum in Equation depends on the average maximum reward for each task. In order to obtain this we use an overly pessimistic bound on γ and the concentrability coefficients, however this approximation is not too loose if the MDPs are sufficiently similar. We bound the task-averaged approximation error ε avg at each AVI iteration k involved in following a derivation similar to the one proposed by , obtaining: Theorem 3. Let µ, W, H and F be defined as above and assume 0 ∈ H and f = 0, ∀f ∈ F. Then for δ > 0 with probability at least 1 − δ in the draw ofZ ∼ T t=1 µ n t we have that Remarks The assumptions 0 ∈ H and f = 0 for all f ∈ F are not essential for the proof and are only needed to simplify the . For reasonable function classes, the Gaussian complexity If sup w w(X) and sup h,w h(w(X)) can be uniformly bounded, then they are O(√ nT). For some function classes, the Gaussian average of Lipschitz quotients O(·) can be bounded independently from the number of samples. Given these assumptions, the first and the fourth term of the right hand side of Equation, which represent respectively the cost of learning the meta-state space w and the task-specific f mappings, are both O(1 / √ n). The second term represents the cost of learning the multi-task representation h and is O(1 / √ nT), thus vanishing in the multi-task limit T → ∞. The third term can be removed if ∀h ∈ H, ∃p 0 ∈ P: h(p) = 0; even when this assumption does not hold, this term can be ignored for many classes of interest, e.g. neural networks, as it can be arbitrarily small. The last term to be bounded in is the minimum average approximation error ε * avg at each AVI iteration k. Recalling that the task-averaged approximation error is defined as in, applying Theorem 5.3 by we obtain: with C AE defined as in. Final remarks The bound for MTRL is derived by composing the in Theorems 2 and 3, and Lemma 4. The above highlight the advantage of learning a shared representation. The bound in Theorem 2 shows that a small approximation error is critical to improve the convergence towards the optimal action-value function, and the bound in Theorem 3 shows that the cost of learning the shared representation at each AVI iteration is mitigated by using multiple tasks. This is particularly beneficial when the feature representation is complex, e.g. deep neural networks. As stated in the remarks of Equation, the benefit of MTRL is evinced by the second component of the bound, i.e. the cost of learning h, which vanishes with the increase of the number of tasks. Obviously, adding more tasks require the shared representation to be large enough to include all of them, undesirably causing the term sup h,w h(w(X)) in the fourth component of the bound to increase. This introduces a tradeoff between the number of features and number of tasks; however, for Figure 1: (a) The architecture of the neural network we propose to learn T tasks simultaneously. The w t block maps each input x t from task µ t to a shared set of layers h which extracts a common representation of the tasks. Eventually, the shared representation is specialized in block f t and the output y t of the network is computed. Note that each block can be composed of arbitrarily many layers. a reasonable number of tasks the number of features used in the single-task case is enough to handle them, as we show in some experiments in Section 5. Notably, since the AVI/API framework provided by provides an easy way to include the approximation error of a generic function approximator, it is easy to show the benefit in MTRL of the bound in Equation. Despite being just multi-task extensions of previous works, our are the first one to theoretically show the benefit of sharing representation in MTRL. Moreover, they serve as a significant theoretical motivation, besides to the intuitive ones, of the practical algorithms that we describe in the following sections. We want to empirically evaluate the benefit of our theoretical study in the problem of jointly learning T different tasks µ t, introducing a neural network architecture for which our bounds hold. Following our theoretical framework, the network we propose extracts representations w t from inputs x t for each task µ t, mapping them to common features in a set of shared layers h, specializing the learning of each task in respective separated layers f t, and finally computing the output (Figure 1(a) ). The idea behind this architecture is not new in the literature. For instance, similar ideas have already been used in DQN variants to improve exploration on the same task via bootstrapping and to perform MTRL . The intuitive and desirable property of this architecture is the exploitation of the regularization effect introduced by the shared representation of the jointly learned tasks. Indeed, unlike learning a single task that may end up in overfitting, forcing the model to compute a shared representation of the tasks helps the regression process to extract more general features, with a consequent reduction in the variance of the learned function. This intuitive justification for our approach, joins the theoretical benefit proven in Section 3. Note that our architecture can be used in any MTRL problem involving a regression process; indeed, it can be easily used in value-based methods as a Q-function regressor, or in policy search as a policy regressor. In both cases, the targets are learned for each task µ t in its respective output block f t. Remarkably, as we show in the experimental Section 5, it is straightforward to extend RL algorithms to their multi-task variants only through the use of the proposed network architecture, without major changes to the algorithms themselves. To empirically evince the effect described by our bounds, we propose an extension of FQI , that we call MFQI, for which our AVI bounds apply. Then, to empirically evaluate our approach in challenging RL problems, we introduce multi-task variants of two well-known DRL algorithms: DQN and DDPG , which we call Multi Deep Q-Network (MDQN) and Multi Deep Deterministic Policy Gradient (MDDPG) respectively. Note that for these methodologies, our AVI and API bounds hold only with Figure 2: Discounted cumulative reward averaged over 100 experiments of DQN and MDQN for each task and for transfer learning in the Acrobot problem. An epoch consists of 1, 000 steps, after which the greedy policy is evaluated for 2, 000 steps. The 95% confidence intervals are shown. the simplifying assumption that the samples are i.i.d.; nevertheless they are useful to show the benefit of our method also in complex scenarios, e.g. MuJoCo . We remark that in these experiments we are only interested in showing the benefit of learning multiple tasks with a shared representation w.r.t. learning a single task; therefore, we only compare our methods with the single task counterparts, ignoring other works on MTRL in literature. Experiments have been developed using the MushroomRL library , and run on an NVIDIA R DGX Station TM and Intel R AI DevCloud. Refer to Appendix B for all the details and our motivations about the experimental settings. As a first empirical evaluation, we consider FQI, as an example of an AVI algorithm, to show the effect described by our theoretical AVI bounds in experiments. We consider the Car-On-Hill problem as described in , and select four different tasks from it changing the mass of the car and the value of the actions (details in Appendix B). Then, we run separate instances of FQI with a single task network for each task respectively, and one of MFQI considering all the tasks simultaneously. Figure 1(b) shows the L 1 -norm of the difference between Q * and Q π K averaged over all the tasks. It is clear how MFQI is able to get much closer to the optimal Q-function, thus giving an empirical evidence of the AVI bounds in Theorem 2. For completeness, we also show the advantage of MFQI w.r.t. FQI in performance. Then, in Figure 1 (c) we provide an empirical evidence of the benefit of increasing the number of tasks in MFQI in terms of both quality and stability. As in , our MDQN uses separate replay memories for each task and the batch used in each training step is built picking the same number of samples from each replay memory. Furthermore, a step of the algorithm consists of exactly one step in each task. These are the only minor changes to the vanilla DQN algorithm we introduce, while all other aspects, such as the use of the target network, are not modified. Thus, the time complexity of MDQN is considerably lower than vanilla DQN thanks to the learning of T tasks with a single model, but at the cost of a higher memory complexity for the collection of samples for each task. We consider five problems with similar state spaces, sparse rewards and discrete actions: Cart-Pole, Acrobot, Mountain-Car, Car-On-Hill, and Inverted-Pendulum. The implementation of the first three problems is the one provided by the OpenAI Gym library , while Car-On-Hill is described in and Inverted-Pendulum in. Figure 2(a) shows the performance of MDQN w.r.t. to vanilla DQN that uses a single-task network structured as the multi-task one in the case with T = 1. The first three plots from the left show good performance of MDQN, which is both higher and more stable than DQN. In Car-On-Hill, MDQN is slightly slower than DQN to reach the best performance, but eventually manages to be more stable. Finally, the Inverted-Pendulum experiment is clearly too easy to solve for both approaches, but it is still useful for the shared feature extraction in MDQN. The described provide important hints about the better quality of the features extracted by MDQN w.r.t. DQN. To further demonstrate this, we evaluate the performance of DQN on Acrobot, arguably the hardest of the five problems, using a single-task network with the shared parameters in h initialized with the weights of a multi-task network trained with MDQN on the other four problems. Arbitrarily, the pre-trained weights can be adjusted during the learning of the new task or can be kept fixed and only the remaining randomly initialized parameters in w and f are trained. From Figure 2(b), the advantages of initializing the weights are clear. In particular, we compare the performance of DQN without initialization w.r.t. DQN with initialization in three settings: in Unfreeze-0 the initialized weights are adjusted, in NoUnfreeze they are kept fixed, and in Unfreeze-10 they are kept fixed until epoch 10 after which they start to be optimized. Interestingly, keeping the shared weights fixed shows a significant performance improvement in the earliest epochs, but ceases to improve soon. On the other hand, the adjustment of weights from the earliest epochs shows improvements only compared to the uninitialized network in the intermediate stages of learning. The best are achieved by starting to adjust the shared weights after epoch 10, which is approximately the point at which the improvement given by the fixed initialization starts to lessen. In order to show how the flexibility of our approach easily allows to perform MTRL in policy search algorithms, we propose MDDPG as a multi-task variant of DDPG. As an actor-critic method, DDPG requires an actor network and a critic network. Intuitively, to obtain MDDPG both the actor and critic networks should be built following our proposed structure. We perform separate experiments on two sets of problems with similar continuous state and action spaces: the first set includes Inverted-Pendulum, Inverted-Double-Pendulum, and Inverted-Pendulum-Swingup as implemented in the pybullet library, whereas the second set includes Hopper-Stand, Walker-Walk, and Half-Cheetah-Run as implemented in the. Figure 3(a) shows a relevant improvement of MDDPG w.r.t. DDPG in the pendulum tasks. Indeed, while in Inverted-Pendulum, which is the easiest problem among the three, the performance of MDDPG is only slightly better than DDPG, the difference in the other two problems is significant. The advantage of MDDPG is confirmed in Figure 3 (c) where it performs better than DDPG in Hopper and equally good in the other two tasks. Again, we perform a TL evaluation of DDPG in the problems where it suffers the most, by initializing the shared weights of a single-task network with the ones of a multi-task network trained with MDDPG on the other problems. Figures 3(b) and 3(d) show evident advantages of pre-training the shared weights and a significant difference between keeping them fixed or not. Our work is inspired from both theoretical and empirical studies in MTL and MTRL literature. In particular, the theoretical analysis we provide follows previous about the theoretical properties of multi-task algorithms. For instance, and prove the theoretical advantages of MTL based on linear approximation. More in detail, derives bounds on MTL when a linear approximator is used to extract a shared representation among tasks. , which we considered in this work, describes similar that extend to the use of non-linear approximators. Similar studies have been conducted in the context of MTRL. Among the others, and give theoretical proofs of the advantage of learning from multiple MDPs and introduces new algorithms to empirically support their claims, as done in this work. Generally, contributions in MTRL assume that properties of different tasks, e.g. dynamics and reward function, are generated from a common generative model. About this, interesting analyses consider Bayesian approaches; for instance assumes that the tasks are generated from a hierarchical Bayesian model, and likewise considers the case when the value functions are generated from a common prior distribution. Similar considerations, which however does not use a Bayesian approach, are implicitly made in , , and also in this work. In recent years, the advantages of MTRL have been empirically evinced also in DRL, especially exploiting the powerful representational capacity of deep neural networks. For instance, and propose to derive a multi-task policy from the policies learned by DQN experts trained separately on different tasks. compares to a therein introduced variant of DQN, which is very similar to our MDQN and the one in , showing how their method overcomes it in the Atari benchmark. Further developments, extend the analysis to policy search , and to multi-goal RL . addresses the problem of balancing the learning of multiple tasks with a single deep neural network proposing a method that uniformly adapts the impact of each task on the training updates of the agent. We have theoretically proved the advantage in RL of using a shared representation to learn multiple tasks w.r.t. learning a single task. We have derived our extending the AVI/API bounds to MTRL, leveraging the upper bounds on the approximation error in MTL provided in. The of this analysis show that the error propagation during the AVI/API iterations is reduced according to the number of tasks. Then, we proposed a practical way of exploiting this theoretical benefit which consists in an effective way of extracting shared representations of multiple tasks by means of deep neural networks. To empirically show the advantages of our method, we carried out experiments on challenging RL problems with the introduction of multi-task extensions of FQI, DQN, and DDPG based on the neural network structure we proposed. As desired, the favorable empirical confirm the theoretical benefit we described. A PROOFS Proof of Theorem 2. We compute the average expected loss across tasks: with γ = max VI,ρ,ν (K; t, r), and R max,avg = 1 /T T t=1 R max,t. Considering the term 1 /T Using Jensen's inequality: So, now we can write as avg (ε avg,0, . . ., ε avg,K−1 ; r) with ε avg,k = 1 /T T t=1 ε t,k and E avg (ε avg,0, . . ., ε avg,K−1 ; r) = Proof of Lemma 4. Let us start from the definition of optimal task-averaged risk: where Q * t,k, with t ∈ [1, T], are the minimizers of ε avg,k. Consider the taskť such thatť we can write the following inequality: By the application of Theorem 5.3 by to the right hand side, and defining Qť k,i ν, we obtain: Squaring both sides yields the : We start by considering the bound for the API framework: ν, we have: where with E(ε 0, . . ., ε K−1 ; r) = K−1 k=0 α 2r k ε k, the three coefficients c PI1,ρ,ν, c PI2,ρ,ν, c PI3,ρ,ν, the distributions ρ and ν, and the series α k are defined as in. From Theorem 5, by computing the average loss across tasks and pushing inside the average using Jensen's inequality, we derive the API bounds averaged on multiple tasks. ν, we have: avg (ε avg,0, . . ., ε avg,K−1 ; r) with Proof of Theorem 6. The proof is very similar to the one for AVI. We compute the average expected loss across tasks: with ε avg,k = 1 /T T t=1 ε t,k and E avg (ε avg,0, . . ., ε avg,K−1 ; r) = K−1 k=0 α 2r k ε avg,k. Proof of Theorem 3. Let w * 1,..., w * T, h * and f * 1,..., f * T be the minimizers of ε * avg, then: We proceed to bound the three components individually: • C can be bounded using Hoeffding's inequality, with probability 1 − δ /2 by ln(2 /δ) /(2nT), as it contains only nT random variables bounded in the interval; Firstly, to bound L(F), let y, y ∈ R KT n, where y = (y ti) with y ti ∈ R K and y = (y ti) with y ti ∈ R K. We can write the following: whence L(F) ≤ L(F). Then, we bound: Then, since it is possible to bound the Euclidean diameter using the norm of the supremum value in the set, we bound D(S) ≤ 2 sup h,w h(w(X)) and D(W (X)) ≤ 2 sup w∈W T w(X). E sup g∈F γ, g(y) − g(y) = E sup f ∈F T ti γ ti (f t (y ti) − f t (y ti)) whence O(F) ≤ √ To minimize the last term, it is possible to choose y 0 = 0, as f = 0, ∀f ∈ F, ing in min y∈Y G(F(y)) = G(F) = 0. Then, substituting in, and recalling that G(S) ≤ G(S):"

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: We introduce MTLAB, a new algorithm for learning multiple related tasks with strong theoretical guarantees. Its key idea is to perform learning sequentially over the data of all tasks, without interruptions or restarts at task boundaries. Predictors for individual tasks are derived from this process by an additional online-to-batch conversion step. By learning across task boundaries, MTLAB achieves a sublinear regret of true risks in the number of tasks. In the lifelong learning setting, this leads to an improved generalization bound that converges with the total number of samples across all observed tasks, instead of the number of examples per tasks or the number of tasks independently. At the same time, it is widely applicable: it can handle finite sets of tasks, as common in multi-task learning, as well as stochastic task sequences, as studied in lifelong learning. In recent years, machine learning has become a core technology in many commercially relevant applications. One observation in this context was that real-world learning tasks often do not occur in isolation, but rather as collections or temporal sequences of many, often highly related tasks. Examples include click-through rate prediction for online ads, personalized voice recognition for smart devices, or handwriting recognition of different languages. Multi-task learning BID3 has been developed exactly to handle such situations. It is based on an intuitive idea that sharing information between tasks should help the learning process and therefore lead to improved prediction quality. In practice, however, this is not guaranteed and multi-task learning can even lead to a reduction of prediction quality, so called negative transfer. The question when negative transfer occurs and how it can be avoided has triggered a surge of research interest to better understanding the theoretical properties of multi-task learning, as well as related research areas, such as lifelong learning BID1 BID9, where more and more tasks occur sequentially, and task curriculum learning, where the order in which to learn tasks needs to be determined. In this work, we describe a new approach to multi-task learning that has strong theoretical guarantees, in particular improving the rate of convergence over some previous work. Our core idea is to decouple the process of predictor learning from the task structure. This is also the main difference of our approach to previous work, which typically learned one predictor for each task. We treat the available data for all tasks as parts of a single large online-learning problem, in which individual tasks simply correspond to subsets of the data stream that is processed. To obtain predictors for the individual tasks, we make use of online-to-batch conversion methods. We name the method MTLAB (multi-task learning across boundaries).Our main contribution is a sublinear bound on the task regret of MTLAB with true risks. As a corollary, we show that MTLAB improves the existing convergence rates in the case of lifelong learning. From the regret-type bounds, we derive high probability bounds on the expected risk of each task, which constitutes a second main contribution of our work. For real-world problems, not all tasks might be related to all previous ones. Our third contribution is a theoretically well-founded, yet practical, mechanism to avoid negative transfer in this case: we show that by splitting the set of tasks into homogeneous groups and using MTLAB to learn individual predictors on each of the ing subsequences of samples, one obtains the same strong guarantees for each of the learned predictors while avoiding negative transfer. In this section we present the main notation and introduce the MTLAB approach to information transfer between tasks. We face a sequence of tasks k 1,..., k n,..., where each k t from a task environment K, and the sequence is a random realization of a stochastic process over K. Note that this general formulation includes the situations most commonly studied in the literature: the case of finitely many fixed tasks (in which case the distribution over the tasks sequence is a delta peak) and the lifelong learning setting with i.i.d. BID1 BID9 or non-i.i.d. tasks.All tasks share the same input set X, output set Y, and hypothesis set H. Each task k t, however, has its own associated joint probability distribution, D t, over X × Y, conditioned on k t. Whenever we observe a task k t, we receive a set S t = {(x t,i, y t,i)} mt i=1 sampled i.i.d. from the task distribution D t, and we are given a loss function, DISPLAYFORM0 that measures the quality of predictions. Alternatively, one can assume that all tasks share the same, a priori known, loss function. Learning a task k t means to identify a hypothesis h ∈ H with as small as possible per-task risk er t (h), which is defined as DISPLAYFORM1 The PAC-Bayes framework, originated in BID7 BID12, studies the performance of stochastic (Gibbs) predictors. A stochastic predictor is defined by a probability distribution Q over the hypotheses set. For any Gibbs predictor with a distribution Q we define the corresponding true risk of a predictor as DISPLAYFORM2 As described in the introduction, we do not require that data for all tasks is available at the same time. Instead, we adopt an online learning protocol for tasks: at step t we observe the dataset S t for task k t, and we output the distributionQ t. Our first goal is, at any step n, to bound the regret of a learned sequence of predictorsQ 1,...,Q n with respect to any fixed reference distribution Q from some set, ∆, of distributions, i.e. DISPLAYFORM0 Note that the regret is defined using true risks, that we do not observe, in contrast to empirical risks. This makes the problem setting very different from the traditional online learning where the empirical performance is considered. The main idea of MTLAB is to run an online learning algorithm on the samples from all tasks, essentially ignoring the task structure of the problem, and then use a properly defined online-to-batch conversion to obtain predictors for the individual tasks. In this paper, we work with Input: decision set ∆, initial distribution P, learning rate η Initialization: Q 1,0 = P At any time point t = 1, 2,...: BID4 run on the level of samples. Let P be some prior distribution over H. We set Q 1,0 = P and, once we receive a dataset S t = {(x t,1, y t,1),..., (x t,mt, y t,mt)} on step t, we compute a sequence of predictors Q t,i each being a solution to DISPLAYFORM1 DISPLAYFORM2 for all i = 1,..., m t with η > 0. Afterwards, the algorithm outputs a predictorQ t = 1 mt mt i=1 Q t,i for task t, and sets Q t+1,0 = Q t,mt, to be used as a starting distribution for the next task. We call the above procedure MTLAB (multi-task learning across task boundaries) and summarize it in Figure 1. Our first main is a regret bound for the true risks of the sequence of distributions that it produces. Theorem 1. Letm = n/(n t=1 1/m t) be the harmonic mean of m 1,..., m n and let P be a fixed prior distribution that is chosen independently of the data. The predictors produced by MTLAB satisfy with probability 1 − δ (over the random training sets) uniformly over Q ∈ ∆ DISPLAYFORM3 Corollary. Set η = m n. Then, with probability 1 − δ, it holds uniformly over DISPLAYFORM4 To put this into perspective, we compare it to the average regret bounds given in BID0, where the goal is to find the best possible data representation for tasks. Even though the settings are a bit different, it gives a good idea of the qualitative nature of our . BID0 provides O(DISPLAYFORM5) bound (if all tasks are of the same size m) that can be sometimes improved to O(DISPLAYFORM6 In either case, convergence happen only in the regime when the number of tasks and the amount of data for each task both tend to infinity. In contrast to this, the right hand side of inequality converges to zero even if only one of the two quantities grows, so in particular for the most common case that the number of tasks grows to infinity, but the amount of data per task remains bounded. The examples of real-world implementations of MTLAB are provided in the supplementary material. We obtain further insight into the behavior of MTLAB by comparing it to the situation in which each task is learned independently. A more traditional PAC-Bayes bound (e.g. BID6) states that with probability 1 − δ the following inequality holds for all Q DISPLAYFORM0 This inequality suggests a learning algorithm, namely to minimize the upper bound with respect to Q. In principle, MTLAB is based on a similar objective, but it acts on the sample level and it automatically provides relevant prior distributions for each task. Thereby it is able to achieve better guarantee than one could get by combining separate bounds of the form for multiple tasks. The bound of Theorem 1 holds for any stochastic process over the tasks. In particular, it holds in special case where tasks are sampled independently from a hyper distribution over the task environment, which is usually called lifelong learning BID1 BID9. In this setting, we have a fixed distribution T over K, and the sequence k 1,..., k n is an i.i.d. sample from this distribution. One can then define the lifelong risk as DISPLAYFORM0 where D k and k are the distribution and loss function for a task k, respectively. The risk of the Gibbs predictor is then DISPLAYFORM1..,Q n be the output of MTLAB, then we define the corresponding batch solution asQ n = 1 n n t=1Q t and observe DISPLAYFORM2 Using Theorem 1 we obtain the following guarantee. Theorem 2. In the lifelong learning setting, if we run MT-LAB with η = √m √ n, for any fixed prior distribution P that is chosen independently from the data, with probability 1 − δ DISPLAYFORM3 Typical for this setting, such as shown in BID9 BID5 BID0, show the additive convergence rate O(DISPLAYFORM4, which goes to zero only in the case of infinite data and infinite tasks. In contrast, the generalization error for MTLAB converges in the most realistic scenario of finite data per task and increasing number of tasks. The of the previous section provide guarantees on MTLAB's multi-task regret. In this section we compliment those by presenting a modification that provides guarantees for individual risks of each task. The detailed proofs of all statements can be found in the supplementary material. As a start, let us consider a bound that can be obtained immediately from Theorem 1. We make use of the following notion of relatedness between tasks that is commonly used in the field of domain adaptation BID2 . Definition 1. For a fixed hypothesis class H, the discrepancy between tasks k i and k j is defined as DISPLAYFORM0 The following theorem is an immediate corollary of Theorem 1. Theorem 3. Let P be a fixed prior distribution that is chosen independently of the data. LetQ t be a sequence of predictors produced by MTLAB run with η = m n and let Q n = 1 n n t=1Q t . Then the following inequality holds with probability 1 − δ, uniformly over Q ∈ ∆ DISPLAYFORM1 DISPLAYFORM2 This bound resembles the guarantees typical in the setting of learning from drifting distributions BID8 . It converges if 1 n n i=1 disc(k i, k n) → 0 with n, so if either tasks are identical to each other, or if tasks get suitably more similar on average with growing n. This is a good example of possible negative transfer: when the previous tasks are not related to the current one as measured by the discrepancies, the average discrepancy term will prevent the bound from convergence. The main question is if we can avoid the negative transfer and improve upon the bound of Theorem 3 in the case when 1 n n i=1 disc(k i, k n) does not vanish over time. Consider, for example, a simple case of two alternating tasks, i.e. DISPLAYFORM3 If we split the sequence of tasks into two subsequences, one for tasks with even and one for tasks with odd indices, and then run MT-LAB separately for each sequence, we could nevertheless guarantee the convergence of the error rate for the ing procedure. Unfortunately, it is rather easy to construct examples in which convergence to zero is not achievable, even with the best possible split of the sequence of tasks into subsequences. Consequently, we redefine our goal to prove error rates that converge below a given threshold ε. We present an online algorithm, MTLAB.MS (for MTLAB with Multiple Sequences), that splits the tasks into subsequences on the fly given some distance dist(k i, k j) between tasks. MTLAB.MS keeps a representative task for each subsequence, and we use the distances to the representatives to decide which subsequence to extend with the new task, or if a new subsequence needs to be initialized. Pseudo-code for MTLAB.MS is provided in Algorithm 2. The notationQ, P = MTLAB(S, P) denotes a single run of MTLAB that takes a dataset S, runs its learning procedure starting from distribution P and outputs two distributions: the final distribution P to be used in the subsequent runs and the aggregate distributionQ that is a final predictor for the task. Further notation used are: I n are the indices of the tasks in the subsequence chosen at step n, s n = |I n | is the size of this subsequence,m n is the harmonic average of the sizes of tasks in the chosen subsequence and η n is the learning rate of MTLAB associated with the chosen subsequence. The following theorem shows that if MTLAB.MS could be run with the task discrepancies as distances, it would, for any given threshold ε, yield subsequences with generalization error below ε. Theorem 4. Let P be a fixed prior distribution that is chosen independently of the data. If we run MTLAB.MS with dist(k i, k j) = disc(k i, k j), we get with probability 1 − δ, DISPLAYFORM4 This theorem works when the transfer algorithm uses a fixed learning rate η for each subsequence. It is possible to prove a similar statement for the case when the parameters are optimized for the length of each subsequence using the machinery developed in BID13. However, the final statement gets more complicated and adds little to the discussions in the current paper. Therefore, we leave this extension for future work. Input: task distance dist, prior distribution P, threshold ε Initialization: set of representative tasks R = ∅ set of priors P = ∅ At any time point t = 1, 2,...:• receive dataset S t.• set I = {r ∈ R : dist(k r, k t) ≤ ε} • if I = ∅ then -add t to the set of representatives R -set P(t) = P • choose the closest representatives r = argmin DISPLAYFORM5 • run the transfer algorithm: DISPLAYFORM6 • set P(r) = P • outputQ t Figure 2. MTLAB.MS algorithm Theorem 4 confirms that it is possible to avoid effects of negative transfer by carefully choosing the tasks we transfer knowledge from at each step. MTLAB.MS is a computationally efficient way of doing this. In practice, however, the true discrepancy values are unknown. The most direct method to determine the right subsequence for each task is to estimate the discrepancies from the data and use the estimates in the MTLAB.MS algorithm. In the supplementary material we detail two approaches for discrepancy estimation: a) using a part of the labelled training data and b) using separate unlabelled datasets. In both cases it is possible to prove the statements similar to Theorem 4. We introduced a new and widely applicable algorithm for sequentially learning of multiple tasks. By performing learning across tasks boundaries it is able to achieve a sublinear regret bound and improves the convergence rates in the lifelong learning scenario. MTLAB's way of not interrupting or restarting the learning process at task boundaries in faster convergence rates than what can be achieved by learning individual predictors for each task: in particular, the generalization error decreases with the product of the number of tasks and the number of samples per task, instead of separately in each of these quantities. We also introduced a mechanism for the situation when the tasks to be learned are not all related to each other. We show that by constructing suitable subsequences of task, the convergence properties can hold even in this case.

represent the tldr for finding the corresponding scientific article

A new algorithm for online multi-task learning that learns without restarts at the task borders

represent the short summary for finding the corresponding scientific article (long)!

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.