Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The key challenge in semi-supervised learning is how to effectively leverage unlabeled data to improve learning performance. The classical label propagation method, despite its popularity, has limited modeling capability in that it only exploits graph information for making predictions. In this paper, we consider label propagation from a graph signal processing perspective and decompose it into three components: signal, filter, and classifier. By extending the three components, we propose a simple generalized label propagation (GLP) framework for semi-supervised learning. GLP naturally integrates graph and data feature information, and offers the flexibility of selecting appropriate filters and domain-specific classifiers for different applications. Interestingly, GLP also provides new insight into the popular graph convolutional network and elucidates its working mechanisms. Extensive experiments on three citation networks, one knowledge graph, and one image dataset demonstrate the efficiency and effectiveness of GLP. The success of deep learning and neural networks comes at the cost of large amount of training data and long training time. Semi-supervised learning BID37 BID8 ) is interesting and important as it can leverage ample available unlabeled data to aid supervised learning, thus greatly saving the cost, trouble, and time for human labeling. Many researches have shown that when used properly, unlabeled data can significantly improve learning performance BID38 BID16. The key challenge for semi-supervised learning is how to effectively leverage the information of unlabeled data, such as graph structures and data features. Label propagation BID39 BID36 BID2 is arguably the most popular method for graph-based semi-supervised learning. As a simple and effective tool, it has been widely used in many scientific research fields and has found numerous industrial applications. Given a non-oriented graph G = (V, W, X) with n = |V| vertices, a nonnegative symmetric affinity matrix W ∈ R n×n + encoding edge weights, and a feature matrix X ∈ R n×m which contains an mdimensional feature vector of each vertex. For semi-supervised classification, only a small subset of vertices are labeled, and the goal is to predict the labels of other vertices. Denote by Y ∈ {0, 1} n×l the labeling matrix 1 with l being the number of classes. The objective of of label propagation (LP) is to find a prediction (embedding) matrix Z ∈ R n×l which agrees with Y while being smooth on the graph such that nearby vertices have similar embeddings: DISPLAYFORM0 where α is a balancing parameter, L = D − W is the graph Laplacian 2 and D is the degree matrix. The term enforcing smoothness is called graph Laplacian regularization or Tikhonov regularization. Solving the quadratic regularization framework gives the prediction of LP.As LP makes predictions only based on graph information (W), its performance depends on whether the underlying graph structure can well represent the class information of data -vertices in the same 1 If the label of vertex vi is known, then Y (i, :) is a one-hot embedding of vi with yij = 1 if vi belongs to the j-th class and yij = 0 otherwise. If the label of vertex vi is not given, then Y (i, :) is a vector of all zeros.2 Other variants such as the normalized Laplacian matrices are also applicable.cluster tend to have same labels. For some applications such as social network analysis, data exhibits a natural graph structure. For some other applications such as image or text classification, data may come in a vector form, and a graph is usually constructed using data features. Nevertheless, in many cases, graphs only partially encode data information. Take document classification in a citation network as an example, the citation links between documents form a graph which represents their citation relation, and each document is represented as a bag-of-words feature vector which describes its content. To correctly classify a document, both the citation relations (W) and the content information (X) need to be taken into account, as they contain different aspects of document information. However, in this case, LP can only exploit the graph information to make predictions without using any of the feature information, thus ing in poor performance. To go beyond the limit of LP and jointly model graph and feature information, a common approach is to train a supervised learner to classify data features while regularizing the classifier using graph information. Manifold regularization BID1 trains a support vector machine with a graph Laplacian regularizer. Deep semi-supervised embedding BID32 and Planetoid BID34 ) train a neural network with an embedding-based regularizer. The recently proposed graph convolutional neural networks BID16 ) adopts a different approach by integrating graph and feature information in each of its convolutional layer, which is coupled with a projection layer for classification. In this paper, we extends the modeling capability of LP in the context of graph signal processing. Casted in the spectral domain, LP can be interpreted as low-pass graph filtering BID10 BID11. In light of this, we decompose LP into three components: graph signal, graph filter, and classifier. By naturally extending the three components, we propose a generalized label propagation (GLP) framework for semi-supervised learning. In GLP, a low-pass graph filter is applied on vertex features to produce smooth features, which are then fed to a supervised learner for classification. After filtering, the data features within each class are more similar and representative, making it possible to train a good classifier with few labeled examples. GLP not only extends LP to incorporate vertex features in a simple way, but also offers the flexibility of designing appropriate graph filters and adopting domain-specific classifiers for different semisupervised applications. The popular graph convolutional networks (GCN) BID16 is closely related to GLP. In fact, GCN without internal ReLUs is a special case of GLP with a certain graph filter and a multilayer perceptron classifier. When revisited under the GLP framework, it makes clear the working mechanisms of GCN including its design of convolutional filter and model parameter setting. Extensive experiments on citation networks, knowledge graphs, and image datasets show substantial improvement of GLP over GCN and other baselines for semi-supervised classification, confirming the effectiveness of this simple and flexible framework. The rest of the paper is organized as follows. Section 2 interprets LP in the context of graph signal processing. Section 3 presents the proposed GLP framework. Section 4 revisits GCN under GLP. Section 5 discusses the design of graph filters for GLP. Section 6 presents experimental . Section 7 discusses related works. Finally, section 8 concludes the paper. In this section, we provide a spectral view of LP in the context of graph signal processing. In graph signal processing BID24, the eigenvectors and eigenvalues of the graph Laplacian play the role of Fourier basis and frequencies in parallel with classical harmonic analysis. The graph Laplacian matrix can be eigen-decomposed as: L = ΦΛΦ −1, where Λ = diag(λ 1, · · ·, λ n) are the eigenvalues in an increasing order, i.e., 0 = λ 1 ≤ · · · ≤ λ n, and Φ = (φ 1, · · ·, φ n) are the associated orthogonal eigenvectors. Note that the row normalized graph Laplacian L r = D −1 L and the symmetrically normalized graph Laplacian L s = D A graph signal is a real-valued function f: V → R defined on the vertex set of a graph. Denote by f = (f (v 1), · · ·, f (v n)) a graph signal in a vector form. Consider (φ i) 1≤i≤n as basis functions. Any graph signal f can be decomposed into a linear combination of the basis functions: DISPLAYFORM0 where c = (c 1, · · ·, c n) and c i is the coefficient of φ i. The magnitude of the coefficient |c i | represents the strength of the basis function φ i presented in the signal f.A graph filter is defined as a matrix G ∈ R n×n. G is linear shift-invariant BID22, if and only if there exists an function p(·): R → R, satisfying G = Φp(Λ)Φ −1, where DISPLAYFORM1 It is well known that the basis functions associated with lower frequencies (smaller eigenvalues) are smoother BID38, as the smoothness of φ i can be measured by λ i: DISPLAYFORM2 This indicates that a smooth signal f should contain more low-frequency components than highfrequency components. To produce a smooth signal, the graph filter G should be able to preserve the low-frequency components in f while filtering out the high-frequency components. By Eq., we havef DISPLAYFORM3 In the filtered signalf, the coefficient c i of the basis function φ i is scaled by p(λ i). To preserve the low-frequency components and remove the high-frequency components, p(λ i) should amplify c i when λ i is small and suppress c i when λ i is large. Simply put, p(·) should behave like a low-pass filter in classical harmonic analysis. The prediction (embedding) matrix of LP can be obtained by taking the derivative of the unconstrained quadratic optimization problem in Eq. and setting it to zero: DISPLAYFORM0 With the prediction matrix Z, each unlabeled vertex v i is usually classified by simply comparing the elements in Z(i, :). In some methods, a normalization scheme may be applied on the columns of Z first before the comparison BID39.Casted in the context of graph signal processing, LP can be decomposed into three components: signal, filter, and classifier. By Eq., the input signal matrix of LP is the labeling matrix Y, where it has l channels and each column Y (:, i) can be considered as a graph signal. In Y (:, i), only the labeled vertices in class i have value 1 and others 0.The graph filter used in LP is DISPLAYFORM1 with frequency response function DISPLAYFORM2 Note that this also holds for the normalized graph Laplacians. As shown in FIG2, the frequency response function of LP is low-pass. For any α > 0, p(λ i) is near 1 when λ i is close to 0 and p(λ i) decreases and approaches 0 as λ i increases. Applying the filter on signal Y (:, i), it will produce a smooth signal Z(:, i) in which vertices of the same class have similar values and vertices in class i have larger values than others under the cluster assumption. The balancing parameter α controls the degree of the graph Laplacian regularization. When α increases, the filter becomes more low-pass (FIG2) and will produce smoother embeddings. Finally, LP applies a nonparametric classifier on the embeddings to classify the unlabeled vertices, i.e., the label of an unlabeled vertex v i is given by y i = arg max j Z(i, j). We propose a generalized label propagation (GLP) framework by naturally generalizing the three components of LP for semi-supervised classification:• Signal: Use the feature matrix X instead of the labeling matrix Y as input signal.• Filter: The filter G can be any low-pass, linear, shift-invariant filter.• Classifier: The classifier can be any classifer trained on the embeddings of labeled vertices. GLP consists of two steps. First, a low-pass, linear, shift-invariant graph filter G is applied on the feature matrix X to obtain a smooth feature matrixX ∈ R n×m: DISPLAYFORM0 The next step is to train a supervised classifier (e.g., multilayer perceptron, convolutional neural networks, support vector machines, etc.) with the filtered features of labeled data, and then apply the classifier on the filtered features of unlabeled data to predict their labels. GLP naturally combines graph and feature information in Eq. FORMULA9, and allows taking advantage of a powerful supervised classifier. The rationale behind GLP is to learn representative feature vectors of each class for easing the downstream classification task. After filtered by G, vertices in the same class are expected to have more similar and representative features, which makes it much easier to train a good classifier with very few samples. Consider an extreme case that each class is a connected component of the graph. In this case, we can learn perfect features by an extremely low-pass filter G, whose spectrum p(·) is unit impulse function, i.e., p = 1 and p(λ) = 0 if λ = 0. We can compute G = Φp(Λ)Φ −1 in the spatial domain. In particular, G ij = 1 l k if v i and v j are of the same class, otherwise G ij = 0, where l k is the number of labeled samples in class k. After filtered by G, vertices in the same class will have an identical feature vector which is its class mean. Then any classifier that can correctly classify the labeled data will achieve 100% accuracy on the unlabeled data, and only one labeled example per class is needed to train the classifier. In this section, we show that the graph convolutional networks (GCN) for semi-supervised classification can be interpreted under the GLP framework, which explains its implicit design features including the number of layers, the choice of the normalized graph Laplacian, and the renormalization trick on the convolutional filter. Graph Convolutional Networks. The GCN model contains three steps. First, a renormalization trick is applied on the adjacency matrix W by adding an self-loop to each vertex, which in a new adjacency matrixW = W + I with the degree matrixD = D + I. After that, symmetrically normalizeW and getW s =D DISPLAYFORM0 where H (t) is the matrix of activations in the t-th layer and DISPLAYFORM1 is the trainable weight matrix in layer t, and σ is the activation function, e.g., ReLU(·) = max(0, ·). The graph convolution is defined by multiplying the input of each layer with the renormalized adjacency matrixW s from the left, i.e.,W s H (t). The convoluted features are then fed into a projection matrix Θ (t). Third, stack two layers up and apply a softmax function on the output features to produce a prediction matrix: DISPLAYFORM2 and train the model using the cross-entropy loss over the labeled instances. The graph convolution in each layer of the GCN model actually performs feature smoothing with a low-pass filterW s = I −L s, whereL s is the symmetrically normalized graph Laplacian of the graph with extra self-loops. Suppose thatL s can be eigen-decomposed asL s = ΦΛΦ −1, then we have I −L s = Φ(I −Λ)Φ −1. The frequency response function of the filter is DISPLAYFORM0 Clearly, as shown in FIG2, this function is linear and low-pass on the interval, but not on, as it amplifies the eigenvalues near 2.Interestingly, by removing the activation function ReLU in Eq., we can see that GCN is a special case of GLP, where the input signal is X, the filter isW 2 s, and the classifier is a two-layer multi-layer perceptron (MLP).Why the Normalized Graph Laplacian. Note that the eigenvalues of the normalized Laplacians L s and L r all fall into interval , while the unnormalized Laplacian L has eigenvalues in [0, +∞]. If using the unnormalized graph Laplacian, the response function in Eq. will amplify eigenvalues in [2, +∞], which will introduce noise and undermine performance. Why Two Convolutional Layers. In Eq., the GCN model stacks two convolutional layers. Without the activation function, the feature matrix is actually filtered by I −L s twice, which is equivalent to be filtered by (I −L s) 2 with response function (1 − λ) 2. As we can see from FIG2, DISPLAYFORM1 2 is more low-pass than (1 − λ) by suppressing the eigenvalues in the mid-range of harder, which explains why GCNs with two convolutional layers perform better than those with only one. Why the Renormalization Trick. The effect of the renormalization trick is illustrated in FIG4, where the response functions on the eigenvalues of L s andL s on the Cora citation network are plotted. We can see that by adding a self-loop to each vertex, the range of eigenvalues shrink from to [0, 1.5], thus avoiding amplifying eigenvalues near 2 and reducing noise. This explains why the renormalization trick works. In this section, we discuss the design and computation of low-pass graph filters for GLP.Auto-Regressive. The Auto-Regressive (AR) filter is the one used in LP: DISPLAYFORM0 Actually p ar is an auto-regressive filter of order one BID29. We have shown p ar is low-pass in section 2.2. However, the computation of p ar involves matrix inversion, which is also computationally expensive with complexity O(n 3). Fortunately, we can circumvent this problem by approximating p ar using its polynomial expansion: DISPLAYFORM1 We can then computeX = p ar (L)X iteratively with DISPLAYFORM2 and letX = 1 1+α X (k). Empirically, we find that k = 4α is enough to get a good approximation. Hence, the computational complexity is reduced to O(nmα + N mα) (note that X is of size n × m), where N is the number of nonzero entries in L, and N n 2 when the graph is sparse. Renormalization. The renormalization (RNM) filter is an exponential function of the renormalized adjacency filter used in GCN: DISPLAYFORM3 We have shown in section 4.1 that although the response function p rnm (λ) = (1 − λ) k is not lowpass, the renormalization trick shrinks the range of eigenvalues ofL and makes p rnm resemble a low-pass filter. The exponent parameter k controls the low-pass effect of p rnm. When k = 0, p rnm is all-pass. When k increases, p rnm becomes more low-pass. Note that for a sparse graph, (I −L) is a sparse matrix. Hence, the fastest way to computeX = p rnm (L)X is to left multiply X by (I −L) repeatedly for k times, which has the computational complexity O(N mk).Random Walk. We also propose to design a random walk (RW) filter: DISPLAYFORM4 We call p rw the random walk filter because DISPLAYFORM5 is a stochastic matrix of a lazy random walk which at each step returns to the current state with probability 1 2, and DISPLAYFORM6 is the k-step transition probability matrix. Similarly, we can derive the response function of p rw as DISPLAYFORM7 Note that L r has the same eigenvalues with L s, with range. Unlike the RNM, p rw is a typical low-pass filter on, as shown in FIG2. We can also see in FIG2 that the curves of (1−λ) 2 and (1 − 1 2 λ) 4 are very close, implying that to have the same level of low-pass effect, k in p rw should be set twice as large as in p rnm. This may be explained by the fact that the two functions (1 − λ) DISPLAYFORM8 2k have the same derivative k at λ = 0. On the computation side, RW has the same complexity O(N mk) as RNM.An important issue of filter design for GLP is how to control the strength of filters by setting parameters such as α and k. Intuitively, when labeled data is scarce, it would be desirable for the filtered features of each instance to be closer to its class mean and be more representative of its own class. Hence, in this case, α and k should be set large to produce smoother features. However, oversmoothing usually in inaccurate class boundaries. Therefore, when the amount of labeled data is reasonably large, α and k should be set relatively small to preserve feature diversity in order to learn more accurate class boundaries. Datasets In this section, we test GLP on three citation networks -Cora, CiteSeer and PubMed BID23, one knowledge graph -NELL BID5, and one handwritten digit image dataset -MNIST BID18. Dataset discriptions are provided in Appendix A. Baselines On citation networks and NELL, we compare GLP against GCN BID16, LP BID33, multi-layer perceptron (MLP), Planetoid BID34, DeepWalk BID21, manifold regularization (ManiReg) BID1, semi-supervised embedding (SemiEmb) BID32, and iterative classification algorithm (ICA) BID23. On MNIST, we compare GLP against GCN, LP, MLP, and convolutional neural networks (CNN).Experimental Setup We test GLP with RNM, RW and AR filters (section 5) on all the datasets. We use MLP as the classifier for GLP on citation networks and NELL, and use CNN as the classifier on MNIST. Guided by our analysis in section 5, the filter parameters k and α should be set large with small label rate and set small with large label rate. We use fixed parameters k = 10 for RNM, k = 20 for RW, and α = 20 for AR when label rate is less than or equal to 2%, and set them to 5, 10, 10 respectively otherwise. We follow BID16 to set the parameters of MLP, including learning rate, dropout, and weight decay. To make sure GLP works in practice and for more fair comparison with baselines, we do not use a validation set for classifier model selection as in BID16, instead we select the classifier with the highest training accuracy in 200 steps. Results of GLP and GCN on all the datasets are reported without using a validation set, except that on NELL, we also report the with validation (on the right of "/"). More implementation details are provided in Appendix B due to space limitations. Performance of GLP The are summarized in TAB0, where the top 3 classification accuracies are highlighted in bold. Overall, GLP performs the best on all the datasets. On citation networks, with 20 labels per class, GLP performs comparably with GCN and outperforms other baselines by a considerable margin. With 4 labels per class, GLP significantly outperforms all baselines including GCN. On NELL, GLP wtih RW and RNM filters consistently outperforms the best baseline Planetoid for each setting, and outperforms other baselines including GCN by a large margin. Note that GLP achieves this performance without using any additional validation set. The performance of GLP (and GCN) will be further boosted with validation, as shown on the right of "/". On MNIST, GLP consistently outperforms all baselines for every setting. The running times of GLP and some other baselines are also reported in TAB0. GLP runs much faster than GCN on most datasets, except for NELL, on which the running times of GLP with two filters are similar with GCN. More discussions about running times are included in Appendix E.Results Analysis Compared with LP and DeepWalk which only use graph information, the large performance gains of GLP clearly comes from leveraging both graph and feature information. Compared with purely supervised MLP and CNN which are trained on raw features, the performance gains of GLP come from the unsupervised feature filtering. FIG5 visualizes the raw and filtered features (by RNM filter) of Cora projected by t-SNE (Van der BID30 . The filtered features exhibit a much more compact cluster structure, thus making classification much easier. In Appendix C, we show that feature filtering improves the accuracy of various classifiers significantly. Compared with GCN and other baselines which use both graph and feature information, the performance gains of GLP come in two folds. First, GLP allows using stronger filters to extract higher level data representations to improve performance when label rate is low, which can be easily achieved by increasing the filter parameters k and α, as shown in FIG5 . But this cannot be easily achieved in CNN) 94.1 (5.1s) 95.3 (6.7s) 95.6 (8.9s) GLP (AR, CNN) 94.1 (7.2s) 95.5 (8.8s) 95.8 (11.1s) GCN. As each convolutional layer of GCN is coupled with a projection layer, to increase smoothness one needs to stack many layers, and a deep GCN is difficult to train. Second, GLP allows adopting domain-specific classifiers such as CNN to deal with vision tasks. As shown in TAB2, the performance of CNN trained on raw features of labeled data is very competitive and grows fast. Due to space limitations, we include the stability analysis of GLP in Appendix D. Many graph-based semi-supervised learning methods adopt a common assumption that nearby vertices are likely to have same labels. One idea is to learn smooth low-dimensional embedding of data points by using Markov random walks BID25, Laplacian eigenmaps BID0, spectral kernels BID7 BID35, and context-based methods BID21. Another idea hinges on graph partition, where the cuts should agree with the labeled data and be placed in low density regions BID3 BID39 BID13 BID4. Perhaps the most popular idea is to formulate a quadratic regularization framework to explicitly enforce the consistency with the labeled data and the cluster assumption, which is known as label propagation BID36 BID6 BID2 BID17.To leverage more data information to improve predictions, a variety of methods proposed to jointly model data feature and graph information. BID36 proposed to combine label propagation with external classifiers by attaching a "dongle" vertex to each unlabeled vertex. Iterative classification algorithm BID23 iteratively classifies an unlabeled vertex using its neighbors' labels and features. Manifold regularization BID1, deep semi-supervised embedding BID32, and Planetoid BID34 regularize a supervised classifier with a Laplacian regularizer or an embedding-based regularizer. Graph convolutional networks BID16 combine graph and feature information in convolutional layers, which is actually doing Laplacian smoothing on data features. Follow-up works include graph attention networks BID31, attention-based graph neural network BID28, and graph partition neural networks BID20.The idea of feature smoothing has been widely used in computer graphics community for fairing 3D surface BID27 a; BID9. BID12 proposed manifold denoising which uses feature smoothing as a preprocessing step for running a label propagation algorithm, i.e, the denoised features are used to construct a better graph for LP. This method is still "onedimensional", as it cannot use the preexisting graph information in data such as citation networks. In contrast, the proposed GLP and the GCN frameworks are "two-dimensional". In this paper, we have proposed a simple, flexible, and efficient framework GLP for semi-supervised learning, and demonstrated its effectiveness theoretically and empirically. GLP offers new insights into existing methods and opens up possible avenues for new methods. An important direction for future research is the design and selection of graph filters for GLP in different application scenarios. Other directions include making GLP readily applicable to inductive problems, developing faster algorithms for GLP, and applying GLP to solve large-scale real-world problems. We include dataset descriptions, experimental details, supplementary experiments, stability analysis, and running time analysis here. Citation networks BID23 are networks that record documents' citation relationship. In citation networks, vertices are documents and edges are citation links. A pair of vertices are connected by an undirected edge if and only if one cites another. Each vertex is associated with a feature vector, which encodes the document content. In the three citation networks we tested on, CiteSeer, Cora and PubMed, feature vectors are 0/1 vectors that have the same length as the dictionary size and indicate whether a word appears in a document. The statistics of datasets are summarized in TAB3.Never Ending Language Learning (NELL) BID5 ) is a knowledge graph introduced by Carlson et al.. Yang et al. extracted an entity classification dataset from NELL, and converted the knowledge graph into a single relation graph. For each relation type r, they created two new vertices r 1 and r 2 in the graph. For each triplet (e 1, r, e 2), they created two edges (e 1, r 1) and (e 2, r 2). We follow BID16 to extend the features by assigning a unique one-hot representation for every relation vertex, ing in a 61,278-dimensional sparse feature vector for each vertex. Dataset statistics are also provided in TAB3.MNIST contains 70,000 images of handwritten digits from 0 to 9 of size 28 × 28. Each image is represented by a dense 784-dimensional vector where each dimension is a gray intensity pixel value. A 5-NN graph is constructed based on the Euclidean distance between images. If the i-th image is within the j-th image's 5 nearest neighbors or vice versa, then w ij = w ji = 1, otherwise w ij = w ji = 0. We provide more experimental details here for the sake of reproduction. Parameters We set k = 10 for RNM, k = 20 for RW, and α = 20 for AR, if label rate is less or equal than 2%; otherwise, we set them to 5, 10, 10 respectively. Networks On citation networks, we follow BID16 to use a two-layer MLP with 16 hidden units for citation networks, 0.01 learning rate, 0.5 dropout rate, and 5 × 10 −4 L2 regularization. On NELL, we also follow BID16 to use 64 hidden units, 10 −5 L2 regularization, 0.1 dropout rate and two layer-structure. On MNIST, we use 256 hidden units, 0.01 learning rate, 0.5 dropout rate, and 5 × 10 −4 L2 regularization. The CNN we use consists of six layers, whose structure is specified in TAB4. For CNN, we use 0.003 learning rate and 0.5 dropout. All of MNIST are averaged over 10 runs. We train all networks using Adam BID14.Baselines Results of some baselines including ManiReg, SemiEmb, DeepWalk, ICA, Planetoid are taken from BID16, except for the 4-labels-per-class setting, for which we run BID34. All other are reported by us. To demonstrate the benefit of GLP, we compare training various supervised classifiers with raw and filtered features. The classifiers include support vector machine (SVM), decision tree (DT), logistic regression (LR), and multilayer perceptron (MLP). The are summarized in TAB5. We can see that for all classifiers and on all datasets, there is a huge improvement in classification accuracy with the smooth features produced by the three filters we proposed. This clearly demonstrates the advantage of filtered features over raw features. In this experiment, we use 0.01 learning rate and 5 × 10 −4 L 2 regularization for LR. For SVM, we use the RBF kernel with γ = 1/n and 1.0 L 2 regularization. For DT, we use Gini impurity as quality measure. We use the same parameters for MLP as described in Appendix B. We test how the filter parameters k and α influence the performance of GLP. Figs. 4 to 6 plot the classification accuracies of GLP with different k and α on three citation networks, with 4 labels per class. Consistent with our analysis in section 5, the classification accuracy of GLP first increases and then decreases as k and α increases. The shows that GLP consistently outperforms GCN for a wide range of k and α.

Represent the short summary for finding the corresponding scientific article (long)

We extend the classical label propation methods to jointly model graph and feature information from a graph filtering perspective, and show connections to the graph convlutional networks.

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications. It supports TensorFlow and is available open source. As deep learning has become mainstream, research on aspects like architectures BID15 BID16 BID48 BID50 BID41 and hardware BID33 BID9 ) has exploded, and helped professionalize the field. In comparison, the optimization routines used to train deep nets have arguable changed only little. Comparably simple first-order methods like SGD BID38, its momentum variants (MOMENTUM) BID34 BID31 and ADAM BID20 remain standards BID14 BID19. The low practical relevance of more advanced optimization methods is not for lack of research, though. There is a host of papers proposing new ideas for acceleration of first-order methods BID13 BID49 BID54 BID12 BID3 BID24 BID37, incorporation of second-order information BID27 BID28 BID5 BID8, and automating optimization BID43 BID25 BID39, to name just a few. One problem is that these methods are algorithmically involved and difficult to reproduce by practitioners. If they are not provided in packages for popular frameworks like TENSORFLOW, PYTORCH etc., they get little traction. Another problem, which we hope to address here, is that new optimization routines are often not convincingly compared to simpler alternatives in research papers, so practitioners are left wondering which of the many new choices is the best (and which ones even really work in the first place).Designing an empirical protocol for deep learning optimizers is not straightforward, and the corresponding experiments can be time-consuming. This is partly due to the idiosyncrasies of the domain:• Generalization: While the optimization algorithm (should) only ever see the training-set, the practitioner cares about performance of the trained model on the test set. Worse, in some important application domains, the optimizer's loss function is not the objective we ultimately care about. For instance in image classification, the real interest may be in the percentage of correctly labeled images, the accuracy. Since this 0-1 loss is infeasible in practice BID26, a surrogate loss function is used instead. So which score should actually be presented in a comparison of optimizers? Train loss, because that is what the optimizer actually works on; test loss, because an over-fitting optimizer is useless, or test accuracy, because that's what the human user cares about?• Stochasticity: Sub-sampling (batching) the data-set to compute estimates of the loss function and its gradient introduces stochasticity. Thus, when an optimizer is run only once on a given problem, its performance may be misleading due to random fluctuations. The same stochasticity also causes many optimization algorithms to have one or several tuning parameters (learning rates, etc.). How should an optimizer with two free parameter be compared in a fair way with one that has only one, or even no free parameters?• Realistic Settings, Fair Competition: There is a widely-held belief that popular standards like MNIST and CIFAR-10 are too simplistic to serve as a realistic place-holder for a contemporary combination of large-scale data set and architecture. While this worry is not unfounded, researchers, ourselves included, have sometimes found it hard to satisfy the demands of reviewers for ever new data sets and architectures. Finding and preparing such data sets and building a reasonable architecture for them is time-consuming for researchers who want to focus on their novel algorithm. Even when this is done, one then has to not just run one's own algorithm, but also various competing baselines, like SGD, MOMENTUM, ADAM, etc. This step does not just cost time, it also poses a risk of bias, as the competition invariably receives less care than one's own method. Reviewers and readers can never be quite sure that an author has not tried a bit too much to make their own method look good, either by choosing a convenient training problem, or by neglecting to tune the competition. To address these problems, we propose an extensible, open-source benchmark specifically for optimization methods on deep learning architectures. We make the following three contributions:• A protocol for benchmarking stochastic optimizers. Section 2 discusses and recommends best practices for the evaluation of deep learning optimizers. We define three key performance indicators: final performance, speed, and tunability, and suggest means of measuring all three in practice. We provide evidence that it is necessary to show the of multiple runs in order to get a realistic assessment. Finally, we strongly recommend reporting both loss and accuracy, for both training and test set, when demonstrating a new optimizer as there is no obvious way those four learning curves are connected in general.• DEEPOBS 1, a deep learning optimizer benchmark suite. We have distilled the above ideas into an open-source python package, written in TENSORFLOW BID0, which automates most of the steps presented in section 2. The package currently provides over twenty off-the-shelf test problems across four application domains, including image classification and natural language processing, and this collection can be extended and adapted as the field makes progress. The test problems range in complexity from stochastic two dimensional functions to contemporary deep neural networks capable of delivering near state-of-the-art on data sets such as IMAGENET. The package is easy to install in python, using the pip toolchain. It automatically downloads data sets, sets up models, and provides a back-end to automatically produce L A T E X code that can directly be included in academic publications. This automation does not just save time, it also helps researchers to create reproducible, comparable, and interpretable .• Benchmark of popular optimizers From the collection of test problems, two sets, of four simple ("small") and four more demanding ("large") problems, respectively, are selected as a core set of benchmarks. Researchers can design their algorithm in rapid iterations on the simpler set, then test on the more demanding set. We argue that this protocol saves time, while also reducing the risk of over-fitting in the algorithm design loop. The package also provides realistic baselines for the most popular optimizers on those test problems. In Section 4 we report on the performance of SGD, SGD with momentum (MOMENTUM) and ADAM on the small and large benchmarks (this also demonstrates the output of the benchmark). For each optimizer we perform an exhaustive but realistic hyperparameter search. The best performing are provided with DEEPOBS and can be used as a fair performance metric for new optimizers without the need to compute these baselines again. We invite the authors of other algorithms to add their own method to the benchmark (via a git pull-request). We hope that the benchmark will offer a common platform, allowing researchers to publicise their algorithms, giving practitioners a clear view on the state of the art, and helping the field to more rapidly make progress. To our knowledge, there is currently no commonly accepted benchmark for optimization algorithms that is well adapted to the deep learning setting. This impression is corroborated by a more or less random sample of recent research papers on deep learning optimization BID13 BID54 BID20 BID28 BID12 BID3 BID24 BID37, whose empirical sections follow no joint standard (beyond a popularity of the MNIST data set). There are a number of existing benchmarks for deep learning as such. However, they do not focus on the optimizer. Instead, they are either framework or hardwarespecific, or cover deep learning as a holistic process, wrapping together architecture, hardware and training procedure, The following are among most popular ones: DAWNBench The task in this challenge is to train a model for IMAGENET, CIFAR-10 or SQUAD BID35 as quickly as possible to a specified validation accuracy, tuning the entire tool-chain from architecture to hardware and optimizer BID10.MLPerf is another holistic benchmark similar to DAWNBench. It has two different rule sets; only the'open' set allows a choice of optimization algorithm BID30. DLBS is a benchmark focused on the performance of deep learning models on various hardware systems with various software .DeepBench tests the speed of hardware for the low-level operations of deep learning, like matrix products and convolutions .Fathom is another hardware-centric benchmark, which among other things assesses how computational resources are spent.TBD focuses on the performance of three deep learning frameworks BID56.None of these benchmarks are good test beds for optimization research. BID42 defined unit tests for stochastic optimization. In contrast to the present work, they focus on small-scale problems like quadratic bowls and cliffs. In the context of deep learning, these problems provide unit tests, but do not give a realistic impression of an algorithm's performance in practice. This section expands the discussion from section 1 of design desiderata for a good benchmark protocol, and proposes ways to nevertheless arrive at an informative, fair, and reproducible benchmark. The optimizer's performance in a concrete training run is noisy, due to the random sampling of mini-batches and initial parameters. There is an easy remedy, which nevertheless is not universally adhered to: Optimizers should be run on the same problem repeatedly with different random seeds, and all relevant quantities should be reported as mean and standard deviation of these samples. This allows judging the statistical significance of small performance differences between optimizers, and exposes the "variability" of performance of an optimizer on any given problem. The obvious reason why researchers are reluctant to follow this standard is that it requires substantial computational effort. DEEPOBS alleviates this issue in two ways: It provides functionality to conveniently run multiple runs of the same setting with different seeds. More importantly, it provides stored baselines of popular optimizers, freeing computational resources to collect statistics rather than baselines. Training a machine learning system is more than a pure optimization problem. The optimizers' immediate objective is training loss, but the users' interest is in generalization performance, as estimated on a held-out test set. It has been observed repeatedly that in deep learning, different optimizers of similar training-set performance can have surprisingly different generalization (e.g.). Moreover, the loss function is regularly just a surrogate for the metric the user is ultimately interested in. In classification problems, for example, we are interested in classification accuracy, but this is infeasible to optimize directly. Thus, there are up to four relevant metrics to consider: training loss, test loss, training accuracy and test accuracy. We strongly recommend reporting all four of these to give a comprehensive assessment of a deep learning optimizer. For hyperparameter tuning, we use test accuracy or, if that is not available, test loss, as the criteria. We also use them as the performance metrics in TAB2.For empirical plots, many authors compute train loss (or accuracy) only on mini-batches of data, since these are computed during training anyway. But these mini-batch quantities are subject to significant noise. To get a decent estimate of the training-set performance, whenever we evaluate on the test set, we also evaluate on a larger chunk of training data, which we call a train eval set. In addition to providing a more accurate estimate, this allows us to "switch" the architecture to evaluation mode (e.g. dropout is not used during evaluation). Relevant in practice is not only the quality of a solution, but also the time required to reach it. A fast optimizer that finds a decent albeit imperfect solution using a fraction of other methods' resources can be very relevant in practice. Unfortunately, since learning curves have no parametric form, there is no uniquely correct way to define "time to convergence". In DEEPOBS, we take a pragmatic approach and measure the time it takes to reach an "acceptable" convergence performance, which is individually defined for each test problem from the baselines SGD, MOMENTUM and ADAM each with their best hyperparameter setting. Arguably the most relevant measure of speed would be the wall-clock time to reach this convergence performance. However, wall-clock runtime has well-known drawbacks, such as dependency on hardware or weak reproducibility. So many authors report performance against gradient evaluations, since these often dominate the total computational costs. However, thiscan hide large per-iteration overhead. We recommend first measuring wall-clock time of both the new competitor and SGD on one of the small test problems for a few iterations, and computing their ratio. This computation, which can be done automatically using DEEPOBS, can be done sequentially on the same hardware. One can then report performance against the products of iterations and per-iteration cost relative to SGD.For many first-order optimization methods, such as SGD, MOMENTUM or ADAM, the choice of hyperparameters does not affect the runtime of the algorithm. However, more evolved optimization methods, e.g. ones that dynamically estimate the Hessian, the hyperparameters can influence the runtime significantly. In those cases, it is suggested to repeat the runtime estimate for different hyperparameters. Almost all deep learning optimizers expose tunable hyperparameters, e.g., step sizes or averaging constants. The ease of tuning these hyperparameters is a relevant characteristic of an optimization method. How does one "fairly" compare optimizers with tunable hyperparameters?A full analysis of the effects of an optimizer's hyperparameters on its performance and speed is tedious, especially since they often interact. Even a simpler sensitivity analysis requires a large number of optimization runs, which are infeasible for most users. Such analyses also do not take into account if hyperparameters have default values that work for almost all optimization problems and therefore require no tuning in general. Instead we recommend that authors find and report the bestperforming hyperparameters for each test problem. Since DEEPOBS covers multiple test problems, the spread of these best choices gives a good impression of the required tuning. Additionally, we suggest reporting the relative performance of the hyperparameter settings used during this tuning process FIG5 shows an example). Doing so yields a characterization of tunability without additional computations. For the baselines presented in this paper, we chose a simple log-grid search to tune the learning rate. While this is certainly not an optimal tuning method, and more sophisticated methods exists (e.g. BID4, BID45), it is nevertheless used often in practice and reveals interesting properties about the optimizers and their tunability. Other tuning methods can be used with DEEPOBS however, this would require recomputing the baselines as well. DEEPOBS supports authors in adhering to good scientific practice by removing various moral hazards. The baseline for popular optimizers (whose hyperparameters have been tuned by us or, in the future, the very authors of the competing methods) avoid "starving" the competition of attention. When using different hyperparameter tuning methods, it is necessary to allocate the same computational budget for all methods in particular when comparing optimization methods of varying number of hyperparameters. The fixed set of test problems provided by the benchmark makes it impossible to (knowingly or subconsciously) cherry-pick problems tuned to a new method. And finally, the fact that the benchmark spreads over multiple such problem sets constitutes a mild but natural barrier to "overfit" the optimizer method to established data sets and architectures (like MNIST). Performances of the most popular optimizers..tex files of learning curves for new optimizer and the baselines. DEEPOBS provides the full stack required for rapid, reliable, and reproducible benchmarking of deep learning optimizers. At the lowest level, a data loading (§3.1) module automatically loads and preprocesses data sets downloaded from the net. These are combined with a list of models (§3.2) to define test problems. At the core of the library, runners (§3.3) take care of the actual training, and log a multitude of statistics, e.g., training loss or test accuracy. Baselines (§3.4) are provided for a collection of competitors. They currently include the popular choices SGD (raw, and with MOMENTUM) and ADAM, but we invite authors of other methods to contribute their own. The visualization (§3.6) script maps the to L A T E X output. Future releases of DEEPOBS will include a version number that follows the pattern MA-JOR.MINOR.PATCH, where MAJOR versions will differ in the selection of the benchmark sets, MINOR versions signify changes that could affect the . PATCHES will not affect the benchmark . All obtained with the same MAJOR.MINOR version of DEEPOBS will be directly comparable, all with the same MAJOR version will compare on the same problems. We now give a brief overview of the functionality; the full documentation can be found online. Excluding IMA-GENET, the downloaded data sets require less than one GB of disk space. The DEEPOBS data loading module then performs all necessary processing of the data sets to return inputs and outputs for the deep learning model (e.g. images and labels for image classification). This processing includes splitting, shuffling, batching and data augmentation. The data loading module can also be used to build new deep learning models that are not (yet) part of DEEPOBS. Together, data set and model define a loss function and thus an optimization problem. TAB1 provides an overview of the data sets and models included in DEEPOBS. We selected problems for diversity of task as well as the difficulty of the optimization problem itself. The list includes popular image classification models on data sets like MNIST, CIFAR-10 or IMAGENET, but also models for natural language processing and generative models. Additionally, three two-dimensional problems and an ill-conditioned quadratic problem are included. These simple tests can be used as illustrative toy problems to highlight properties of an algorithm and perform sanity-checks. Over time, we plan to expand this list when hardware and research progress renders small problems out of date, and introduces new research directions and more challenging problems. The runners of the DEEPOBS package handle training and the logging of statistics measuring the optimizers performance. For optimizers following the standard TensorFlow optimizer API it is enough to provide the runners with a list of the optimizer's hyperparameters. We provide a template for this, as well as an example of including a more sophisticated optimizer that can't be described as a subclass of the TensorFlow optimizer API. DEEPOBS also provides realistic baselines for, currently, the three most popular optimizers in deep learning, SGD, MOMENTUM, and ADAM. These allow comparing a newly developed algorithm to the competition without computational overhead, and without risk of conscious or unconscious bias against the competition. Section 4 describes how these baselines were constructed and discusses their performance. Baselines for further optimizers will be added when authors provide the optimizer's code, assuming the method perform competitively. Currently, baselines are available for all test problems in the small and large benchmark set; we plan to provide baselines for the full set of models in the near future.3.5 ESTIMATE RUNTIME DEEPOBS provides an option to quickly estimate the runtime overhead of a new optimization method compared to SGD. It measures the ratio of wall-clock time between the new optimizer and SGD. By default this ratio is measured on five runs each, for three epochs, on a fully connected network on MNIST. However, this can be adapted to a setting which fairly evaluates the new optimizer, as some optimizers might have a high initial cost that amortizes over many epochs. The DEEPOBS visualization module reduces the overhead for the preparation of , and simultaneously standardizes the presentation, making it possible to include a comparably large amount of information in limited space. The module produces.tex files with pgfplots-code for all learning curves for the proposed optimizer as well as the most relevant baselines (section 4 includes an example of this output). For the baseline provided with DEEPOBS, we evaluate three popular deep learning optimizers (SGD, MOMENTUM and ADAM) on the eight test problems that are part of the small (problems P1 to P4) and large (problems P5 to P8) benchmark set (cf. TAB1). The learning rate α was tuned for each optimizer and test problem individually, by evaluating on a logarithmic grid from α min = 10 −5 to α max = 10 2 with 36 samples. Once the best learning rate has been determined, we run those settings ten times with different random seeds. While we are using a log grid search, researchers are free to use any other hyperparameter tuning method, however this would require re-running the baselines as well. FIG4 shows the learning curves of the eight problems in the small and large benchmark set. TAB2 summarizes the from both benchmark sets. We focus on three main observations, which corroborate widely-held beliefs and support the case for an extensive and standardized benchmark. There is no optimal optimizer for all test problems. While ADAM compares favorably on most test problems, in some cases the other optimizers are considerably better. This is most notable on CIFAR-100, where MOMENTUM is significantly better then the other two. The connection between the four learning metrics is non-trivial. Looking at P6 and P7 we note that the optimizers rank differently on train vs. test loss. However, there is no optimizerthat universally generalizes better than the others; the generalization performance is evidently problem-dependent. The same holds for the generalization from loss to accuracy (e.g. P3 or P6).ADAM is somewhat easier to tune. Between the eight test problems, the optimal learning rate for each optimizer varies significantly. FIG5 shows the final performance against learning rate for each of the eight test problems. There is no significant difference between the three optimizers in terms of their learning rate sensitivity. However, in most cases, the order of magnitude of the optimal learning rate for ADAM is in the order of 10 (with the exception of P1), while for SGD and MOMENTUM this spread is slightly larger. Deep learning continues to pose a challenging domain for optimization algorithms. Aspects like stochasticity and generalization make it challenging to benchmark optimization algorithms against each other. We have discussed best practices for experimental protocols, and presented the DEEPOBS package, which provide an open-source implementation of these standards. We hope that DEEPOBS can help researchers working on optimization for deep learning to build better algorithms, by simultaneously making the empirical evaluation simpler, yet also more reproducible and fair. By providing a common ground for methods to be compared on, we aim to speed up the development of deep-learning optimizers, and aid practitioners in their decision for an algorithm.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We provide a software package that drastically simplifies, automates, and improves the evaluation of deep learning optimizers.

MAML is great, but it has many problems, we solve many of those problems and as a result we learn most hyper parameters end to end, speed-up training and inference and set a new SOTA in few-shot learning


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Pre-trained word embeddings are the primary method for transfer learning in several Natural Language Processing (NLP) tasks. Recent works have focused on using unsupervised techniques such as language modeling to obtain these embeddings. In contrast, this work focuses on extracting representations from multiple pre-trained supervised models, which enriches word embeddings with task and domain specific knowledge. Experiments performed in cross-task, cross-domain and crosslingual settings indicate that such supervised embeddings are helpful, especially in the lowresource setting, but the extent of gains is dependent on the nature of the task and domain. Named entity recognition, semantic role labelling, relation extraction etc. can be thought of as primary tasks necessary for solving high level tasks like question answering, summarization etc. However, labelling large amounts of data at this granularity is not only prohibitively expensive, but also unscalable. Given that high performance models for these tasks already exist, it is desirable to leverage them for other language understanding tasks. Next, consider the domain adaptation setting where some domains have a lot of data, while others do not. A model for a low-resource domain would benefit from information in expert models trained on other data rich domains. Finally, consider the setting of cross-lingual adaptation, a common problem for personal assistants expanding to more languages. As the number of languages increases, it becomes unfeasible to obtain human annotated data. Again, the need to adapt to low-resource languages can be met by leveraging models that already exist for high-resource languages. Motivated by the above scenarios, we propose a simple method to transfer supervised knowledge, from multiple sources, in an easy to implement manner. In our approach, this knowledge is extracted from source models in the form of contextual word embeddings. We treat preexisting models as embedding extractors, which are used to extract token level representations for an input sentence. These representations are then combined via a task specific convex combination. Unsupervised transfer learning methods such as ELMo have shown great success for a variety of tasks BID15. While they have the advantage of being trained on very large corpora, the training objectives are unsupervised. We show that in low-resource settings especially, leveraging representations from multiple pre-trained supervised models in related tasks, domains or languages can prove to be beneficial. The common way of supervised transfer learning via fine-tuning can transfer information only from a single source task BID11. One way to incorporate information from multiple external sources is via multi-task learning BID5 BID17. The limitations of multitask learning are the need for labelled data for the source models, longer training times and complex design decisions (weighing the losses for each task, sampling strategies, and choice of architecture). In contrast, our plug-and-play approach is simple and does not assume availability of source model data at training time. Finally, our approach also provides some interpretability (through the parameters of the convex combination) into which source tasks or domains are important for which other tasks and domains. Our work aligns most with the following three directions of research. Unsupervised transfer learning Embeddings such as GloVe and FastText have become an integral part of the modern NLP pipeline BID13; BID0. Over the last year, language model based deep contextualized embedding methods such as ELMo have shown substantial improvements over their shallow counterparts, heralding a new era of word representations BID15. In terms of modelling approach, our work is similar to BID7, where the authors use multiple existing models for domain adaptation for spoken language understanding. In comparison, our work focuses not just on the domain adaptation, but also the cross-task and cross-lingual settings. In another work, BID1 create metaembeddings from multiple embeddings like GloVe, Fasttext etc. Most deep learning models can be thought of as having an encoder E and decoder D. For example in a Deep-SRL model BID6, stacked bidirectional LSTM constitutes E, while D is the softmax layer. Assume K existing supervised models either for different tasks or different domains M 1,..., M K and corresponding encoders E 1,..., E K. Given a sentence of N tokens (t 1, t 2, ..., t N), we feed these tokens to the K different encoders and get K different representations for each token. We denote the encoder output of the kth model for the nth token by h k n. Each encoder generates representations specialized for the task, domain, or language it was trained for. Since our approach assumes no explicit information about the encoders of the model, they can be of varying dimensions and use different underlying architectures. Evidently, they would also be in different vector spaces and therefore we first use a projection layer to bring all of them in the same vector space. The parameters of these projection layers W 1,... W K are learned along with the target model parameters. DISPLAYFORM0 For inclusion in a downstream model, we aggregate the projection layer output of all the different source models into one vector. Several aggregation schemes can be employed: pooling, convex combination, attention etc. We choose the simple yet interpretable convex combination approach, as described below. This technique is similar to one used by ELMo BID15. We use a softmax normalized weight s k corresponding to each of the different representations of the word, add them up and use a scalar parameter γ that scales up the whole vector. The embedding O n for the nth word comes out to be: DISPLAYFORM0 This approach adds K + 1 trainable parameters to the model. An advantage of combining the representations in this manner is that the size of the embedding is fixed irrespective of the number of source models used. Once we get a combined representation, it can be used in the target model just like any other embedding. In our experiments, we concatenate these embeddings with traditional GloVe or ELMo embeddings. We use the proposed supervised contextual embeddings along with GloVe and ELMo embeddings in three knowledge transfer settings. Cross-task transfer In this setting, we transfer knowledge to a target task from models trained on multiple source tasks. We transfer into Semantic Role Labeling (SRL) task using Constituency Parsing (CP), Dependency Parsing (DP) and Named Entity Recognition (NER) as source tasks. The choice of SRL as a target task, with source embeddings from CP, DP and NER models, is inspired by the popular use of explicit syntactic parsing features for SRL. We use OntoNotes 5.0 BID16 dataset to train the SRL target tasks. We use the stacked alternating LSTM architechture for SRL as per BID6. On the source side, the DP model is based on BID4 and CP on BID18. For most of the source models, we use off-the-shelf, pre-trained models provided by AllenNLP 1. We refer readers to BID15 for further description of model architectures for the various tasks. Cross-domain transfer Here, we study the applicability of our method in the cross-domain setting. The target task is same as the source tasks, but instead, the domains of the source and target models are different. For this set of experiments, our task is NER and we use the OntoNotes 5.0 dataset which comes with annotations for multiple domains. Though NER is an easier task, we chose it as the target task for the cross-domain setting as even state of the art NER models may perform poorly for a data-scarce domain. We choose the target domain as web blogs and the source domains are newswire, broadcast conversation, telephone conversation, magazines and broadcast news. Note that the samples in the validation and test sets are also limited to the web blogs domain only. We use an LSTM-CRF architechture with 1 LSTM layer for NER as per BID14.Cross-lingual transfer From the CoNLL shared tasks, we obtain NER datasets for English, Spanish, German and Dutch Tjong BID20. We consider two scenarios with German and Spanish as the target languages and the remaining 3 as source languages. To facilitate the input of sentences into models from other languages with different scripts, we rely on crosslingual embeddings provided by MUSE Conneau et al. (2017b). The NER model architecture is the same as the one used for the cross-domain experiments. To study the effectiveness of our approach in the low resource setting, in addition to the full datasets, we also run experiments on smaller training subsets. Similar to BID12, we create random subsets of 1,000 and 5,000 samples to simulate a low resource setting. In all the aforementoiend settings, the source task models are trained on their complete datasets. Hyperparameters We use the Adam optimizer (lr=0.001) for all our experiments. We run our target models for 50 epochs in SRL tasks and 75 epochs for NER tasks. Batch size is kept at 8 for the 1k data setting and 16 for 5k data setting. The dimensions of the GloVe and ELMo embeddings are 100 and 1024 respectively. The output dimension of the projection layer in all settings for supervised embeddings is 300. Cross-task SRL (with GloVe and ELMo in 1k, 5k and full data settings) have been tabulated in TAB1 has the for cross-domain NER and TAB2 shows the for crosslingual transfer on NER. All the reported numbers are F1 scores. Cross-task SRL With GloVe embeddings, adding the supervised embeddings gives us significant improvements in F1 scores ∼ 5% for 1k and ∼ 7% for 5k examples. When we use the entire dataset, adding supervised embeddings provides no performance gains. Examining the learned source task weights in the 1k setting, we find that weights for CP, DP and NER have values 0.41, 0.41 and 0.18 respectively which shows that SRL benefits greatly from syntactic tasks like CP and DP. This is in agreement with SRL state-of-the-art models BID19 and BID8 which rely on syntactic features. When we replace GloVe with ELMo representations, we see that the baseline model improves by over ∼ 13%, showing that ELMo representations are indeed very strong. But adding supervised embeddings in the 1k setting further improves upon the ELMo baseline by over ∼ 5%. A similar improvement of ∼ 5% can be seen in the 5k setting as well. Our model shows comparable performance as the baseline when we use the entire dataset. These suggest that the proposed supervised contextual embeddings further bring about improvements over already strong language model features in a low-resource setting. This reinforces the learning that when sufficient data is available, supervised signals do not provide information that the model cannot learn by itself from the data alone. Cross-domain NER Supervised embeddings provide an impressive 4% improvement over the GloVe baseline with both 1,000 and 5,000 samples. Even when we replace GloVe with ELMo, we see an improvement of 3%, indicating that the benefits of using knowledge from other domains is orthogonal to what ELMo can offer. However, the gains vanish when the full dataset is used, suggesting that knowledge from other domains is particularly useful in the very low-resource setting. However, if sufficient data is available, the model has enough resources to build upon generic word embeddings. It is also interesting to note that for this dataset, GloVe based models outperform their ELMo counterparts. This is probably due to the mismatch in the data used to train ELMo (formal language from the 1 billion word corpus) as opposed to the NER dataset which consists of informal language used in web blogs. Cross-lingual NER We observe substantial gains by exploiting information present in other languages. For both German and Spanish the performance gains are highest when number of samples is 1,000, thus validating the suitability of the proposed method for transfer to very low-resource settings. Even when full dataset is used, we see gains over 1% for both languages. We propose supervised contextual embeddings, an easy way to incorporate supervised knowledge from multiple pre-existing models. We perform experiments in the cross-task, cross-domain and cross-lingual setups and find that the proposed embeddings are particularly useful in the lowresource setting. Our work points to the potential of such embeddings in various downstream tasks in different transfer learning settings. Future work includes incorporating more tasks, domains and languages, and understanding the relationships among them. These explorations would build towards our larger vision of building a more complete taxonomy of transfer learning dependencies among NLP tasks, domains and languages.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

extract contextual embeddings from off-the-shelf supervised model. Helps downstream NLP models in low-resource settings

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the article for finding a single-sentence TLDR

Question: We build a theoretical framework for understanding practical meta-learning methods that enables the integration of sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms in order to provide within-task performance guarantees. Our approach improves upon recent analyses of parameter-transfer by enabling the task-similarity to be learned adaptively and by improving transfer-risk bounds in the setting of statistical learning-to-learn. It also leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure. Meta-learning, or learning-to-learn (LTL) BID26, has recently re-emerged as an important direction for developing algorithms capable of performing well in multitask learning, changing environments, and federated settings. By using the data of numerous training tasks, meta-learning algorithms seek to perform well on new, potentially related test tasks without using many samples from them. Successful modern approaches have also focused on exploiting the capacity of deep neural networks, whether by learning multi-task data representations passed to simple classifiers BID25 or by neural control of the optimization algorithms themselves BID23.Because of its simplicity and flexibility, a common approach is that of parameter-transfer, in which all tasks use the same class of Θ-parameterized functions f θ: X → Y; usually a shared global model φ ∈ Θ is learned that can then be used to train task-specific parameters. In gradient-based meta-learning (GBML) BID11, φ is a metainitialization such that a few stochastic gradient steps on a Preliminary work. Under review by the International Conference on Machine Learning (ICML). Do not distribute. few samples from a new task suffice to learn a good taskspecific model. GBML is now used in a variety of LTL domains such as vision BID18 BID21 BID17, federated learning BID7, and robotics BID0. However, its simplicity also raises many practical and theoretical questions concerning what task-relationships it is able to exploit and in which settings it may be expected to succeed. While theoretical LTL has a long history BID4 BID19 BID22, there has recently been an effort to understand GBML in particular. This has naturally lead to online convex optimization (OCO) , either directly BID12 BID16 or via online-to-batch conversion to statistical LTL BID16 BID9. These efforts all consider learning a shared initialization of a descent method; BID12 then prove learnability of a metalearning algorithm while BID16 and BID9 give meta-test-time performance guarantees. However, this line of work has so far considered at most a very restricted, if natural, notion of task-similarity -closeness to a single fixed point in the parameter space. We introduce a new theoretical framework, Averaged-Regret Upper-Bound Analysis (ARUBA), that enables the derivation of meta-learning algorithms that can provably take advantage of much more sophisticated task-structure. Expanding significantly upon the work of BID16, ARUBA treats meta-learning as the online learning of a sequence of losses that each upper bound the regret on a single task. These bounds frequently have convenient functional forms that are (a) nice enough for us to easily draw on the existing OCO literature and (b) strongly dependent on both the task-data and the meta-initialization, thus encoding task-similarity in a mathematically accessible way. Using ARUBA we provide new or dramatically improved meta-learning algorithms in the following settings:• Adaptive Meta-Learning: A major drawback of previous work is the reliance on knowing the task-similarity beforehand to set the learning rate BID12 or regularization BID9, or the use of a suboptimal guess-and-tune approach based on the doubling trick BID16. ARUBA yields a simple and efficient gradient-based algorithm that eliminates the need to guess the task-similarity by learning it on-the-fly.• Statistical LTL: ARUBA allows us to leverage powerful in online-to-batch conversion BID27 BID15 to derive new upper-bounds on the transfer risk when using GBML for statistical LTL BID4, including fast rates in the number of tasks when the task-similarity is known and fully highprobability guarantees for a class of losses that includes linear regression. These improve directly upon the guarantees of BID16 and BID9 for similar or identical GBML algorithms.• LTL in Dynamic Environments: Many practical applications of GBML include settings where the optimal initialization may change over time due to a changing taskenvironment BID0. However, current theoretical work on GBML has only considered learning a fixed initialization BID12 BID9. ARUBA reduces the problem of meta-learning in changing environments to a dynamic regret-minimization problem, for which there exists a vast array of online algorithms with provable guarantees.• Meta-Learning the Task Geometry: A recurring theme in parameter-transfer LTL is the idea that certain model weights, such as those encoding a shared representation, are common to all tasks, whereas others, such as those performing a task-specific classification, need to be updated on each one. However, by simply using a fixed initialization we are forced to re-learn this structure on every task. Using ARUBA we provide an algorithm that can learn and take advantage of such structure by adaptively determining which directions in parameter-space need to be updated. We further provide a fully adaptive, per-coordinate variant that may be viewed as an analog for Reptile BID21 of the Meta-SGD modification of MAML BID11 BID18, which learns a per-coordinate learning rate; in addition to its provable guarantees, our version is more efficient and can be applied to a variety of GBML methods. In the current paper we provide in Section 2 an introduction to ARUBA and use it to show guarantees for adaptive and statistical LTL. We defer our theory for meta-learning in dynamic environments and of different task-geometries, as well as our empirical , to the full version of the paper. Theoretical Learning-to-Learn: The statistical analysis of LTL as learning over a task-distribution was formalized by BID4 and expanded upon by BID19. Recently, several works have built upon this theory to understand modern LTL, either from a PAC-Bayesian perspective BID2 or in the ridge regression setting with a learned kernel BID8. However, due to the nature of the data, tasks, and algorithms involved, much effort has been devoted to the online setting, often through the framework of lifelong learning BID22 BID3 BID1. The latter work considers a many-task notion of regret similar to our own in order to learn a shared data representations, although our algorithms are significantly more practical. Very recently, BID6 also developed a more efficient online approach to learning a linear embedding of the data. However, such work is related to popular shared-representation methods such as ProtoNets BID25, whereas we consider the parameter-transfer setting of GBML.Gradient-Based Meta-Learning: GBML developed from the model-agnostic meta-learning (MAML) algorithm of BID11 and has been widely used in practice BID18 BID0 BID21 BID14 ). An expressivity was shown for MAML by BID10, proving that the metalearner could approximate any permutation-invariant learning algorithm given enough data and a specific neural network architecture. Under strong-convexity and smoothness assumptions and using a fixed learning rate, BID12 show that the MAML meta-initialization is learnable, albeit via a somewhat impractical Follow-the-Leader (FTL) method. In contrast to these efforts, BID16 and BID9 focus on providing finite-sample meta-test-time performance guarantees in the convex setting, the former for the SGD-based Reptile algorithm of BID21 and the latter for a more strongly-regularized variant. Our work improves upon these analyses by considering the case when the learning rate, a proxy for the task-similarity, is not known beforehand as in BID12 and BID9 but must be learned online; BID16 do consider an unknown task-similarity but use a rough doubling-trick-based approach that considers the absolute deviation of the task-parameters from the meta-initialization and is thus average-case suboptimal and sensitive to outliers. Furthermore, ARUBA can handle more sophisticated and dynamic notions of task-similarity and in certain settings can provide better statistical guarantees than those of BID16 and BID9. Following the setup of BID1, we consider a sequence of tasks t = 1,..., T; each task has rounds i = 1,..., m, on each of which we see a loss function DISPLAYFORM0 In the online setting, our goal will be to design algorithms taking actions θ t,i ∈ Θ that in small task-averaged regret (TAR) BID16, which averages the within-task regret over t ∈ [T]: DISPLAYFORM1 This quantity measures within-task performance by dynamically comparing to the best action on individual tasks. A common approach in this setting is to run an online algorithm, such as online gradient descent (OGD) with learning rate η t > 0 and initialization φ t ∈ Θ, on each task t: DISPLAYFORM2 The meta-learning problem is then reduced to determining which learning rate and initialization to use on each task t. Specific cases of this setup include the Reptile method of BID21 and the algorithms in several recent theoretical analyses BID1 BID16 BID9. The observation that enables the in the current paper is the fact that the online algorithms of interest in few-shot learning and meta-learning often have existing regret guarantees that depend strongly on both the parameters and the data; for example, the withintask regret of OGD for G-Lipschitz convex losses is DISPLAYFORM3 for θ * t the optimal parameter in hindsight. Whereas more sophisticated adaptive methods for online learning attempt to reduce this dependence on initialization, in our setting each task does not have enough data to do so. Instead we can observe that if the upper boundR t (φ t, η t) ≥ R t on the task-t regret is low on average over t ∈ [T] then the TAR of the actions θ t,i due to running OGD initialized at φ t with learning rate η t at each task t will also be low, i.e. DISPLAYFORM4 Often this upper-boundR t will have a nice functional form; for example, the OGD bound above is jointly convex in the learning rate η t and the initialization φ t. Then standard OCO can be applied directly. While this approach was taken implicitly by BID16, and indeed is related to earlier work on adaptive bound optimization for online learning BID20, in this work we make explicit this framework, which we call Averaged-Regret Upper-Bound Analysis (ARUBA), and showcase its usefulness in deriving a variety of new in both online and batch LTL. Specifically, our approach will reduce LTL to the online learning of a sequence of regret upper-boundsR 1 (x),...,R T (x), where x parameterizes the within-task algorithms. The ing guarantees will then have the generic form DISPLAYFORM5 Thus as T → ∞ the algorithm competes with the best parameterization x, which encodes the task-relatedness through the task-data-dependence ofR t.Algorithm 1: General form of meta-learning algorithm we study. TASK η,φ corresponds to online mirror descent (OMD) or follow-the-regularized-leader (FTRL) with initialization φ ∈ Θ, learning rate η > 0, and regularization R: Θ → R. META is follow-the-leader (FTL). META is some OCO algorithm. Set meta-initialization φ 1 ∈ Θ and learning rate η 1 > 0. DISPLAYFORM6 Our first is an adaptive algorithm for a simple notion of task-similarity that serves also to demonstrate how our framework may be applied. We consider tasks t = 1,..., T whose optimal actions θ * t are close to some unknown global φ * ∈ Θ according to some metric. For 2 -distance this assumption was made, explicitly or implicitly, by BID12 and BID9; BID16 also consider the case of a Bregman divergence B R (θ * t ||φ *) for 1-strongly-convex R: Θ → R BID5, with R(·) = B R (θ * t ||φ *) of the task-parameters; for OCO methods V is proportional to the learning rate or the inverse of the regularization coefficient, which were fixed by BID12 and BID9. BID16 instead used the doubling trick to learn the maximum deviation max t B R (θ * t ||φ *) ≥ V, which is suboptimal and extremely sensitive to outliers. We first formalize the setting we consider, extensions of which will also be used for later :Setting 2.1. Each task t ∈ [T] has m convex loss functions t,i Θ → R that are G-Lipschitz on average. Let θ * t ∈ arg min θ∈Θ mt i=1 t,i (θ) be the minimum-norm optimal fixed action for task t. We will consider variants of Algorithm 1, in which a parameterized OCO method TASK η,φ is run within-task and two OCO methods, META and META, are run in the outer loop to determine the learning rate η > 0 and initialization φ ∈ Θ. We provide the following guarantee: Theorem 2.1. In Setting 2.1 Algorithm 1 achieves TAR DISPLAYFORM0 where D 2 = max t B R (θ * t ||φ t) and R T is the regret of META on a sequence f 1,..., f T of functions of form DISPLAYFORM1 Proof Sketch. The proof follows from the well-known re- x + x. This is nontrivial, as while the functions are convex they are non-Lipschitz near 0. However, using strongly-convex coupling once more one can show that using the actions of FTL on the modified loss functionsf t (x) = by proving the exp-concavity off t and using the Exponentially-Weighted Online Optimization (EWOO) algorithm of BID13, which can be implemented efficiently in this single-dimensional case, instead of FTL. We thus have the following corollary: DISPLAYFORM2 DISPLAYFORM3 e. the mean and squared average deviation of the optimal task parameters, we have an asymptotic per-task regret of V G √ m, which is much better than the minimax-optimal single-task guarantee DG √ m when V D, i.e. when the tasks are on-average close in parameter space. As in BID16 and assuming a quadratic growth condition on each task, in the full version we extend this to the case when θ * t is not known and either the last or average within-task iterate is used to perform the meta-updates. An important motivation for studying LTL via online learning has been to provide batch-setting bounds on the transfer risk BID1 BID9. While BID16 provide an in-expectation bound on the expected transfer risk of any low-TAR algorithm, their cannot exploit the many stronger in the online-tobatch conversion literature. Following the classical distribution over task-distributions setup of BID4, ARUBA yields strong bounds on the expected transfer risk in the general case of convexR, as well as fast rates in the stronglyconvex case using BID15 and high probability bounds for linear regression using BID27. Theorem 2.2. Let convex losses t,i: Θ → be sampled i.i.d. P t ∼ Q, {t,i} i ∼ P m t for some distribution Q over task distributions P t. If the losses are given to an algorithm with averaged regret upper-boundR T that on each task runs an algorithm with regret upper-boundR t (s t) a convex, nonnegative, and B √ m-bounded function of the state s t of the algorithm at the beginning of time t then we have the following bound on the expected transfer risk: 2, 6αB √ m} αmT log 8 log T δ If the losses satisfy a certain self-bounding property then we have a high probability bound on the transfer risk itself: DISPLAYFORM0 DISPLAYFORM1 T m log 2 δ + 3ρ + 2 m log 2 δ w.p. 1 − δ for some ρ > 0.In the case of a known task-similarity, when we know the expected task-parameter deviation V and can fix the learning rate in Algorithm 1 accordingly, the above yields DISPLAYFORM2 This can be compared to of BID9, where the last term only decreases as T. 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 Adaptive Gradient-Based Meta-Learning Methods Zinkevich, M. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on.

Practical adaptive algorithms for gradient-based meta-learning with provable guarantees.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.