Question: Represent the text to find a very short summary (1-2 sentences max)

Question: One of the most notable contributions of deep learning is the application of convolutional neural networks (ConvNets) to structured signal classification, and in particular image classification. Beyond their impressive performances in supervised learning, the structure of such networks inspired the development of deep filter banks referred to as scattering transforms. These transforms apply a cascade of wavelet transforms and complex modulus operators to extract features that are invariant to group operations and stable to deformations. Furthermore, ConvNets inspired recent advances in geometric deep learning, which aim to generalize these networks to graph data by applying notions from graph signal processing to learn deep graph filter cascades. We further advance these lines of research by proposing a geometric scattering transform using graph wavelets defined in terms of random walks on the graph. We demonstrate the utility of features extracted with this designed deep filter bank in graph classification of biochemistry and social network data (incl. state of the art in the latter case), and in data exploration, where they enable inference of EC exchange preferences in enzyme evolution. Over the past decade, numerous examples have established that deep neural networks (i.e., cascades of linear operations and simple nonlinearities) typically outperform traditional "shallow" models in various modern machine learning applications, especially given the increasing Big Data availability nowadays. Perhaps the most well known example of the advantages of deep networks is in computer vision, where the utilization of 2D convolutions enable network designs that learn cascades of convolutional filters, which have several advantages over fully connected network architectures, both computationally and conceptually. Indeed, in terms of supervised learning, convolutional neural networks (ConvNets) hold the current state of the art in image classification, and have become the standard machine learning approach towards processing big structured-signal data, including audio and video processing. See, e.g., Goodfellow et al. (2016, Chapter 9) for a detailed discussion. Beyond their performances when applied to specific tasks, pretrained ConvNet layers have been explored as image feature extractors by freezing the first few pretrained convolutional layers and then retraining only the last few layers for specific datasets or applications (e.g., BID47 BID33 . Such transfer learning approaches provide evidence that suitably constructed deep filter banks should be able to extract task-agnostic semantic information from structured data, and in some sense mimic the operation of human visual and auditory cortices, thus supporting the neural terminology in deep learning. An alternative approach towards such universal feature extraction was presented in BID28, where a deep filter bank, known as the scattering transform, is designed, rather than trained, based on predetermined families of distruptive patterns that should be eliminated to extract informative representations. The scattering transform is constructed as a cascade of linear wavelet transforms and nonlinear complex modulus operations that provides features with guaranteed invariance to a predetermined Lie group of operations such as rotations, translations, or scaling. Further, it also provides Lipschitz stability to small diffeomorphisms of the inputted signal. Scattering features have been shown to be effective in several audio (e.g., BID6 BID0 BID27 and image (e.g., BID7 BID40 BID34 processing applications, and their advantages over learned features are especially relevant in applications with relatively low data availability, such as quantum chemistry (e.g., BID15 BID35 .Following the recent interest in geometric deep learning approaches for processing graph-structured data (see, for example, BID4 and references therein), we present here a generalization of the scattering transform from Euclidean domains to graphs. Similar to the Euclidean case, our construction is based on a cascade of bandpass filters, defined in this case using graph signal processing BID38 notions, and complex moduli, which in this case take the form of absolute values (see Sec. 3). While several choices of filter banks could generally be used with the proposed cascade, we focus here on graph wavelet filters defined by lazy random walks (see Sec. 2). These wavelet filters are also closely related to diffusion geometry and related notions of geometric harmonic analysis, e.g. the diffusion maps algorithm of BID10 and the associated diffusion wavelets of BID11. Therefore, we call the constructed cascade geometric scattering, which also follows the same terminology from geometric deep learning. We note that similar attempts at generalizing the scattering transform to graphs have been presented in BID9 as well as BID49 and BID17. The latter two works are most closely related to the present paper. In them, the authors focus on theoretical properties of the proposed graph scattering transforms, and show that such transforms are invariant to graph isomorphism. The geometric scattering transform that we define here also possesses the same invariance property, and we expect similar stability properties to hold for the proposed construction as well. However, in this paper we focus mainly on the practical applicability of geometric scattering transforms for graph-structured data analysis, with particular emphasis on the task of graph classification, which has received much attention recently in geometric deep learning (see Sec. 4) In supervised graph classification problems one is given a training database of graph/label pairs DISPLAYFORM0 ⊂ G × Y sampled from a set of potential graphs G and potential labels Y. The goal is to use the training data to learn a model f: G → Y that associates to any graph G ∈ G a label y = f (G) ∈ Y. These types of databases arise in biochemistry, in which the graphs may be molecules and the labels some property of the molecule (e.g., its toxicity), as well as in various types of social network databases. Until recently, most approaches were kernel based methods, in which the model f was selected from the reproducing kernel Hilbert space generated by a kernel that measures the similarity between two graphs; one of the most successful examples of this approach is the Weisfeiler-Lehman graph kernel of BID37. Numerous feed forward deep learning algorithms, though, have appeared over the last few years. In many of these algorithms, task based (i.e., dependent upon the labels Y) graph filters are learned from the training data as part of the larger network architecture. These filters act on a characteristic signal x G that is defined on the vertices of any graph G, e.g., x G may be a vector of degrees of each vertex (we remark there are also edge based algorithms, such as BID20 and references within, but these have largely been developed for and tested on databases not considered in Sec. 4). Here, we propose an alternative to these methods in the form of a geometric scattering classifier (GSC) that leverages graph-dependent (but not label dependent) scattering transforms to map each graph G to the scattering features extracted from x G. Furthermore, inspired by transfer learning approaches such as BID33, we consider treatment of our scattering cascade as frozen layers on x G, either followed by fully connected classification layers (see FIG2), or fed into other classifiers such as SVM or logistic regression. We note that while the formulation in Sec. 3 is phrased for a single signal x G, it naturally extends to multiple signals by concatenating their scattering features. In Sec. 4.1 we evaluate the quality of the scattering features and ing classification by comparing it to numerous graph kernel and deep learning methods over 13 datasets (7 biochemistry ones and 6 social network ones) commonly studied in related literature. In terms of classification accuracy on individual datasets, we show that the proposed approach obtains state of the art on two datasets and performs competitively on the rest, despite only learning a classifier that come after the geometric scattering transform. Furthermore, while other methods may excel on specific datasets, when considering average accuracy: within social network data, our proposed GSC outperforms all other methods; in biochemistry or over all datasets, it outperforms nearly all feed forward neural network approaches, and is competitive with state of the art of graph kernels BID26 and graph recurrent neural networks BID41. We regard this as crucial in establishing the universality of graph features extracted by geometric scattering, as they provide an effective task-independent representation of analyzed graphs. Finally, to establish their unsupervised qualities, in Sec. 4.2 we use geometric scattering features extracted from enzyme data BID2 to infer emergent patterns of enzyme commission (EC) exchange preferences in enzyme evolution, validated with established knowledge from BID12. We define graph wavelets as the difference between lazy random walks that have propagated at different time scales, which mimics classical wavelet constructions found in BID29 as well as more recent constructions found in BID11. The underpinnings for this construction arise out of graph signal processing, and in particular the properties of the graph Laplacian. Let G = (V, E, W) be a weighted graph, consisting of n vertices V = {v 1, . . ., v n}, edges E ⊆ {(v, v m): 1 ≤, m ≤ n}, and weights W = {w(v, v m) > 0: (v, v m) ∈ E}. Note that unweighted graphs are considered as a special case, by setting w (v, v m DISPLAYFORM0 and zero otherwise, where we use the notation A(v, v m) to denote the (, m) entry of the matrix A so as to emphasize the correspondence with the vertices in the graph and to reserve sub-indices for enumerating objects. Define the (weighted) degree of vertex v as DISPLAYFORM1 The graph Laplacian is a symmetric, real valued positive semi-definite matrix, and thus has n nonnegative eigenvalues. Furthermore, if we set 0 = (0, . . ., 0)T to to be the n × 1 vector of all zeroes, and 1 = (1, . . ., 1)T to be the analogous vector of all ones, then it is easy to see that L1 = 0. Therefore 0 is an eigenvalue of L and we write the n eigenvalues of L as 0 = λ 0 ≤ λ 1 ≤ · · · ≤ λ n−1 with corresponding n × 1 orthonormal eigenvectors 1/ √ n = ϕ 0, ϕ 1,..., ϕ n−1. If the graph G is connected, then λ 1 > 0. In order to simplify the following discussion we assume that this is the case, although the discussion below can be amended to include disconnected graphs as well. Since ϕ 0 is constant and every other eigenvector is orthogonal to ϕ 0, it is natural to view the eigenvectors ϕ k as the Fourier modes of the graph G, with a frequency magnitude √ λ k. Let x: V → R be a signal defined on the vertices of the graph G, which we will consider as an n × 1 vector with entries x(v). It follows that the Fourier transform of x can be defined as x(k) = x · ϕ k, where x · y is the standard dot product. This analogy is one of the foundations of graph signal processing and indeed we could use this correspondence to define wavelet operators on the graph G, as in BID22. Rather than follow this path, though, we instead take a related path similar to BID11; BID17 by defining the graph wavelet operators in terms of random walks defined on G, which will avoid diagonalizing L and will allow us to control the "spatial" graph support of the filters directly. Define the n × n transition matrix of a lazy random random walk as P = 1 2 D −1 A + I. Note that the row sums of P are all one and thus the entry P(v, v m) corresponds to the transition probability of walking from vertex v to v m in one step. Powers of P run the random walk forward, so that in particular P t (v, v m) is the transition probability of walking from v to v m in exactly t steps. We will use P as a left multiplier, in which case P acts a diffusion operator. To understand this idea more precisely, first note that a simple calculation shows that P1 = 1 and furthermore if the graph G is connected, every other eigenvalue of P is contained in. Note in particular that L and P share the eigenvector 1. It follows that P t x responds most significantly to the zero frequency x of x while depressing the non-zero frequencies of x (where the frequency modes are defined in terms of the graph Laplacian L, as described above). On the spatial side, the value P t x(v) is the weighted average of x(v) with all values x(v m) such that v m is within t steps of v in the graph G.High frequency responses of x can be recovered in multiple different fashions, but we utilize multiscale wavelet transforms that group the non-zero frequencies of G into approximately dyadic bands. As shown in Mallat (2012, Lemma 2.12), wavelet transforms are provably stable operators in the Euclidean domain, and the proof of Zou & Lerman (2018, Theorem 5.1) indicates that similar on graphs may be possible. Furthermore, the multiscale nature of wavelet transforms will allow the ing geometric scattering transform (Sec. 3) to traverse the entire graph G in one layer, which is valuable for obtaining global descriptions of G. Following BID11, define the n × n diffusion wavelet matrix at the scale 2 j as DISPLAYFORM2 Since P t 1 = 1 for every t, we see that Ψ j 1 = 0 for each j ≥ 1. Thus each Ψ j x partially recovers x(k) for k ≥ 1. The value Ψ j x(v) aggregates the signal information x(v m) from the vertices v m. Instead, it responds to sharp transitions or oscillations of the signal x within the neighborhood of v with radius 2 j (in terms of the graph path distance). Generally, the smaller j the higher the frequencies Ψ j x recovers in x. These high frequency wavelet coefficients up to the scale 2 J are denoted by: DISPLAYFORM3 Since 2 J controls the maximum scale of the wavelet, in the experiments of Sec. 4 we select J such that 2 J ∼ diam(G). FIG0 plots the diffusion wavelets at different scales on two different graphs. A geometric wavelet scattering transform follows a similar construction as the (Euclidean) wavelet scattering transform of BID28, but leverages a graph wavelet transform. In this paper we utilize the wavelet transform defined in of the previous section, but remark that in principle any graph wavelet transform could be used (see, e.g., BID49 . In Sec. 3.1 we define the graph scattering transform, in Sec. 3.2 we discuss its relation to other recently proposed graph scattering constructions BID17 BID49, and in Sec. 3.3 we describe several of its desirable properties as compared to other geometric deep learning algorithms on graphs. Machine learning algorithms that compare and classify graphs must be invariant to graph isomorphism, i.e., re-indexations of the vertices and corresponding edges. A common way to obtain invariant graph features is via summation operators, which act on a signal x = x G that can be defined on any graph G, e.g., DISPLAYFORM0 The geometric scattering transform, which is described in the remainder of this section, follows such an approach. The simplest of such summation operators computes the sum of the responses of the signal x. As described in BID44, this invariant can be complemented by higher order summary statistics of x, the collection of which form statistical moments, and which are also referred to as "capsules" in that work. For example, the unnormalized q th moments of x yield the following "zero" order geometric scattering moments: DISPLAYFORM1 We can also replace with normalized (i.e., standardized) moments of x, in which case we store its mean (q = 1), variance (q = 2), skew (q = 3), kurtosis (q = 4), and so on. In the numerical experiments described in Sec. 4 we take Q = 2, 3, 4 depending upon the database, where Q is chosen via cross validation to optimize classification performance. Higher order moments are not considered as they become increasingly unstable, and we report for both normalized and unnormalized moments. In what follows we discuss the unnormalized moments, since their presentation is simpler and we use them in conjunction with fully connected layers (FCL) for classification purposes, but the same principles also apply to normalized moments (e.g., used with SVM and logistic regression in our classification ). The invariants Sx(q) do not capture the full variability of x and hence the graph G upon which the signal x is defined. We thus complement these moments with summary statistics derived from the wavelet coefficients of x, which in turn will lead naturally to the graph ConvNet structure of the geometric scattering transform. Observe, analogously to the Euclidean setting, that in computing Sx FORMULA3, which is the summation of x(v) over V, we have captured the zero frequency of x since DISPLAYFORM2. Higher order moments of x can incorporate the full range of frequencies in x, e.g. DISPLAYFORM3 2, but they are mixed into one invariant coefficient. We can separate and recapture the high frequencies of x by computing its wavelet coefficients Ψ (J) x, which were defined in. However, Ψ (J) x is not invariant to permutations of the vertex indices; in fact, it is covariant (or equivariant). Before summing the individual wavelet coefficient vectors Ψ j x, though, we must first apply a pointwise nonlinearity. Indeed, define the n × 1 vector d(v) = deg(v), and note that Ψ j x · d = 0 since one can show that d is a left eigenvector of P with eigenvalue 1. If G is a regular graph then d = c1 from which it follows that Ψ j x · 1 = 0. For more general graphs d(v) ≥ 0 for v ∈ V, which implies that for many graphs 1 · d will be the dominating coefficient in an expansion of 1 in an orthogonal basis containing d; it follows that in these cases |Ψ j x · 1| 1.We thus apply the absolute value nonlinearity, to obtain nonlinear covariant coefficients |Ψ (J) x| = {|Ψ j x| : 1 ≤ j ≤ J}. We use absolute value because it is covariant to vertex permutations, nonexpansive, and when combined with traditional wavelet transforms on Euclidean domains, yields a provably stable scattering transform for q = 1. Furthermore, initial theoretical in BID49; BID17 indicate that similar graph based scattering transforms possess certain types of stability properties as well. As in, we extract invariant coefficients from |Ψ j x| by computing its moments, which define the first order geometric scattering moments: DISPLAYFORM4 These first order scattering moments aggregate complimentary multiscale geometric descriptions of G into a collection of invariant multiscale statistics. These invariants give a finer partition of the frequency responses of x. For example, whereas Sx mixed all frequencies of x, we see that Sx(j, 2) only mixes the frequencies of x captured by the graph wavelet Ψ j.First order geometric scattering moments can be augmented with second order geometric scattering moments by iterating the graph wavelet and absolute value transforms, which leads naturally to the structure of a graph ConvNet. These moments are defined as: DISPLAYFORM5 which consists of reapplying the wavelet transform operator Ψ (J) to each |Ψ j x| and computing the summary statistics of the magnitudes of the ing coefficients. The intermediate covariant coefficients |Ψ j |Ψ j x|| and ing invariant statistics Sx(j, j, q) couple two scales 2 j and 2 j within the graph G, thus creating features that bind patterns of smaller subgraphs within G with patterns of larger subgraphs (e.g., circles of friends of individual people with larger community structures in social network graphs). The transform can be iterated additional times, leading to third order features and beyond, and thus has the general structure of a graph ConvNet. The collection of graph scattering moments Sx = {Sx(q), Sx(j, q), Sx(j, j, q)} (illustrated in FIG2) provides a rich set of multiscale invariants of the graph G. These can be used in supervised settings as input to graph classification or regression models, or in unsupervised settings to embed graphs into a Euclidean feature space for further exploration, as demonstrated in Sec. 4. In order to assess the utility of scattering features for representing graphs, two properties have to be considered: stability and capacity. First, the stability property aims to essentially provide an upper bound on distances between similar graphs that only differ by types of deformations that can be treated as noise. This property has been the focus of both BID49 and BID17, and in particular the latter shows that a diffusion scattering transform yields features that are stable to graph structure deformations whose size can be computed via the diffusion framework BID11 ) that forms the basis for their construction. While there are some technical differences between the geometric scattering here and the diffusion scattering in BID17, these constructions are sufficiently similar that we can expect both of them to have analogous stability properties. Therefore, we mainly focus here on the complementary property of the scattering transform capacity to provide a rich feature space for representing graph data without eliminating informative variance in them. We note that even in the classical Euclidean case, while the stability of scattering transforms to deformations can be established analytically BID28, their capacity is typically examined by empirical evidence when applied to machine learning tasks (e.g., BID5 BID39 BID0 . Similarly, in the graph processing settings, we examine the capacity of our proposed geometric scattering features via their discriminaive power in graph data analysis tasks. In Sec. 4.1, we describe extensive numerical experiments for graph classification problems in which our scattering coefficients are utilized in conjunction with several classifiers, namely, fully connected layers (FCL, illustrated in FIG2), support vector machine (SVM), and logistic regression. We note that SVM classification over scattering features leads to state of the art on social network data, as well as outperforming all feed-forward neural network methods in general. Furthermore, for biochemistry data (where graphs represent molecule structures), FCL classification over scattering features outperforms all other feed-forward neural networks, even though we only train the fully connected layers. Finally, to assess the scattering feature space for data representation and exploration, in Sec. 4.2 we examine its qualities when analyzing biochemistry data, with emphasis on enzyme graphs. We show that geometric scattering enables graph embedding in a relatively low dimensional Euclidean space, while preserving insightful properties in the data. Beyond establishing the capacity of our specific construction, these also indicate the viability of graph scattering transforms in general, as universal feature extractors on graph data, and complement the stability established in BID49 and BID17. DISPLAYFORM0 (a) Representative zeroth-, first-, and second-order cascades of the geometric scattering transform for an input graph signal x. The presented cascades, indexed by j, j, q, are collected together to form the set of scattering coefficients Sx defined in eqs.. DISPLAYFORM1 A d ja c e n c y m a tr ix: DISPLAYFORM2 Si gn al ve ct or: DISPLAYFORM3 Diffusion wavelets: DISPLAYFORM4 Fully connected layers: DISPLAYFORM5 We give a brief comparison of geometric scattering with other graph ConvNets, with particular interest in isolating the key principles for building accurate graph ConvNet classifiers. We begin by remarking that like several other successful graph neural networks, the graph scattering transform is covariant or equivariant to vertex permutations (i.e., commutes with them) until the final features are extracted. This idea has been discussed in depth in various articles, including BID25, so we limit the discussion to observing that the geometric scattering transform thus propagates nearly all of the information in x through the multiple wavelet and absolute value layers, since only the absolute value operation removes information on x. As in BID44, we aggregate covariant responses via multiple summary statistics (i.e., moments), which are referred to there as a capsule. In the scattering context, at least, this idea is in fact not new and has been previously used in the Euclidean setting for the regression of quantum mechanical energies in BID16 BID42 and texture synthesis in BID8. We also point out that, unlike many deep learning classifiers (graph included), a graph scattering transform extracts invariant statistics at each layer/order. These intermediate layer statistics, while necessarily losing some information in x (and hence G), provide important coarse geometric invariants that eliminate needless complexity in subsequent classification or regression. Furthermore, such layer by layer statistics have proven useful in characterizing signals of other types (e.g., texture synthesis in BID19 .A graph wavelet transform Ψ (J) x decomposes the geometry of G through the lens of x, along different scales. Graph ConvNet algorithms also obtain multiscale representations of G, but several works, including BID1 and, propagate information via a random walk. While random walk operators like P t act at different scales on the graph G, per the analysis in Sec. 2 we see that P t for any t will be dominated by the low frequency responses of x. While subsequent nonlinearities may be able to recover this high frequency information, the ing transform will most likely be unstable due to the suppression and then attempted recovery of the high frequency content of x. Alternatively, features derived from P t x may lose the high frequency responses of x, which are useful in distinguishing similar graphs. The graph wavelet coefficients Ψ (J) x, on the other hand, respond most strongly within bands of nearly non-overlapping frequencies, each with a center frequency k j that depends on Ψ j.Finally, graph labels are often complex functions of both local and global subgraph structure within G. While graph ConvNets are adept at learning local structure within G, as detailed in BID44 they require many layers to obtain features that aggregate macroscopic patterns in the graph. This is due in large part to the use of fixed size filters, which often only incorporate information from the neighbors of any individual vertex. The training of such networks is difficult due to the limited size of many graph classification databases (see Table 4 in Appendix D). Geometric scattering transforms have two advantages in this regard: (a) the wavelet filters are designed; and (b) they are multiscale, thus incorporating macroscopic graph patterns in every layer/order. To evaluate the proposed geometric scattering features, we test their effectiveness for graph classification on thirteen datasets commonly used for this task. Out of these, seven datasets contain biochemistry graphs that describe molecular structures of chemical compounds, as described in the following works that introduced them: NCI1 and NCI109, BID45 BID14. In these cases, each graph has several associated vertex features x that represent chemical properties of atoms in the molecule, and the classification is aimed to characterize compound properties (e.g., protein types). The other six datasets, which are introduced in BID46, contain social network data extracted from scientific collaborations (COLLAB), movie collaborations (IMDB-B & IMDB-M), and Reddit discussion threads (REDDIT-B, REDDIT-5K, REDDIT-12K). In these cases there are no inherent graph signals in the data, and therefore we compute general node characteristics (e.g., degree, eccentricity, and clustering coefficient) over them, as is considered standard practice in relevant literature (see, for example, BID44 In all cases, we iterate over all graphs in the database and for each one we associate graph-wide features by computing the scattering features of each of the available graph signals (provided or computed), and concatenating the features of all such signals. Then, the full scattering feature vectors of these graphs are passed to a classifier, which is trained from input labels, in order to infer the class for each graph. We consider three classifiers here: neural network with two/three fully connected hidden layers (FCL), SVM with RBF kernel, or logistic regression. We note that the scattering features (computed as described in Sec. 3) are based on either normalized or unnormalized moments over the entire graph. Here we used unnormalized moments for FCL, and normalized ones for other classifiers, but the difference is subtle and similar can be achieved for the other combinations. Finally, we also note that all technical design choices for configuring our geometric scattering or the classifiers were done as part of the cross validation described in Appendix E.We evaluate the classification of our three geometric scattering classification (GSC) settings using ten-fold cross validation (as explained in Appendix E) and compare them to 14 prominent methods for graph classification. Out of these, six are graph kernel methods, namely: WeisfeilerLehman graph kernels (WL, BID37, propagation kernel (PK, BID31, Graphlet kernels BID36, Random walks (RW, BID18, deep graph kernels (DGK, BID46, and Weisfeiler-Lehman optimal assignment kernels (WL-OA, BID26 . Seven other methods are recent geometric feed forward deep learning algorithms, namely: deep graph convolutional neural network , Graph2vec BID30, 2D convolutional neural networks (2DCNN, BID42, covariant compositional networks (CCN, BID24, Patchy-san (PSCN, BID32, with k = 10), diffusion convolutional neural networks (DCNN, BID1, and graph capsule convolutional neural networks (GCAPS-CNN, BID44 . Finally, one method is the recently introduced recurrent neural network autoencoder for graphs (S2S-N2N-PP, BID41 . Following the standard format of reported classification performances for these methods (per their respective references, see also Appendix A), our are reported in the form of average accuracy ± standard deviation (in percentages) over the ten crossvalidation folds. We remark here that many of them are not reported for all datasets, and hence, we mark N/A when appropriate. For brevity, the comparison is reported here in FIG4 in summarized form, as explained below, and in full in Appendix A.Since the scattering transform is independent of training labels, it provides universal graph features that might not be specifically optimal in each individual dataset, but overall provide stable classification . Further, careful examination of the of previous methods (feed forward algorithms in particular) shows that while some may excel in specific cases, none of them achieves the best in all reported datasets. Therefore, to compare the overall classification quality of our GSC methods with related methods, we consider average accuracy aggregated over all datasets, and within each field (i.e., biochemistry and social networks) in the following way. First, out of the thirteen datasets, classification on four datasets (NCI109, ENZYMES, IMDB-M, REDDIT-12K) are reported significantly less frequently than the others, and therefore we discard them and use the remaining nine for the aggregation. Next, to address reported values versus N/A ones, we set an inclusion criterion of 75% reported datasets for each method. This translates into at most one N/A in each individual field, and at most two N/A overall. For each method that qualifies for this inclusion criterion, we compute its average accuracy over reported values (ignoring N/A ones) within each field and over all datasets; this in up to three reported values for each method. The aggregated of our GSC and 13 of the compared methods appears in FIG4. These show that GSC (with SVM) outperforms all other methods on social network data, and in fact as shown Appendinx B, it achieves state of the art on two datasets of this type. Additionally, the aggregated shows that our GSC approach (with FCL or SVM) outperforms all other feed forward methods both on biochemsitry data and overall in terms of universal average accuracy 2. The CCN method is omitted from these aggregated , as its in BID24 are only reported on four biochemistry datasets. For completeness, detailed comparison of GSC with this method, which appears in FIG4, shows that our method outperforms it on two datasets while CCN outperforms GSC on the other two. Geometric scattering essentially provides a task independent representation of graphs in a Euclidean feature space. Therefore, it is not limited to supervised learning applications, and can be also utilized for exploratory graph-data analysis, as we demonstrate in this section. We focus our discussion on biochemistry data, and in particular on the ENZYMES dataset. Here, geometric scattering features can be considered as providing "signature" vectors for individual enzymes, which can be used to explore interactions between the six top level enzyme classes, labelled by their Enzyme Commission (EC) numbers BID2. In order to emphasize the properties of scattering-based feature extraction, rather than downstream processing, we mostly limit our analysis of the scattering feature space to linear operations such as principal component analysis (PCA).We start by considering the viability of scattering-based embedding for dimensionality reduction of graph data. To this end, we applied PCA to our scattering coefficients (computed from unnormalized moments), while choosing the number of principal components to capture 90% explained variance. In the ENZYMES case, this yields a 16 dimensional subspace of the full scattering features space. While the Euclidean notion of dimensionality is not naturally available in the original dataset, we note that graphs in it have, on average, 124.2 edges, 29.8 vertices, and 3 features per vertex, and therefore the effective embedding of the data into R 16 indeed provides a significant dimensionality reduction. Next, to verify the ing PCA subspace still captures sufficient discriminative information with respect to classes in the data, we compare SVM classification on the ing low dimensional vectors to the the full feature space; indeed, projection on the PCA subspace in only a small drop in accuracy from 56.85 ± 4.97 (full) to 49.83 ± 5.40 (PCA). Finally, we also consider the dimensionality of each individual class (with PCA and > 90% exp. variance) in the scattering feature space, as we expect scattering to reduce the variability in each class w.r.t. the full feature space. In the ENZYMES case, individual classes have PCA dimensionality ranging between 6 and 10, which is indeed significantly lower than the 16 dimensions of the entire PCA space. Appendix C summarizes these findings, and repeats the described procedure for two addi- tional biochemistry datasets (from BID45 to verify that these are not unique to the specific ENZYMES dataset, but rather indicate a more general trend for geometric scattering feature spaces. To further explore the scattering feature space, we now use it to infer relations between EC classes. First, for each enzyme e, with scattering feature vector v e (i.e., with Sx for all vertex features x), we compute its distance from class EC-j, with PCA subspace C j, as the projection distance: dist(e, EC-j) = v e − proj Sj v e. Then, for each enzyme class EC-i, we compute the mean distance of enzymes in it from the subspace of each EC-j class as D(i, j) = mean{dist(e, EC-j): e ∈ EC-i}. Appendix C summarizes these distances, as well as the proportion of points from each class that have their true EC as their nearest (or second nearest) subspace in the scattering feature space. In general, 48% of enzymes select their true EC as the nearest subspace (with additional 19% as second nearest), but these proportions vary between individual EC classes. Finally, we use these scatteringbased distances to infer EC exchange preferences during enzyme evolution, which are presented in FIG5 and validated with respect to established preferences observed and reported in BID12. We note that the there is observed independently from the ENZYMES dataset. In particular, the portion of enzymes considered from each EC is different between these data, since BID3 took special care to ensure each EC class in ENZYMES has exactly 100 enzymes in it. However, we notice that in fact the portion of enzymes (in each EC) that choose the wrong EC as their nearest subspace, which can be considered as EC "incoherence" in the scattering feature space, correlates well with the proportion of evolutionary exchanges generally observed for each EC in BID12, and therefore we use these as EC weights in FIG5. Our in FIG5 demonstrate that scattering features are sufficiently rich to capture relations between enzyme classes, and indicate that geometric scattering has the capacity to uncover descriptive and exploratory insights in graph data analysis, beyond the supervised graph classification from Sec 4.1. We presented the geometric scattering transform as a deep filter bank for feature extraction on graphs. This transform generalizes the scattering transform, and augments the theoretical foundations of geometric deep learning. Further, our evaluation on graph classification and data exploration show the potential of the produced scattering features to serve as universal representations of graphs. Indeed, classification with these features with relatively simple classifier models reaches high accuracy on most commonly used graph classification datasets, and outperforms both traditional and recent deep learning feed forward methods in terms of average classification accuracy over multiple datasets. We note that this might be partially due to the scarcity of labeled big data in this field, compared to more traditional ones (e.g., image or audio classification). However, this trend also correlates with empirical for the classic scattering transform, which excels in cases with low data availability. Finally, the geometric scattering features provide a new way for computing and considering global graph representations, independent of specific learning tasks. Therefore, they raise the possibility of embedding entire graphs in Euclidean space and computing meaningful distances between graphs with them, which can be used for both supervised and unsupervised learning, as well as exploratory analysis of graph-structured data. APPENDIX A FULL COMPARISON TABLE DISPLAYFORM0 DISPLAYFORM1 The details of the datasets used in this work are as follows (see the main text in Sec. 3 for references):NCI1 contains 4,110 chemical compounds as graphs, with 37 node features. Each compound is labeled according to is activity against non-small cell lung cancer and ovarian cancer cell lines, and these labels serve as classification goal on this data. NCI109 is similar to NCI1, but with 4,127 chemical compounds and 38 node features. MUTAG consists of 188 mutagenic aromatic and heteroaromatic nitro compounds (as graphs) with 7 node features. The classification here is binary (i.e., two classes), based on whether or not a compound has a mutagenic effect on bacterium. PTC is a dataset of 344 chemical compounds (as graphs) with nineteen node features that are divided into two classes depending on whether they are carcinogenic in rats. PROTEINS dataset contains 1,113 proteins (as graphs) with three node features, where the goal of the classification is to predict whether the protein is enzyme or not. D&D dataset contains 1,178 protein structures (as graphs) that, similar to the previous one, are classified as enzymes or non-enzymes. ENZYMES is a dataset of 600 protein structures (as graphs) with three node features. These proteins are divided into six classes of enzymes (labelled by enzyme commission numbers) for classification. COLLAB is a scientific collaboration dataset contains 5K graphs. The classification goal here is to predict whether the graph belongs to a subfield of Physics. IMDB-B is a movie collaboration dataset with contains 1K graphs. The graphs are generated on two genres: Action and Romance, the classification goal is to predict the correct genre for each graph. IMDB-M is similar to IMDB-B, but with 1.5K graphs & 3 genres: Comedy, Romance, and Sci-Fi. REDDIT-B is a dataset with 2K graphs, where each graph corresponds to an online discussion thread. The classification goal is to predict whether the graph belongs to a Q&A-based community or discussion-based community. REDDIT-5K consists of 5K threads (as graphs) from five different subreddits. The classification goal is to predict the corresponding subreddit for each thread. REDDIT-12K is similar to REDDIT-5k, but with 11,929 graphs from 12 different subreddits. Table 4 summarizes the size of available graph data (i.e., number of graphs, and both max & mean number of vertices within graphs) in these datasets, as previously reported in the literature. Graph signals for social network data: None of the social network datasets has ready-to-use node features. Therefore, in the case of COLLAB, IMDB-B, and IMDB-M, we use the eccentricity, degree, and clustering coefficients for each vertex as characteristic graph signals. In the case of REDDIT-B, REDDIT-5K and REDDIT-12K, on the other hand, we only use degree and clustering coefficient, due to presence of disconnected graphs in these datasets. Software & hardware environment: Geometric scattering and related classification code were implemented in Python with TensorFlow. All experiments were performed on HPC environment using an intel16-k80 cluster, with a job requesting one node with four processors and two Nvidia Tesla k80 GPUs.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new feed forward graph ConvNet based on generalizing the wavelet scattering transform of Mallat, and demonstrate its utility in graph classification and data exploration tasks.

Represent the short summary for finding the corresponding scientific article (long)

We study the effect of the embedding complexity in learning domain-invariant representations and develop a strategy that mitigates sensitivity to it.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: OCR is inevitably linked to NLP since its final output is in text. Advances in document intelligence are driving the need for a unified technology that integrates OCR with various NLP tasks, especially semantic parsing. Since OCR and semantic parsing have been studied as separate tasks so far, the datasets for each task on their own are rich, while those for the integrated post-OCR parsing tasks are relatively insufficient. In this study, we publish a consolidated dataset for receipt parsing as the first step towards post-OCR parsing tasks. The dataset consists of thousands of Indonesian receipts, which contains images and box/text annotations for OCR, and multi-level semantic labels for parsing. The proposed dataset can be used to address various OCR and parsing tasks. Optical character recognition (OCR) is a technique for converting images of characters into digitized texts. Recently, deep learning in computer vision domain has significantly improved the performances of OCR. Nonetheless, there is still huge room for improvement, especially concerning the tasks simultaneously linked to natural language processing (NLP) as well. In particular, post-OCR parsing is currently one of the most important, yet challenging problems in both OCR and NLP community. The goal of post-OCR parsing is to predict pre-defined semantic labels from the given OCR. Researchers from both domains have long tried to tackle the problem and collected a significant amount of data sets independently. However, since it is a specialized task, the datasets contain critical limitations to provide proper supervision. The OCR datasets typically do not have parsing class labels for the extracted texts. The parsing datasets usually contain error-free and well-ordered digitized texts in contrast to the erroneous outcomes from OCR process. We can add synthetic noise to the parsing data, but the distribution and error patterns could be different from the OCR errors, which would inevitably lead to the degradation of generalization performance. Over the past few years, a few post-OCR parsing datasets have been made public through post OCR challenges. For example, ICDAR 2019 Post-OCR Challenge introduced the Scanned Receipts OCR and Information Extraction (SROIE) dataset. It provides receipt images of texts and two types of annotations for OCR and parsing problem: box-level text annotations for OCR, and document-level parse annotations for parsing. Although the availability of both OCR and parsing information have given rise to active research within the field, it still possesses some shortcomings, e.g., limited data size and lack of box-level parsing annotations. Considering that only hundreds of samples are provided in the SROIE dataset, weak document-level annotations could not provide enough supervision for training a model with satisfactory performance. In this paper, we introduce a novel dataset called CORD, which stands for a Consolidated Receipt Dataset for post-OCR parsing. To the best of our knowledge, this is the first publicly available dataset which includes both box-level text and parsing class annotations. The parsing class labels are provided in two-levels. The eight superclasses include store, payment, menu, subtotal, and total. The eight superclasses are subdivided into 54 subclasses e.g., store has nine subclasses including name, address, telephone, and fax. Furthermore, it also provides line annotations for the serialization task which is a newly emerging problem as a combination of the two tasks. Current semantic parsing techniques can handle only well-ordered texts. Texts obtained by OCR, however, are in two-dimensional space, thus we need an appropriate serialization technique for mapping obtained texts into one-dimensional space. In our experiments, serialization has a significant impact on parsing performance. To recapitulate briefly, the key contributions of our paper are as follows: • We introduce a novel and large-scale receipt dataset that can be used for OCR and parsing tasks, from task-specific to end-to-end. • Our dataset provides multi-level labels for weakly and strongly supervised parsing tasks. The dataset and descriptions will be available on https://github.com/clovaai/cord at the time of publication. 2 Data Acquisition We collected over 11,000 Indonesian receipt images through crowd-sourcing. Receipts have been obtained from shops and restaurants. In general, crowd-sourcing involves providing a guideline and annotation tool for crowd workers. For making guidelines, we first sampled hundreds of receipts and analyzed their common structures. The sampled receipts are carefully examined and then dominant and useful parse categories were defined. The main parse classes consist of store information, payment information, menu, void menu, subtotal, void total, total. Since these structures are not easily perceptible for the human mind, we carried out pilot annotation and redesigned parse categories repeatedly. Finally, we created two guidelines for the OCR and parse annotation. For more efficient annotation, we developed a web-based annotation tool and provided it to crowd workers. The annotation tool has two types of user accounts: annotator and inspector. When the annotator completes the annotation on an image, the inspector checks the annotation and determines whether the annotator complies with the guidelines or not. If the annotation is not done properly, the corresponding image is reassigned to another annotator. Although a relatively small amount, receipts may have sensitive information, e.g., customer's name, number of credit/debit card, transaction date/time. We thoroughly inspected the image and then removed the information by blurring in the image and deleting the corresponding field in the JSONformatted file. 3 Data Specification The receipts dataset consists of more than 11,000 image and JSON pairs. An example of image and json pair is shown in Figure 1. The ground truth has three main attributes, meta, the region of The valid line field has crucial information for post-OCR parsing. The quad field contains four coordinates of quadrilateral, and the text field has the incorporating text of the corresponding box. quad and text fields are used for OCR detection/localization and recognition task, respectively. Note that only optically identifiable text instances are annotated. For parsing tasks, there are three additional fields, category, is_key, and row_id. The category indicates parse class label. Note that the details of parse classes are explained on the Section 3.2. The row_id is an index of the line. The text instances which have the same row_id are on the same line. As represented in Figure 1, BORNGA and EXPRESS have the same row_id since they are placed next to each other. The is_key flag is used to identify words that act as a key to other text elements. For example, the is_key value of the text BORNGA is 0 since it does not act as a key but a value. On the other hand, the text Total has an is_key value of 1 because it acts as a key to another text element, 45,500. We present dataset statistics in Table 1. A dataset consists of eight superclasses: store, payment, menu, void menu, subtotal, void total, total, and etc. These eight superclasses are divided into 54 subclasses, e.g., the superclass menu has 16 subclasses containing menu name, quantity, unit price, discount price, submenu. Figure 2 represents the top 20 class labels from 1,000 randomly sampled data examples. The most frequent class label is the menu, especially menu name, price, and count. In this paper, we introduce a novel receipt dataset for a unified OCR-parsing task. The proposed dataset can be exploited not only each task-specific task but also end-to-end approaches. A receipt is a complex type of document. It contains many numbers and symbols as well as plain text, and it also has a complex text layout such as a tablet-shaped layout. Besides, receipt images acquired through the OCR process have various types of optical noise that originated from wrinkles and warpage. Due to these characteristics, the proposed data is more suitable for the OCR-parsing task than the synthetically generated dataset in terms of generalization performance. As mentioned in Section 3.2, our dataset provides multi-level class labels, for eight superclasses and 54 subclasses. It can be used to carry out weakly supervised learning as well as strongly supervised learning. Note that we will first release 1,000 samples which consist of 800 (train), 100 (dev), and 100 (test) data examples, and the remaining data will be published in sequence. The exposure of sensitive information can cause a legal problem, so we will carefully examine and remove sensitive information.

Represent the short summary for finding the corresponding scientific article (long)

We introduce a large-scale receipt dataset for post-OCR parsing tasks.

We introduce a lightweight architecture for named entity recognition and carry out incremental active learning, which is able to match state-of-the-art performance with just 25% of the original training data.


Question: Represent the scientific article to find a short TLDR

Question: The growth in the complexity of Convolutional Neural Networks (CNNs) is increasing interest in partitioning a network across multiple accelerators during training and pipelining the backpropagation computations over the accelerators. Existing approaches avoid or limit the use of stale weights through techniques such as micro-batching or weight stashing. These techniques either underutilize of accelerators or increase memory footprint. We explore the impact of stale weights on the statistical efficiency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization and keeps memory overhead modest. We use 4 CNNs (LeNet-5, AlexNet, VGG and ResNet) and show that when pipelining is limited to early layers in a network, training with stale weights converges and in models with comparable inference accuracies to those ing from non-pipelined training on MNIST and CIFAR-10 datasets; a drop in accuracy of 0.4%, 4%, 0.83% and 1.45% for the 4 networks, respectively. However, when pipelining is deeper in the network, inference accuracies drop significantly. We propose combining pipelined and non-pipelined training in a hybrid scheme to address this drop. We demonstrate the implementation and performance of our pipelined backpropagation in PyTorch on 2 GPUs using ResNet, achieving speedups of up to 1.8X over a 1-GPU baseline, with a small drop in inference accuracy. Modern Convolutional Neural Networks (CNNs) have grown in size and complexity to demand considerable memory and computational resources, particularly for training. This growth makes it sometimes difficult to train an entire network with a single accelerator (; ; . Instead, the network is partitioned among multiple accelerators, typically by partitioning its layers among the available accelerators, as shown in Figure 1 for an example 8-layer network. The 8 layers are divided into 4 computationally-balanced partitions, P 0 ...P 3 and each partition is mapped to one of the 4 accelerators, A 0 ...A 3 . Each accelerator is responsible for the computations associated with the layers mapped to it. However, the nature of the backpropagation algorithm used to train CNNs is that the computations of a layer are performed only after the computations of the preceding layer in the forward pass of the algorithm and only after the computations of the succeeding layer in the backward pass. Further, the computations for one batch of input data are only performed after the computations of the preceding batch have updated the parameters (i.e., weights) of the network. These dependences underutilize the accelerators, as shown by the space-time diagram in Figure 2; only one accelerator can be active at any given point in time. The underutilization of accelerators can be alleviated by pipelining the computations of the backpropagation algorithm over the accelerators (; ; . That is, by overlapping the computations of different input data batches using the multiple accelerators. However, pipelining causes an accelerator to potentially use weights that are yet to be updated by an accelerator further down in the pipeline. The use of such stale weights can negatively affect the statistical efficiency of the network, preventing the convergence of training or producing a model with lower inference accuracy. Common wisdom is that the use of stale weights must either be avoided, e.g., with the use of microbatches , be constrained to ensure the consistency of the weights within an accelerator using stashing , or by limiting the use of pipelining to very small networks . However, these approaches either underutilize accelerators or inflate memory usage to stash multiple copies of weights . In this paper we question this common wisdom and explore pipelining that allows for the full utilization of accelerators while using stale weights. This in a pipelining scheme that, compared to existing schemes, is simpler to implement, fully utilizes the accelerators and has lower memory overhead. We evaluate this pipelining scheme using 4 CNNs: LeNet-5 (trained on MNIST), AlexNet, VGG and ResNet (all trained on CIFAR-10). We analyze the impact of weight staleness and show that if pipelining is limited to early layers in the network, training does converge and the quality of the ing models is comparable to that of models obtained with non-pipelined training. For the 4 networks, the drop in accuracy is 0.4%, 4%, 0.83% and 1.45%, respectively. However, inference accuracies drop significantly when the pipelining is deeper in the network. While this is not a limitation since the bulk of computations that can benefit from pipelining are in the early convolutional layers, we address this through a hybrid scheme that combines pipelined and non-pipelined training to maintain inference accuracy while still delivering performance improvement. Evaluation shows that our pipelined training delivers a speedup of up to 1.8X on a 2-GPU system. The remainder of this paper is organized as follows. Section 2 briefly describes the backpropagation for training of CNNs. Section 3 details our pipelining scheme. Section 4 describes how non-pipelined and pipelined backpropagation are combined. Section 5 highlights some of the implementation details. Experimental evaluation is presented in Section 6. Related work is reviewed in Section 7. Finally, Section 8 gives concluding remarks and directions for future work. The backpropagation algorithm consists of two passes: a forward pass that calculates the output error and a backward pass that calculates the error gradients and updates the weights of the network. The two passes are performed for input data one mini-batch at a time. In the forward pass, a mini-batch is fed into the network, propagating from the first to the last layer. At each layer l, the activations of the layer, denoted by x (l), are computed using the weights of the layer, denoted by W (l). When the output of the network (layer L) x (L) is produced, it is used with the true data label to obtain a training error e for the mini-batch. In the backward pass, the error e is propagated from the last to the first layer. The error gradients with respect to pre-activations of layer l, denoted by δ (l), are calculated. Further, the error gradients with respect to weights of layer l, ∂e ∂W (l), are computed using the activations from layer l − 1 (i.e., x (l−1) ) and δ (l). Subsequently, δ (l) is used to calculate the δ (l−1). When ∂e ∂W (l) is computed for every layer, the weights are updated using the error gradients. In the forward pass, the activations of the layer l, x (l), cannot be computed until the activations of the previous layers, i.e., x (l−1), are computed. In backward pass, ∂e ∂W (l) can only be computed once x (l−1) and δ (l) have been computed. Moreover, δ (l) depends on δ (l+1). Finally, for a given mini-batch the backward pass cannot be started until the forward pass is completed and the error e has been determined. The above dependences ensure that the weights of the layers are updated using the activations and error gradients calculated from the same batch of training data in one iteration of the backpropagation algorithm. Only when the weights are updated is the next batch of training data fed into the Figure 3: Pipelined Backpropagation Algorithm network. These dependences limit parallelism when a network is partitioned across multiple accelerators and allow only one accelerator to be active at any point. This in under-utilization of the accelerators. It is this limitation that pipelining addresses. We illustrate our pipelined backpropagation implementation with the L layer network shown in Figure 3, using conceptual pipeline registers. Two registers are inserted between layers l and l + 1; one register for the forward pass and a second for the backward pass. The forward register stores the activations of layer l (x (l) ). The backward register stores the gradients δ (l+1) of layer l + 1. This defines a 4-stage pipelined backpropagation. The forward pass for layers 1 to l forms forward stage FS 1. The forward pass for layers l + 1 to L form forward stage FS 2. Similarly, the backwards pass for layers l + 1 to L and 1 to l form backward stages BKS 1 and BKS 2 respectively. The forward and backward stages are executed in a pipelined fashion on 3 accelerators: one for FS 1, one for both FS 2 and BKS 1, and one for BKS 2 1. In cycle 0, mini-batch 0 is fed to FS 1. The computations of the forward pass are done as in the traditional non-pipelined implementation. In cycle 1, layer l activations x (l) are fed to FS 2 and mini-batch 1 is fed to FS 1. In cycle 2, the error for mini-batch 0 computed in FS 2 is directly fed to BKS 1, the activations of layer l x (l) are forwarded to FS 2 and mini-batch 2 is fed to FS 1. This pipelined execution is illustrated by the space-time diagram in Figure 4 for 5 mini-batches. The figure depicts the mini-batch processed by each accelerator cycles 0 to 6. At steady state, all the accelerators are active in each cycle of execution. The above pipelining scheme utilizes weights in FS 1 that are yet to be updated by the errors calculated by FS 2 and BKS 1. At steady state, the activations of a mini-batch in FS 1 are calculated using weights that are 2 execution cycles old, or 2 cycles stale. This is reflected in Figure 4 by indicating the weights used by each forward stage and the weights updated by each backward stage. The weights of a forward stage are subscripted by how stale they are (-ve subscripts). Similarly, the weights updated by a backward stage are subscripted by how delayed they are (+ve subscripts). Further, since the updates of the weights by BKS 2 requires activations calculated for the same mini-batch in FS 1 for all layers in the stage, it is necessary to save these activations until the error gradients with respect to the weights are calculated by BKS 2. Only when the weights are updated using the gradients can these activations be discarded. In the general case, we use K pairs of pipeline registers (each pair consisting of a forward register and a backward register) inserted between the layers of the network. We describe the placement of the register pairs by the Pipeline Placement Vector, PPV = (p 1, p 2, ..., p K), where p i represents the layer number after which a pipeline register pair is inserted. Such a placement creates (K + 1) forward stages, labeled FS i, i = 1, 2,..., K + 1 and (K + 1) backward stages, labeled BKS i, i = 1, 2,..., K + 1. Forward stage FS i and backward stage BKS K−i+2 correspond to the same set of layers. Specifically, stage FS i contains layers p i + 1 to p i+1, inclusive. We assign each forward stage and each backward stage to an accelerator, with the exception of the FS K+1 and backward stage BKS 1, which are assigned to the same accelerator to reduce weight staleness by an execution cycle. In total 2K + 1 accelerators are used. We quantify weight staleness as follows. A forward stage FS i and backward stage BKS K−i+2 use the same weights that are 2(K − i + 1) cycles old. A forward stage FS i must store the activations of On the one hand, the above pipelined execution allows a potential speedup of 2K + 1 over the nonpipelined implementation, keeping all the accelerators active at steady state. On the other hand, the use of stale weights may prevent training convergence or may in a model that has an inferior inference accuracy. Further, it requires an increase in storage for activations. Our goal is to assess the benefit of this pipelined execution and the impact of its down sides. Hybrid training combines pipelined training with non-pipelined training. We start with pipelined training and after a number of iterations, we switch to non-pipelined training. This can address drops in inference accuracy of ing models because of weight staleness, but it reduces the performance benefit since during non-pipelined training, the accelerators are under-utilized. The extent of the speedup obtained by hybrid training with a given number of accelerators is determined by the number of iterations used for pipelined and non-pipelined training. Assume that n np iterations are used to reach the best inference accuracy for non-pipelined training, and that in hybrid training, n p iterations (n p ≤ n np) are pipelined followed by n np − n p iterations of non-pipelined training to reach the same inference accuracy as non-pipelined training. The speedup of hybrid training with respect to the non-pipelined training with 2K + 1 accelerators is n np /(n p /(2K + 1) + (n np − n p)). For large K, then using Amdahl's law, the speedup approaches an upper bound of n np /(n np − n p). We implement pipelined training in two ways: simulated in Caffe , where the whole training process is performed on one process with no parallelism, and actual with parallelism across accelerators in PyTorch . The simulated execution is used to analyze statistical convergence, inference accuracy and impact of weight staleness unconstrained by parallelism and communication overhead. The actual execution is used to report performance and PyTorch is used instead of Caffe to leverage its support for collective communication protocols and its flexibility in partitioning a network across multiple accelerators. Both Caffe and PyTorch have no support for pipelined training. Thus both were extended to provide such support. We develop a custom Caffe layer in Python, which we call a Pipeline Manager Layer (PML), to facilitate the simulated pipelining. During the forward pass, a PML registers the input from a previous layer and passes the activation to the next layer. It also saves the activations for the layers connected to it to be used in the backward pass. During the backward pass, a PML passes the appropriate error gradients. It uses the corresponding activations saved during the forward pass to update weights and generate error gradients for the previous stage, using existing weight update mechanisms in Caffe. To implement actual hardware-accelerated pipelined training, we partition the network onto different accelerators (GPUs), each running its own process. Asynchronous sends and receives are used for data transfers, but all communication must go through the host CPU, since point-to-point communication between accelerators is not supported in PyTorch. This increases communication overhead. Similar to the PMLs in Caffe, the activations computed on one GPU are copied to the next GPU (via the CPU) in the forward pass and the error gradients are sent (again via the CPU) to the preceding GPU during the backward pass. The GPUs are running concurrently, achieving pipeline parallelism. Simulated pipelining is evaluated on a machine with one Nvidia GTX1060 GPU with 6 GB of memory and an Intel i9-7940X CPU with 64 GB of RAM. The performance of actual pipelining is evaluated using two Nvidia GTX1060 GPUs, each with 6 GB of memory, hosted in an Intel i7-9700K machine with 32 GB of RAM. We use four CNNs in our evaluation: LeNet-5 with minor variations to the hyperparameters, as described in Appendix 8. We evaluate the effectiveness of pipelined training in terms of its training convergence and its Top-1 inference accuracy, compared to those of the non-pipelined training. We use the speedup to evaluate performance improvements. It is defined as the ratio of the training time of the non-pipelined implementation on single communication-free GPU to the training time of the pipelined training. 6.2 TRAINING CONVERGENCE AND INFERENCE ACCURACY Figure 5 shows the improvements in the inference accuracies for both pipelined and non-pipelined training as a function of the number of training iterations (each iteration corresponds to a minibatch). The pipelined training is done using 4, 6, 8 and 10 stages. Table 1 shows where the registers are inserted in the networks using their PPV defined in Section 3. Figure 5 shows that for all the networks, both pipelined and non-pipelined training have similar convergence patterns. They converge in more or less the same number of iterations for a given number of pipeline stages, albeit to different inference accuracies. This indicates that our approach to pipelined training with stale weights does converge, similar to non-pipelined training. Table 2 shows the inference accuracy obtained after up to 30,000 iterations of training. For LeNet-5, the inference accuracy drop is within 0.5%. However, for the other networks, there is a small drop in inference accuracy with 4 and 6 stages. AlexNet has about 4% drop in inference accuracy, but for VGG-16 the inference accuracy drop is within 2.4%, and for ResNet-20 the accuracy drop is within 3.5%. Thus, the ing model quality is comparable to that of a non-pipelining-trained model. However, with deeper pipelining (i.e., 8 and 10 stages), inference accuracy significantly drops. There is a 12% and a 8.5% inference accuracy drop for VGG-16 and ResNet-20 respectively. In this case, the model quality is not comparable to that of the non-pipelined training. This confirm what is reported in the literature and can be attributed to the use of stale weights. Below we further explore the impact of stale weights on inference accuracy. We wish to better understand the impact of the number of pipeline stages and the location of these stages in the network on inference accuracy. We focus on ResNet-20 because of its relatively small size and regular structure. It consists of 3 residual function groups with 3 residual function blocks within each group. In spite of this relatively small size and regular structure, it enables us to create pipelines with up to 20 stages by inserting pipeline register pairs within residual function blocks. We conduct two experiments. In the first, we increase the number of pipeline stages (from earlier layers to latter layers) and measure the inference accuracy of the ing model. The are shown in Table 3, which gives the inference accuracy of pipelined training after 100,000 iterations, as the number of pipeline stages increases. The 8-stage pipelined training is created by a PPV of, and the subsequent pipeline schemes are created by adding pipeline registers after every 2 layers after layer 7. Clearly, the greater the number stages, the worse is the ing model quality. The number of stale weights used in the pipelined training increases as the number of pipeline stages increases. Thus, Figure 6 depicts the inference accuracy as a function of the percentage of weights that are stale. The curve labeled "Increasing Stages" shows that the drop in inference accuracy increases as the percentage of stale weights increases. In the second experiment, we investigate the impact of the degree of staleness (Section 3). Only one pair of pipeline registers is inserted. The position of this register slides from the beginning of the network to its end. At every position, the percentage of stale weights remains the same as in the first experiment, but all stale weights have the same degree of staleness. The of this experiment is shown by the curve labeled "Sliding Stage" in Figure 6. The curve shows the inference accuracy also drops as the percentage of stale weights increases. However, it also indicates that the drop of inference accuracy remains more or less the same as in the first experiment in which the degree of staleness is higher. Thus, the percentage of stale weight appears to be what determines the drop in inference accuracy and not the degree of staleness of the weights. The percentage of stale weight is determined by where the last pair of pipeline registers are placed in the network. It is the position of this pair that determines the loss in inference accuracy. Therefore, it is desirable to place this last pair of registers as early as possible in the network so as to minimize the drop in inference accuracy. While at first glance this may seem to limit pipelining, it is important to note that the bulk of computations in a CNN is in the first few convolutional layers in the network. Inserting pipeline registers for these early layers can in both a large number of stages that are computationally balanced. For example, our profiling of the runtime of ResNet-20 shows that the first three residual functions take more than 50% of the training runtime. This favors more pipeline stages at the beginning of the network. Such placement has the desirable effect of reducing the drop in inference accuracy while obtaining relatively computationally balanced pipeline stages. We demonstrate the effectiveness of hybrid training using only ResNet-20 for brevity. Figure 7 shows the inference accuracy for 20K iterations of pipelined training followed by either 10K or 20K iterations of non-pipelined training. This inference accuracy is compared to 30K iterations of either non-pipelined or pipelined training with PPV. The figure demonstrates that hybrid training converges in a similar manner to both pipelined and non-pipelined training. Table 4 shows the ing inference accuracies. The table shows that the 20K+10K hybrid training produces a model with accuracy that is comparable to that of the non-pipelined model. Further, with an additional 10K iterations of non-pipelined training, the model quality is slightly better than that of the non-pipelined model. This demonstrates the effectiveness of hybrid training. We implement 4-stage pipelined training ResNet-20/56/110/224/362 on a 2-GPU system. Each GPU is responsible for one forward stage and one backward stage. Thus, the maximum speedup that can be obtained is 2. We train every ResNet for 200 epochs. Table 5 shows the inference accuracies with and without pipelining as model as the measured speedups of pipelined training over the non-pipelined one. The table indicates that the quality of the models produced by pipelined training is comparable to those achieved by the simulated pipelining on Caffe. The table further shows that speedup exists for all networks. Indeed, for ResNet-362, the speedup is 1.82X. This is equivalent to about 90% utilization for each GPU. The table also reflects that as the networks get larger, the speedup improves. This reflects that with larger networks, the ratio of computation to communication overhead is higher, leading to better speedups. Moreover, we combine the 4-stage pipelined training described above with non-pipelined training to demonstrate the performance of hybrid training. We train every ResNet using pipelined training for 100 epochs and follow it up by 100 epochs of non-pipelined training. Because the maximum speedup for the pipelined training is 2 and only half the training epochs is accelerated, the maximum speedup for this hybrid training is s = t/(t/2 + t/4) = 1.33, where t is the training time of non- Table 5 shows the inference accuracies and speedup of the hybrid training for each ResNet and validates that hybrid training can produce a model quality that is comparable to the baseline non-pipelined training while speeding up the training process. As network size grows, the speedup reaches 1.29X, approaching the theoretical limit 1.33X. Pipelined training requires the saving of intermediate activations, as described earlier in Section 3, leading to an increase in memory footprint. This increase in memory is a function of not only the placement of the pipeline registers, but also of the network architecture and the number of inputs in a mini-batch (batch size). We calculate the memory usage of the 4-stage pipelined ResNet training above to show that this increase is modest for our pipelining scheme. Specifically, we use torchsummary in PyTorch to report memory usage for weights and activations for a network and calculate the additional memory required by the additional copies of activations. The are shown in Table 6. Assuming a batch size of 128, the percentage increase in size is close to 60% except for ResNet-20. We compare our pipelined training scheme with two key existing systems: PipeDream and GPipe . We do so on three aspects: the pipelining scheme, performance and memory usage. We believe that PipeDream and GPipe are representative of existing key approaches that implement pipelined training, including Decoupled Backpropagation (DDG) (b) and Feature Replay (FR) (a) (discussed in Section 7). Our pipelining scheme is simpler than that of PipeDream and GPipe in that we do not require weight stashing nor do we divide mini-batches into micro-batches. This leads to less communication overhead, and is amicable to rapid realization in machine learning framework such as PyTorch or in actual hardware such as Xilinx's xDNN FPGA accelerators . Our pipelining scheme, as PipeDream, eliminates bubbles that exist in the pipeline leading to better performance. For example, we obtain a speedup of 1.7X for ResNet-110 using 2 GPUs in contrast to GPipe that obtains a speedup of roughly 1.3X for ResNet-101 using 2 TPUs. We also obtain similar performance compared to PipeDream for similar networks. When the number of pipeline stages grows, pipeline bubbles exhibits more negative effect on performance shown in GPipe on a 4-partition pipelined ResNet-101 using 4 TPUs as its bubble overhead doubled compared to that of the 2-partition pipelined ResNet-101. Our scheme uses less memory compared to PipeDream, although it introduces more memory overhead compared to GPipe. PipeDream saves intermediate activations during training, as we do. However, it also saves multiple copies of a network's weights for weight stashing. The memory footprint increase due to this weight stashing depends on the network architecture, including the number of weights and activations, as well as on the size of the mini-batch. For example, for VGG-16 trained on CIFAR-10 with a mini-batch size of 128 using a 4-stage, pipelined training, we estimate our pipelining methodology to use 49% less memory compared PipeDream. Similarly for VGG-16 trained on ImageNet and a mini-batch size of 32, our scheme uses 29% less memory. We estimate the memory increase due to weight stashing also using tourchsummary. There has been considerable work that explores parallelism in the training of deep neural networks. There are several approaches to exploiting parallelism. One approach is to exploit data parallelism (; ; ; ; ;), in which each accelerator obtains a full copy of the model and processes different mini-batches of training data simultaneously. At the end of each training iteration, the gradients produced by all accelerators are aggregated and used to update weights for all copies of the model, synchronously or asynchronously . A centralized parameter server is usually used to facilitate data communication . Although the training is performed in parallel, the communication overhead can be significant . A second approach is to exploit model parallelism (; ; ; ;). In this approach, a model is partitioned onto different accelerators (; ; ; ;). Each accelerator is only responsible for updating the weights for the portion of the model assigned to it. This approach is often used when a model is large and cannot fit into the memory of a single accelerator. However, because of the data dependences described in Section 2, only one accelerator is active during the training process, ing in under-utilization of accelerators resources. Moreover, inter-layer activations and gradients across two consecutive stages needs to be communicated during training, adding more overhead to the entire process. Pipelined parallelism addresses the under-utilization of accelerators resources for the training of large models. There have been a few studies that explore pipelined parallelism (; ; ; ; b ;a), which we review in this section. PipeDream implements pipelined training for large neural networks such as VGG-16, Inception-v3 and S2VT across multiple GPUs. However, in their implementation, they limited the usage of stale weights by weight stashing, i.e., keeping multiple versions of network parameters (weights) during training. This increases the memory footprint of training. In contrast, we do not maintain multiple copies of weights during training, therefore reducing the memory footprint of pipelined training. GPipe implements a library in Tensorflow to enable pipelined parallelism for the training of large neural networks. GPipe pipelines micro-batches within each mini-batch to keep the gradients consistently accumulated. This eliminates the use of stale weight during training, but it does so at the expense of "pipeline bubbles" at steady state. GPipe utilizes these bubbles to reduce the memory footprint by re-computing forward activations instead of storing them. In contrast, our work has no pipeline bubble and thus dedicates computing resources to compute forward pass and backward pass only once during each training iteration. Huo et al. (b) implement decoupled backpropagation (DDG) using delayed gradient updates. They show that DDG guarantees convergence through a rigorous convergence analysis. Similar to PipeDream, DDG uses multiple copies of the weights and thus increases memory footprint. Further, DDG pipelines only the backward pass of training, leaving forward pass un-pipelined. Huo et al. (a) follow up by proposing feature replay (FR) that re-computes activations during backward pass, similar to GPipe, ing less memory footprint and improved inference accuracy than DDG. In contrast, we pipeline both forward and backward pass without maintaining multiple copies of weights or re-computing forward activations during backward pass. Thus, in summary, our work contrasts to the above work on pipelined training, in that we use pipelining with unconstrained stale weights, ing in full pipeline utilization with a modest increase in memory usage. We extend earlier work by studying the impact of weights staleness on the quality of the model. We show that it is effective to use stale weights if the pipelining is in early layers, which is where the bulk of computations exist. Further we also extend earlier work through hybrid training, which combines both pipelined and non-pipelined training. We compare the performance and memory footprint increase of our scheme to existing work in Section 6.7. We evaluate pipelined execution of backpropagation for the training of CNNs in a way that fully utilizes accelerators, achieving a speedup of 1.82X on the 2-GPU system, and does not significantly increase memory usage, unlike previous work. We show that pipelining training with stale weights does converge. Further, we show that the inference accuracies of the ing models are comparable to those of models obtained with traditional backpropagation, but only when pipelining is implemented in the early layers of the network, with inference accuracy drop within 1.45% on 4-stage pipelined training except for AlexNet. This does not limit the benefit of pipelining since the bulk of computations is in the early convolutional layers. When pipelining is implemented deeper in the network, the inference accuracies do drop significantly, but we can compensate for this drop by combining pipelined with non-pipelined training, albeit with lower performance gains, obtaining model quality with an average of 0.19% better than the baseline in inference accuracies for ResNets. This work can be extended in a number of directions. One direction is to evaluate the approach with a larger number of accelerators since pipelined parallelism is known to scale naturally with the number of accelerators. Another is to evaluate the approach on larger datasets, such as ImageNet. Finally, pipelined parallelism lends itself to hardware implementation. Thus, another direction for future work is to evaluate pipelined parallelism using Field Programmable Gate Array (FPGA) or ASIC accelerators. LeNet-5 is trained on the MNIST dataset with Stochastic Gradient Descent (SGD) using a learning rate of 0.01 with inverse learning policy, a momentum of 0.9, a weight decay of 0.0005 and a minibatch size of 100 and for 30,000 iterations. The progression of inference accuracy during training is recorded with 300 tests AlexNet is trained on the CIFAR-10 dataset with SGD with Nesterov momentum using a learning rate of 0.001 that is decreased by 10x twice during training, a momentum of 0.9, a weight decay of 0.004 and a mini-batch size of 100 for 250,000 iterations. One test is performed every epoch to record the progression of inference accuracy. VGG-16 is trained on CIFAR-10 dataset with SGD with Nesterov momentum using a learning rate starting at 0.1 that is decreased by half every 50 epochs during training, a momentum of 0.9, a weight decay of 0.0005 and a mini-batch size of 100 for 250,000. Since it is relatively more difficult to train VGG-16 compared to other models, batch normalization and dropout are used during training throughout the network. One test is performed every epoch to record the progression of inference accuracy. ResNet is trained on CIFAR-10 dataset with SGD using a learning rate starting at 0.1 and 0.01 for non-pipelined and pipelined training respectively, that is decreased by 10x twice during training, a momentum of 0.9, a weight decay of 0.0001 and a mini-batch size of 128 for 100,000 iterations. Batch normalization is used during training throughout the network. One test is performed every 100 iterations to record the progression of inference accuracy. For the baseline non-piplined training, ResNet-20/56/110/224/362 is trained on CIFAR-10 dataset for 200 epochs with SGD using a learning rate of 0.1 that is decreased by a factor of 10 twice (at epoch 100 and 150), a momentum of 0.9, a weight decay of 0.0001 and a mini-batch size of 128. Batch normalization is used during training throughout the network. This set of hyperparameters can be found at https://github.com/akamaster/pytorch resnet cifar10. For the 4-stage pipelined training, the hyperparameters are the same as the non-pipelined baseline, except for the BKS 2 learning rate.

Accelerating CNN training on a Pipeline of Accelerators with Stale Weights

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.