Question: Represent the article for finding a single-sentence TLDR

Question: The rise of graph-structured data such as social networks, regulatory networks, citation graphs, and functional brain networks, in combination with resounding success of deep learning in various applications, has brought the interest in generalizing deep learning models to non-Euclidean domains. In this paper, we introduce a new spectral domain convolutional architecture for deep learning on graphs. The core ingredient of our model is a new class of parametric rational complex functions (Cayley polynomials) allowing to efficiently compute spectral filters on graphs that specialize on frequency bands of interest. Our model generates rich spectral filters that are localized in space, scales linearly with the size of the input data for sparsely-connected graphs, and can handle different constructions of Laplacian operators. Extensive experimental show the superior performance of our approach on spectral image classification, community detection, vertex classification and matrix completion tasks. In many domains, one has to deal with large-scale data with underlying non-Euclidean structure. Prominent examples of such data are social networks, genetic regulatory networks, functional networks of the brain, and 3D shapes represented as discrete manifolds. The recent success of deep neural networks and, in particular, convolutional neural networks (CNNs) BID19 have raised the interest in geometric deep learning techniques trying to extend these models to data residing on graphs and manifolds. Geometric deep learning approaches have been successfully applied to computer graphics and vision; BID3 a); BID24, brain imaging BID18, and drug design BID10 problems, to mention a few. For a comprehensive presentation of methods and applications of deep learning on graphs and manifolds, we refer the reader to the review paper BID4.Related work. The earliest neural network formulation on graphs was proposed by BID11 and BID27, combining random walks with recurrent neural networks (their paper has recently enjoyed renewed interest in BID20 ; BID30). The first CNN-type architecture on graphs was proposed by BID5. One of the key challenges of extending CNNs to graphs is the lack of vector-space structure and shift-invariance making the classical notion of convolution elusive. Bruna et al. formulated convolution-like operations in the spectral domain, using the graph Laplacian eigenbasis as an analogy of the Fourier transform BID29 ). BID13 used smooth parametric spectral filters in order to achieve localization in the spatial domain and keep the number of filter parameters independent of the input size. BID8 proposed an efficient filtering scheme using recurrent Chebyshev polynomials applied on the Laplacian operator. BID17 simplified this architecture using filters operating on 1-hop neighborhoods of the graph. BID0 proposed a Diffusion CNN architecture based on random walks on graphs. BID24 (and later,) proposed a spatial-domain generalization of CNNs to graphs using local patch operators represented as Gaussian mixture models, showing a significant advantage of such models in generalizing across different graphs. In BID25, spectral graph CNNs were extended to multiple graphs and applied to matrix completion and recommender system problems. Main contribution. In this paper, we construct graph CNNs employing an efficient spectral filtering scheme based on Cayley polynomials that enjoys similar advantages of the Chebyshev filters BID8 ) such as localization and linear complexity. The main advantage of our filters over BID8 is their ability to detect narrow frequency bands of importance during training, and to specialize on them while being well-localized on the graph. We demonstrate experimentally that this affords our method greater flexibility, making it perform better on a broad range of graph learning problems. Notation. We use a, a, and A to denote scalars, vectors, and matrices, respectively.z denotes the conjugate of a complex number, Re{z} its real part, and i is the imaginary unit. diag(a 1, . . ., a n) denotes an n×n diagonal matrix with diagonal elements a 1,..., a n. Diag(A) = diag(a 11, . . ., a nn) denotes an n × n diagonal matrix obtained by setting to zero the off-diagonal elements of A. Off(A) = A − Diag(A) denotes the matrix containing only the off-diagonal elements of A. I is the identity matrix and A • B denotes the Hadamard (element-wise) product of matrices A and B. Proofs are given in the appendix. Spectral graph theory. Let G = ({1, . . ., n}, E, W) be an undirected weighted graph, represented by a symmetric adjacency matrix W = (w ij). We define DISPLAYFORM0 We denote by N k,m the k-hop neighborhood of vertex m, containing vertices that are at most k edges away from m. The unnormalized graph Laplacian is an n × n symmetric positive-semidefinite matrix DISPLAYFORM1 In the following, we use the generic notation ∆ to refer to some Laplacian. Since both normalized and unnormalized Laplacian are symmetric and positive semi-definite matrices, they admit an eigendecomposition ∆ = ΦΛΦ, where Φ = (φ 1, . . . φ n) are the orthonormal eigenvectors and Λ = diag(λ 1, . . ., λ n) is the diagonal matrix of corresponding non-negative eigenvalues (spectrum) 0 = λ 1 ≤ λ 2 ≤... ≤ λ n. The eigenvectors play the role of Fourier atoms in classical harmonic analysis and the eigenvalues can be interpreted as (the square of) frequencies. Given a signal f = (f 1, . . ., f n) on the vertices of graph G, its graph Fourier transform is given byf = Φ f. Given two signals f, g on the graph, their spectral convolution can be defined as the element-wise product of the Fourier transforms, f g = Φ (Φ g)•(Φ f) = Φ diag(ĝ 1, . . .,ĝ n)f, which corresponds to the property referred to as the Convolution Theorem in the Euclidean case. Spectral CNNs. BID5 used the spectral definition of convolution to generalize CNNs on graphs, with a spectral convolutional layer of the form DISPLAYFORM2 (Here the n × p and n × q matrices DISPLAYFORM3 represent respectively the p-and q-dimensional input and output signals on the vertices of the graph, (ĝ l,l,1, . . .,ĝ l,l,k) is a k × k diagonal matrix of spectral multipliers representing a learnable filter in the frequency domain, and ξ is a nonlinearity (e.g., ReLU) applied on the vertex-wise function values. Pooling is performed by means of graph coarsening, which, given a graph with n vertices, produces a graph with n < n vertices and transfers signals from the vertices of the fine graph to those of the coarse one. This framework has several major drawbacks. First, the spectral filter coefficients are basis dependent, and consequently, a spectral CNN model learned on one graph cannot be transferred to another graph. Second, the computation of the forward and inverse graph Fourier transforms incur expensive O(n 2) multiplication by the matrices Φ, Φ, as there is no FFT-like algorithms on general graphs. Third, there is no guarantee that the filters represented in the spectral domain are localized in the spatial domain (locality property simulates local reception fields, BID7); assuming k = O(n) Laplacian eigenvectors are used, a spectral convolutional layer requires O(pqk) = O(n) parameters to train. DISPLAYFORM4 To address the latter issues, BID13 argued that smooth spectral filter coefficients in spatially-localized filters (an argument similar to vanishing moments). The filter coefficients are represented asĝ i = g(λ i), where g(λ) is a smooth transfer function of frequency λ. Applying such filter to signal f can be expressed as DISPLAYFORM5 where applying a function to a matrix is understood in the operator functional calculus sense (applying the function to the matrix eigenvalues). BID13 used parametric functions of the form g(λ) = r j=1 α j β j (λ), where β 1 (λ),..., β r (λ) are some fixed interpolation kernels such as splines, and α = (α 1, . . ., α r) are the interpolation coefficients used as the optimization variables during the network training. In matrix notation, the filter is expressed as Gf = Φdiag(Bα)Φ f, where B = (b ij) = (β j (λ i)) is a k × r matrix. Such a construction in filters with r = O parameters, independent of the input size. However, the authors explicitly computed the Laplacian eigenvectors Φ, ing in high complexity. ChebNet. BID8 used polynomial filters represented in the Chebyshev basis DISPLAYFORM6 applied to rescaled frequencyλ ∈ [−1, 1]; here, α is the (r + 1)-dimensional vector of polynomial coefficients parametrizing the filter and optimized for during the training, and Such an approach has several important advantages. First, since g α (∆) = r j=0 α j T j (∆) contains only matrix powers, additions, and multiplications by scalar, it can be computed avoiding the explicit expensive O(n 3) computation of the Laplacian eigenvectors. Furthermore, due to the recursive definition of the Chebyshev polynomials, the computation of the filter g α (∆)f entails applying the Laplacian r times, ing in O(rn) operations assuming that the Laplacian is a sparse matrix with O non-zero elements in each row (a valid hypothesis for most real-world graphs that are sparsely connected). Second, the number of parameters is O as r is independent of the graph size n. Third, since the Laplacian is a local operator affecting only 1-hop neighbors of a vertex and a polynomial of degree r of the Laplacian affects only r-hops, the ing filters have guaranteed spatial localization. DISPLAYFORM7 A key disadvantage of Chebyshev filters is the fact that using polynomials makes it hard to produce narrow-band filters, as such filters require very high order r, and produce unwanted non-local filters. This deficiency is especially pronounced when the Laplacian has clusters of eigenvalues concentrated around a few frequencies with large spectral gap (FIG4, middle right). Such a behavior is characteristic of graphs with community structures, which is very common in many real-world graphs, for instance, social networks. To overcome this major drawback, we need a new class of filters, that are both localized in space, and are able to specialize in narrow bands in frequency. A key construction of this paper is a family of complex filters that enjoy the advantages of Chebyshev filters while avoiding some of their drawbacks. A Cayley polynomial of order r is a real-valued function with complex coefficients, DISPLAYFORM0 where c = (c 0, . . ., c r) is a vector of one real coefficient and r complex coefficients and h > 0 is the spectral zoom parameter, that will be discussed later. A Cayley filter G is a spectral filter defined on real signals f by where the parameters c and h are optimized for during training. Similarly to the Chebyshev filters, Cayley filters involve basic matrix operations such as powers, additions, multiplications by scalars, and also inversions. This implies that application of the filter Gf can be performed without explicit expensive eigendecomposition of the Laplacian operator. In the following, we show that Cayley filters are analytically well behaved; in particular, any smooth spectral filter can be represented as a Cayley polynomial, and low-order filters are localized in the spatial domain. We also discuss numerical implementation and compare Cayley and Chebyshev filters. DISPLAYFORM1 Analytic properties. Cayley filters are best understood through the Cayley transform, from which their name derives. Denote by e iR = {e iθ : θ ∈ R} the unit complex circle. The Cayley transform DISPLAYFORM2 x+i is a smooth bijection between R and e iR \ {1}. The complex matrix C(h∆) DISPLAYFORM3 −1 obtained by applying the Cayley transform to the scaled Laplacian h∆ has its spectrum in e iR and is thus unitary. Since DISPLAYFORM4. Therefore, using 2Re{z} = z + z, any Cayley filter can be written as a conjugateeven Laurent polynomial w.r.t. C(h∆), DISPLAYFORM5 Since the spectrum of C(h∆) is in e iR, the operator C j (h∆) can be thought of as a multiplication by a pure harmonic in the frequency domain e iR for any integer power j, DISPLAYFORM6 A Cayley filter can be thus seen as a multiplication by a finite Fourier expansions in the frequency domain e iR. Since is conjugate-even, it is a (real-valued) trigonometric polynomial. Note that any spectral filter can be formulated as a Cayley filter. Indeed, spectral filters g(∆) are specified by the finite sequence of values g(λ 1),..., g(λ n), which can be interpolated by a trigonometric polynomial. Moreover, since trigonometric polynomials are smooth, we expect low order Cayley filters to be well localized in some sense on the graph, as discussed later. Finally, in definition we use complex coefficients. If c j ∈ R then FORMULA14 is an even cosine polynomial, and if c j ∈ iR then is an odd sine polynomial. Since the spectrum of h∆ is in R +, it is mapped to the lower half-circle by C, on which both cosine and sine polynomials are complete and can represent any spectral filter. However, it is beneficial to use general complex coefficients, since complex Fourier expansions are overcomplete in the lower half-circle, thus describing a larger variety of spectral filters of the same order without increasing the computational complexity of the filter. Spectral zoom. To understand the essential role of the parameter h in the Cayley filter, consider C(h∆). Multiplying ∆ by h dilates its spectrum, and applying C on the maps the non-negative spectrum to the complex half-circle. The greater h is, the more the spectrum of h∆ is spread apart in R +, ing in better spacing of the smaller eigenvalues of C(h∆). On the other hand, the smaller h is, the further away the high frequencies of h∆ are from ∞, the better spread apart are the high frequencies of C(h∆) in e iR (see FIG1). Tuning the parameter h allows thus to'zoom' in to different parts of the spectrum, ing in filters specialized in different frequency bands. Numerical properties. The numerical core of the Cayley filter is the computation of C j (h∆)f for j = 1,..., r, performed in a sequential manner. Let y 0,..., y r denote the solutions of the following linear recursive system, DISPLAYFORM7 Note that sequentially approximating y j in using the approximation of y j−1 in the rhs is stable, since C(h∆) is unitary and thus has condition number 1.Equations FORMULA16 can be solved with matrix inversion exactly, but it costs O(n 3). An alternative is to use the Jacobi method, 1 which provides approximate solutionsỹ j ≈ y j. Let J = −(Diag(h∆ + iI)) −1 Off(h∆ + iI) be the Jacobi iteration matrix associated with equation. For the unnormalized Laplacian, J = (hD + iI) −1 hW. Jacobi iterations for approximating for a given j have the form DISPLAYFORM8 initialized withỹ DISPLAYFORM9. The application of the approximate Cayley filter is given by Gf = r j=0 c jỹj ≈ Gf, and takes O(rKn) operations under the previous assumption of a sparse Laplacian. The method can be improved by normalizing ỹ j 2 = f 2.Next, we give an error bound for the approximate filter. For the unnormalized Laplacian, let DISPLAYFORM10 < 1. For the normalized Laplacian, we assume that (h∆ n + iI) is dominant diagonal, which gives κ = J ∞ < 1. Proposition 1. Under the above assumptions, Proposition 1 is pessimistic in the general case, while requires strong assumptions in the regular case. We find that in most real life situations the behavior is closer to the regular case. It also follows from Proposition 1 that smaller values of the spectral zoom h in faster convergence, giving this parameter an additional numerical role of accelerating convergence. DISPLAYFORM11 Complexity. In practice, an accurate inversion of (h∆ + iI) is not required, since the approximate inverse is combined with learned coefficients, which "compensate", as necessary, for the inversion inaccuracy. In a CayleyNet for a fixed graph, we fix the number of Jacobi iterations. Since the convergence rate depends on κ, that depends on the graph, different graphs may need different numbers of iterations. The convergence rate also depends on h. Since there is a trade-off between the spectral zoom amount h, and the accuracy of the approximate inversion, and since h is a learnable parameter, the training finds the right balance between the spectral zoom amount and the inversion accuracy. We study the computational complexity of our method, as the number of edges n of the graph tends to infinity. For every constant of a graph, e.g d, κ, we add the subscript n, indicating the number of edges of the graph. For the unnormalized Laplacian, we assume that d n and h n are bounded, which gives κ n < a < 1 for some a independent of n. For the normalized Laplacian, we assume that κ n < a < 1. By Theorem 1, fixing the number of Jacobi iterations K and the order of the filter r, independently of n, keeps the Jacobi error controlled. As a , the number of parameters is O, and for a Laplacian modeled as a sparse matrix, applying a Cayley filter on a signal takes O(n) operations. Localization. Unlike Chebyshev filters that have the small r-hop support, Cayley filters are rational functions supported on the whole graph. However, it is still true that Cayley filters are well localized on the graph. Let G be a Cayley filter and δ m denote a delta-function on the graph, defined as one at vertex m and zero elsewhere. We show that Gδ m decays fast, in the following sense: Definition 2 (Exponential decay on graphs). Let f be a signal on the vertices of graph G, 1 ≤ p ≤ ∞, and 0 < < 1. Denote by S ⊆ {1, . . ., n} a subset of the vertices and by S c its complement. We say that the L p -mass of f is supported in S up to if f | S c p ≤ f p, where f | S c = (f l) l∈S c is the restriction of f to S c. We say that f has (graph) exponential decay about vertex m, if there exists some γ ∈ and c > 0 such that for any k, the L p -mass of f is supported in N k,m up to cγ k. Here, N k,m is the k-hop neighborhood of m. 1 We remind that the Jacobi method for solving Ax = b consists in decomposing A = Diag(A) + Off(A) and obtaining the solution iteratively as Remark 3. Note that Definition 2 is analogous to classical exponential decay on Euclidean space: DISPLAYFORM12 DISPLAYFORM13 Theorem 4. Let G be a Cayley filter of order r. Then, Gδ m has exponential decay about m in L 2, with constants c = 2M1 Gδm 2 and γ = κ 1/r (where M and κ are from Proposition 1). Chebyshev as a special case of Cayley. For a regular graph with D = dI, using Jacobi inversion based on zero iterations, we get that any Cayley filter of order r is a polynomial of ∆ in the monomial base h∆−i hd+i j. In this situation, a Chebyshev filter, which is a real valued polynomial of ∆, is a special case of a Cayley filter. Spectral zoom and stability. Generally, both Chebyshev polynomials and trigonometric polynomials give stable approximations, optimal for smooth functions. However, this crude statement is oversimplified. One of the drawbacks in Chebyshev filters is the fact that the spectrum of ∆ is always mapped to [−1, 1] in a linear manner, making it hard to specialize in small frequency bands. In Cayley filters, this problem is mitigated with the help of the spectral zoom parameter h. As an example, consider the community detection problem discussed in the next section. A graph with strong communities has a cluster of small eigenvalues near zero. Ideal filters g(∆) for extracting the community information should be able to focus on this band of frequencies. Approximating such filters with Cayley polynomials, we zoom in to the band of interest by choosing the right h, and then project g onto the space of trigonometric polynomials of order r, getting a good and stable approximation (FIG4, bottom right). However, if we project g onto the space of Chebyshev polynomials of order r, the interesting part of g concentrated on a small band is smoothed out and lost (FIG4, middle right). Thus, projections are not the right way to approximate such filters, and the stability of orthogonal polynomials cannot be invoked. When approximating g on the small band using polynomials, the approximation will be unstable away from this band; small perturbations in g will in big perturbations in the Chebyshev filter away from the band. For this reason, we say that Cayley filters are more stable than Chebyshev filters. Regularity. We found that in practice, low-order Cayley filters are able to model both very concentrated impulse-like filters, and wider Gabor-like filters. Cayley filters are able to achieve a wider range of filter supports with less coefficients than Chebyshev filters FIG3 ), making the Cayley class more regular than Chebyshev. Complexity. Under the assumption of sparse Laplacians, both Cayley and Chebyshev filters incur linear complexity O(n). Besides, the new filters are equally simple to implement as Chebyshev filters; as seen in Eq.7, they boil down to simple sparse matrix-vector multiplications providing a GPU friendly implementation. Experimental settings. We test the proposed CayleyNets reproducing the experiments of Defferrard et al. FORMULA7; BID17; BID24 a) and using ChebNet BID8 ) as our main baseline method. All the methods were implemented in TensorFlow of M. BID21. The experiments were executed on a machine with a 3.5GHz Intel Core i7 CPU, 64GB of RAM, and NVIDIA Titan X GPU with 12GB of RAM. SGD+Momentum and Adam BID16 ) optimization methods were used to train the models in MNIST and the rest of the experiments, respectively. Training and testing were always done on disjoint sets. Community detection. We start with an experiment on a synthetic graph consisting of 15 communities with strong connectivity within each community and sparse connectivity across communities FIG4. Though rather simple, such a dataset allows to study the behavior of different algorithms in controlled settings. On this graph, we generate noisy step signals, defined as f i = 1 + σ i if i belongs to the community, and f i = σ i otherwise, where DISPLAYFORM0 The goal is to classify each such signal according to the community it belongs to. The neural network architecture used for this task consisted of a spectral convolutional layer (based on Chebyshev or Cayley filters) with 32 output features, a mean pooling layer, and a softmax classifier for producing the final classification into one of the 15 classes. The classification accuracy is shown in FIG4 (right, top) along with examples of learned filters (right, bottom). We observe that CayleyNet significantly outperforms ChebNet for smaller filter orders, with an improvement as large as 80%.Studying the filter responses, we note that due to the capability to learn the spectral zoom parameter, CayleyNet allows to generate band-pass filters in the low-frequency band that discriminate well the communities FIG4. Complexity. We experimentally validated the computational complexity of our model applying filters of different order r to synthetic 15-community graphs of different size n using exact matrix inversion and approximation with different number of Jacobi iterations FIG5 center and right, Figure 6 in the appendix). All times have been computed running 30 times the considered models and averaging the final . As expected, approximate inversion guarantees O(n) complexity. We further conclude that typically very few Jacobi iterations are required FIG5, left shows that our model with just one Jacobi iteration outperforms ChebNet for low-order filters on the community detection problem).MNIST. Following BID8; BID24, for a toy example, we approached the classical MNIST digits classification as a learning problem on graphs. Each pixel of an image is a vertex of a graph (regular grid with 8-neighbor connectivity), and pixel color is a signal on the graph. We used a graph CNN architecture with two spectral convolutional layers based on Chebyshev and Cayley filters (producing 32 and 64 output features, respectively), interleaved with pooling layers performing 4-times graph coarsening using the Graclus algorithm (Dhillon et al. FORMULA7 finally a fully-connected layer (this architecture replicates the classical LeNet5, BID19, architecture, which is shown for comparison). MNIST classification are reported in TAB0. CayleyNet (11 Jacobi iterations) achieves the same (near perfect) accuracy as ChebNet with filters of lower order (r = 12 vs 25).Examples of filters learned by ChebNet and CayleyNet are shown in FIG3. 0.1776 +/-0.06079 sec and 0.0268 +/-0.00841 sec are respectively required by CayleyNet and ChebNet for analyzing a batch of 100 images at test time. FORMULA7 ) 86.64 % 47K ChebNet BID8 ) 87.07 % 46K CayleyNet 88.09 % 46KCitation network. Next, we address the problem of vertex classification on graphs using the popular CORA citation graph, BID28. Each of the 2708 vertices of the CORA graph represents a scientific paper, and an undirected unweighted edge represents a citation (5429 edges in total). For each vertex, a 1433-dimensional binary feature vector representing the content of the paper is given. The task is to classify each vertex into one of the 7 groundtruth classes. We split the graph into training (1,708 vertices), validation (500 vertices) and test (500 vertices) sets, for simulating the labeled and unlabeled information. We train ChebNet and CayleyNet with the architecture presented in BID17; BID24 (two spectral convolutional layers with 16 and 7 outputs), DCNN BID0 ) with 2 diffusion layer (10 hidden features and 2 diffusion hops) and GCN BID17 ) with 3 convolutional layer (32 and 16 hidden features). Figure 5 compares ChebNets and CayleyNets, in a number of different settings. Since ChebNets require Laplacians with spectra bounded in [−1, 1], we consider both the normalized Laplacian (the two left figures), and the scaled unnormalized Laplacian (2∆/λ max − I), where ∆ is the unnormalized Laplacian and λ max is its largest eigenvalue (the two right figures). For fair comparison, we fix the order of the filters (top figures), and fix the overall number of network parameters (bottom figures). In the bottom figure, the Cayley filters are restricted to even cosine polynomials by considering only real filter coefficients. TAB1 shows a comparison of the performance obtained with different methods (all architectures roughly present the same amount of parameters). The best CayleyNets consistently outperform the best competitors. Recommender system. In our final experiment, we applied CayleyNet to recommendation system, formulated as matrix completion problem on user and item graphs, BID24. The task is, given a sparsely sampled matrix of scores assigned by users (columns) to items (rows), to fill in the missing scores. The similarities between users and items are given in the form of column and row graphs, respectively. BID24 approached this problem as learning with a Recurrent Graph CNN (RGCNN) architecture, using an extension of ChebNets to matrices defined on multiple graphs in order to extract spatial features from the score matrix; these features are then fed into an Figure 5: ChebNet (blue) and CayleyNet (orange) test accuracies obtained on the CORA dataset for different polynomial orders. Polynomials with complex coefficients (top) and real coefficients (bottom) have been exploited with CayleyNet in the two analysis. Orders 1 to 6 have been used in both comparisons. The best CayleyNet consistently outperform the best ChebNet requiring at the same time less parameters (CayleyNet with order r and complex coefficients requires a number of parameters equal to ChebNet with order 2r).RNN producing a sequential estimation of the missing scores. Here, we repeated verbatim their experiment on the MovieLens dataset BID23 ), replacing Chebyshev filters with Cayley filters. We used separable RGCNN architecture with two CayleyNets of order r = 4 employing 15 Jacobi iterations. The are reported in TAB3. To present a complete comparison we further extended the experiments reported in BID24 by training sRGCNN with ChebNets of order 8, this provides an architecture with same number of parameters as the exploited CayleyNet (23K coefficients). Our version of sRGCNN outperforms all the competing methods, including the previous with Chebyshev filters reported in BID24. sRGCNNs with Chebyshev polynomials of order 4 and 8 respectively require 0.0698 +/-0.00275 sec and 0.0877 +/-0.00362 sec at test time, sRGCNN with Cayley polynomials of order 4 and 15 jacobi iterations requires 0.165 +/-0.00332 sec. BID14; BID31 ) 1.653 GMC BID15 0.996 GRALS BID26 ) 0.945 sRGCNN Cheby,r=4 BID24 0.929 sRGCNN Cheby,r=8 BID24 0.925 sRGCNN Cayley 0.922 In this paper, we introduced a new efficient spectral graph CNN architecture that scales linearly with the dimension of the input data. Our architecture is based on a new class of complex rational Cayley filters that are localized in space, can represent any smooth spectral transfer function, and are highly regular. The key property of our model is its ability to specialize in narrow frequency bands with a small number of filter parameters, while still preserving locality in the spatial domain. We validated these theoretical properties experimentally, demonstrating the superior performance of our model in a broad range of graph learning problems. First note the following classical for the approximation of Ax = b using the Jacobi method: if the initial condition is DISPLAYFORM0 In our case, note that if we start with initial conditionỹ j = 0, the next iteration givesỹ j = b j, which is the initial condition from our construction. Therefore, since we are approximating y j = C(h∆)ỹ j−1 byỹ j =ỹ DISPLAYFORM1 Define the approximation error in C(h∆) j f by DISPLAYFORM2 By the triangle inequality, by the fact that C j (h∆) is unitary, and by DISPLAYFORM3 where the last inequality is due to DISPLAYFORM4 Now, using standard norm bounds, in the general case we have DISPLAYFORM5 The solution of this recurrent sequence is DISPLAYFORM6 If we use the version of the algorithm, in which eachỹ j is normalized, we get by e j ≤ e j−1 + √ nκ K+1. The solution of this recurrent sequence is DISPLAYFORM7 We denote in this case M j = j √ nIn case the graph is regular, we have D = dI. In the non-normalized Laplacian case, DISPLAYFORM8 The spectral radius of ∆ is bounded by 2d. This can be shown as follows. a value λ is not an eigenvalue of ∆ (namely it is a regular value) if and only if (∆ − λI) is invertible. Moreover, the matrix (∆ − λI) is strictly dominant diagonal for any |λ| > 2d. By Levy-Desplanques theorem, any strictly dominant diagonal matrix is invertible, which means that all of the eigenvalues of ∆ are less than 2d in their absolute value. As a , the spectral radius of (dI − ∆) is realized on the smallest eigenvalue of ∆, namely it is |d − 0| = d. This means that the specral radius of J is DISPLAYFORM9. As a DISPLAYFORM10 = κ. We can now continue from to get e j ≤ e j−1 + J K+1 2(1 + e j−1) = e j−1 + κ K+1 (1 + e j−1).As before, we get e j ≤ jκ K+1 + O(κ 2K+2), and e j ≤ jκ K+1 if eachỹ j is normalized. We denote in this case M j = j. In the case of the normalized Laplacian of a regular graph, the spectral radius of ∆ n is bounded by 2, and the diagonal entries are all 1. Equation in this case reads J = h h+i (I − ∆ n), and J has spectral radius h √ h 2 +1. Thus J 2 = h √ h 2 +1= κ and we continue as before to get e j ≤ jκ K+1 and M j = j. In all cases, we have by the triangle inequality Figure 6: Test (above) and training (below) times with corresponding ratios as function of filter order r and graph size n on our community detection dataset.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A spectral graph convolutional neural network with spectral zoom properties.

Represent the TLDR for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the scientific article to find a short TLDR

Question: We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model’s accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy. Semantic segmentation predicts pixel-level annotations of different semantic categories for an image. Despite its performance breakthrough thanks to the prosperity of convolutional neural networks (CNNs) , as a dense structured prediction task, segmentation models commonly suffer from heavy memory costs and latency, often due to stacking convolutions and aggregating multiple-scale features, as well as the increasing input image resolutions. However, recent years witness the fast-growing demand for real-time usage of semantic segmentation, e.g., autonomous driving. Such has motivated the enthusiasm on designing low-latency, more efficient segmentation networks, without sacrificing accuracy notably (; a). The recent success of neural architecture search (NAS) algorithms has shed light on the new horizon in designing better semantic segmentation models, especially under latency of other resource constraints. Auto-DeepLab (a) first introduced network-level search space to optimize resolutions (in addition to cell structure) for segmentation tasks. and adopted pre-defined network-level patterns of spatial resolution, and searched for operators and decoders with latency constraint. Despite a handful of preliminary successes, we observe that the successful human domain expertise in designing segmentation models appears to be not fully integrated into NAS frameworks yet. For example, human-designed architectures for real-time segmentation (; a) commonly exploit multi-resolution branches with proper depth, width, operators, and downsample rates, and find them contributing vitally to the success: such flexibility has not been unleashed by existing NAS segmentation efforts. Furthermore, the trade-off between two (somewhat conflicting) goals, i.e., high accuracy and low latency, also makes the search process unstable and prone to "bad local minima" architecture options. As the well-said quote goes: "those who do not learn history are doomed to repeat it". Inheriting and inspired by the successful practice in hand-crafted efficient segmentation, we propose a novel NAS framework dubbed FasterSeg, aiming to achieve extremely fast inference speed and competitive accuracy. We designed a special search space capable of supporting optimization over multiple branches of different resolutions, instead of a single backbone. These searched branches are adaptively aggregated for the final prediction. To further balance between accuracy versus latency and avoiding collapsing towards either metric (e.g., good latency yet poor accuracy), we design a decoupled and fine-grained latency regularization, that facilitates a more flexible and effective calibration between latency and accuracy. Moreover, our NAS framework can be easily extended to a collaborative search (co-searching), i.e., jointly searching for a complex teacher network and a light-weight student network in a single run, whereas the two models are coupled by feature distillation in order to boost the student's accuracy. We summarize our main contributions as follows: • A novel NAS search space tailored for real-time segmentation, where multi-resolution branches can be flexibility searched and aggregated. • A novel decoupled and fine-grained latency regularization, that successfully alleviates the "architecture collapse" problem in the latency-constrained search. • A novel extension to teacher-student co-searching for the first time, where we distill the teacher to the student for further accuracy boost of the latter. • Extensive experiments demonstrating that FasterSeg achieves extremely fast speed (over 30% faster than the closest manually designed competitor on CityScapes) and maintains competitive accuracy. Human-designed CNN architectures achieve good accuracy performance nowadays (; . However, designing architectures to balance between accuracy and other resource constraints (latency, memory, FLOPs, etc.) requires more human efforts. To free human experts from this challenging trade-off, neural architecture search (NAS) has been recently introduced and drawn a booming interest (; ; ; a; a; ; c; . These works optimize both accuracy and resource utilization, via a combined loss function, or a hybrid reward signal for policy learning), or a constrained optimization formulation. Most existing resource-aware NAS efforts focus on classification tasks, while semantic segmentation has higher requirements for preserving details and rich contexts, therefore posing more dilemmas for efficient network design. Fortunately, previous handcrafted architectures for real-time segmentation have identified several consistent and successful design patterns. ENet adopted early downsampling, and ICNet further incorporated feature maps from multiresolution branches under label guidance. BiSeNet (a) fused a context path with fast downsampling and a spatial path with smaller filter strides. More works target on segmentation efficiency in terms of computation cost ) and memory usage. Their multi-resolution branching and aggregation designs ensure sufficiently large receptive fields (contexts) while preserving high-resolution fine details, providing important clues on how to further optimize the architecture. There have been recent studies that start pointing NAS algorithms to segmentation tasks. AutoDeepLab (a) pioneered in this direction by searching the cells and the networklevel downsample rates, to flexibly control the spatial resolution changes throughout the network. and introduced resource constraints into NAS segmentation. A multi-scale decoder was also automatically searched. However, compared with manually designed architectures, those search models still follow a single-backbone design and did not fully utilize the prior wisdom (e.g., multi-resolution branches) in designing their search spaces. Lastly, we briefly review knowledge distillation , that aims to transfer learned knowledge from a sophisticated teacher network to a light-weight student, to improve the (more efficient) student's accuracy. For segmentation, Liu et al. (2019b) and proposed to leverage knowledge distillation to improve the accuracy of the compact model and speed-up convergence. There was no prior work in linking distillation with NAS yet, and we will introduce the extension of FasterSeg by integrating teacher-student model collaborative search for the first time. 3 FASTERSEG: FASTER REAL-TIME SEGMENTATION Our FasterSeg is discovered from an efficient and multi-resolution search space inspired by previous manual design successes. A fine-grained latency regularization is proposed to overcome the challenge of "architecture collapse" . We then extend our FasterSeg to a teacherstudent co-searching framework, further ing in a lighter yet more accurate student network. The core motivation behind our search space is to search multi-resolution branches with overall low latency, which has shown effective in previous manual design works (; a). Our NAS framework automatically selects and aggregates branches of different resolutions, based on efficient cells with searchable superkernels. Figure 1: The multi-resolution branching search space for FasterSeg, where we aim to optimize multiple branches with different output resolutions. These outputs are progressively aggregated together in the head module. Each cell is individually searchable and may have two inputs and two outputs, both of different downsampling rates (s). Inside each cell, we enable searching for expansion ratios within a single superkernel. . Bottom: BiSeNet (a) Inspired by (a), we enable searching for spatial resolutions within the L-layer cells (Figure 1), where each cell takes inputs from two connected predecessors and outputs two feature maps of different resolutions. Hand-crafted networks for real-time segmentation found multi-branches of different resolutions to be effective (; a). However, architectures explored by current NAS algorithms are restricted to a single backbone. Our goal is to select b (b > 1) branches of different resolutions in this L-layer framework. Specifically, we could choose b different final output resolutions for the last layer of cells, and decode each branch via backtrace (section 3.4). This enables our NAS framework to explore b individual branches with different resolutions, which are progressively "learned to be aggregated" by the head module (Figure 1). We follow the convention to increase the number of channels at each time of resolution downsampling. To enlarge the model capacity without incurring much latency, we first downsample the input image to 1 8 original scale with our stem module, and then set our searchable downsample rates s ∈ {8, 16, 32}. Figure 2 shows that our multi-resolution search space is able to cover existing human-designed networks for real-time segmentation. See Appendix B for branch selection details. As we aim to boost the inference latency, the speed of executing an operator is a direct metric (rather than indirect metrics like FLOPs) for selecting operator candidates O. Meanwhile, as we previously discussed, it is also important to ensure sufficiently large receptive field for spatial contexts. We analyze typical operators, including their common surrogate latency measures (FLOPs, parameter numbers), and their real-measured latency on an NVIDIA 1080Ti GPU with TensorRT library, and their receptive fields, as summarized in Table 1. Compared with standard convolution, group convolution is often used for reducing FLOPs and number of parameters . Convolving with two groups has the same receptive field with a standard convolution but is 13% faster, while halving the parameter amount (which might not be preferable as it reduces the model learning capacity). Dilated convolution has an enlarged receptive field and is popular in dense predictions ). However, as shown in Table 1 (and as widely acknowledged in engineering practice), dilated convolution (with dilation rate 2) suffers from dramatically higher latency, although that was not directly reflected in FLOPs nor parameter numbers. In view of that, we design a new variant called "zoomed convolution", where the input feature map is sequentially processed with bilinear downsampling, standard convolution, and bilinear upsampling. This special design enjoys 40% lower latency and 2 times larger receptive field compared to standard convolution. Our search space hence consists of the following operators: • skip connection • 3×3 conv. • 3×3 conv. ×2 • "zoomed conv.": bilinear downsampling + 3×3 conv. + bilinear upsampling • "zoomed conv. ×2": bilinear downsampling + 3×3 conv. ×2 + bilinear upsampling As mentioned by , network fragmentation can significantly hamper the degree of parallelism, and therefore practical efficiency. Therefore, we choose a sequential search space (rather than a directed acyclic graph of nodes), i.e., convolutional layers are sequentially stacked in our network. In Figure 1, each cell is differentiable 2019a) and will contain only one operator, once the discrete architecture is derived (section 3.4). It is worth noting that we allow each cell to be individually searchable across the whole search space. We further give each cell the flexibility to choose different channel expansion ratios. In our work, we search for the width of the connection between successive cells. That is however non-trivial due to the exponentially possible combinations of operators and widths. To tackle this problem, we propose a differentiably searchable superkernel, i.e., directly searching for the expansion ratio χ within a single convolutional kernel which supports a set of ratios X ⊆ N +. Inspired by (c) and , from slim to wide our connections incrementally take larger subsets of input/output dimensions from the superkernel. During the architecture search, for each superkernel, only one expansion ratio is sampled, activated, and back-propagated in each step of stochastic gradient descent. This design contributes to a simplified and memory-efficient super network and is implemented via the renowned "Gumbel-Softmax" trick (see Appendix C for details). To follow the convention to increase the number of channels as resolution downsampling, in our search space we consider the width = χ × s, where s ∈ {8, 16, 32}. We allow connections between each pair of successive cells flexibly choose its own expansion ratio, instead of using a unified single expansion ratio across the whole search space. Denote the downsample rate as s and layer index as l. To facilitate the search of spatial resolutions, we connect each cell with two possible predecessors' outputs with different downsample rates: Each cell could have at most two outputs with different downsample rates into its successors: The expansion ratio χ j s,l is sampled via "Gumbel-Softmax" trick according to p(χ = χ j s,l) = γ j s,l. Here, α, β, and γ are all normalized scalars, associated with each operator O k ∈ O, each predecessor's output O l−1, and each expansion ratio χ ∈ X, respectively (Appendix D). They encode the architectures to be optimized and derived. Low latency is desirable yet challenging to optimize. Previous works (; observed that during the search procedure, the supernet or search policy often fall into bad "local minimums" where the generated architectures are of extremely low latency but with poor accuracy, especially in the early stage of exploration. In addition, the searched networked tend to use more skip connections instead of choosing low expansion ratios . This problem is termed as "architecture collapse" in our paper. The potential reason is that, finding architectures with extremely low latency (e.g. trivially selecting the most light-weight operators) is significantly easier than discovering meaningful compact architectures of high accuracy. To address this "architecture collapse" problem, we for the first time propose to leverage a fine-grained, decoupled latency regularization. We first achieve the continuous relaxation of latency similar to the cell operations in section 3.1.4, via replacing the operator O in Eqn. 1 and 2 with the corresponding latency. We build a latency lookup table that covers all possible operators to support the estimation of the relaxed latency. Figure 3 demonstrates the high correlation of 0.993 between the real and estimated latencies (see details in appendix E). We argue that the core reason behind the "architecture collapse" problem is the different sensitivities of supernet to operator O, downsample rate s, and expansion ratio χ. Operators like "3×3 conv. ×2" and "zoomed conv." have a huge gap in latency. Similar latency gap (though more moderate) exists between slim and wide expansion ratios. However, downsample rates like "8" and "32" do not differ much, since resolution downsampling also brings doubling of the number of both input and output channels. We quantitatively compared the influence of O, s, and χ towards the supernet latency, by adjusting one of the three aspects and fixing the other two. Taking O as the example, we first uniformly initialize β and γ, and calculate ∆Latency(O) as the gap between the supernet which dominantly takes the slowest operators and the one adopts the fastest. Similar calculations were performed for s and χ. Values of ∆Latency in Table 2 indicate the high sensitivity of the supernet's latency to operators and expansion ratios, while not to resolutions. Figure 4 (a) shows that the unregularized latency optimization will bias the supernet towards light-weight operators and slim expansion ratios to quickly minimize the latency, ending up with problematic architectures with low accuracy. Based on this observation, we propose a regularized latency optimization leveraging different granularities of our search space. We decouple the calculation of supernet's latency into three granularities of our search space (O, s, χ), and regularize each aspect with a different factor: where we by default set w 1 = 0.001, w 2 = 0.997, w 3 = 0.002 1. This decoupled and fine-grained regularization successfully addresses this "architecture collapse" problem, as shown in Figure 4 Knowledge Distillation is an effective approach to transfer the knowledge learned by a large and complex network (teacher T) into a much smaller network (student S). In our NAS framework, we can seamlessly extend to teacher-student cosearching, i.e., collaboratively searching for two networks in a single run (Figure 5). Specifically, we search a complex teacher and light-weight student simultaneously via adopting two sets of architectures in one supernet: (α T, β T) and (α S, β S, γ S). Note that the teacher does not search the expansion ratios and always select the widest one. This extension does not bring any overhead in memory usage or size of supernet since the teacher and student share the same supernet weights W during the search process. Two sets of architectures are iteratively optimized during search (please see details in Appendix F), and we apply the latency constraint only on the student, not on the teacher. Therefore, our searched teacher is a sophisticated network based on the same search space and supernet weights W used by the student. During training from scratch, we apply a distillation loss from teacher T to student S: ) KL denotes the KL divergence. q s i and q t i are predicted logit for pixel i from S and T, respectively. Equal weights (1.0) are assigned to the segmentation loss and this distillation loss. Once the search is completed, we derive our discrete architecture from α, β, and γ: • α, γ: We select the optimum operators and expansion ratios by taking the argmax of α and γ. We shrink the operator "skip connection" to obtain a shallower architecture with less cells. • β: Different from (a), for each cell s,l we consider β 0 and β 1 as probabilities of two outputs from cell s 2,l−1 and cell s,l−1 into cell s,l. Therefore, by taking the l * = argmax l (β 0 s,l), we find the optimum position (cell s,l *) where to downsample the current resolution (It is worth noting that, the multi-resolution branches will share both cell weights and feature maps if their cells are of the same operator type, spatial resolution, and expansion ratio. This design contributes to a faster network. Once cells in branches diverge, the sharing between the branches will be stopped and they become individual branches (See Figure 6). 1 These values are obtained by solving equations derived from Table 2 in order to achieve balanced sensitivities on different granularities: 10.42 × w1 = 0.01 × w2 = 5.54 × w1, s.t. w1 + w2 + w3 = 1. 2 For a branch with two searchable downsampling positions, we consider the argmax over the joint proba- We use the Cityscapes as a testbed for both our architecture search and ablation studies. After that, we report our final accuracy and latency on Cityscapes, CamVid , and BDD (b). In all experiments, the class mIoU (mean Intersection over Union per class) and FPS (frame per second) are used as the metrics for accuracy and speed, respectively. Please see Appendix G for dataset details. In all experiments, we use Nvidia Geforce GTX 1080Ti for benchmarking the computing power. We employ the high-performance inference framework TensorRT v5.1.5 and report the inference speed. During this inference measurement, an image of a batch size of 1 is first loaded into the graphics memory, then the model is warmed up to reach a steady speed, and finally, the inference time is measured by running the model for six seconds. All experiments are performed under CUDA 10.0 and CUDNN V7. Our framework is implemented with PyTorch. The search, training, and latency measurement codes are available at https://github.com/TAMU-VITA/FasterSeg. We consider a total of L = 16 layers in the supernet and our downsample rate s ∈ {8, 16, 32}. In our work we use number of branches b = 2 by default, since more branches will suffer from high latency. We consider expansion ratio χ s,l ∈ X = {4, 6, 8, 10, 12} for any "downsample rate" s and layer l. The multi-resolution branches have 1695 unique paths. For cells and expansion ratios, we have (1 + 4 × 5) (15+14+13) + 5 3 ≈ 3.4 × 10 55 unique combinations. This in a search space in the order of 10 58, which is much larger and challenging, compared with preliminary studies. Architecture search is conducted on Cityscapes training dataset. Figure 6 visualizes the best spatial resolution discovered (FasterSeg). Our FasterSeg achieved mutli-resolutions with proper depths. The two branches share the first three operators then diverge, and choose to aggregate outputs with downsample rates of 16 and 32. Operators and expansion ratios are listed in Table 7 in Appendix I, where the zoomed convolution is heavily used, suggesting the importance of low latency and large receptive field. We conduct ablation studies on Cityscapes to evaluate the effectiveness of our NAS framework. More specifically, we examine the impact of operators (O), downsample rate (s), expansion ratios (χ), and also distillation on the accuracy and latency. When we expand from a single backbone (b = 1) to multi-branches (b = 2), our FPS drops but we gain a much improvement on mIoU, indicating the multiresolution design is beneficial for segmentation task. By enabling the search for expansion ratios (χ), we discover a faster network with FPS 163.9 without sacrificing accuracy (70.5%), which proves that the searchable superkernel gets the benefit from eliminating redundant channels while maintaining high accuracy. This is our student network (S) discovered in our co-searching framework (see below). We further evaluate the efficacy of our teacher-student co-searching framework. After the collaboratively searching, we obtain a teacher architecture (T) and a student architecture (S). As mentioned above, S is searched with searchable expansion ratios (χ), achieving an FPS of 163.9 and an mIoU of 70.5%. In contrast, when we directly compress the teacher (channel pruning via selecting the slimmest expansion ratio) and train with distillation from the well-trained original cumbersome teacher, it only achieved mIoU = 66.1% with only FPS = 146.7, indicating that our architecture cosearching surpass the pruning based compression. Finally, when we adopt the knowledge distillation from the well-trained cumbersome teacher to our searched student, we boost the student's accuracy to 73.1%, which is our final network FasterSeg. This demonstrates that both a student discovered by co-searching and training with knowledge distillation from the teacher are vital for obtaining an accurate faster real-time segmentation model. In this section, we compare our FasterSeg with other works for real-time semantic segmentation on three popular scene segmentation datasets. Note that since we target on real-time segmentation, we measure the mIoU without any evaluation tricks like flipping, multi-scale, etc. 67.7 69.5 37.7 1024×2048 BiSeNet (a) 69.0 68.4 105.8 768×1536 CAS 71.6 70.5 108.0 768×1536 Fast-SCNN 68.6 68.0 123.5 1024×2048 DF1-Seg-d8 (73.1 71.5 163.9 1024×2048 Cityscapes: We evaluate FasterSeg on Cityscapes validation and test sets. We use original image resolution of 1024×2048 to measure both mIoU and speed inference. In Table 4, we see the superior FPS (163.9) of our FasterSeg, even under the maximum image resolution. This high FPS is over 1.3× faster than human-designed networks. Meanwhile, our FasterSeg still maintains competitive accuracy, which is 73.1% on the validation set and 71.5% on the test set. This accuracy is achieved with only Cityscapes fine-annotated images, without using any extra data (coarse-annotated images, ImageNet, etc.). We directly transfer the searched architecture on Cityscapes to train on CamVid. Table 5 reveals that without sacrificing much accuracy, our FasterSeg achieved an FPS of 398.1. This extremely high speed is over 47% faster than the closest competitor in FPS (a), and is over two times faster than the work with the best mIoU. This impressive verifies both the high performance of FasterSeg and also the transferability of our NAS framework. BDD: In addition, we also directly transfer the learned architecture to the BDD dataset. In Table 6 we compare our FasterSeg with the baseline provided by Yu et al. (2018b). Since no previous work has considered real-time segmentation on the BDD dataset, we get 15 times faster than the DRN-D-22 with slightly higher mIoU. Our FasterSeg still preserve the extremely fast speed and competitive accuracy on BDD. ENet 68.3 61.2 ICNet 67.1 27.8 BiSeNet (a) 65.6 269.1 CAS 71.2 169.0 FasterSeg (ours) 71.1 398.1 We introduced a novel multi-resolution NAS framework, leveraging successful design patterns in handcrafted networks for real-time segmentation. Our NAS framework can automatically discover FasterSeg, which achieved both extremely fast inference speed and competitive accuracy. Our search space is intrinsically of low-latency and is much larger and challenging due to flexible searchable expansion ratios. More importantly, we successfully addressed the "architecture collapse" problem, by proposing the novel regularized latency optimization of fine-granularity. We also demonstrate that by seamlessly extending to teacher-student co-searching, our NAS framework can boost the student's accuracy via effective distillation. A STEM AND HEAD MODULE Stem: Our stem module aims to quickly downsample the input image to 1 8 resolution while increasing the number of channels. The stem module consists of five 3 × 3 convolution layers, where the first, second, and fourth layer are of stride two and double the number of channels. Head: As shown in Figure 1, feature map of shape (C 2s × H × W) is first reduced in channels by a 1 × 1 convolution layer and bilinearly upsampled to match the shape of the other feature map (C s × 2H × 2W). Then, two feature maps are concatenated and fused together with a 3 × 3 convolution layer. Note that we not necessarily have C 2s = 2C s because of the searchable expansion ratios. Since our searchable downsample rates s ∈ {8, 16, 32} and the number of selected branches b = 2, our supernet needs to select branches of three possible combinations of resolutions: {8, 16}, {8, 32}, and {16, 32}. For each combination, branches of two resolutions will be aggregated by our head module. Our supernet selects the best b branches based on the criterion used in: where m is a searched model aggregating b branches, with accuracy ACC(m) and latency LAT (m). w is the weight factor defined as: We empirically set α = β = −0.07 and the target latency T = 8.3 ms in our work. Formally, suppose we have our set of expansion ratios X ⊆ N +, and we want to sample one ratio χ from X. For each χ i we have an associated probability γ i, where helps us approximate differentiable sampling. We first sample a "Gumbel-Noise" ). We set the temperature parameter τ = 1 in our work. D NORMALIZED SCALARS α, β, γ α, β, and γ are all normalized scalars and implemented as softmax. They act as probabilities associating with each operator O k ∈ O, each predecessor's output O l−1, and each expansion ratio χ ∈ X, respectively: where s is downsample rate and l is index of the layer in our supernet. We build a latency lookup table that covers all possible situations and use this lookup table as building blocks to estimate the relaxed latency. To verify the continuous relaxation of latency, we randomly sample networks of different operators/downsample rates/expansion ratios out of the supernet M, and measured both the real and estimated latency. We estimate the network latency by accumulating all latencies of operators consisted in the network. In Figure 3, we can see the high correlation between the two measurements, with a correlation coefficient of 0.993. This accurate estimation of network latency benefits from the sequential design of our search space. Given our supernet M, the overall optimization target (loss) during architecture search is: We adopt cross-entropy with "online-hard-element-mining" as our segmentation loss L seg. Lat(M) is the continuously relaxed latency of supernet, and λ is the balancing factor. We set λ = 0.01 in our work. As the architecture α, β, and γ are now involved in the differentiable computation graph, they can be optimized using gradient descent. Similar in (a), we adopt the first-order approximation , randomly split our training dataset into two disjoint sets trainA and trainB, and alternates the optimization between: H ARCHITECTURE SEARCH IMPLEMENTATIONS As stated in the second line of Eqn. 2, a stride 2 convolution is used for all s → 2s connections, both to reduce spatial size and double the number of filters. Bilinear upsampling is used for all upsampling operations. We conduct architecture search on the Cityscapes dataset. We use 160 × 320 random image crops from half-resolution (512 × 1024) images in the training set. Note that the original validation set or test set is never used for our architecture search. When learning network weights W, we use SGD optimizer with momentum 0.9 and weight decay of 5×10 −4. We used the exponential learning rate decay of power 0.99. When learning the architecture parameters α, β, andγ, we use Adam optimizer with learning rate 3×10 −4. The entire architecture search optimization takes about 2 days on one 1080Ti GPU. In Table 7 we list the operators (O) and expansion ratios (χ) selected by our FasterSeg. The downsample rates s in Table 7 and Figure 6 match. We have the number of output channels c out = s × χ. We observed that the zoomed convolution is heavily used, suggesting the importance of low latency and large receptive field.

Represent the TLDR for finding the corresponding scientific article

We present a real-time segmentation model automatically discovered by a multi-scale NAS framework, achieving 30% faster than state-of-the-art models.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.