Question: Represent the article for finding a single-sentence TLDR

Question: We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model. This approach is in contrast with most frameworks of conditional GANs used in application today, which use the conditional information by concatenating the (embedded) conditional vector to the feature vectors. With this modification, we were able to significantly improve the quality of the class conditional image generation on ILSVRC2012 (ImageNet) dataset from the current state-of-the-art , and we achieved this with a single pair of a discriminator and a generator. We were also able to extend the application to super-resolution and succeeded in producing highly discriminative super-resolution images. This new structure also enabled high quality category transformation based on parametric functional transformation of conditional batch normalization layers in the generator. Generative Adversarial Networks (GANs) BID5 are a framework to construct a generative model that can mimic the target distribution, and in recent years it has given birth to arrays of state-of-the-art algorithms of generative models on image domain BID23; BID30 BID20. The most distinctive feature of GANs is the discriminator D(x) that evaluates the divergence between the current generative distribution p G (x) and the target distribution q(x) BID5 BID16. The algorithm of GANs trains the generator model by iteratively training the discriminator and generator in turn, with the discriminator acting as an increasingly meticulous critic of the current generator. Conditional GANs (cGANs) are a type of GANs that use conditional information BID13 for the discriminator and generator, and they have been drawing attention as a promising tool for class conditional image generation BID17, the generation of the images from text BID20 BID30, and image to image translation BID10 BID31. Unlike in standard GANs, the discriminator of cGANs discriminates between the generator distribution and the target distribution on the set of the pairs of generated samples x and its intended conditional variable y. To the authors' knowledge, most frameworks of discriminators in cGANs at the time of writing feeds the pair the conditional information y into the discriminator by naively concatenating (embedded) y to the input or to the feature vector at some middle layer BID13 BID2 BID20 BID30 BID18 BID22 BID3 BID24. We would like to however, take into account the structure of the assumed conditional probabilistic models underlined by the structure of the discriminator, which is a function that measures the information theoretic distance between the generative distribution and the target distribution. By construction, any assumption about the form of the distribution would act as a regularization on the choice of the discriminator. In this paper, we propose a specific form of the discriminator, a form motivated by a probabilistic model in which the distribution of the conditional variable y given x is discrete or uni-modal continuous distributions. This model assumption is in fact common in many real world applications, including class-conditional image generation and super-resolution. As we will explain in the next section, adhering to this assumption will give rise to a structure of the discriminator that requires us to take an inner product between the embedded condition vector y and the feature vector (Figure 1d). With this modification, we were able to significantly improve the quality of the class conditional image generation on 1000-class ILSVRC2012 dataset BID21 with a single pair of a discriminator and generator (see the generated examples in Figure 2). Also, when we applied our model of cGANs to a super-resolution task, we were able to produce high quality super-resolution images that are more discriminative in terms of the accuracy of the label classifier than the cGANs based on concatenation, as well as the bilinear and the bicubic method. Let us denote the input vector by x and the conditional information by y 1. We also denote the cGAN discriminator by D(x, y; θ):= A(f (x, y; θ)), where f is a function of x and y, θ is the parameters of f, and A is an activation function of the users' choice. Using q and p to designate the true distributions and the generator model respectively, the standard adversarial loss for the discriminator is given by: DISPLAYFORM0 with A in D representing the sigmoid function. By construction, the nature of the'critic' D significantly affects the performance of G. A conventional way of feeding y to D until now has been to concatenate the vector y to the feature vector x, either at the input layer BID13 BID2 BID22, or at some hidden layer BID20 BID30 BID18 BID3 BID24 ) (see Figure 1a and Figure 1b). We would like to propose an alternative to this approach by observing the form of the optimal solution BID5 for the loss function, Eq., can be decomposed into the sum of two log likelihood ratios: DISPLAYFORM1 Now, we can model the log likelihood ratio r(y|x) and r(x) by some parametric functions f 1 and f 2 respectively. If we make a standing assumption that p(y|x) and q(y|x) are simple distributions like those that are Gaussian or discrete log linear on the feature space, then, as we will show, the parametrization of the following form becomes natural: DISPLAYFORM2 where V is the embedding matrix of y, φ(·, θ Φ) is a vector output function of x, and ψ(·, θ Ψ) is a scalar function of the same φ(x; θ Φ) that appears in f 1 (see Figure 1d). The learned parameters θ = {V, θ Φ, θ Ψ} are to be trained to optimize the adversarial loss. From this point on, we will refer to this model of the discriminator as projection for short. In the next section, we would like to elaborate on how we can arrive at this form. In this section, we will begin from specific, often recurring models and show that, with certain regularity assumption, we can write the optimal solution of the discriminator objective function in the form of. Let us first consider the a case of categorical variable. Assume that y is a categorical variable taking a value in {1, . . ., C}, which is often common for a class conditional image generation task. The most popular model for p(y|x) is the following log linear model: DISPLAYFORM0 where DISPLAYFORM1 is the partition function, and φ: DISPLAYFORM2 is the input to the final layer of the network model. Now, we assume that the target distribution q can also be parametrized in this form, with the same choice of φ. This way, the log likelihood ratio would take the following form; DISPLAYFORM3 If we make the values of (v . Now, if we can put together the normalization constant − (log Z q (φ(x)) − log Z p (φ(x))) and r(x) into one expression ψ(φ(x)), we can rewrite the equation above as DISPLAYFORM4 by using y to denote a one-hot vector of the label y and using V to denote the matrix consisting of the row vectors v c. Most notably, this formulation introduces the label information via an inner product, as opposed to concatenation. The form is indeed the form we proposed in.We can also arrive at the form for unimodal continuous distributions p(y|x) as well. Let y ∈ R d be a d-dimensional continuous variable, and let us assume that conditional q(y|x) and p(y|x) are both given by Gaussian distributions, so that q(y|x) = N (y|µ q (x), Λ −1 q ) and DISPLAYFORM5 Then the log density ratio r(y|x) = log(q(y|x)/p(y|x)) is given by: DISPLAYFORM6 where ψ(φ(x)) represents the terms independent of y. Now, if we assume that Λ q = Λ p:= Λ, we can ignore the quadratic term. If we further express Λ q W q − Λ p W p in the form V, we can arrive at the form again. Indeed, however, the way that this regularization affects the training of the generator G is a little unclear in its formulation. As we have repeatedly explained, our discriminator measures the divergence between the generator distribution p and the target distribution q on the assumption that p(y|x) and q(y|x) are relatively simple, and it is highly possible that we are gaining stability in the training process by imposing a regularity condition on the divergence measure. Meanwhile, however, the actual p(y|x) can only be implicitly derived from p(x, y) in computation, and can possibly take numerous forms other than the ones we have considered here. We must admit that there is a room here for an important theoretical work to be done in order to assess the relationship between the choice of the function space for the discriminator and training process of the generator. As described above, is a form that is true for frequently occurring situations. In contrast, incorporation of the conditional information by concatenation is rather arbitrary and can possibly include into the pool of candidate functions some sets of functions for which it is difficult to find a logical basis. Indeed, if the situation calls for multimodal p(y|x), it might be smart not to use the model that we suggest here. Otherwise, however, we expect our model to perform better; in general, it is preferable to use a discriminator that respects the presumed form of the probabilistic model. Still another way to incorporate the conditional information into the training procedure is to directly manipulate the loss function. The algorithm of AC-GANs BID17 use a discriminator (D 1) that shares a part of its structure with the classifier(D 2), and incorporates the label information into the objective function by augmenting the original discriminator objective with the likelihood score of the classifier on both the generated and training dataset (see Figure 1c). Plug and Play Generative models (PPGNs) BID15 is another approach for the generative model that uses an auxiliary classifier function. It is a method that endeavors to make samples from p(x|y) using an MCMC sampler based on the Langevin equation with drift terms consisting of the gradient of an autoencoder prior p(x) and a pretrained auxiliary classifier p(y|x). With these method, one can generate a high quality image. However, these ways of using auxiliary classifier may unwittingly encourage the generator to produce images that are particularly easy for the auxiliary classifier to classify, and deviate the final p(x|y) from the true q(x|y). In fact, BID17 reports that this problem has a tendency to exacerbate with increasing number of labels. We were able to reproduce this phenomena in our experiments; when we implemented their algorithm on a dataset with 1000 class categories, the final trained model was able to generate only one image for most classes. Nguyen et al.'s PPGNs is also likely to suffer from the same problem because they are using an order of magnitude greater coefficient for the term corresponding to p(y|x) than for the other terms in the Langevin equation. In order to evaluate the effectiveness of our newly proposed architecture for the discriminator, we conducted two sets of experiments: class conditional image generation and super-resolution on ILSVRC2012 (ImageNet) dataset BID21. For both tasks, we used the ResNet BID8 based discriminator and the generator used in BID6, and applied spectral normalization BID14 to the all of the weights of the discriminator to regularize the Lipschitz constant. For the objective function, we used the following hinge version of the standard adversarial loss BID12 BID28 DISPLAYFORM0 where the last activation function A of D is identity function. p(z) is standard Gaussian distribution and G(z, y) is the generator network. For all experiments, we used Adam optimizer with hyper-parameters set to α = 0.0002, β 1 = 0, β 2 = 0.9. We updated the discriminator five times per each update of the generator. We will use concat to designate the models (Figure 1b) 2, and use projection to designate the proposed model (Figure 1d). The ImageNet dataset used in the experiment of class conditional image generation consisted of 1,000 image classes of approximately 1,300 pictures each. We compressed each images to 128×128 pixels. Unlike for AC-GANs 3 we used a single pair of a ResNet-based generator and a discriminator. Also, we used conditional batch normalization BID4 BID1 for the generator. As for the architecture of the generator network used in the experiment, please see FIG2 for more detail. Our proposed projection model discriminator is equipped with a'projection layer' that takes inner product between the embedded one-hot vector y and the intermediate output (FIG2). As for the structure of the the concat model discriminator to be compared against, we used the identical bulk architecture as the projection model discriminator, except that we removed the projection layer from the structure and concatenated the spatially replicated embedded conditional vector y to the output of third ResBlock. We also experimented with AC-GANs as the current state of the art model. For AC-GANs, we placed the softmax layer classifier to the same structure shared by concat and projection. For each method, we updated the generator 450K times, and applied linear decay for the learning rate after 400K iterations so that the rate would be 0 at the end. For the comparative experiments, we trained the model for 450K iterations, which was ample for the training of concat to stabilize. AC-GANs collapsed prematurely before the completion of 450K iterations, so we reported the from the peak of its performance (80K iterations). For all experiments throughout, we used the training over 450K iterations for comparing the performances. On a separate note, our method continued to improve even after 450K. We therefore also reported the inception score and FID of the extended training (850K iterations) for our method exclusively. See the table 1 for the exact figures. We used inception score BID23 for the evaluation of the visual appearance of the generated images. It is in general difficult to evaluate how'good' the generative model is. Indeed, however, either subjective or objective, some definite measures of'goodness' exists, and essential two of them are'diversity' and the sheer visual quality of the images. One possible candidate for quantitative measure of diversity and visual appearance is FID BID9. We computed FID between the generated images and dataset images within each class, and designated the values as intra FIDs. More precisely, FID BID9 measures the 2-Wasserstein distance between the two distributions q y and p y, and is given by F (q y, p y) = µ qy − µ py 2 2 + trace C qy + C py − 2(C qy C py) 1/2, where {µ qy, C qy}, {µ py, C py} are respectively the mean and the covariance of the final feature vectors produced by the inception model BID25 from the true samples and generated samples of class y. When the set of generated examples have collapsed modes, the trace of C py becomes small and the trace term itself becomes large. In order to compute C qy we used all samples in the training data belonging to the class of concern, and used 5000 generated samples for the computation of C py. We empirically observed in our experiments that intra FID is, to a certain extent, serving its purpose well in measuring the diversity and the visual quality. To highlight the effectiveness of our inner-product based approach (projection) of introducing the conditional information into the model, we compared our method against the state of the art ACGANs as well as the conventional incorporation of the conditional information via concatenation 2 in the preliminary experiments of the image geneation task on CIFAR-10 BID27 and CIFAR-100 BID27, we confirmed that hidden concatenation is better than input concatenation in terms of the inception scores. For more details, please see TAB1 in the appendix section.3 For AC-GANs, the authors prepared a pair of discriminator and generator for each set classes of size 10. at hidden layer (concat). As we can see in the training curves Figure 3, projection outperforms inception score than concat throughout the training. TAB0 compares the intra class FIDs and the inception Score of the images generated by each method. The shown here for the AC-GANs is that of the model at its prime in terms of the inception score, because the training collapsed at the end. We see that the images generated by projection have lower intra FID scores than both adversaries, indicating that the Wasserstein distance between the generative distribution by projection to the target distribution is smaller. For the record, our model performed better than other models on the CIFAR10 and CIFAR 100 as well (See Appendix A). FIG8 and 10b shows the set of classes for which (a) projection yielded with better intra FIDs than the concat and (b) the reverse. From the top, the figures are listed in descending order of the ratio between the intra FID score between the two methods. Note that when the concat outperforms projection it only wins by a slight margin, whereas the projection outperforms concat by large margin in the opposite case. A quick glance on the cases in which the concat outperforms the projection suggests that the FID is in fact measuring the visual quality, because both sets looks similar to the human eyes in terms of appearance. FIG3 shows an arbitrarily selected set of yielded by AC-GANs from variety of zs. We can clearly observe the mode-collapse on this batch. This is indeed a tendency reported by the inventors themselves BID17. ACGANs can generate easily recognizable (i.e classifiable) images, but at the cost of losing diversity and hence at the cost of constructing a generative distribution that is significantly different from the target distribution as a whole. We can also assess the low FID score of projection from different perspective. By construction, the trace term of intra FID measures the degree of diversity within the class. Thus, our on the intra FID scores also indicates that that our projection is doing better in reproducing the diversity of the original. The GANs with the concat discriminator also suffered from mode-collapse for some classes (see FIG4). For the set of images generated by projection, we were not able to detect any notable mode-collapse. FIG5 shows the samples generated with the projection model for the classes on which the cGAN achieved lowest intra FID scores (that is the classes on which the generative distribution were particularly close to target conditional distribution), and FIG5 the reverse. While most of the images listed in FIG5 are of relatively high quality, we still observe some degree of mode-collapse. Note that the images in the classes with high FID are featuring complex objects like human; that is, one can expect the diversity within the class to be wide. However, we note that we did not use the most complicated neural network available for the experiments presented on this paper, because we prioritized the completion of the training within a reasonable time frame. It is very possible that, by increasing the complexity of the model, we will be able to further improve the visual quality of the images and the diversity of the distribution. In Appendix D, we list images of numerous classes generated by cGANs trained with our projection model. With our new architecture, we were also able to successfully perform category morphism. When there are classes y 1 and y 2, we can create an interpolated generator by simply mixing the parameters of conditional batch normalization layers of the conditional generator corresponding to these two classes. FIG6 shows the output of the interpolated generator with the same z. Interestingly, the combination is also yielding meaningful images when y 1 and y 2 are significantly different. Fine-tuning with the pretrained model on the ILSVRC2012 classification task. As we mentioned in Section 4, the authors of Plug and Play Generative model (PPGNs) BID15 were able to improve the visual appearance of the model by augmenting the cost function with that of the label classifier. We also followed their footstep and augmented the original generator loss with an additional auxiliary classifier loss. As warned earlier regarding this type of approach, however, this type of modification tends to only improve the visual performance of the images that are easy for the pretrained model to classify. In fact, as we can see in Appendix B, we were able to improve the visual appearance the images with the augmentation, but at the cost of diversity. We also evaluated the effectiveness of in its application to the super-resolution task. Put formally, the super-resolution task is to infer the high resolution RGB image of dimension DISPLAYFORM0 This task is very much the case that we presumed in our model construction, because p(y|x) is most likely unimodal even if p(x|y) is multimodal. For the super-resolution task, we used the following formulation for discriminator function: DISPLAYFORM1 where F (φ(x; θ Φ)) = V * φ(x; θ Φ) where V is a convolutional kernel and * stands for convolution operator. Please see FIG3 in the appendix section for the actual network architectures we used (a) Generated images on the class with'low' FID scores.(b) generated images on the class with'high' FID scores. for this task. For this set of experiments, we constructed the concat model by removing the module in the projection model containing the the inner product layer and the accompanying convolution layer altogether, and simply concatenated y to the output of the ResBlock preceding the inner product module in the original. As for the resolutions of the image datasets, we chose R H = 128 and R L = 32, and created the low resolution images by applying bilinear downsampling on high resolution images. We updated the generators 150K times for all methods, and applied linear decay for the learning rate after 100K iterations so that the final learning rate was 0 at 150K-th iteration. FIG7 shows the of our super-resolution. The bicubic super-resolution is very blurry, and concat is suffering from excessively sharp and rough edges. On the other hand, the edges of the images generated by our projection method are much clearer and smoother, and the image itself is much more faithful to the original high resolution images. In order to qualitatively compare the performances of the models, we checked MS-SSIM BID29 and the classification accuracy of the inception model on the generated images using the validation set of the ILSVRC2012 dataset. As we can see in Table 2, our projection model was able to achieve high inception accuracy and high MS-SSIM when compared to bicubic and concat. Note that the performance of superresolution with concat model even falls behind those of the bilinear and bicubic super-resolutions 878 -in terms of the inception accuracy. Also, we used projection model to generate multiple batches of images with different random values of z to be fed to the generator and computed the average of the logits of the inception model on these batches (MC samples). We then used the so-computed average logits to make prediction of the labels. With an ensemble over 10 seeds (10 MC in Table 2), we were able to improve the inception accuracy even further. This indicates that our GANs are learning the super-resolution as an distribution, as opposed to deterministic function. Also, the success with the ensemble also suggests a room for a new way to improve the accuracy of classification task on low resolution images. Any specification on the form of the discriminator imposes a regularity condition for the choice for the generator distribution and the target distribution. In this research, we proposed a model for the discriminator of cGANs that is motivated by a commonly occurring family of probabilistic models. This simple modification was able to significantly improve the performance of the trained generator on conditional image generation task and super-resolution task. The presented in this paper is strongly suggestive of the importance of the choice of the form of the discriminator and the design A OF CLASS CONDITIONAL IMAGE GENERATION ON CIFAR-10 AND CIFAR-100As a preliminary experiment, we compared the performance of conditional image generation on CIFAR-10 and CIFAR-100 3. For the discriminator and the generator, we reused the same architecture used in BID14 for the task on CIFAR-10. For the adversarial objective functions, we used, and trained both machine learners with the same optimizer with same hyper parameters we used in Section 5. For our projection model, we added the projection layer to the discriminator in the same way we did in the ImageNet experiment (before the last linear layer). Our projection model achieved better performance than other methods on both CIFAR-10 and CIFAR-100. Concatenation at hidden layer (hidden concat) was performed on the output of second ResBlock of the discriminator. We tested hidden concat as a comparative method in our main experiments on ImageNet, because the concatenation at hidden layer performed better than the concatenation at the input layer (input concat) when the number of classes was large (CIFAR-100).To explore how the hyper-parameters affect the performance of our proposed architecture, we conducted hyper-parameter search on CIFAR-100 about the Adam hyper-parameters (learning rate α and 1st order momentum β 1) for both our proposed architecture and the baselines. Namely, we varied each one of these parameters while keeping the other constant, and reported the inception scores for all methods including several versions of concat architectures to compare. We tested with concat module introduced at (a) input layer, (b) hidden layer, and at (c) output layer. As we can see in Figure 11, our projection architecture excelled over all other architectures for all choice of the parameters, and achieved the inception score of 9.53. Meanwhile, concat architectures were able to achieve all 8.82 at most. The best concat model in term of the inception score on CIFAR-100 was the hidden concat with α = 0.0002 and β 1 = 0, which turns out to be the very choice of the parameters we picked for our ImageNet experiment. In this experiment, we followed the footsteps of Plug and Play Generative model (PPGNs) BID15 and augmented the original generator loss with an additional auxiliary classifier loss. In particular, we used the losses given by: DISPLAYFORM0 wherep pre (y|x) is the fixed model pretrained for ILSVRC2012 classification task. For the actual experiment, we trained the generator with the original adversarial loss for the first 400K updates, and used the augmented loss for the last 50K updates. For the learning rate hyper parameter, we adopted the same values as other experiments we described above. For the pretrained classifier, we used ResNet50 model used in BID7. FIG0 compares the generated by vanilla objective function and the generated by the augmented objective function. As we can see in TAB2, we were able to significantly outperform PPGNs in terms of inception score. However, note that the images generated here are images that are easy to classify. The method with auxiliary classifier loss seems effective in improving the visual appearance, but not in training faithful generative model. ReLU Conv 3x3ReLU (a) ResBlock architecture for the discriminator. Spectral normalization BID14 was applied to each conv layer. BID4 BID1 was used in place of the standard batch normalization layer. For the ResBlock in the generator for the super resolution tasks that implements the upsampling, the random vector z was fed to the model by concatenating the vector to the embedded low resolution image vector y prior to the first convolution layer within the block. For the procedure of downsampling and upsampling, we followed the implementation by BID6. For the discriminator, we performed downsampling (average pool) after the second conv of the ResBlock. For the generator, we performed upsampling before the first conv of the ResBlock. For the ResBlock that is performing the downsampling, we replaced the identity mapping with 1x1 conv layer followed by downsampling to balance the dimension. We did the essentially same for the Resblock that is performing the upsampling, except that we applied the upsampling before the 1x1 conv.

Represent the TLDR for finding the corresponding scientific article.

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model.

Represent the short summary for finding the corresponding scientific article (long)

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question:
represent the scientific article to find a short tldr

------

Examples:
'The rise of graph-structured data such as social networks, regulatory networks, citation graphs, and functional brain networks, in combination with resounding success of deep learning in various applications, has brought the interest in generalizing deep learning models to non-Euclidean domains. In this paper, we introduce a new spectral domain convolutional architecture for deep learning on graphs. The core ingredient of our model is a new class of parametric rational complex functions (Cayley polynomials) allowing to efficiently compute spectral filters on graphs that specialize on frequency bands of interest. Our model generates rich spectral filters that are localized in space, scales linearly with the size of the input data for sparsely-connected graphs, and can handle different constructions of Laplacian operators. Extensive experimental show the superior performance of our approach on spectral image classification, community detection, vertex classification and matrix completion tasks. In many domains, one has to deal with large-scale data with underlying non-Euclidean structure. Prominent examples of such data are social networks, genetic regulatory networks, functional networks of the brain, and 3D shapes represented as discrete manifolds. The recent success of deep neural networks and, in particular, convolutional neural networks (CNNs) BID19 have raised the interest in geometric deep learning techniques trying to extend these models to data residing on graphs and manifolds. Geometric deep learning approaches have been successfully applied to computer graphics and vision; BID3 a); BID24, brain imaging BID18, and drug design BID10 problems, to mention a few. For a comprehensive presentation of methods and applications of deep learning on graphs and manifolds, we refer the reader to the review paper BID4.Related work. The earliest neural network formulation on graphs was proposed by BID11 and BID27, combining random walks with recurrent neural networks (their paper has recently enjoyed renewed interest in BID20 ; BID30). The first CNN-type architecture on graphs was proposed by BID5. One of the key challenges of extending CNNs to graphs is the lack of vector-space structure and shift-invariance making the classical notion of convolution elusive. Bruna et al. formulated convolution-like operations in the spectral domain, using the graph Laplacian eigenbasis as an analogy of the Fourier transform BID29 ). BID13 used smooth parametric spectral filters in order to achieve localization in the spatial domain and keep the number of filter parameters independent of the input size. BID8 proposed an efficient filtering scheme using recurrent Chebyshev polynomials applied on the Laplacian operator. BID17 simplified this architecture using filters operating on 1-hop neighborhoods of the graph. BID0 proposed a Diffusion CNN architecture based on random walks on graphs. BID24 (and later,) proposed a spatial-domain generalization of CNNs to graphs using local patch operators represented as Gaussian mixture models, showing a significant advantage of such models in generalizing across different graphs. In BID25, spectral graph CNNs were extended to multiple graphs and applied to matrix completion and recommender system problems. Main contribution. In this paper, we construct graph CNNs employing an efficient spectral filtering scheme based on Cayley polynomials that enjoys similar advantages of the Chebyshev filters BID8 ) such as localization and linear complexity. The main advantage of our filters over BID8 is their ability to detect narrow frequency bands of importance during training, and to specialize on them while being well-localized on the graph. We demonstrate experimentally that this affords our method greater flexibility, making it perform better on a broad range of graph learning problems. Notation. We use a, a, and A to denote scalars, vectors, and matrices, respectively.z denotes the conjugate of a complex number, Re{z} its real part, and i is the imaginary unit. diag(a 1, . . ., a n) denotes an n×n diagonal matrix with diagonal elements a 1,..., a n. Diag(A) = diag(a 11, . . ., a nn) denotes an n × n diagonal matrix obtained by setting to zero the off-diagonal elements of A. Off(A) = A − Diag(A) denotes the matrix containing only the off-diagonal elements of A. I is the identity matrix and A • B denotes the Hadamard (element-wise) product of matrices A and B. Proofs are given in the appendix. Spectral graph theory. Let G = ({1, . . ., n}, E, W) be an undirected weighted graph, represented by a symmetric adjacency matrix W = (w ij). We define DISPLAYFORM0 We denote by N k,m the k-hop neighborhood of vertex m, containing vertices that are at most k edges away from m. The unnormalized graph Laplacian is an n × n symmetric positive-semidefinite matrix DISPLAYFORM1 In the following, we use the generic notation ∆ to refer to some Laplacian. Since both normalized and unnormalized Laplacian are symmetric and positive semi-definite matrices, they admit an eigendecomposition ∆ = ΦΛΦ, where Φ = (φ 1, . . . φ n) are the orthonormal eigenvectors and Λ = diag(λ 1, . . ., λ n) is the diagonal matrix of corresponding non-negative eigenvalues (spectrum) 0 = λ 1 ≤ λ 2 ≤... ≤ λ n. The eigenvectors play the role of Fourier atoms in classical harmonic analysis and the eigenvalues can be interpreted as (the square of) frequencies. Given a signal f = (f 1, . . ., f n) on the vertices of graph G, its graph Fourier transform is given byf = Φ f. Given two signals f, g on the graph, their spectral convolution can be defined as the element-wise product of the Fourier transforms, f g = Φ (Φ g)•(Φ f) = Φ diag(ĝ 1, . . .,ĝ n)f, which corresponds to the property referred to as the Convolution Theorem in the Euclidean case. Spectral CNNs. BID5 used the spectral definition of convolution to generalize CNNs on graphs, with a spectral convolutional layer of the form DISPLAYFORM2 (Here the n × p and n × q matrices DISPLAYFORM3 represent respectively the p-and q-dimensional input and output signals on the vertices of the graph, (ĝ l,l,1, . . .,ĝ l,l,k) is a k × k diagonal matrix of spectral multipliers representing a learnable filter in the frequency domain, and ξ is a nonlinearity (e.g., ReLU) applied on the vertex-wise function values. Pooling is performed by means of graph coarsening, which, given a graph with n vertices, produces a graph with n < n vertices and transfers signals from the vertices of the fine graph to those of the coarse one. This framework has several major drawbacks. First, the spectral filter coefficients are basis dependent, and consequently, a spectral CNN model learned on one graph cannot be transferred to another graph. Second, the computation of the forward and inverse graph Fourier transforms incur expensive O(n 2) multiplication by the matrices Φ, Φ, as there is no FFT-like algorithms on general graphs. Third, there is no guarantee that the filters represented in the spectral domain are localized in the spatial domain (locality property simulates local reception fields, BID7); assuming k = O(n) Laplacian eigenvectors are used, a spectral convolutional layer requires O(pqk) = O(n) parameters to train. DISPLAYFORM4 To address the latter issues, BID13 argued that smooth spectral filter coefficients in spatially-localized filters (an argument similar to vanishing moments). The filter coefficients are represented asĝ i = g(λ i), where g(λ) is a smooth transfer function of frequency λ. Applying such filter to signal f can be expressed as DISPLAYFORM5 where applying a function to a matrix is understood in the operator functional calculus sense (applying the function to the matrix eigenvalues). BID13 used parametric functions of the form g(λ) = r j=1 α j β j (λ), where β 1 (λ),..., β r (λ) are some fixed interpolation kernels such as splines, and α = (α 1, . . ., α r) are the interpolation coefficients used as the optimization variables during the network training. In matrix notation, the filter is expressed as Gf = Φdiag(Bα)Φ f, where B = (b ij) = (β j (λ i)) is a k × r matrix. Such a construction in filters with r = O parameters, independent of the input size. However, the authors explicitly computed the Laplacian eigenvectors Φ, ing in high complexity. ChebNet. BID8 used polynomial filters represented in the Chebyshev basis DISPLAYFORM6 applied to rescaled frequencyλ ∈ [−1, 1]; here, α is the (r + 1)-dimensional vector of polynomial coefficients parametrizing the filter and optimized for during the training, and Such an approach has several important advantages. First, since g α (∆) = r j=0 α j T j (∆) contains only matrix powers, additions, and multiplications by scalar, it can be computed avoiding the explicit expensive O(n 3) computation of the Laplacian eigenvectors. Furthermore, due to the recursive definition of the Chebyshev polynomials, the computation of the filter g α (∆)f entails applying the Laplacian r times, ing in O(rn) operations assuming that the Laplacian is a sparse matrix with O non-zero elements in each row (a valid hypothesis for most real-world graphs that are sparsely connected). Second, the number of parameters is O as r is independent of the graph size n. Third, since the Laplacian is a local operator affecting only 1-hop neighbors of a vertex and a polynomial of degree r of the Laplacian affects only r-hops, the ing filters have guaranteed spatial localization. DISPLAYFORM7 A key disadvantage of Chebyshev filters is the fact that using polynomials makes it hard to produce narrow-band filters, as such filters require very high order r, and produce unwanted non-local filters. This deficiency is especially pronounced when the Laplacian has clusters of eigenvalues concentrated around a few frequencies with large spectral gap (FIG4, middle right). Such a behavior is characteristic of graphs with community structures, which is very common in many real-world graphs, for instance, social networks. To overcome this major drawback, we need a new class of filters, that are both localized in space, and are able to specialize in narrow bands in frequency. A key construction of this paper is a family of complex filters that enjoy the advantages of Chebyshev filters while avoiding some of their drawbacks. A Cayley polynomial of order r is a real-valued function with complex coefficients, DISPLAYFORM0 where c = (c 0, . . ., c r) is a vector of one real coefficient and r complex coefficients and h > 0 is the spectral zoom parameter, that will be discussed later. A Cayley filter G is a spectral filter defined on real signals f by where the parameters c and h are optimized for during training. Similarly to the Chebyshev filters, Cayley filters involve basic matrix operations such as powers, additions, multiplications by scalars, and also inversions. This implies that application of the filter Gf can be performed without explicit expensive eigendecomposition of the Laplacian operator. In the following, we show that Cayley filters are analytically well behaved; in particular, any smooth spectral filter can be represented as a Cayley polynomial, and low-order filters are localized in the spatial domain. We also discuss numerical implementation and compare Cayley and Chebyshev filters. DISPLAYFORM1 Analytic properties. Cayley filters are best understood through the Cayley transform, from which their name derives. Denote by e iR = {e iθ : θ ∈ R} the unit complex circle. The Cayley transform DISPLAYFORM2 x+i is a smooth bijection between R and e iR \ {1}. The complex matrix C(h∆) DISPLAYFORM3 −1 obtained by applying the Cayley transform to the scaled Laplacian h∆ has its spectrum in e iR and is thus unitary. Since DISPLAYFORM4. Therefore, using 2Re{z} = z + z, any Cayley filter can be written as a conjugateeven Laurent polynomial w.r.t. C(h∆), DISPLAYFORM5 Since the spectrum of C(h∆) is in e iR, the operator C j (h∆) can be thought of as a multiplication by a pure harmonic in the frequency domain e iR for any integer power j, DISPLAYFORM6 A Cayley filter can be thus seen as a multiplication by a finite Fourier expansions in the frequency domain e iR. Since is conjugate-even, it is a (real-valued) trigonometric polynomial. Note that any spectral filter can be formulated as a Cayley filter. Indeed, spectral filters g(∆) are specified by the finite sequence of values g(λ 1),..., g(λ n), which can be interpolated by a trigonometric polynomial. Moreover, since trigonometric polynomials are smooth, we expect low order Cayley filters to be well localized in some sense on the graph, as discussed later. Finally, in definition we use complex coefficients. If c j ∈ R then FORMULA14 is an even cosine polynomial, and if c j ∈ iR then is an odd sine polynomial. Since the spectrum of h∆ is in R +, it is mapped to the lower half-circle by C, on which both cosine and sine polynomials are complete and can represent any spectral filter. However, it is beneficial to use general complex coefficients, since complex Fourier expansions are overcomplete in the lower half-circle, thus describing a larger variety of spectral filters of the same order without increasing the computational complexity of the filter. Spectral zoom. To understand the essential role of the parameter h in the Cayley filter, consider C(h∆). Multiplying ∆ by h dilates its spectrum, and applying C on the maps the non-negative spectrum to the complex half-circle. The greater h is, the more the spectrum of h∆ is spread apart in R +, ing in better spacing of the smaller eigenvalues of C(h∆). On the other hand, the smaller h is, the further away the high frequencies of h∆ are from ∞, the better spread apart are the high frequencies of C(h∆) in e iR (see FIG1). Tuning the parameter h allows thus to'zoom' in to different parts of the spectrum, ing in filters specialized in different frequency bands. Numerical properties. The numerical core of the Cayley filter is the computation of C j (h∆)f for j = 1,..., r, performed in a sequential manner. Let y 0,..., y r denote the solutions of the following linear recursive system, DISPLAYFORM7 Note that sequentially approximating y j in using the approximation of y j−1 in the rhs is stable, since C(h∆) is unitary and thus has condition number 1.Equations FORMULA16 can be solved with matrix inversion exactly, but it costs O(n 3). An alternative is to use the Jacobi method, 1 which provides approximate solutionsỹ j ≈ y j. Let J = −(Diag(h∆ + iI)) −1 Off(h∆ + iI) be the Jacobi iteration matrix associated with equation. For the unnormalized Laplacian, J = (hD + iI) −1 hW. Jacobi iterations for approximating for a given j have the form DISPLAYFORM8 initialized withỹ DISPLAYFORM9. The application of the approximate Cayley filter is given by Gf = r j=0 c jỹj ≈ Gf, and takes O(rKn) operations under the previous assumption of a sparse Laplacian. The method can be improved by normalizing ỹ j 2 = f 2.Next, we give an error bound for the approximate filter. For the unnormalized Laplacian, let DISPLAYFORM10 < 1. For the normalized Laplacian, we assume that (h∆ n + iI) is dominant diagonal, which gives κ = J ∞ < 1. Proposition 1. Under the above assumptions, Proposition 1 is pessimistic in the general case, while requires strong assumptions in the regular case. We find that in most real life situations the behavior is closer to the regular case. It also follows from Proposition 1 that smaller values of the spectral zoom h in faster convergence, giving this parameter an additional numerical role of accelerating convergence. DISPLAYFORM11 Complexity. In practice, an accurate inversion of (h∆ + iI) is not required, since the approximate inverse is combined with learned coefficients, which "compensate", as necessary, for the inversion inaccuracy. In a CayleyNet for a fixed graph, we fix the number of Jacobi iterations. Since the convergence rate depends on κ, that depends on the graph, different graphs may need different numbers of iterations. The convergence rate also depends on h. Since there is a trade-off between the spectral zoom amount h, and the accuracy of the approximate inversion, and since h is a learnable parameter, the training finds the right balance between the spectral zoom amount and the inversion accuracy. We study the computational complexity of our method, as the number of edges n of the graph tends to infinity. For every constant of a graph, e.g d, κ, we add the subscript n, indicating the number of edges of the graph. For the unnormalized Laplacian, we assume that d n and h n are bounded, which gives κ n < a < 1 for some a independent of n. For the normalized Laplacian, we assume that κ n < a < 1. By Theorem 1, fixing the number of Jacobi iterations K and the order of the filter r, independently of n, keeps the Jacobi error controlled. As a , the number of parameters is O, and for a Laplacian modeled as a sparse matrix, applying a Cayley filter on a signal takes O(n) operations. Localization. Unlike Chebyshev filters that have the small r-hop support, Cayley filters are rational functions supported on the whole graph. However, it is still true that Cayley filters are well localized on the graph. Let G be a Cayley filter and δ m denote a delta-function on the graph, defined as one at vertex m and zero elsewhere. We show that Gδ m decays fast, in the following sense: Definition 2 (Exponential decay on graphs). Let f be a signal on the vertices of graph G, 1 ≤ p ≤ ∞, and 0 < < 1. Denote by S ⊆ {1, . . ., n} a subset of the vertices and by S c its complement. We say that the L p -mass of f is supported in S up to if f | S c p ≤ f p, where f | S c = (f l) l∈S c is the restriction of f to S c. We say that f has (graph) exponential decay about vertex m, if there exists some γ ∈ and c > 0 such that for any k, the L p -mass of f is supported in N k,m up to cγ k. Here, N k,m is the k-hop neighborhood of m. 1 We remind that the Jacobi method for solving Ax = b consists in decomposing A = Diag(A) + Off(A) and obtaining the solution iteratively as Remark 3. Note that Definition 2 is analogous to classical exponential decay on Euclidean space: DISPLAYFORM12 DISPLAYFORM13 Theorem 4. Let G be a Cayley filter of order r. Then, Gδ m has exponential decay about m in L 2, with constants c = 2M1 Gδm 2 and γ = κ 1/r (where M and κ are from Proposition 1). Chebyshev as a special case of Cayley. For a regular graph with D = dI, using Jacobi inversion based on zero iterations, we get that any Cayley filter of order r is a polynomial of ∆ in the monomial base h∆−i hd+i j. In this situation, a Chebyshev filter, which is a real valued polynomial of ∆, is a special case of a Cayley filter. Spectral zoom and stability. Generally, both Chebyshev polynomials and trigonometric polynomials give stable approximations, optimal for smooth functions. However, this crude statement is oversimplified. One of the drawbacks in Chebyshev filters is the fact that the spectrum of ∆ is always mapped to [−1, 1] in a linear manner, making it hard to specialize in small frequency bands. In Cayley filters, this problem is mitigated with the help of the spectral zoom parameter h. As an example, consider the community detection problem discussed in the next section. A graph with strong communities has a cluster of small eigenvalues near zero. Ideal filters g(∆) for extracting the community information should be able to focus on this band of frequencies. Approximating such filters with Cayley polynomials, we zoom in to the band of interest by choosing the right h, and then project g onto the space of trigonometric polynomials of order r, getting a good and stable approximation (FIG4, bottom right). However, if we project g onto the space of Chebyshev polynomials of order r, the interesting part of g concentrated on a small band is smoothed out and lost (FIG4, middle right). Thus, projections are not the right way to approximate such filters, and the stability of orthogonal polynomials cannot be invoked. When approximating g on the small band using polynomials, the approximation will be unstable away from this band; small perturbations in g will in big perturbations in the Chebyshev filter away from the band. For this reason, we say that Cayley filters are more stable than Chebyshev filters. Regularity. We found that in practice, low-order Cayley filters are able to model both very concentrated impulse-like filters, and wider Gabor-like filters. Cayley filters are able to achieve a wider range of filter supports with less coefficients than Chebyshev filters FIG3 ), making the Cayley class more regular than Chebyshev. Complexity. Under the assumption of sparse Laplacians, both Cayley and Chebyshev filters incur linear complexity O(n). Besides, the new filters are equally simple to implement as Chebyshev filters; as seen in Eq.7, they boil down to simple sparse matrix-vector multiplications providing a GPU friendly implementation. Experimental settings. We test the proposed CayleyNets reproducing the experiments of Defferrard et al. FORMULA7; BID17; BID24 a) and using ChebNet BID8 ) as our main baseline method. All the methods were implemented in TensorFlow of M. BID21. The experiments were executed on a machine with a 3.5GHz Intel Core i7 CPU, 64GB of RAM, and NVIDIA Titan X GPU with 12GB of RAM. SGD+Momentum and Adam BID16 ) optimization methods were used to train the models in MNIST and the rest of the experiments, respectively. Training and testing were always done on disjoint sets. Community detection. We start with an experiment on a synthetic graph consisting of 15 communities with strong connectivity within each community and sparse connectivity across communities FIG4. Though rather simple, such a dataset allows to study the behavior of different algorithms in controlled settings. On this graph, we generate noisy step signals, defined as f i = 1 + σ i if i belongs to the community, and f i = σ i otherwise, where DISPLAYFORM0 The goal is to classify each such signal according to the community it belongs to. The neural network architecture used for this task consisted of a spectral convolutional layer (based on Chebyshev or Cayley filters) with 32 output features, a mean pooling layer, and a softmax classifier for producing the final classification into one of the 15 classes. The classification accuracy is shown in FIG4 (right, top) along with examples of learned filters (right, bottom). We observe that CayleyNet significantly outperforms ChebNet for smaller filter orders, with an improvement as large as 80%.Studying the filter responses, we note that due to the capability to learn the spectral zoom parameter, CayleyNet allows to generate band-pass filters in the low-frequency band that discriminate well the communities FIG4. Complexity. We experimentally validated the computational complexity of our model applying filters of different order r to synthetic 15-community graphs of different size n using exact matrix inversion and approximation with different number of Jacobi iterations FIG5 center and right, Figure 6 in the appendix). All times have been computed running 30 times the considered models and averaging the final . As expected, approximate inversion guarantees O(n) complexity. We further conclude that typically very few Jacobi iterations are required FIG5, left shows that our model with just one Jacobi iteration outperforms ChebNet for low-order filters on the community detection problem).MNIST. Following BID8; BID24, for a toy example, we approached the classical MNIST digits classification as a learning problem on graphs. Each pixel of an image is a vertex of a graph (regular grid with 8-neighbor connectivity), and pixel color is a signal on the graph. We used a graph CNN architecture with two spectral convolutional layers based on Chebyshev and Cayley filters (producing 32 and 64 output features, respectively), interleaved with pooling layers performing 4-times graph coarsening using the Graclus algorithm (Dhillon et al. FORMULA7 finally a fully-connected layer (this architecture replicates the classical LeNet5, BID19, architecture, which is shown for comparison). MNIST classification are reported in TAB0. CayleyNet (11 Jacobi iterations) achieves the same (near perfect) accuracy as ChebNet with filters of lower order (r = 12 vs 25).Examples of filters learned by ChebNet and CayleyNet are shown in FIG3. 0.1776 +/-0.06079 sec and 0.0268 +/-0.00841 sec are respectively required by CayleyNet and ChebNet for analyzing a batch of 100 images at test time. FORMULA7 ) 86.64 % 47K ChebNet BID8 ) 87.07 % 46K CayleyNet 88.09 % 46KCitation network. Next, we address the problem of vertex classification on graphs using the popular CORA citation graph, BID28. Each of the 2708 vertices of the CORA graph represents a scientific paper, and an undirected unweighted edge represents a citation (5429 edges in total). For each vertex, a 1433-dimensional binary feature vector representing the content of the paper is given. The task is to classify each vertex into one of the 7 groundtruth classes. We split the graph into training (1,708 vertices), validation (500 vertices) and test (500 vertices) sets, for simulating the labeled and unlabeled information. We train ChebNet and CayleyNet with the architecture presented in BID17; BID24 (two spectral convolutional layers with 16 and 7 outputs), DCNN BID0 ) with 2 diffusion layer (10 hidden features and 2 diffusion hops) and GCN BID17 ) with 3 convolutional layer (32 and 16 hidden features). Figure 5 compares ChebNets and CayleyNets, in a number of different settings. Since ChebNets require Laplacians with spectra bounded in [−1, 1], we consider both the normalized Laplacian (the two left figures), and the scaled unnormalized Laplacian (2∆/λ max − I), where ∆ is the unnormalized Laplacian and λ max is its largest eigenvalue (the two right figures). For fair comparison, we fix the order of the filters (top figures), and fix the overall number of network parameters (bottom figures). In the bottom figure, the Cayley filters are restricted to even cosine polynomials by considering only real filter coefficients. TAB1 shows a comparison of the performance obtained with different methods (all architectures roughly present the same amount of parameters). The best CayleyNets consistently outperform the best competitors. Recommender system. In our final experiment, we applied CayleyNet to recommendation system, formulated as matrix completion problem on user and item graphs, BID24. The task is, given a sparsely sampled matrix of scores assigned by users (columns) to items (rows), to fill in the missing scores. The similarities between users and items are given in the form of column and row graphs, respectively. BID24 approached this problem as learning with a Recurrent Graph CNN (RGCNN) architecture, using an extension of ChebNets to matrices defined on multiple graphs in order to extract spatial features from the score matrix; these features are then fed into an Figure 5: ChebNet (blue) and CayleyNet (orange) test accuracies obtained on the CORA dataset for different polynomial orders. Polynomials with complex coefficients (top) and real coefficients (bottom) have been exploited with CayleyNet in the two analysis. Orders 1 to 6 have been used in both comparisons. The best CayleyNet consistently outperform the best ChebNet requiring at the same time less parameters (CayleyNet with order r and complex coefficients requires a number of parameters equal to ChebNet with order 2r).RNN producing a sequential estimation of the missing scores. Here, we repeated verbatim their experiment on the MovieLens dataset BID23 ), replacing Chebyshev filters with Cayley filters. We used separable RGCNN architecture with two CayleyNets of order r = 4 employing 15 Jacobi iterations. The are reported in TAB3. To present a complete comparison we further extended the experiments reported in BID24 by training sRGCNN with ChebNets of order 8, this provides an architecture with same number of parameters as the exploited CayleyNet (23K coefficients). Our version of sRGCNN outperforms all the competing methods, including the previous with Chebyshev filters reported in BID24. sRGCNNs with Chebyshev polynomials of order 4 and 8 respectively require 0.0698 +/-0.00275 sec and 0.0877 +/-0.00362 sec at test time, sRGCNN with Cayley polynomials of order 4 and 15 jacobi iterations requires 0.165 +/-0.00332 sec. BID14; BID31 ) 1.653 GMC BID15 0.996 GRALS BID26 ) 0.945 sRGCNN Cheby,r=4 BID24 0.929 sRGCNN Cheby,r=8 BID24 0.925 sRGCNN Cayley 0.922 In this paper, we introduced a new efficient spectral graph CNN architecture that scales linearly with the dimension of the input data. Our architecture is based on a new class of complex rational Cayley filters that are localized in space, can represent any smooth spectral transfer function, and are highly regular. The key property of our model is its ability to specialize in narrow frequency bands with a small number of filter parameters, while still preserving locality in the spatial domain. We validated these theoretical properties experimentally, demonstrating the superior performance of our model in a broad range of graph learning problems. First note the following classical for the approximation of Ax = b using the Jacobi method: if the initial condition is DISPLAYFORM0 In our case, note that if we start with initial conditionỹ j = 0, the next iteration givesỹ j = b j, which is the initial condition from our construction. Therefore, since we are approximating y j = C(h∆)ỹ j−1 byỹ j =ỹ DISPLAYFORM1 Define the approximation error in C(h∆) j f by DISPLAYFORM2 By the triangle inequality, by the fact that C j (h∆) is unitary, and by DISPLAYFORM3 where the last inequality is due to DISPLAYFORM4 Now, using standard norm bounds, in the general case we have DISPLAYFORM5 The solution of this recurrent sequence is DISPLAYFORM6 If we use the version of the algorithm, in which eachỹ j is normalized, we get by e j ≤ e j−1 + √ nκ K+1. The solution of this recurrent sequence is DISPLAYFORM7 We denote in this case M j = j √ nIn case the graph is regular, we have D = dI. In the non-normalized Laplacian case, DISPLAYFORM8 The spectral radius of ∆ is bounded by 2d. This can be shown as follows. a value λ is not an eigenvalue of ∆ (namely it is a regular value) if and only if (∆ − λI) is invertible. Moreover, the matrix (∆ − λI) is strictly dominant diagonal for any |λ| > 2d. By Levy-Desplanques theorem, any strictly dominant diagonal matrix is invertible, which means that all of the eigenvalues of ∆ are less than 2d in their absolute value. As a , the spectral radius of (dI − ∆) is realized on the smallest eigenvalue of ∆, namely it is |d − 0| = d. This means that the specral radius of J is DISPLAYFORM9. As a DISPLAYFORM10 = κ. We can now continue from to get e j ≤ e j−1 + J K+1 2(1 + e j−1) = e j−1 + κ K+1 (1 + e j−1).As before, we get e j ≤ jκ K+1 + O(κ 2K+2), and e j ≤ jκ K+1 if eachỹ j is normalized. We denote in this case M j = j. In the case of the normalized Laplacian of a regular graph, the spectral radius of ∆ n is bounded by 2, and the diagonal entries are all 1. Equation in this case reads J = h h+i (I − ∆ n), and J has spectral radius h √ h 2 +1. Thus J 2 = h √ h 2 +1= κ and we continue as before to get e j ≤ jκ K+1 and M j = j. In all cases, we have by the triangle inequality Figure 6: Test (above) and training (below) times with corresponding ratios as function of filter order r and graph size n on our community detection dataset.' == 'A spectral graph convolutional neural network with spectral zoom properties.' != 'We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.'

Question: Learning theory tells us that more data is better when minimizing the generalization error of identically distributed training and test sets. However, when training and test distribution differ, this distribution shift can have a significant effect. With a novel perspective on function transfer learning, we are able to lower bound the change of performance when transferring from training to test set with the Wasserstein distance between the embedded training and test set distribution. We find that there is a trade-off affecting performance between how invariant a function is to changes in training and test distribution and how large this shift in distribution is. Empirically across several data domains, we substantiate this viewpoint by showing that test performance correlates strongly with the distance in data distributions between training and test set. Complementary to the popular belief that more data is always better, our highlight the utility of also choosing a training data distribution that is close to the test data distribution when the learned function is not invariant to such changes. Imagine there are two students who are studying for an exam. Student A studies by diligently learning the class material by heart. Student B studies by learning the underlying reasons for why things are the way they are. Come test day, student A is only able to answer test questions that are very similar to the class material while student B has no trouble answering different looking questions that follow the same reasoning. Distilled from this example, we note there is a trade-off between how "well" a student studied, i.e., how indifferent the student is to receiving exercise or test questions, and how close the test questions are to the exercise questions. While most machine learning work studies the generalization error, i.e., the error when testing on different samples from the same distribution, we do not take the match of train and test distribution as given. In fact, it appears that the distance between train and test distribution may be critical for successful "generalization". Following a similar line of thought, devised a distribution measurement to select only features that do not vary from one domain to another. In contrast, we are interested in linking performance directly to the distance between train and test distribution. Invariance to distribution shifts: We say that a function is invariant to a given input perturbation when the corresponding output does not change with the perturbation. This is desirable when trying to achieve robustness to irrelevant data variations which are called nuisances . As outlined by; , the "optimal" learned function from input to output is maximally invariant to all data variations that do not contain information about the output. To the extent to which a learner reacts to such nuisance variations, which carry no information about the output, it will incur a performance change in expectation. The difficulty lies in knowing what can be ignored and what cannot. Similarity between training and test distribution: Another strategy would be to ensure that the training and test distribution match which has been investigated in a number of diverse settings ). Variations of this theme were encountered by , where they show that networks are able to fit random labels perfectly, yet understandably fail to generalize to the test set of the correct label distribution. We frame the learning problem of training on a training set then shifting to test on a test set as a transfer learning problem that goes beyond the usual generalization which assumes the same distribution for training and testing. Based on our analysis, we aim to explicitly express the trade-off of the distance between both data set distributions given by the Wasserstein distance, which measures how close training and testing distribution are, and how invariant the learned function is to the training to test distribution shift. Joined together this is expressed via the Wasserstein distance between training and test samples embedded in the feature space of the learned function. Our experiments show a strong negative linear correlation between the distribution distances and network performance. This corroborates the notion that as long as the true function class is not found, it is best to ensure that training and test distribution are close. While this seems intuitive, it may often be overlooked in the hunt for more data. The trade-off and effect on performance between invariance to distribution shifts and the magnitude of such distribution shifts has been an active field of research, albeit mostly implicitly. Given the correct invariance, changes in distribution do not have a strong effect and learning "generalizes". What has not been investigated so far, to the best of our knowledge, is a more systematic treatment of measuring distribution distances and invariances of learned functions. These assumptions currently remain mainly implicit in the literature. Invariance: Invariance is a key component of Deep Learning with many works focusing on increasing invariance by incorporating structure into their models (; ; ; ; van der). Other works apply GANs , e.g., to domain adaptation problems, to map source and target distributions to the same space while being invariant to occurring distribution shifts. As stand in for many similar works, employ invariance to improve visual adaptation to domain changes. In this work, rather than trying to beat a benchmark, we aim to establish the quantitative and qualitative trade-offs between invariance and distribution shift. In contrast to adapting the structure of a model, many works achieve invariance through data augmentation (b; ; Hernández-García et al., 2019). While most augmentations are handpicked based on domain knowledge, automated methods based on GANs have been successfully applied to data augmentation . Any of these transformations can be considered a canonization , i.e., a generalization of data normalization to group actions, by removing unwanted nuisances to increase robustness. Many other techniques such as regularization , and dimension reduction can also be considered part of the umbrella of invariance learning (; ;). In this work, we simply take the inherent invariance of learned embeddings to approximate the invariance of our function approximator and focus on the distributional shift from train to test distribution. Distribution shifts: Shifts in data set distribution can take a number of forms where either the input distribution p(x), the target distribution p(y) or a combination thereof changes. Likewise imbalanced datasets , methods of over and undersampling , and domain shifts can be described in this framework. By knowing how the distribution changes from one setting to another, were able to define an operating domain for different vision sensors and employ this for a semi-supervised learning setup. Part of evaluating distribution shifts is measuring these probability distribution distances. A popular use for this is the Maximum Mean Discrepancy (MMD) distance that has found application in anomaly detection and two-sample testing . Other distances such as the Wasserstein distance have made an impact in computational graphics modeling and in generating realistic-looking synthetic images using GANs . In this work, we apply the Wasserstein distance as a powerful tool to investigate the performance drop when moving from train to test samples. Before jumping ahead, we gain an intuitive understanding of the behavior of a neural network in the case of mismatched train and test distributions. In particular we are interested in the effect of distribution shift from training to test distribution and the effect of invariance of the learned function to such changes. Looking at this through a different lens, we also investigate what impact the type of function we are approximating, and the type of function we are approximating with can have. First, we run experiments detailing changes in the input distribution as illustrated in the top row of Fig. 1. From left to right, we increase input values x by adding a scalar α. As can be readily observed, the quality of the function approximation deteriorates under such a distribution shift. If on the other hand, we normalize the input data by subtracting the mean of the distribution, the function approximation remains unchanged as seen in the second row of Fig. 1. This added invariance through normalization made the function approximation robust to the given distribution shift. Secondly, we run function approximation experiments in which we have a training set in one interval and a test set in another. We focus on what happens when the true function class is the same as the one used for approximation and what happens when this is not the case. We observe in the third row of Fig. 1 that when the correct function is found, as in the second and third image, the approximated function works even in a domain in which it was not trained. On the other hand, the first plot of the third row and the plots in the fourth row show that if the true function is not found exactly, then outside of the training domain errors accumulate. Further, we conclude from the bottom row in Fig. 1 that a oversized training domain (bottom right plot) can hurt performance on the test domain, when compared to the middle plot with overlapping train and test domains. Our takeaway is that invariance indeed influences how distribution shifts affect changes in training and testing performance. We however also note that this topic is more nuanced. The difficulty lies in the fact that finding the "correct" invariances also means finding the true function as visualized in the third row of Fig. 1. These smaller experiments make us expect that it may be unlikely to find the true invariances of an underlying functional relationship. In such a case, ensuring that train and test distribution match thus becomes a viable option to enforce a small change from train to test performance. Noted at the bottom of the spaces are the assumed underlying metrics of the space. We begin our investigation by stating that learning is a process that maps a training dataset D train with distribution p train (x, y) to a learned function approximation Fig. 2. In deep learning this is usually achieved by minimizing a loss function on the training set using stochastic gradient descent . Of interest however is the performance of the learned function on a testing dataset D test with possibly different distribution p test (x, y). By modeling the learned relationship f (p train, x) ∈ Y as a function on the data distribution p train (x, y), we are then able to obtain a relationship between changes in the data distribution and changes in the learned function and its performance. Theorem 1. We are given two dataset distributions p 1 (x, y) and p 2 (x, y) of input data x ∈ X and output y ∈ Y situated in the probability space with Wasserstein metric W 2 (·, ·) and assume that f (p, x): P × X → Y is the function of a function family F continuous in the probability space P that minimizes the norm loss L(f (p 1, x), y) = ||f (p 1, x) − y|| 2 when the data is distributed according to p 1 (x, y). Then the expected absolute difference in loss for distribution p 2 has the following lower bound. where denotes the average value of along the integral as part of the intermediate value theorem. Proof. First we note that we can write the learned function for distribution p 2 as the function learned for distribution p 1 plus the line integral of the change of the function along the Wasserstein geodesic between p 1 and p 2. Then the change in loss is lower bounded as follows: From the above lower bound in Ineq. 1, we can deduce two ways of achieving a minimal change in loss when moving from training distribution p 1 to testing distribution p 2. Either the difference in distribution from training set to test set is small W 2 (p 1, p 2) ≈ 0. In this case, by assuming the learned function fits the training distribution perfectly and is not ill-conditioned, we can deduce that also the test distribution is fit perfectly, or The function f (p) does not change when the dataset distribution changes from p 1 to p 2 by being invariant to such changes s.t. E p2 Clearly, if the learned function cannot distinguish between, i.e. is invariant to, changes in training and test set, then even a shift of distribution to the test set cannot alter the quality of the performance. If on the other hand, as may be more common, the function class is not invariant, then the difference in distribution from training to test set becomes decisive. Note that since we regard changes in performance, it is vital to already have a good performance on the training distribution. From the above analysis, we believe that it may be essential to employ training sets that are as close as possible in distribution to the test set. If this is the case one does not need to worry about finding the right function class with the right invariances to approximate the true function. In the case where the learned function has "too much invariance", the function is not noticing some of the relevant variations in the data, e.g. in the extreme a constant would not change from training to testing distribution. While its performance does not change from training to test distribution, its training performance already suffers from unnecessary invariance. If on the other hand, the function is not invariant enough it might fit the training set well yet lose performance when switching to the test distribution. Instead of either calculating the distribution distance on the raw data (W 2 (p 1, p 2)), we compute the distance on an embedding of the raw data that aims to mirror the distances the function f (·) would perceive. This has the advantage of avoiding to compute demanding high dimensional distribution distances from raw data on the one hand, and having to train networks for both training and test distribution on the other hand. Additionally, it may be hard to properly compare the distribution of a function trained on the training distribution vs. one trained on the test distribution since training and test set oftentimes are of very different size. To measure distribution distances, we employ the Fréchet distance (FD) (Fréchet, 1957), also called the 2-Wasserstein distance , by assuming all probabilities to be normally distributed. The FD is a distance measure between two normal distributions, also adopted in the Fréchet Inception Distance (FID) to evaluate the quality difference of real and synthetic images. Instead of comparing raw data, we use the metric by exploiting domain-relevant features using an embedding network such as the inception network . The FD of two normal distributions N (µ 1, Σ 1) and N (µ 2, Σ 2) is given by: show that the distance correlates well with human judgement in the vision domain, it can also be applied to other domains if a domain relevant model replaces the inception network. We apply the FD to measure the mismatch of two sets in the following sections. Next, we aim to show that the distribution distance between training and test distributions matters in many practical problems. We underline this perspective by running experiments in text classification, vision data augmentation, and speech separation. Crucial for these experiments is that we ensure that the output distribution stays the same while only the input distribution changes. In the following we offer the general experimental setup for each dataset. For more comprehensive implementation details please refer to appendices B and C. We begin our investigation in text classification. To this end we employ different subsets of a large text classification dataset which classifies product reviews into binary good or bad product ratings. To measure the effect of distribution shift, we train our rating classifier on texts of one product category and test it on another. This yields many pairwise train/test distances and performances. Specifically, we use the Amazon Review dataset which contains text reviews and ratings divided into various product categories. In our experiments, we select 9 categories of the 5-core (at least 5 reviews per item) subsets. As input we use the review texts and as output we create binary rating categories of good or bad by splitting the rating categories into negative and positive labels. Reviews with a rating of 3 are removed. To infer the FD score between datasets categories, we use pretrained GloVe embeddings of size 100 and 300. The word embeddings of each sentence are averaged to obtain a single embedding for each review. From Fig. 3a, we can see that the classification accuracy of the binarized ratings decreases the further the training set is away from the test set as measured by the FD score. The relationship correlates strongly (Pearson correlation -0.78), and notably there is no high accuracy for large FD scores. Next we examine the effect of distribution shift in vision classification tasks. Special about this problem is that the task is rarely the same across many datasets. Instead we study distribution shifts that are created when applying data augmentation techniques . We study the problem on the popular CIFAR10 , Street View House Numbers (SVHN) , and Fashion MNIST dataset . As a classifica- tion network, we apply a simple convolutional neural network (see appendix B for more details) and use the Inception-v3 network to compute the FD score. In order to create distribution shifts, we apply data augmentation techniques such as flipping, changing contrast, random cropping, and adding salt and pepper noise. We report the ing classification accuracies and FD scores in Tab. 1. Intriguing about the displayed in Tab. 1 is that yet again we see that larger FD scores lead to worse performance. The numbers however are more nuanced than in the previous text experiments. The example of salt and pepper noise shows how nonlinear the relationship between distribution distance and performance can be. While the performance does decrease with increasing distance, it does so at a different scale than observed for the other data augmentation techniques. Nevertheless, these suggest a direct application of our insights: Selecting the augmentation methods applied based on the FD score between the augmented training set and a validation set. We leave an empirical validation of this use case to future work. As a final experiment, we venture into speech separation which deals with separating the speech of 2 or more speakers from audio data. This task is unusual because, through the combinatorial way the dataset is generated, it allows for very large datasets. This lets us gauge the influence of distribution shift especially when more than sufficient data is available. Supervised speech separation requires mixed speaker samples of at least two speakers and the corresponding single speaker streams as labels. In the monaural (single-channel) case, one can simply add two single speaker streams to create overlap. The ing combinatorial possibilities create large datasets. We collect data from the Libri Speech Corpus , the Voxceleb2 data set , Common Voice data set and the TED-LIUM speech corpus . We balance the number of speakers and time per speaker for the first four data sets, keeping 707 speakers for each data set and 7.5 minutes per speaker leading to roughly 88 hours in the training set. For the test set, we sample 10 hours from each data set, from a held back set of data with new speakers. We create mixes from equal datasets since they have more consistent recording settings, which resemble real world situations more closely. We adopt the strongest performing configuration of the Conv-TasNet model with an SDRi score of 15.6 on the 2-mix WSJ dataset. Identically to , we use 4 second long samples with a sampling rate of 8k Hz. We report scores as a SDR improvement score (SDRi), obtained through the difference between the SDR of the processed and unprocessed sample. In place of the inception network, we leverage the encoded features of a speaker embedding network to compute FD scores. Detailed hyperparameters and layer configurations are reported in the appendix in Tab. 2. Again, as depicted in Fig. 4, we find a strong correlation (-0.61) between loss in performance and distribution distance between different training and testing distribution pairs. It appears that larger data sets ing in better performance is not the whole story. Fig. 4a shows that a large combined dataset performs worse on the individual test sets than a model trained only on the corresponding training set. The presented experiments substantiate theorem 1 by showing a strong correlation between network performance and data distribution distances, especially visible in Fig. 3. Following popular wisdom, one would be led to believe that more data is all you need. The presented theory and experiments however clearly detail that, while the amount of data is important, ensuring that train and test distribution are close may be similarly significant to perform well on the test set. From the small-scale and real-world experiments we are left with the startling observation that, frequently, neural networks do not find the "true" functional relationship between input and output. If this were the case, distribution shifts between training and testing should have a smaller impact. Whether this problem can be remedied by finding richer function classes or whether it may be inherently unsolvable will have to be investigated. An important aspect of this work is how we measure distribution distances. By using a representation network, we obtain low dimensional embeddings and reduce the effect of noisy data. This embedding is however in itself limited by its features, training data, and objective. To apply the insights of this work, it will therefore be paramount to carefully choose an embedding for each dataset and task, whose features are able to meaningfully model the various data shifts a desired learning algorithm would react to. As an example, a word embedding trained only on English texts will not provide meaningful on other languages and hence is useless for modeling a distribution shift. Through this work, we emphasize the consequences of using models for predictions, which do not share the invariances of the true functional relationship. In this case, data distribution shifts lead to a deterioration of performance. As a remedy to this issue, we propose applying the Fréchet distance to measure the distance of the dataset distributions to infer the degree of mismatch. With this measure, we can deduce important criteria to choose training sets, select data augmentation techniques, and help optimize networks and their invariances. We believe that making the problem explicit and having a way to measure progress through the FD score may allow for a new wave of innovative ideas on how to address generalization under data shifts. The appendix comprises extended details of the work that may add to the understanding without being critical to follow the main text. The following specifies the experimental details of the small-scale insight experiments from section 3. Throughout these experiments, a model needs to approximate a function f (x) given a point x ∈ R, which is sampled from the uniform distribution. We limit sampling of x during training to a specific interval [a, b] and add gaussian noise from N (0, 0.4) to the training labels. We further elaborate the experiments based on the rows of Fig. 1. In rows one and two of Fig. 1, a model is trained to approximate the function f (x) = e −x cos(2πx) on the training interval [−2, 2]. During testing, we shift the test interval by the scalar α = [0, −0.5, 0.5] (images from left to right respectively). Additionally, we apply a zero mean normalization to the input data in the experiments of the second row. Note that the ground truth is invariant to a change in the input distribution and thus stays consistent. In the third row of Fig. 1, x is drawn from Uniform[−6, 6] during training, while the test set samples lie outside of the training domain in [−10, −6] and. The functions from left to right are given as follows: For all task we use the same neural network as a model. We train a two-layer feedforward neural network with 8 hidden units in its first layer followed by a ReLU activation function and a linear layer with one output unit in its second layer. In total the network consists of 25 parameters including biases, which we optimize using the mean squared error and Adam optimizer . To complete the description of the experiment in section 5.1, we offer a more detailed description of the preprocessing and network details. Before a model can use the text data from the amazon review dataset, several processing steps are necessary to convert text to numeric representations. Each review contains several information fields. However, we only utilize the overall score and the review text, from which we remove capitalization, punctuation and multiple digit numbers. The overall score is split into positive and negative reviews, while ratings of 3 are removed. The vocabulary size is limited to 10,000 words ing in less frequent words being replaced by the < UNK > symbol. To achieve consistent sentence lengths, we cut or pad reviews to a fixed length of 256 with padded words set to the symbol < PAD >. As a last processing step, we tokenize the training data and apply the mapping to the paired test set. We report the complete classification accuracies and FD scores in Tab. 3 between each category used as a training and test set. The classification network consists of an embedding, a global average pooling, and three dense layers of size 32, 16 and 1, respectively. ReLUs are applied after the first two dense layers and a Sigmoid activation function is applied to the last layer. We use Adam and the binary cross-entropy loss for optimization with a learning rate of 0.01. Again we offer complementary details to the vision classification experiments with data augmentation. Especially, we provide more details on the examined datasets, augmentations, and classification network. The CIFAR10 dataset consists of 32 × 32 colored natural images divided into 10 classes. The training and test set consists of 50,000 images and 10,000 images, respectively. The Street View House Numbers (SVHN) dataset, also contains 10 categories, one for each digit between 1 and 10. For SVHN, training set and test set consist of 73,257 images and 26,032 images, respectively. The Fashion MNIST dataset consists of 60,000 train and 10,000 test images. The dataset contains grayscale images of clothes again classified into 10 classes. For data augmentation we offer complementary details. Random cropping reduces the image size to 24 × 24 and is only applied to the middle of images in the test set. We differentiate between only cropping and cropping and resize, in which the images are rescaled back to 32 × 32. However, the test set remains unchanged when rescaling. For salt and pepper noise, we set distorted pixels equiprobably to black or white with a probability of 33%. For random contrast adjustment, we set the upper and lower limit to 1.8 and 0.2, respectively. The classification network is based on a CNN structure. Convolutional layers of size 64 use a kernel size of 5×5 followed by a ReLu activation function, a max pooling layer with a kernel size of 3×3 and stride 2, and a local response normalization (a) with a depth radius of 4, a bias of 1, α equal to 0.001 9, and β set to 0.75. After two blocks three fully-connected layers of size 384, 192 and 10 are used, with ReLu activations after the first two and a softmax layer following the last layer. For optimization, we use the cross-entropy loss and gradient descent with a initial learning rate of 0.1 and an exponential decay rate of 0.1. Additionally, we use exponential moving average on the weights with a decay factor of 0.9999. In order to compute FD scores, we use the Inception-v3 network, which is available pretrained in the Tensorflow library . Images are resized to 299 × 299 to fit the input dimension of the network, which was pretrained on Imagenet . The ing feature representations are retrieved from the pool-3 layer with 2048 output dimensions. Making the description of the speech separation experiments more comprehensive, we offer the following additional implementation details for the speech embedding network for which we adopt the generalized end-to-end loss (GE2E) proposed by. We train a network build on convolutional and LSTM layers. The inputs are based on 3 second long samples transformed into the Short-Time Fourier Transform (STFT) spectrogram and the Mel-frequency cepstral coefficients (MFCC). For the spectrogram creation, we use a hop size of 6 ms and a window length of 23 ms. We apply a simple 1-dimensional convolutional layer with a kernel size of 1 to the MFCC to match the number of frequency bins of the STFT, such that we can use the concatenation of both as an input to our network. Tab. 2 summarizes the network following the input, predicting a 100 dimensional speaker embedding. Table 2: Detailed specifications of the speaker embedding network trained with the GE2E Loss applied to calculate FD scores in section 5.3. The network is based on the densely connected network , fully-connected layers (FC), LSTMs , and Statistical Pooling . As an optimizer, we use Adam with a learning rate of 0.0001. Filter Sizes Filters

represent this text to find a much much longer article about it with say hundreds-thousands of words.

The Frechet Distance between train and test distribution correlates with the change in performance for functions that are not invariant to the shift.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We introduce a new procedural dynamic system that can generate a variety of shapes that often appear as curves, but technically, the figures are plots of many points. We name them spiroplots and show how this new system relates to other procedures or processes that generate figures. Spiroplots are an extremely simple process but with a surprising visual variety. We prove some fundamental properties and analyze some instances to see how the geometry or topology of the input determines the generated figures. We show that some spiroplots have a finite cycle and return to the initial situation, whereas others will produce new points infinitely often. This paper is accompanied by a JavaScript app that allows anyone to generate spiroplots.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A new, very simple dynamic system is introduced that generates pretty patterns; properties are proved and possibilities are explored

GAN representations are examined in detail, and sets of representation units are found that control the generation of semantic concepts in the output.