Question: Represent the scientific article to find a short TLDR:

Question: We propose a non-adversarial feature matching-based approach to train generative models. Our approach, Generative Feature Matching Networks (GFMN), leverages pretrained neural networks such as autoencoders and ConvNet classifiers to perform feature extraction. We perform an extensive number of experiments with different challenging datasets, including ImageNet. Our experimental demonstrate that, due to the expressiveness of the features from pretrained ImageNet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art for challenging benchmarks such as CIFAR10 and STL10. One of the key research focus in unsupervised learning is the training of generative methods that can model the observed data distribution. Good progress has been made in recent years with the advent of new approaches such as generative adversarial networks (GANs) BID10 and variational autoencoders (VAE) BID19 which use deep neural networks as building blocks. Both methods have advantages and disadvantages, and a significant number of recent works focus on addressing their issues BID20 BID5. While the main disadvantage of VAEs is the generation of blurred images, the main issue with GANs is the training instability due to the adversarial learning. Feature matching has been explored to improve the stability of GANs BID41. The key idea in feature matching GANs (FM-GANs) is to use the discriminator network as a feature extractor, and guide the generator to generate data that matches the feature statistics of the real data. Concretely, the objective function of the generator in FM-GAN consists in minimizing the mean squared error of the average features of a minibatch of generated data and a minibatch of real data. The features are extracted from one single layer of the discriminator. FM-GAN is somewhat similar to methods that use maximum mean discrepancy (MMD) BID11 BID12. However, while in FM-GAN the objective is to match the mean of the extracted features, in MMD-based generative models BID9, one normally aims to match all the moments of the two distributions using a Gaussian kernel. Although MMD-based generative models have strong theoretical guarantees, these models normally perform much worse than GANs on challenging benchmarks BID23.In this work, we focus on answering the following research question: can we train effective generative models by performing feature matching on features extracted from a pretrained neural networks? In other words, we would like to know if adversarial training of the feature extractor together with the generator is a requirement for training effective generators. Towards answering this question, we propose Generative Feature Matching Networks (GFMN), a new feature matching-based approach to train generative models that uses features from pretrained neural networks, breaking away from the problematic min/max game completely. Some interesting properties of the proposed method include: the loss function is directly correlated to the generated image quality; mode collapsing is not an issue; the same pretrained feature extractor can be used across different datasets; and both supervised (classifiers) and unsupervised (autoencoder) models can be used as feature extractors. We perform an extensive number of experiments with different challenging datasets, including ILSVRC2012 (henceforth Imagenet) BID37. We demonstrate that, due to the expressiveness of the features from pretrained Imagenet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art for challenging benchmarks such as CIFAR10 and STL10. Moreover, we show that the same feature extractor is effective across different datasets. The main contributions of this work can be summarized as follows: We propose a new effective feature matching-based approach to train generative models that does not use adversarial learning, have stable training and achieves state-of-the-art ; We propose an ADAM-based moving average method that allows effective training with small minibatches; Our extensive quantitative and qualitative experimental demonstrate that pretrained autoencoders and deep convolutional net (DCNN) classifiers can be effectively used as feature extractors for the purpose of learning generative models. Let G be the generator implemented as a neural network with parameters θ, and let E be a pretrained neural network with L hidden layers. Our proposed approach consists of training G by minimizing the following loss function: DISPLAYFORM0 where: ||.|| 2 is the L 2 loss; x is a real data point sampled from the data generating distribution p data; z ∈ R nz is a noise vector sampled from the normal distribution N (0, I nz); E j (x), denotes the output vector/feature map of the hidden layer j from E; M ≤ L is the number of hidden layers used to perform feature matching. In practice, we train G by sampling mini-batches of true data and generated (fake) data and optimizing the parameters θ using stochastic gradient descent (SGD) with backpropagation. The network E is used for the purpose of feature extraction only and is kept fixed during the training of G. A natural choice of unsupervised method to train a feature extractor is the autoencoder framework. The decoder part of an AE consists exactly of an image generator that uses features extracted by the encoder. Therefore, by design, the encoder network should be a good feature extractor for the purpose of generation. Let E and D be the encoder and the decoder networks with parameters φ and ψ, respectively. We pretrain the autoencoder using mean squared error (MSE): DISPLAYFORM0 or the Laplacian pyramid loss BID25: DISPLAYFORM1 where L j (x) is the j-th level of the Laplacian pyramid representation of x. The Laplacian pyramid loss provides better signal for learning the high frequencies of the images and overcome some of the known issues of the blurry images that one would get with a simple MSE loss. recently demonstrated that the Lap 1 loss produces better than L 2 loss for both autoencoders and generative models. Another attractive feature of the autoencoder framework is that the decoder network can be used to initialize the parameters of the generator, which can make the training of the generator easier by starting in a region closer to the data manifold. We use this option in our experiments and show that it leads to significantly better . Different past work has shown the usefulness and power of the features extracted from DCNNs pretrained on classification tasks BID42. In particular, features from DCNNs pretrained on ImageNet BID37 have demonstrated an incredible value for a different number of tasks. In this work, we perform experiments where we use different DCCNs pretrained on ImageNet to play the role of the feature extractor E. Our hypothesis is that ImageNet-based features are powerful enough to allow the successful training of (cross-domain) generators by feature matching. From feature matching loss to moving averages. In order to train with the (mean) feature matching loss, one would need large mini-batches for generating a good estimate of the mean features. When using images larger than 32×32 and DCNNs that produce millions of features, this can easily in memory issues. To alleviate this problem we propose to use moving averages of the difference of means of real and generated data. Instead of computing the (memory) expensive feature matching loss in Eq. 1, we propose to keep moving averages v j of the difference of feature means at layer j between real and generated data so that DISPLAYFORM0 where N is the minibatch size. Using these moving averages we replace the loss given in Eq. 1 by DISPLAYFORM1 where v j is a moving average on ∆ j, the difference of the means of the features extracted by the j-th layer of E: DISPLAYFORM2 The moving average formulation of features matching given in Eq. 2, gives a major advantage on the naive formulation of Eq. 1, since we can now rely on v j to get a better estimate of the population feature means of real and generated data while using a small minibatch of size N. In order to obtain a similar using the feature matching loss given in Eq. 1, one would need a minibatch with a large size N, which becomes problematic as the number of features becomes large. ADAM moving average: from SGD to ADAM updates. Note that for a rate α, the moving average v j has the following update: DISPLAYFORM3 It is easy to see that the moving average is a gradient descent update on the following loss: DISPLAYFORM4 Hence, writing the gradient update with learning rate α we have equivalently: DISPLAYFORM5 With this interpretation of the moving average we propose to get a better estimate of the moving average using the ADAM optimizer BID18 on the loss of the moving average given in Eq. 4, such that DISPLAYFORM6 function is computed as follows: DISPLAYFORM7 where x is the gradient for the loss function in Eq. 4, t is the iteration number, m t is the first moment vector at iteration t, u t is the second moment vector at iteration t, β 1 =.9, β 2 =.999 and = 10 −8 are constants. m 0 and u 0 are initialized as proposed by BID18. We refer the reader to BID18 for a more detailed description of the ADAM optimizer. This moving average formulation, which we call ADAM Moving Average (AMA) promotes stable training when using small minibatches. The main advantage of AMA over simple moving average (MA) is in its adaptive first order and second order moments that ensures a stable estimation of the moving averages v j. In fact, this is a non stationary estimation since the mean of the generated data changes in the training, and it is well known that ADAM works well for such online and non stationary losses BID18.In Section 4.2.3 we provide experimental supporting: The memory advantage that the AMA formulation of feature matching offers over the naive implementation of feature matching; The stability advantage and improved generation that AMA allows when compared to the naive implementation of the simple MA. Features from DCNNs pretrained on ImageNet have been used frequently to perform transfer learning in many computer vision tasks BID16. Some previous work uses DCNN features in the context of image generation and transformation. BID7 combines feature based loss with adversarial loss to improve image quality of variational autoencoders (VAE) BID19. BID17 proposes a feature based loss that uses features from different layers of the VGG-16 neural network and is effective for image transformation task such as style transfer and super-resolution. BID17 confirms the findings of BID28 that the initial layers of the network are more related to content while the last layers are more related to style. Our proposed approach is closely related to the recent body of work on MMD-based generative models BID9 BID23 BID2 BID36. In fact, our method is a type of MMD where we (only) match the first moment of the transformed data. Among the approaches reported in the literature, the closest to our method is the Generative Moment Matching Network + Autoencoder (GMMN+AE) proposed by. In GMMN+AE, the objective is to train a generator G that maps from a prior uniform distribution to the latent code learned by a pretrained AE. To generate a new image, one samples a noise vector z from the prior, maps it the AE latent space using G, then uses the (frozen) decoder to map from the AE latent space to the image space. One key difference in our approach is that our generator G maps from the z space directly to the data space, such as in GANs BID10. Additionally the dimensionality of the feature space that we use to perform distribution matching is orders of magnitude larger than the dimensionality of the latent code normally used in GMMN+AE. BID23 demonstrate that GMMN+AE is not competitive with GANs for challenging datasets such as CIFAR10. Recent MMD-based generative models have demonstrated state-of-the-art with the use of adversarial learning to train the MMD kernel as a replacement of the fixed Gaussian kernel in GMMN BID23 BID2. Additionally, BID36 recently proposed a method to perform online learning of the moments while training the generator. Our proposed method differs from these previous approaches where we use a frozen pretrained feature extractor to perform moment matching. proposed the Generative Latent Optimization (GLO) model, a generative approach that jointly optimizes the model parameters and the noise input vectors z. GLO models obtain competitive for CelebA and LSUN datasets without using adversarial training. also demonstrated that the Laplacian pyramid loss is an effective way to improve the performance of non-adversarial methods that use reconstruction loss. Our work relates also to plug and play generative models of BID33 where a pretrained classifier is used to sample new images, using MCMC sampling methods. Our work is also related to AE-based generative models variational autoencoder (VAE) BID19, adversarial autoencoder (AAE) BID29 and Wasserstein autoencoder (WAE) BID40. While in VAE and WAE a penalty is used to impose a prior distribution on the hidden code vector of the AE, in AAE an adversarial training procedure is used for that purpose. In our method, the aim is to get a generative model out of a pretrained autoencoder. We fix the pretrained encoder to be the discriminator in a GAN like setting. Another recent line of work that involves the use of AEs in generative models consists in applying AEs to improve GANs stability. BID43 proposed an energy based approach where the discriminator is replaced by an autoencoder. BID41 augments the training loss of the GAN generator by including a feature reconstruction loss term that is computed as the mean squared error of a set of features extracted by the discriminator and their reconstructed version. The reconstruction is performed using an AE trained on the features extracted by the discriminator for the real data. Datasets: We evaluate our proposed approach on MNIST (60k training, 10k test images, 10 classes), CIFAR10 BID21 ) (50k training, 10k test images, 10 classes), STL10 BID6 ) (5K training, 8k test images, 100k unlabeled, 10 classes), CelebA BID26 ) (200k images) and different portions of ImageNet BID37 datasets. MNIST and STL10 images are rescaled to 32×32, while CelebA and ImageNet images are rescaled to 64×64. CelebA images are center cropped to 160×160 before rescaling. In our experiments with all datasets but ImageNet, our generator G uses a DCGAN-like architecture. For CIFAR10, STL10 and CelebA, we use two extra layers as commonly used in previous works BID13. For ImageNet, we use a Resnet-based generator such as the one in BID30. More details about the architectures can be found in Appendix A.2.Autoencoder Features: For most AE experiments, we use an encoder network whose architecture is similar to the discriminator in DCGAN (strided convolutions). We use batch normalization and ReLU non-linearity after each convolution. We set the latent code size to 8, 128, 128, and 512 for MNIST, CIFAR10, STL10 and CelebA, respectively. To perform feature extraction, we get the output of each ReLU in the network as well as the output of the very last layer, the latent code. Additionally, we also perform some experiments where the encoder uses a VGG13 architecture. The decoder network D uses a network architecture similar to our generator G. More details in Appendix A.2.Classifier Features: We perform our experiments on classifier features with VGG19 BID39 and Resnet18 networks BID14 which we pretrained using the whole ImageNet dataset with 1000 classes. More details about the pretrained ImageNet classifiers can be found in Appendices A.2 and A.3. We train GFMN with an ADAM optimizer and keep most of the hyperparameters fixed for the different datasets. We use n z = 100 and minibatch size 64. When using autoencoder features, we set the learning rate to 5 × 10 −6 when G is initialized with D, and to 5 × 10 −5 when it is not. When using features from ImageNet classifiers, we set set the learning rate to 1 × 10 −4. We use ADAM moving average (Sec. 2.4) in all reported experiments. In this section, we present experimental on the use of pretrained encoders as feature extractors. The first two rows of in Tab. 1 show GFMN performance in terms of Inception Score (IS) and Fréchet Inception Distance (FID) BID15 for CIFAR10 in the case where the (DCGAN-like) encoder is used as feature extractor. The use of a pretrained decoder D to initialize the generator gives a significant boost in both IS and FID. A visual comparison that corroborates the quantitative can be found in Appendix A.5. In Figures 1a, 1b, and 1c, we present random samples generated by GFMN when trained with MNIST, CIFAR10 and CelebA datasets, respectively. For each dataset, we train its respective AE using the (unlabeled) training set. DISPLAYFORM0 Figure 1: Generated samples from GFMN using pretrained encoder as feature extractor. The last six rows in Tab. 1 present the IS and FID for our best configurations that use ImageNet pretrained VGG19 and Resnet18 as feature extractors. Yhere is a large boost in performance when ImageNet classifiers are used as feature extractors instead of autoencoders. Despite the classifiers being trained on data from a different domain (ImageNet vs. CIFAR10), the classifier features are significantly more effective. In all cases, the use of an initialized generator improves the . However, the improvements are much less significant when compared to the one obtained for the encoder feature extractor. We perform an additional experiment where we use simultaneously VGG19 and Resnet18 as feature extractors, which increases the number of features to 832K. This last configuration gives the best performance for both CIFAR10 and STL10. FIG0 show random samples from the GFMN VGG19+Resnet18 model, where no init. of the generator is used. In Tab. 2, we report IS and FID for increasing number of layers (i.e. number of features) in our extractors VGG19 and Resnet18. We select up to 16 layers for VGG19 and 17 layers for Resnet18, which means that we excluded the output of fully connected layers. Using more layers dramatically improves the performance of both feature extractors, reaching (IS) peak performance when the maximum number of layers is used. The in Tab. 1 are better than the ones in Tab. 2 because, for the former, we trained the models for a longer number of epochs. All models in Tab. 2 are trained for 391K generator updates, while VGG19 and Resnet18 models in Tab. 1 are trained for 1.17M updates (we use small learning rates). Note that for both feature extractors, the features are ReLU activation outputs. As a , the encodings may be quite sparse. Figs. 2d, 2e and 2f show generated images when 1, 3, and 9 layers are used for feature matching, respectively (more in Appendix A.7).In order to check if the number of features is the main factor for the performance, we performed an experiment where we trained an autoencoder whose encoder network uses a VGG13 architecture. This encoder produces a total of 244K features. We pretrained the autoencoder we both CIFAR10 and ImageNet datasets, so to compare the impact of the autoencoder training set size. The for this experiment are in the 3rd and 4th rows of Tab. 1 (Encoder (VGG13)). Although there is some improvement in both IS and FID, specially when using the Encoder pretrained with ImageNet, the boost is not comparable with the one obtained by using a VGG19 classifier. In other words, features from classifiers are significantly more informative than autoencoder features for the purpose of training generators by feature matching. In this section, we present experimental that evidence the advantage of our proposed ADAM moving average (AMA) over the simple moving average (MA). The main benefit of AMA is the promotion of stable training when using small minibatches. The ability to train with small minibatches is essential due to GFMN's need for large number of features from DCNNs, which becomes a challenge in terms of GPU memory usage. For instance, our Pytorch BID34 implementation of GFMN can only handle minibatches of size up to 160 when using VGG19 as a feature extractor and image size 64×64 on a Tesla K40 GPU w/ 12GB of memory. A more optimized implementation minimizing Pytorch's memory overhead could, in principle, handle somewhat larger minibatch sizes (as could a more recent Tesla V100 w/ 16 GB). However, if we increase the image size or the feature extractor size, the memory footprint increases quickly and we will always run out of memory when using larger minibatches, regardless of implementation or hardware. For the experiments presented in this section, we use CelebA as the training set, and the feature extractor is the encoder from an autoencoder that follows a DCGAN-like architecture. We use this feature extractor because it is smaller than VGG19/Resnet18 and allows for minibatches of size up to 512 for image size 64×64. FIG1 shows generated images from GFMN when trained with either MA or our proposed AMA. For MA, we present generated images for GFMN trained with four different batch sizes: 64, 128, 256 and 512 (Figs. 3a, 3b, 3c and 3d, respectively). For AMA, we show for two different minibatch sizes: 64 and 512 (Figs. 3e and 3f, respectively). We can note that the minibatch size has a huge impact in the quality of generated images when training with MA. With minibatches smaller than 512 FIG1 ), almost all images generated by GFMN trained with MA are quite damaged. On the other hand, when using AMA, GFMN generates much better images even with minibatch size 64 FIG1. For AMA, increasing the minibatch size from 64 to 512 FIG1 does not seem to improve the quality of generated images for the given dataset and feature extractor. In Appendix A.9, we show a comparison between MA and AMA when VGG19 ImageNet classifier is used as a feature extractor. A minibatch size of 64 is used for that experiment. We can see in FIG9 that AMA also has a very positive effect in the quality of generated images when a stronger feature extractor is used. An alternative for training with larger minibatches would be the use of multi-GPU, multi-node setups. However, performing large scale experiments is beyond the scope of the current work. Moreover, many practitioners do not have access to a GPU cluster, and the availability of methods that can work on a single GPU with small memory footprint is essential. An important advantage of GFMN over adversarial methods is its training stability. FIG2 shows the evolution of the generator loss per epoch with some generated examples for an experiment where AMA is used. There is a clear correlation between the quality of generated images and the loss. Moreover, mode collapsing was not observed in our experiments with AMA. In order to evaluate the performance of GFMN for an even more challenging dataset, we trained GFMN VGG19 with different portions of the ImageNet dataset. FIG3 shows some (cherry picked) images generated by GFMN VGG19 trained on the ImageNet subset that contains different dog breeds (same as used in FIG0). The are quite impressive given that we perform unconditional generation. FIG3 presents (randomly sampled) images generated by GFMN VGG19 trained with the daisy portion of ImageNet. More can be found in Appendix A.1. In Tab. 3, we compare GFMN with different adversarial and non-adversarial approaches for CIFAR10 and STL10. The table includes for recent models that, like ours, use a DCGAN-like (or CNN) architecture in the generator and do not use CIFAR10/STL10 labels while training the generator. Despite using a frozen cross-domain feature extractor, GFMN outperforms the other systems in terms of FID for both datasets, and achieves the best IS for CIFAR10.We performed additional experiments with a WGAN-GP architecture where: the discriminator is a VGG19 or a Resnet18; the discriminator is pretrained on ImageNet; the generator is pretrained on CIFAR10 through autoencoding. The objective of the experiment is to evaluate if WGAN-GP can benefit from DCNN classifiers pretrained on ImageNet. Although we tried different hyperparameter combinations, we were not able to successfully train WGAN-GP with VGG19 or Resnet18 discriminators. More details about this experiment in Appendix A.8. This work is driven towards answering the question of whether one can train effective generative models by performing feature matching on features extracted from pretrained neural networks. The goal is to avoid adversarial training, breaking away from the problematic min/max game completely. According to our experimental , the answer to our research question is yes. We achieve successful non-adversarial training of generative feature matching networks by introducing different key ingredients: a more robust way to compute the moving average of the mean features by using ADAM optimizer, which allows us to use small minibatches; the use of features from all layers of pretrained neural networks; the use of features from multiple neural networks at the same time (VGG19 + Resnet18); and the initialization of the generator network. Our quantitative in Tab. 3 show that GFMN achieves better or similar compared to the state-of-the-art Spectral GAN (SN-GAN) BID30 for both CIFAR10 and STL10. This is an impressive for a non-adversarial feature matching-based approach that uses pretrained cross-domain feature extractors and has stable training. When compared to other MMD approaches BID23 3.47±.03 GMMN+AE BID23 3.94±.04 VAE BID27 5.62 MMD GAN BID23 6.17±.07 MMD dist GAN BID2 6.39±.04 40.2 / -WGAN BID30 6.41±.11 42.6 / -7.57±.10 64.2 MMDrq GAN BID2 6.51±.03 39.9 / -WGAN-GP BID30 6.68±.06 40.2 / -8.42±.13 55.1 / -McGAN 6.97±.10 SN-GANs BID30 7.58±.12 25.5 / -8.79±.14 43.2 / -MoLM-1024 BID36 7.55±.08 25.0 / 20.3 MoLM-1536 BID36 7.90±.10 23.3 / 18.9 BID9 BID23 BID2 BID36, GFMN presents important distinctions (some of them already listed in Sec. 3) which make it an attractive alternative. Compared to GMMN and GMMN+AE, we can see in TAB2 that GFMN achieves far better . In Appendix A.10, we also show a qualitative comparison between GFMN and GMMN . The main reason why GFMN are significantly better than GMMN is because GFMN uses a strong, robust kernel function (a pretrained DCNN), which, together with our AMA trick, allows the training with small minibatches. On the other hand, the Gaussian kernel used in GMMN requires a very large minibatch size in order to work well, which is impractical due to memory limitations and computational cost. Compared to recent adversarial MMD methods (MMD GAN) BID23 BID2 ) GFMN also presents significantly better while avoiding the problematic min/max game. GFMN achieves similar to the Method of Learned Moments (MoLM) BID36, while using a much smaller number of features to perform matching. The best performing model from BID36, MoLM-1536, uses around 42 million moments to train the CIFAR10 generator, while our best GFMN model uses around 850 thousand moments/features only, almost 50x less. In other words, MoLM-1536 can be used in large-scale environments only, while GFMN can be used in single GPU environments. One may argue that the best from GFMN are obtained with feature extractors that were trained in a supervised manner (classifiers). However, there are two important points to note: we use a cross domain feature extractor and do not use labels from the target datasets (CIFAR10, STL10, MNIST, CelebA); since the accuracy of the classifier does not seem to be the most important factor for generating good features (VGG19 classifier produces better features although it is less accurate than Resnet18, see Appendix A.3); we are confident that GFMN will also achieve state-of-the-art when trained with features from classifiers trained using unsupervised methods such as the one recently proposed by BID4. In this work, we introduced GFMN, an effective non-adversarial approach to train generative models. GFMNs are demonstrated to achieve state-of-the-art while avoiding the challenge of defining and training an adversarial discriminator. Our feature extractors can be easily obtained and provide for a robust and stable training of our generators. Some interesting open questions include: what type of feature extractors other than classifiers and auto-encoders are good for GFMN? What architecture designs are better suited for the purpose of feature extraction in GFMN?A APPENDIX We trained GFMN VGG19 with different portions of the ImageNet dataset using images of size 64×64. Although we adopted this image size due to speedup and memory efficient purposes, GFMN is also effective to generate images of larger sizes. FIG4 shows (randomly sampled) images generated by GFMN VGG19 trained with the following ImageNet portions: pizza FIG4, daisy FIG4, breeds of dogs FIG4 and Persian cats FIG4. Note that in this experiment the generators are trained from scratch, there is not initialization of the generators. While pizza, daisy and Persian cats consist in single ImageNet classes, the breeds of dogs portion (same as use in BID43) consists in multiple classes and therefore is a more challenging task since we doing unconditional generation. For the experiments with ImageNet, we use a Resnet generator similar to the one used by BID30. In TAB4 we detail the neural net architectures used in our experiments. In both DCGAN-like generator and discriminator, an extra layer is added when using images of size 64×64. In VGG19 architecture, after each convolution, we apply batch normalization and ReLU. The Resnet generator is used for ImageNet experiments only. −1 learning rate, 0.9 momentum term, and weight decay set to 5 × 10 −4. We pick models with best top 1 accuracy on the validation set over 100 epochs of training; 29.14% for VGG19 (image size 32×32), and 39.63% for Resnet18 (image size 32×32). When training the classifiers we use random cropping and random horizontal flipping for data augmentation. When using VGG19 and Resnet18 as feature extractors in GFMN, we use features from the output of each ReLU that follows a conv. layer, for a total of 16 layers for VGG and 17 for Resnet18. We evaluate our models using two quantitative metrics: Inception Score (IS) and Fréchet Inception Distance (FID) BID15. We followed the same procedure used in previous work to calculate IS BID30 BID36. For each trained generator, we calculate the IS for randomly generated 5000 images and repeat this procedure 10 times (for a total of 50K generated images) and report the average and the standard deviation of the IS.We compute FID using two sample sizes of generated images: 5K and 50K. In order to be consistent with previous works BID30 BID36 and be able to directly compare our quantitative with theirs, the FID is computed as follows:• CIFAR10: the statistics for the real data are computed using the 50K training images. This (real data) statistics are used in the FID computation of both 5K and 50K samples of generated images. This is consistent with both BID30 and BID36 procedure to compute FID for CIFAR10 experiments.• STL10: when using 5K generated images, the statistics for the real data are computed using the set of 5K (labeled) training images. This is consistent with the FID computation of BID30. When using 50K generated images, the statistics for the real data are computed using a set of 50K images randomly sampled from the unlabeled STL10 dataset. FID computation is repeated 3 times and the average is reported. There is very small variance in the FID . The use a of pretrained decoder D to initialize the generator gives a good boost in performance when the feature extractor is the decoder from a pretrained autoencoder (see Sec. 4.2.1). In Fig. 7, we show a visual comparison that demonstrates the effect of using the pretrained decoder D to initialize the generator. The generators for the images in the first row (7a, 7b, 7c) were not initialized, while the generators for the images in the second row (7d, 7e, 7f) were initialized. For the three datasets, we can see a significant improvement in image quality when the generator is initialized with the decoder. We use the Laplacian pyramid loss to train CIFAR10 and CelebA AEs. However, L2 loss gives almost as good as Lap1 loss. DISPLAYFORM0 Figure 7: Generated samples from GFMN using pretrained encoder as feature extractor. Visual comparison of models generated without (top row) and with (bottom row) initialization of the generation. The experimental in Sec. 4.2.2 demonstrate that cross-domain feature extractors based on DCNNs classifiers are very successful. For instance, we successfully used a VGG19 pretrained on ImageNet to train a GFMN generator for the CelebA dataset. Here, we investigate the impact of the pretraining of autoencoder-based feature extractors in a cross-domain setting. The objective is to further verify if GFMN is dependent on pretraining the autoencoder feature extractors on the same training data where the generator is trained. In Tab. 6, we show the for different combinations of cross-domain feature extractors and G initialization for STL10 and CIFAR10. The subscript indicates which dataset was used for pretraining. We can see in Tab. 6 that, CIFAR10 using E STL and D STL has similar performance (even better IS) to using E CIFAR and D CIFAR. There is a performance drop when using E CIFAR and D CIFAR to train a STL10 generator. We believe this drop is related to the training set size. STL10 contains 100K (unlabeled) training examples while CIFAR10 contains 50K training images. FIG6 shows generated images from generators that were trained with a different number of layers employed to feature matching. In all the in FIG6, the VGG19 network was used to perform feature extraction. We can see a significant improvement in image quality when more layers are used. The objective of the experiments presented in this section is to evaluate if WGAN-GP can benefit from DCNN classifiers pretrained on ImageNet. In the experiments, we used a WGAN-GP architecture where: the discriminator is a VGG19 or a Resnet18; the discriminator is pretrained on ImageNet; the generator is pretrained on CIFAR10 through autoencoding. Although we tried different hyperparameter combinations, we were not able to successfully train WGAN-GP with VGG19 or Resnet18 discriminators. Indeed, the discriminator, being pretrained on ImageNet, can quickly learn to distinguish between real and fake images. This limits the reliability of the gradient information from the discriminator, which in turn renders the training of a proper generator extremely challenging or even impossible. This is a well-known issue with GAN training BID10 where the training of the generator and discriminator must strike a balance. This phenomenon is covered in ) Section 3 (illustrated in their FIG0) as one motivation for work like Wassertein GANs. If a discriminator can distinguish perfectly between real and fake early on, the generator cannot learn properly and the min/max game becomes unbalanced, having no good discriminator gradients for the generator to learn from, producing degenerate models. FIG8 shows some examples of images generated by the unsuccessfully trained models. In this appendix, we present a comparison between the simple moving average (MA) and ADAM moving average (AMA) for the case where VGG19 ImageNet classifier is used as a feature extractor. This experiment uses a minibatch size of 64. We can see in FIG9 that AMA has a very positive effect in the quality of generated images. GFMN trained with MA produces various images with some sort of crossing line artifacts. A.10 VISUAL COMPARISON BETWEEN GFMN AND GMMN GENERATED IMAGES. FIG10 shows a visual comparison between images generated by GFMN FIG10 ) and Generative Moment Matching Networks (GMMN) FIG10 ). GMMN generated images were obtained from BID23. In this appendix, we present a visual comparison between images generated by sampling directly from decoders of pretrained autoencoders, and images generated by GFMN generators which were initialized by the decoders. In FIG0 the images in the top row FIG0 and 12c) were generated by decoders trained using CelebA, CIFAR10 and STL10, respectively. Images in the bottom row FIG0 and 12f) were generated using GFMN generators that were initialized with CelebA, CIFAR10 and STL10 decoders, respectively. As expected, sampling from the decoder produces completely noisy images because the latent space is not aligned with the prior distribution p z. GFMN uses the pretrained decoder as a better starting point and learns an effective implicit generative model, as we can see in Figs. 12d, 12e and 12f. Nevertheless, as demonstrated in Sec. 4.2.1, GFMN is also very effective without generator initialization, specially when using VGG19/Resnet18 feature extractors. Therefore, generator initialization is an interesting positive feature of GFMN, but not an essential aspect of the method. In this appendix, we assess whether GFMN is impacted by computing the mean features of the real data in a minibatch-wise fashion (computed in the minibatch and carried along with a moving average) vs. computing in a global manner (pre-computing the mean features using the whole training dataset, and keeping it fixed during the training). Note that this is for the real data only, for the fake data, we need to use a moving average because its mean features change a lot throughout the training process. In FIG1, we show generated images from GFMN trained with either simple Moving Average (MA) (13a, 13b, 13d and 13e) or Adam Moving Average (AMA) (13c and 13f). For MA, two minibatch sizes (mbs) are used: 64 and 128. Images in the top row were generated by models that perform feature matching using the minibatch-wise mean of the features from the real data, while the models that generated the images in the bottom row used global mean (gm) features computed in the whole CelebA dataset. We can see in FIG1 that using the global mean features does not improve the performance when training with MA, and also does not seem to have any impact when training with AMA.A.13 AUTOENCODER FEATURES VS. VGG19 FEATURES FOR CELEBA.In this appendix, we present a comparison in image quality for autoencoder features vs. VGG19 features for the CelebA dataset. We show for both simple moving average (MA) and ADAM moving average (AMA), for both cases we use a minibatch size of 64. In Fig. 14, we show generated images from GFMN trained with either VGG19 features (top row) or autoencoder (AE) features (bottom row). We show images generated by GFMN models trained with simple moving average (MA) and Adam moving average (AMA). We can note in the images that, although VGG19 features are from a cross-domain classifier, they lead to much better generation quality than AE features, specially for the MA case. FIG1. For MA, two minibatch sizes (mbs) are used: 64 and 128. Images in the top row were generated by models that perform feature matching using the minibatch-wise mean of the features from the real data, while the models that generated the images in the bottom row used global mean (gm) features computed in the whole CelebA dataset. FIG2: Generated images from GFMN trained with either VGG19 features (top row) or autoencoder (AE) features (bottom row). We show images generated by GFMN models trained with simple moving average (MA) and Adam moving average (AMA). Although VGG19 features are from a cross-domain classifier, they perform much better than AE features, specially for the MA case.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A new non-adversarial feature matching-based approach to train generative models that achieves state-of-the-art results.

Represent the TLDR for finding the corresponding scientific article.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: We propose a novel architecture for k-shot classification on the Omniglot dataset. Building on prototypical networks, we extend their architecture to what we call Gaussian prototypical networks. Prototypical networks learn a map between images and embedding vectors, and use their clustering for classification. In our model, a part of the encoder output is interpreted as a confidence region estimate about the embedding point, and expressed as a Gaussian covariance matrix. Our network then constructs a direction and class dependent distance metric on the embedding space, using uncertainties of individual data points as weights. We show that Gaussian prototypical networks are a preferred architecture over vanilla prototypical networks with an equivalent number of parameters. We report consistent with state-of-the-art performance in 1-shot and 5-shot classification both in 5-way and 20-way regime on the Omniglot dataset. We explore artificially down-sampling a fraction of images in the training set, which improves our performance. Our experiments therefore lead us to hypothesize that Gaussian prototypical networks might perform better in less homogeneous, noisier datasets, which are commonplace in real world applications. Humans are able to learn to recognize new object categories on a single or small number of examples. This has been demonstrated in a wide range of activities from hand-written character recognition BID10, and motor control BID2 ), to acquisition of high level concepts BID9. Replicating this kind of behavior in machines is the motivation for studying few-shot learning. Parametric deep learning has been performing well in settings with abundance of data. In general, deep learning models have a very high functional expressivity and capacity, and rely on being slowly, iteratively trained in a supervised regime. An influence of a particular example within the training set is therefore small, as the training is designed to capture the general structure of the dataset. This prevents rapid introduction of new classes after training. BID15 In contrast, few-shot learning requires very fast adaptation to new data. In particular, k-shot classification refers to a regime where classes unseen during training must be learned using k labeled examples. Non-parametric models, such as k-nearest neighbors (kNN) do not overfit, however, their performance strongly depends on the choice of distance metric. BID0 Architectures combining parametric and non-parametric models, as well as matching training and test conditions, have been performing well on k-shot classification recently. In this paper we develop a novel architecture based on prototypical networks used in BID16, and train it and test it on the Omniglot dataset BID9. Vanilla prototypical networks map images into embedding vectors, and use their clustering for classification. They divide a batch into support, and query images, and use the embedding vectors of the support set to define a class prototype -a typical embedding vector for a given class. Proximity to these is then used for classification of query images. Our model, which we call the Gaussian prototypical network, maps an image into an embedding vector, and an estimate of the image quality. Together with the embedding vector, a confidence region around it is predicted, characterized by a Gaussian covariance matrix. Gaussian prototypical networks learn to construct a direction and class dependent distance metric on the embedding space. We show that our model is a preferred way of using additional trainable parameters compared to increasing dimensionality of vanilla prototypical networks. Our goal is to show that by allowing our model to express its confidence in individual data points, we reach better . We also experiment with intentionally corrupting parts of our dataset in order to explore the extendability of our method to noisy, inhomogeneous real world datasets, where weighting individual data points might be crucial for performance. We report, to our knowledge, performance consistent with state-of-the-art in 1-shot and 5-shot classification both in 5-way and 20-way regime on the Omniglot dataset. BID9 By studying the response of our model to partially down-sampled data, we hypothesize that its advantage might be more significant in lower quality, inhomogeneous datasets. This paper is structured as follows: We describe related work in Section 2. We then proceed to introduce our methods in Section 3. The episodic training scheme is also presented there. We discuss the Omniglot dataset in Section 4, and our experiments in Section 5. Finally, our are presented in Section 6. Non-parametric models, such as k-nearest neighbors (kNN), are ideal candidates for few-shot classifiers, as they allow for incorporation of previously unseen classes after training. However, they are very sensitive to the choice of distance metric. BID0 Using the distance in the space of inputs directly (e.g. raw pixel values) does not produce high accuracies, as the connection between the image class and its pixels is very non-linear. A straightforward modification in which a metric embedding is learned and then used for kNN classification has yielded good , as demonstrated by BID5, BID18, BID8, and BID1. An approach using matching networks has been proposed in BID17, in effect learning a distance metric between pairs of images. A noteworthy feature of the method is its training scheme, where each mini-batch (called an episode) tries to mimic the data-poor test conditions by sub-sampling the number of classes as well as numbers of examples in each. It has been demonstrated that such an approach improves performance on few-shot classification. BID17 We therefore use it as well. Instead of learning on the dataset directly, it has recently been proposed BID13 to train an LSTM BID6 to predict updates to a few-shot classifier given an episode as its input. This approach is referred to as meta-learning. Meta-learning has been reaching high accuracies on Omniglot BID9, as demonstrated by BID4, and BID12. A task-agnostic meta-learner based on temporal convolutions has been proposed in BID11, currently outperforming other approaches. Combinations of parametric and non-parametric methods have been the most successful in few-shot learning recently. BID16 BID14 BID7 Our approach is specific to classification of images, and does not attempt to solve the problem via meta-learning. We build on the model presented in BID16, which maps images into embedding vectors, and uses their clustering for classification. The novel feature of our model is that it predicts its confidence about individual data points via a learned, image-dependent covariance matrix. This allows it to construct a richer embedding space to which images are projected. Their clustering under a direction and class-dependent distance metric is then used for classification. FIG1 A diagram of the function of the Gaussian prototypical network. An encoder maps an image into a vector in the embedding space (dark circles). A covariance matrix is also output for each image (dark ellipses). Support images are used to define the prototypes (stars), and covariance matrices (light ellipses) of the particular class. The distances between centroids, and encoded query images, modified by the total covariance of a class, are used to classify query images. The distances are shown as dashed gray lines for a particular query point. In this paper, we build on prototypical networks described in BID16. We extend their architecture to what we call a Gaussian prototypical network, allowing the model to reflect quality of individual data points (images) by predicting their embedding vectors as well as confidence regions around them, characterized by a Gaussian covariance matrix. A vanilla prototypical network comprises an encoder that maps an image into an embedding vector. A batch contains a subset of available training classes. In each iteration, images for each class are randomly split into support, and query images. The embeddings of support images are used to define class prototypes -embedding vectors typical for the class. The proximity of query image embeddings to the class prototypes is used for classification. The encoder architectures of vanilla and Gaussian prototypical networks do not differ. The key difference is the way encoder outputs are interpreted, used, and how the metric on the embedding space is constructed. The function of the Gaussian network is presented in FIG1. We use a multi-layer convolutional neural network without an explicit, final fully connected layer to encode images into high-dimensional Euclidean vectors. For a vanilla prototypical network, the encoder is a function taking an image I and transforming it into a vector x as DISPLAYFORM0 where H and W are the height and width of the input image, and C is the number of its channels. D is the embedding dimension of our vector space which is a hyperparameter of the model. θ are the trainable weights of the encoder. For a Gaussian prototypical network, the output of the encoder is a concatenation of an embedding vector x ∈ R D and real vector s raw ∈ R D S relevant to the covariance matrix Σ ∈ R D×D. Therefore DISPLAYFORM1 where D S is the number of degrees of freedom of the covariance matrix. We explore three variants of the Gaussian prototypical network: a) Radius covariance estimate. D S = 1 and only a single real number s raw ∈ R 1 is generated per image. As such the covariance matrix has the form Σ = diag (σ, σ, . . ., σ), where σ is calculated from the raw encoder output s raw. The confidence estimate is therefore not direction-sensitive. b) Diagonal covariance estimate. D S = D and the dimension of the covariance estimate is the same as of the embedding space. s raw ∈ R D is generated per image. Therefore the covariance matrix has the form Σ = diag (σ), where σ is calculated from the raw encoder output s raw. This allows the network to express direction-dependent confidence about a data point. c) Full covariance estimate. A full covariance matrix is output per data point. This method proved to be needlessly complex for the tasks given and therefore was not explored further. We were using 2 encoder architectures: 1) a small architecture, and 2) a big architecture. The small architecture corresponded to the one used in BID16, and we used it to validate our own experiments with respect. The big architecture was used to see the effect of an increased model capacity on accuracy. As a basic building block, we used the sequence of layers in Equation 3. DISPLAYFORM2 Both architectures were composed of 4 such blocks stacked together. We explored 4 different methods of translating the raw covariance matrix output of the encoder into an actual covariance matrix. Since we primarily deal with the inverse of the covariance matrix S = Σ −1, we were predicting it directly. Let the relevant part of the raw encoder output be S raw. The methods are as follows: a) S = 1 + softplus (S raw), where softplus(x) = log (1 + e x) and it is applied componentwise. Since softplus(x) > 0, this guarantees S > 1 and the encoder can only make data points less important. The value of S is also not limited from above. Both of these approaches prove beneficial for training. Our best models used this regime for initial training. b) S = 1 + sigmoid (S raw), where sigmoid(x) = 1/ (1 + e −x) and it is applied componentwise. Since sigmoid(x) > 0, this guarantees S > 1 and the encoder can only make data points less important. The value of S is bounded from above, as S < 2, and the encoder is therefore more constrained. c) S = 1 + 4 sigmoid (S raw) and therefore 1 < S < 5. We used this to explore the effect of the size of the domain of covariance estimates on performance.d) S = offset + scale × softplus (S raw /div), where offset, scale, and div are initialized to 1.0 and are trainable. Our best models used this regime for late-stage training, as it is more flexible and data-driven than a). A key component of the prototypical model is the episodic training regime described in BID16 and modeled on BID17. During training, a subset of N c classes is chosen from the total number of classes in the training set (without replacement). For each of these classes, N s support examples are chosen at random, as well as N q query examples. The encoded embeddings of the support examples are used to define where a particular class prototype lies in the embedding space. The distances between the query examples and positions of class prototypes are used to classify the query examples and to calculate loss. For the Gaussian prototypical network, the covariance of each embedding point is estimated as well. A diagram of the process is shown in FIG1.For a Gaussian prototypical network, the radius or a diagonal of a covariance matrix is output together with the embedding vector (more precisely its raw form is, as detailed in Section 3.1). These are then used to weight the embedding vectors corresponding to support points of a particular class, as well as to calculate a total covariance matrix for the class. The distance d c (i) from a class prototype c to a query point i is calculated as DISPLAYFORM0 where p c is the centroid, or prototype, of the class c, and DISPLAYFORM1 is the inverse of its class covariance matrix. The Gaussian prototypical network is therefore able to learn class and directiondependent distance metric in the embedding space. We found that the speed of training and its accuracy depend strongly on how distances are used to construct a loss. We conclude that the best option is to work with linear Euclidean distances, i.e. d c (i). The specific form of the loss function used is presented in Algorithm 1. A critical part of a prototypical network is the creation of a class prototype from the available support points of a particular class. We propose a variance- where • denotes a component-wise multiplication, and the division is also component-wise. The diagonal of the inverse of the class covariance matrix is then calculated as DISPLAYFORM0 This corresponds to the optimal combination of Gaussians centered on the individual points into an overall class Gaussian, hence the name of the network. The elements of s are effectively 1/σ 2. Equations 5 and 6 therefore correspond to weighting by 1/σ 2. The full algorithm is described in Algorithm 1. To estimate the accuracy of a model on the test set, we classify the whole test set for every number of support points N s = k in the range k ∈ [1, ..19]. The number of query points for a particular k is therefore N q = 20 − N s, as Omniglot provides 20 examples of each class. The accuracies are then aggregated, and for a particular stage of the model training a k-shot classification accuracy as a function of k is determined. Since we are not using a designated validation set, we ensure our impartiality by considering the test for the 5 highest training accuracies, and calculate their mean and standard deviation. By doing that, we prevent optimizing our for the test set, and furthermore obtain error bounds on the ing accuracies. We evaluate our models in 5-way and 20-way test classification to directly compare to existing literature. We used the Omniglot dataset. BID9 Omniglot contains 1623 character classes from 50 alphabets (real and fictional) and 20 hand-written, gray-scale, 105 × 105 pixel examples of each. We down-sampled them to 28 × 28 × 1, subtracted their mean, and inverted them. We were using the recommended split to 30 training alphabets, and 20 test alphabets, as suggested by BID9, and used by BID16. The training set included overall 964 unique character classes, and the test set 659 of them. There was no class overlap between the training and test datasets. We did not use a separate validation set as we did not fine-tune hyperparameters and chose the best performing model based on training accuracies alone (see Section 3.4).To extend the number of classes, we augmented the dataset by rotating each character by 90•, 180•, and 270•, and defined each rotation to be a new character class on its own. The same approach is used in BID17, and BID16. An example of an augmented character is shown in FIG2. This increased the number of classes 4-fold. In total, there were 77,120 images Let the image i embedding vector be x and its true class y for class c in classes DISPLAYFORM0 summed over c end for L ← L/batch size loss per batch for comparability end for •, 180•, and 270 •. Each rotation is then defined as a new class. This enhances the number of classes, but also introduces degeneracies for symmetric characters.in the training set, and 52,720 images in the test set. Due to the rotational augmentation, characters that have a rotational symmetry were nonetheless defined as multiple classes. As even a hypothetical perfect classifier would not be able to differentiate e.g. the character "O" from a rotated "O", 100 % accuracy was not reachable. We conducted a large number of few-shot learning experiments on the Omniglot dataset. For Gaussian prototypical networks, we explored different embedding space dimensionalities, ways of generating the covariance matrix, and encoder capacities (see Section 3.1 for details). We also compared them to vanilla prototypical networks, and showed that our Gaussian variant is a favorable way of using additional trainable parameters compared to increasing embedding space dimensionality. We found that predicting a single number per embedding point (the radius method in Section 3.1) works the best on Omniglot. In general, we explored the size of the encoder (small, and big, as described in Section 3), the Gaussian/vanilla prototypical network comparison, the distance metric (cosine, √ L 2, L 2, and L 2 2), the number of degrees of freedom of the covariance matrix in the Gaussian networks (radius, and diagonal estimates, see Section 3.1), and the dimensionality of the embedding space. We also explored augmenting the input dataset by down-sampling a subset of it to encourage the usage of covariance estimates by the network, and found that this improves (k > 1)-shot performance. We were using the Adam optimizer with an initial learning rate of 2 × 10 −3. We halved the learning rate every 2000 episodes ≈ 30 epochs. All our models were implemented in TensorFlow, and ran on a single NVidia K80 GPU in Google Cloud. The training time of each model was less than a day. We trained our models with N c = 60 classes (60-way classification) at training time, and tested on N ct = 20 classes (20-way) classification. For our best-performing models, we also conducted a final N ct = 5 (5-way) classification test to compare our to literature. During training, each class present in the mini-batch comprised N s = 1 support points, as we found that limiting the number of support points leads to better accuracies. This could intuitively be understood as matching the training regime to the test regime, as done in BID17. The remaining N q = 20 − N s = 19 images per class were used as query points. We verified, provided that the covariance estimate is not needlessly complex, that using encoder outputs as covariance estimates is more advantageous than using the same number of parameters as additional embedding dimension. This holds true for the radius estimate (i.e. one real number per embedding vector), however, the diagonal estimate does not seem to help with performance (keeping the number of parameters equal). This effect is shown in FIG3. The best performing model was initially trained on the undamaged dataset for 220 epochs. The training then continued with 1.5% of images down-sampled to 24 × 24, 1.0% down-sampled to 20 × 20, and 0.5% down-sampled to 16 × 16 for 100 epochs. Then with 1.5% down-sampled to 23 × 23 and 1.0% down-sampled to 17 × 17 for 20 epochs, and 1.0% down-sampled to 23 × 23 for 10 epochs. These choices were quite arbitrary and not optimized over. The purposeful damage to the dataset encouraged usage of the covariance estimate and increased (k > 1)-shot , as demonstrated in Figure 4.The comparison of our models to from literature is presented in Table 1. To our knowledge, our models perform consistently with state-of-the-art in 1-shot and 5-shot classification both in 5-way and 20-way regime on the Omniglot dataset. In 5-shot 5-way classification in particular, we are reaching very close to perfect performance (99.73 ± 0.04 %) and therefore conclude that a more complex dataset is needed for further few-shot learning algorithms development. In order to validate our assumption that the Gaussian prototypical network outperforms the vanilla version due to its ability to predict covariances of individual embedded images and therefore the Figure 4: The effect of down-sampling a part of the training set on k-shot test accuracy. The network trained on purposefully damaged data outperforms the one trained on unmodified data, as it learns to utilize covariance estimates better. Table 1: The of our experiments as compared to other papers. To our knowledge, our models perform consistently with state-of-the-art in 1-shot and 5-shot classification both in 5-way and 20-way regime on the Omniglot dataset. 5-way Model 1-shot 5-shot 1-shot 5-shot Matching networks BID17 93.8% 98.5% 98.1% 98.9% Matching networks BID17 93.5 % 98.7 % 97.9% 98.7% Neural statistician BID3 93.2 % 98.1 % 98.1% 99.5% Prototypical network BID16 96.0 % 98.9 % 98.8% 99. 7% 98.7 ± 0.4% 99.9 ± 0.3% BID12 98.9 % TCML BID11 97.64 ± 0.30% 99.36 ± 0.18% 98.96 ± 0.20% 99.75 ± 0.11% Gauss (radius) (ours) 97.02 ± 0.40% 99.16 ± 0.11% 99.02 ± 0.11% 99.66 ± 0.04% Gauss (radius) damage (ours) 96.94 ± 0.31% 99.29 ± 0.09% 99.07 ± 0.07% 99.73 ± 0.04% possibility to down-weight them, we studied the distribution of predicted values of s (see Section 3.1 for details) for our best performing network for undamaged and damaged test data. The network was trained on partially down-sampled training data. For the undamaged test set, the vast majority of covariance estimates took the same value, indicating that the network did not use its ability to down-weight data points. However, for a partially downsampled test set, the distribution of magnitudes of covariance estimates got significantly broader. We interpret this as a confirmation that the network learned to put less emphasis on down-sampled images. A comparison of both distributions is shown in Figure 5. In this paper we proposed Gaussian prototypical networks for few-shot classification -an improved architecture based on prototypical networks BID16. We tested our models on the Omniglot dataset, and explored different approaches to generating a covariance matrix estimate together with an embedding vector. We showed that Gaussian prototypical networks outperform vanilla prototypical networks with a comparable number of parameters, and therefore that our architecture choice is beneficial. We found that estimating a single real number on top of an embedding vector works better than estimating a diagonal, or a full covariance matrix. We suspect that lower quality, less homogeneous datasets might prefer a more complex covariance matrix estimate. Contrary to BID16, we found that the best are obtained by training in the 1-shot regime. Our are consistent with state-of-the-art in 1-shot and 5-shot classification both in Figure 5: Predicted covariances for the original test set and a partially down-sampled version of it. The Gaussian network learned to down-weight damaged examples by predicting a higher s, as apparent from the heavier tail of the yellow distribution. The distributions are aligned together, as only the difference between the leading edge and a value influence classification.5-way and 20-way regime on the Omniglot dataset. Especially for 5-way classification, our are very close to perfect performance. We got better accuracies (in particular for (k > 1)-shot classification) by artificially down-sampling fractions of our training dataset, encouraging the network to fully utilize covariance estimates. We hypothesize that the ability to learn the embedding as well as its uncertainty would be even more beneficial for poorer-quality, heterogeneous datasets, which are commonplace in real world applications. There, down-weighting some data points might be crucial for faithful classification. This is supported by our experiments with down-sampling Omniglot.

A novel architecture for few-shot classification capable of dealing with uncertainty.

Represent the short summary for finding the corresponding scientific article (long)

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the article for finding a single-sentence TLDR

Question: We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters. Convolutional Neural Networks (CNNs) have powerful pattern-recognition capabilities that have recently given dramatic improvements in important tasks such as image classification BID13. However, as CNNs are increasingly being applied in real-world, safety-critical domains, their vulnerability to adversarial examples BID27 BID15, and their poor uncertainty estimates are becoming increasingly problematic. Bayesian inference is a theoretically principled and demonstrably successful BID26 BID7 framework for learning in the face of uncertainty, which may also help to address the problems of adversarial examples BID9. Unfortunately, Bayesian inference in CNNs is extremely difficult due to the very large number of parameters, requiring highly approximate factorised variational approximations BID1 BID8, or requiring the storage BID16 ) of large numbers of posterior samples (; BID19 .Other methods such as those based on Gaussian Processes (GPs) are more amenable to Bayesian inference, allowing us to compute the posterior uncertainty exactly BID24. This raises the question of whether it might be possible to combine the pattern-recognition capabilities of CNNs with exact probabilistic computations in GPs. Two such approaches exist in the literature. First, deep convolutional kernels parameterise a GP kernel using the weights and biases of a CNN, which is used to embed the input images into some latent space before computing their similarity. The CNN parameters of the ing kernel then have to be optimised by gradient descent. However, the large number of kernel parameters in the CNN reintroduces the risk of overconfidence and overfitting. To avoid this risk, we need to infer a posterior over the CNN kernel parameters, which is as difficult as directly inferring a posterior over the parameters of the original CNN. Second, it is possible to define a convolutional GP BID22 or a Furthermore, we show that two properties of the GP kernel induced by a CNN allow it to be computed very efficiently. First, in previous work it was necessary to compute the covariance matrix for the output of a single convolutional filter applied at all possible locations within a single image BID22, which was prohibitively computationally expensive. In contrast, under our prior, the downstream weights are independent with zero-mean, which decorrelates the contribution from each location, and implies that it is necessary only to track the patch variances, and not their covariances. Second, while it is still necessary to compute the variance of the output of a convolutional filter applied at all locations within the image, the specific structure of the kernel induced by the CNN means that the variance at every location can be computed simultaneously and efficiently as a convolution. Finally, we empirically demonstrate the performance increase coming from adding translationinvariant structure to the GP prior. Without computing any gradients, and without augmenting the training set (e.g. using translations), we obtain 0.84% error rate on the MNIST classification benchmark, setting a new record for nonparametric GP-based methods. For clarity of exposition, we will treat the case of a 2D convolutional NN. The applies straightforwardly to nD convolutions, dilated convolutions and upconvolutions ("deconvolutions"), since they can be represented as linear transformations with tied coefficients (see Fig. 1). The network takes an arbitrary input image X of height H and width DISPLAYFORM0 Each row, which we denote x 1, x 2,..., x C, corresponds to a channel of the image (e.g. C = 3 for RGB), flattened to form a vector. The first activations A (X) are a linear transformation of the inputs. For i ∈ {1, . . ., C }: DISPLAYFORM1 We consider a network with L hidden layers. The other activations of the network, from A (X) up to A (L+1) (X), are defined recursively: i,j corresponds to applying the filter to the µth convolutional patch of the channel x j. DISPLAYFORM2 represents the flattened jth channel of the image that from applying a convolutional filter to φ(A (X)).The structure of the pseudo-weight matrices W where it does, as illustrated in Fig. 1.The outputs of the network are the last activations, A (L+1) (X). In the classification or regression setting, the outputs are not spatially extended, so we have H (L+1) = D (L+1) = 1, which is equivalent to a fully-connected output layer. In this case, the pseudo-weights W Finally, we define the prior distribution over functions by making the filters U i,j and biases b i be independent Gaussian random variables (RVs). For each layer, channels i, j and locations within the filter x, y: DISPLAYFORM0 Note that, to keep the activation variance constant, the weight variance is divided by the number of input channels. The weight variance can also be divided by the number of elements of the filter, which makes it equivalent to the NN weight initialisation scheme introduced by BID10. We follow the proofs by BID18 and BID20 to show that the output of the CNN described in the previous section, A (L+1), defines a GP indexed by the inputs, X. Their proof BID18 proceeds by applying the multivariate Central Limit Theorem (CLT) to each layer in sequence, i.e. taking the limit as N → ∞, then N → ∞ etc, where N is the number of hidden units in layer. By analogy, we sequentially apply the multivariate CLT by taking the limit as the number of channels goes to infinity, i.e. C → ∞, then C → ∞ etc. While this is the simplest approach to taking the limits, other potentially more realistic approaches also exist BID20.The fundamental quantity we consider is a vector formed by concatenating the feature maps (or equivalently channels), a DISPLAYFORM0 This quantity (and the following arguments) can all be extended to the case of countably finite numbers of input points. Induction base case. For any pair of data points, X and X the feature-maps corresponding to the jth channel, aj (X, X) have a multivariate Gaussian joint distribution. This is because each element is a linear combination of shared Gaussian random variables: the biases, b DISPLAYFORM1 where 1 is a vector of all-ones. While the elements within a feature map display strong correlations, different feature maps are independent and identically distributed (iid) conditioned on the data (i.e. a( FORMULA1 i (X, X) and a Induction step. Consider the feature maps at the th layer, a j (X, X), to be iid multivariate Gaussian RVs (i.e. for j = j, a j (X, X) and a j (X, X) are iid). Our goal is to show that, taking the number of channels at layer to infinity (i.e. C → ∞), the same properties hold at the next layer (i.e. all feature maps, a ( +1) i (X, X), are iid multivariate Gaussian RVs). Writing eq. for two training examples, X and X, we obtain, DISPLAYFORM2 We begin by showing that a (+1) i (X, X) is a multivariate Gaussian RV. The first term is multivariate Gaussian, as it is a linear function of b DISPLAYFORM3, which is itself iid Gaussian. We can apply the multivariate CLT to show that the second term is also Gaussian, because, in the limit as C → ∞, it is the sum of infinitely many iid terms: a j (X, X) are iid by assumption, and W (+1) i,j are iid by definition. Note that the same argument applies to all feature maps jointly, so all elements of A (+1) (X, X) (defined by analogy with eq. 4) are jointly multivariate Gaussian. Following BID18, to complete the argument, we need to show that the output feature maps are iid, i.e. a DISPLAYFORM4 To show that they are independent, remember that a (+1) i (X, X) and a (+1) i (X, X) are jointly Gaussian, so it is sufficient to show that they are uncorrelated, and we can show that they are uncorrelated because the weights, W (+1) i,j are independent with zero-mean, eliminating any correlations that might arise through the shared RV, φ(a j (X, X)). In the appendix, we consider the more complex case where we take limits simultaneously. Here we derive a computationally efficient kernel corresponding to the CNN described in the previous section. It is surprising that we can compute the kernel efficiently because the feature maps, Published as a conference paper at ICLR 2019 a i (X), display rich covariance structure due to the shared convolutional filter. Computing and representing these covariances would be prohibitively computationally expensive. However, in many cases we only need the variance of the output, e.g. in the case of classification or regression with a final dense layer. It turns out that this propagates backwards through the convolutional network, implying that for every layer, we only need the "diagonal covariance" of the activations: the covariance between the corresponding elements of a DISPLAYFORM0 A GP is completely specified by its mean and covariance (kernel) functions. These give the parameters of the joint Gaussian distribution of the RVs indexed by any two inputs, X and X. For the purposes of computing the mean and covariance, it is easiest to consider the network as being written entirely in index notation, DISPLAYFORM0 where and + 1 denote the input and output layers respectively, j and i ∈ {1, . . ., C ( +1) } denote the input and output channels, and ν and µ ∈ {1, . . ., H ( +1) D (+1) } denote the location within the input and output channel or feature-maps. The mean function is thus easy to compute DISPLAYFORM1 i,j,µ,ν have zero mean, and W (+1) i,j,ν,µ are independent of the activations at the previous layer, φ(A j,ν (X)). Now we show that it is possible to efficiently compute the covariance function. This is surprising because for many networks, we need to compute the covariance of activations between all pairs of locations in the feature map (i.e. C A DISPLAYFORM2 and this object is extremely high-dimensional, DISPLAYFORM3 However, it turns out that we only need to consider the "diagonal" covariance, (i.e. we only need C A DISPLAYFORM4 This is true at the output layer (L + 1): in order to achieve an output suitable for classification or regression, we use only a single output location H (L+1) = D (L+1) = 1, with a number of "channels" equal to the number of of outputs/classes, so it is only possible to compute the covariance at that single location. We now show that, if we only need the covariance at corresponding locations in the outputs, we only need the covariance at corresponding locations in the inputs, and this requirement propagates backwards through the network. Formally, as the activations are composed of a sum of terms, their covariance is the sum of the covariances of all those underlying terms, DISPLAYFORM5 As the terms in the covariance have mean zero, and as the weights and activations from the previous layer are independent, DISPLAYFORM6 using Eq., or some other nonlinearity. DISPLAYFORM7 +1) }; using Eq.. 6: end for 7: Output the scalar K DISPLAYFORM8 The weights are independent for different channels: DISPLAYFORM9 can eliminate the sums over j and ν: DISPLAYFORM10 The µth row of W DISPLAYFORM11 is zero for indices ν that do not belong to its convolutional patch, so we can restrict the sum over ν to that region. We also define v g (X, X), to emphasise that the covariances are independent of the output channel, j. The variance of the first layer is DISPLAYFORM12 And we do the same for the other layers, DISPLAYFORM13 where DISPLAYFORM14 is the covariance of the activations, which is again independent of the channel. The elementwise covariance in the right-hand side of Eq. can be computed in closed form for many choices of φ if the activations are Gaussian. For each element of the activations, one needs to keep track of the 3 distinct entries of the bivariate covariance matrix between the inputs, K DISPLAYFORM0 For example, for the ReLU nonlinearity (φ(x) = max(0, x)), one can adapt BID5 in the same way as Matthews et al. (2018a, section 3) to obtain DISPLAYFORM1 where θ DISPLAYFORM2 We now have all the pieces for computing the kernel, as written in Algorithm 1.Putting together Eq. and Eq. FORMULA1 gives us the surprising that the diagonal covariances of the activations at layer + 1 only depend on the diagonal covariances of the activations at layer. This is very important, because it makes the computational cost of the kernel be within a constant factor of the cost of a forward pass for the equivalent CNN with 1 filter per layer. Thus, the algorithm is more efficient that one would naively think. A priori, one needs to compute the covariance between all the elements of a Furthermore, the particular form for the kernel (eq. 1 and eq. 2) implies that the required variances and covariances at all required locations can be computed efficiently as a convolution. DISPLAYFORM0 The induction step in the argument for GP behaviour from Sec. 2.2 depends only on the previous activations being iid Gaussian. Since all the activations are iid Gaussian, we can add skip connections between the activations of different layers while preserving GP behaviour, e.g. A (+1) and A DISPLAYFORM0 where s is the number of layers that the skip connection spans. If we change the NN recursion (Eq. 2) to DISPLAYFORM1 then the kernel recursion (Eq. 11) becomes DISPLAYFORM2 This way of adding skip connections is equivalent to the "pre-activation" shortcuts described by BID11. Remarkably, the natural way of adding residual connections to NNs is the one that performed best in their empirical evaluations. We evaluate our kernel on the MNIST handwritten digit classification task. Classification likelihoods are not conjugate for GPs, so we must make an approximation, and we follow BID18, in re-framing classification as multi-output regression. The training set is split into N = 50000 training and 10000 validation examples. The regression targets Y ∈ {−1, 1} N ×10 are a one-hot encoding of the example's class: y n,c = 1 if the nth example belongs to class c, and −1 otherwise. Training is exact conjugate likelihood GP regression with noiseless targets Y BID24. First we compute the N ×N kernel matrix K xx, which contains the kernel between every pair of examples. Then we compute K −1 xx Y using a linear system solver. The test set has N T = 10000 examples. We compute the N T × N matrix K x * x, the kernel between each test example and all the training examples. The predictions are given by the row-wise maximum of K x * x K −1 xx Y. For the "ConvNet GP" and "Residual CNN GP", (Table 1) we optimise the kernel hyperparameters by random search. We draw M random hyperparameter samples, compute the ing kernel's performance in the validation set, and pick the highest performing run. The kernel hyperparameters are: σ 2 b, σ 2 w; the number of layers; the convolution stride, filter sizes and edge behaviour; the nonlinearity (we consider the error function and ReLU); and the frequency of residual skip connections (for Residual CNN GPs). We do not retrain the model on the validation set after choosing hyperparameters. Table 1: MNIST classification . #samples gives the number of kernels that were randomly sampled for the hyperparameter search. "ConvNet GP" and "Residual CNN GP" are random CNN architectures with a fixed filter size, whereas "ResNet GP" is a slight modification of the architecture by BID11. Entries labelled "SGD" used stochastic gradient descent for tuning hyperparameters, by maximising the likelihood of the training set. The last two methods use parametric neural networks. The hyperparameters of the ResNet GP were not optimised (they were fixed based on the architecture from b).The "ResNet GP" (Table 1) is the kernel equivalent to a 32-layer version of the basic residual architecture by BID10. The differences are: an initial 3 × 3 convolutional layer and a final dense layer instead of average pooling. We chose to remove the pooling because computing its output variance requires the off-diagonal elements of the filter covariance, in which case we could not exploit the efficiency gains described in Sec. 3.3.We found that the despite it not being optimised, the 32-layer ResNet GP outperformed all other comparable architectures (Table 1), including the NNGP in BID18, which is state-ofthe-art for non-convolutional networks, and convolutional GPs (van der ; BID14 . That said, our have not reached state-of-the-art for methods that incorporate a parametric neural network, such as a standard ResNet and a Gaussian process with a deep neural network kernel BID3.To check whether the GP limit is applicable to relatively small networks used practically (with of the order of 100 channels in the first layers), we randomly sampled 10, 000 32-layer ResNets, with 3, 10, 30 and 100 channels in the first layers, and, following the usual practice for ResNets we increase the number the number of hidden units when we downsample the feature maps. The probability density plots show a good match around 100 channels (FIG6, which matches a more sensitive graphical procedure based on quantile-quantile plots FIG6 . Notably, even for only 30 channels, the moments match closely FIG6). For comparison, typical ResNets use from 64 BID10 ) to 192 channels in their first layers. We believe that this is because the moment propagation equations only require the Gaussianity assumption for propagation through the relu, and presumably this is robust to non-Gaussian input activations. Computational efficiency. Asymptotically, computing the kernel matrix takes O(N 2 LD) time, where L is the number of layers in the network and D is the dimensionality of the input, and inverting the kernel matrix takes O(N 3). As such, we expect that for very large datasets, inverting the kernel matrix will dominate the computation time. However, on MNIST, N 3 is only around a factor of 10 larger than N 2 LD. In practice, we found that it was more expensive to compute the kernel matrix than to invert it. For the ResNet kernel, the most expensive, computing K xx, and K xx * for validation and test took 3h 40min on two Tesla P100 GPUs. In contrast, inverting K xx and computing validation and test performance took 43.25 ± 8.8 seconds on a single Tesla P100 GPU. Van der Wilk et al. BID22 ) also adapted GPs to image classification. They defined a prior on functions f that takes an image and outputs a scalar. First, draw a function g ∼ GP(0, k p (X, X)). Then, f is the sum of the output of g applied to each of the convolutional patches. Their approach is also inspired by convolutional NNs, but their kernel k p is applied to all pairs of patches of X and X. This makes their convolutional kernel expensive to evaluate, requiring, and 100 channels in their first layers. A Comparison of the empirical and limiting probability densities. B A more sensitive test of Gaussianity is a quantile-quantile plot, which shows converges with 100 channels. C The moments (variances and covariances) for 100 training inputs shows gives a good match for all numbers of channels.inter-domain inducing point approximations to remain tractable. The kernels in this work, directly motivated by the infinite-filter limit of a CNN, only apply something like k p to the corresponding pairs of patches within X and X (Eq. 10). As such, the CNN kernels are cheaper to compute and exhibit superior performance (Table 1), despite the use of an approximate likelihood function. BID14 define a prior over functions by stacking several GPs with van der Wilk's convolutional kernel, forming a "Deep GP" BID6. In contrast, the kernel in this paper confines all hierarchy to the definition of the kernel, and the ing GPs is shallow. BID3 improved deep kernel learning. The inputs to a classic GP kernel k (e.g. RBF) are preprocessed by applying a feature extractor g (a deep NN) prior to computing the kernel: k deep (X, X):= k(g(X; θ), g(X, θ)). The NN parameters are optimised by gradient ascent using the likelihood as the objective, as in standard GP kernel learning (, Chapter 5). Since deep kernel learning incorporates a state-of-the-art NN with over 10 6 parameters, we expect it to perform similarly to a NN applied directly to the task of image classification. At present both CNNs and deep kernel learning display superior performance to the GP kernels in this work. However, the kernels defined here have far fewer parameters (around 10, compared to their 10 6). also suggests that a CNN exhibits GP behaviour. However, they take the infinite limit with respect to the filter size, not the number of filters. Thus, their infinite network is inapplicable to real data which is always of finite dimension. Finally, there is a series of papers analysing the mean-field behaviour of deep NNs and CNNs which aims to find good random initializations, i.e. those that do not exhibit vanishing or exploding gradients or activations BID25 ). Apart from their very different focus, the key difference to our work is that they compute the variance for a single training-example, whereas to obtain the GPs kernel, we additionally need to compute the output covariances for different training/test examples . We have shown that deep Bayesian CNNs with infinitely many filters are equivalent to a GP with a recursive kernel. We also derived the kernel for the GP equivalent to a CNN, and showed that, in handwritten digit classification, it outperforms all previous GP approaches that do not incorporate a parametric NN into the kernel. Given that most state-of-the-art neural networks incorporate structure (convolutional or otherwise) into their architecture, the equivalence between CNNs and GPs is potentially of considerable practical relevance. In particular, we hope to apply GP CNNs in domains as widespread as adversarial examples, lifelong learning and k-shot learning, and we hope to improve them by developing efficient multi-layered inducing point approximation schemes. The key technical issues in the proof (and the key differences between BID18 BID21 arise from exactly how and where we take limits. In particular, consider the activations as being functions of the activities at the previous layer, DISPLAYFORM0 Now, there are two approaches to taking limits. First, both our argument in the main text, and the argument in BID18 is valid if we are able to take limits "inside" the network, DISPLAYFORM1 However, BID20 b) argue that is preferable to take limits "outside" the network. In particular, BID21 take the limit with all layers simultaneously, DISPLAYFORM2 where C = C (n) goes to infinity as n → ∞. That said, similar technical issues arise if we take limits in sequence, but outside the network. In the main text, we follow BID18 in sequentially taking the limit of each layer to infinity (i.e. C → ∞, then C → ∞ etc.). This dramatically simplified the argument, because taking the number of units in the previous layer to infinity means that the inputs from that layer are exactly Gaussian distributed. However, BID21 argue that the more practically relevant limit is where we take all layers to infinity simultaneously. This raises considerable additional difficulties, because we must reason about convergence in the case where the previous layer is finite. Note that this section is not intended to stand independently: it is intended to be read alongside BID21, and we use several of their without proof. Mirroring Definition 3 in BID21, we begin by choosing a set of "width" functions, C (n), for ∈ {1, . . ., L} which all approach infinity as n → ∞. In BID21, these functions described the number of hidden units in each layer, whereas here they describe the number of channels. Our goal is then to extend the proofs in BID21 (in particular, of theorem 4), to show that the output of our convolutional networks converge in distribution to a Gaussian process as n → ∞, with mean zero and covariance given by the recursion in Eqs..The proof in BID21 has three main steps. First, they use the Cramér-Wold device, to reduce the full problem to that of proving convergence of scalar random variables to a Gaussian with specified variance. Second, if the previous layers have finite numbers of channels, then the channels a j (X) and a j (X) are uncorrelated but no longer independent, so we cannot apply the CLT directly, as we did in the main text. Instead, they write the activations as a sum of exchangeable random variables, and derive an adapted CLT for exchangeable (rather than independent) random variables BID0. Third, they show that moment conditions required by their exchangeable CLT are satisfied. To extend their proofs to the convolutional case, we begin by defining our networks in a form that is easier to manipulate and as close as possible to Eq. in BID21, DISPLAYFORM0 DISPLAYFORM1 where, DISPLAYFORM2 The first step is to use the Cramér-Wold device (Lemma 6 in BID21, which indicates that convergence in distribution of a sequence of finite-dimensional vectors is equivalent to convergence on all possible linear projections to the corresponding real-valued random variable. Mirroring Eq. 25 in BID21, we consider convergence of random vectors, f DISPLAYFORM3 where L ⊂ X × N × {1, . . ., H D } is a finite set of tuples of data points and channel indicies, i, and indicies of elements within channels/feature maps, µ. The suffix [n] indicates width functions that are instantiated with input, n. Now, we must prove that these projections converge in distribution a Gaussian. We begin by defining summands, as in Eq. 26 in BID21, DISPLAYFORM4 such that the projections can be written as a sum of the summands, exactly as in Eq. 27 in BID21, DISPLAYFORM5 Now we can apply the exchangeable CLT to prove that T (L, α) [n] converges to the limiting Gaussian implied by the recursions in the main text. To apply the exchangeable CLT, the first step is to mirror Lemma 8 in BID21, in showing that for each fixed n and ∈ {2, . . ., L + 1}, the summands, γ j (L, α) [n] are exchangeable with respect to the index j. In particular, we apply de Finetti's theorem, which states that a sequence of random variables is exchangeable if and only if they are i.i.d. conditional on some set of random variables, so it is sufficient to exhibit such a set of random variables. Mirroring Eq. 29 in BID21, we apply the recursion, k,ξ (x)[n]: k ∈ {1, . . ., C ( −2) }, ξ ∈ {1, . . ., H ( −2) D (−2) }, x ∈ L X, where L X is the set of input points in L.The exchangeable CLT in Lemma 10 in BID21 indicates that T (L, α) [n] converges in distribution to N 0, σ 2 * if the summands are exchangeable (which we showed above), and if three conditions hold, DISPLAYFORM6 Condition a) follows immediately as the summands are uncorrelated and zero-mean. Conditions b) and c) are more involved as convergence in distribution in the previous layers does not imply convergence in moments for our activation functions. We begin by considering the extension of Lemma 20 in BID21, which allow us to show conditions b) and c) above, even in the case of unbounded but linearly enveloped nonlinearities (Definition 1 in BID21 . Lemma 20 states that the eighth moments of f (t)i,µ (x)[n] are bounded by a finite constant independent of n ∈ N. We prove this by induction. The base case is trivial, as f where g (−1) j∈{1,...,C ( −1) (n)},ν∈µth patch (x)[n] is the set of post-nonlinearities corresponding to j ∈ {1, . . ., C ( −1) (n)} and ν ∈ µth patch. Following BID21, observe that,. The x-axis gives GP prediction for the label probability. The points give corresponding proportion of test points with that label, and the bars give the proportion of training examples in each bin. DISPLAYFORM7

We show that CNNs and ResNets with appropriate priors on the parameters are Gaussian processes in the limit of infinitely many convolutional filters.

Represent the TLDR for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.