Question: Represent the scientific article to find a short TLDR

Question: Deep Neural Networks (DNNs) are known for excellent performance in supervised tasks such as classification. Convolutional Neural Networks (CNNs), in particular, can learn effective features and build high-level representations that can be used for classification, but also for querying and nearest neighbor search. However, CNNs have also been shown to suffer from a performance drop when the distribution of the data changes from training to test data. In this paper we analyze the internal representations of CNNs and observe that the representations of unseen data in each class, spread more (with higher variance) in the embedding space of the CNN compared to representations of the training data. More importantly, this difference is more extreme if the unseen data comes from a shifted distribution. Based on this observation, we objectively evaluate the degree of representation’s variance in each class by applying eigenvalue decomposition on the within-class covariance of the internal representations of CNNs and observe the same behaviour. This can be problematic as larger variances might lead to mis-classification if the sample crosses the decision boundary of its class. We apply nearest neighbor classification on the representations and empirically show that the embeddings with the high variance actually have significantly worse KNN classification performances, although this could not be foreseen from their end-to-end classification . To tackle this problem, we propose Deep Within-Class Covariance Analysis (DWCCA), a deep neural network layer that significantly reduces the within-class covariance of a DNN’s representation, improving performance on unseen test data from a shifted distribution. We empirically evaluate DWCCA on two datasets for Acoustic Scene Classification (DCASE2016 and DCASE2017). We demonstrate that not only does DWCCA significantly improve the network’s internal representation, it also increases the end-to-end classification accuracy, especially when the test set exhibits a slight distribution shift. By adding DWCCA to a VGG neural network, we achieve around 6 percentage points improvement in the case of a distribution mismatch. Convolutional Neural Networks (CNNs) are the state of the art in many supervised learning tasks such as classification, and using the power of convolutional layers, CNNs can learn useful features that are often superior to engineered features, and build internal representations that can achieve high classification performance. It has been shown that CNNs have a surprising ability to fit data, so much so that they can even perfectly learn from data with random labels BID32. But of course, memorising the training data is not sufficient: a model is expected to generalize to unseen data points. Additionally, a robust model has to be able to not only deal with unseen data points that are similar to the training set, but also cope with unseen data points that may come from a slightly different distribution than the training data (distribution mismatch). When there is a distribution shift between the training and test sets, robustness of the model's representation becomes more important as it has to classify or embed data points that are quite different from the ones it has observed in the training set. In this paper, we investigate this by using a well-known DNN architecture (VGG BID28) that is adapted for audio classification BID9 and is widely used among researchers. We evaluate VGG on data with as well as without distribution mismatch and observe that while VGG exhibits a reasonable performance on the data without distribution mismatch, its performance significantly drops when tested on data from a shifted distribution. We start by analyzing the internal representations of the network by using visualisations. As will be seen in the first (a-c) and the 3rd rows (g-i) of FIG2, the network's internal representations in each class spread more in the embedding space for the unseen data (validation or test) compared to the training data. This is even more extreme when the unseen data comes from a shifted distribution (i).For an objective evaluation of the amount of the representation's variance in each class, we compute the within-class covariance of the representations of the network for each class, and we apply eigenvalue decomposition to compute the eigenvalues of each class's covariance matrix. We then report the sorted eigenvalues of the within-class covariance of the representations in Figure 3. As the blue curves show, the eigenvalues in unseen data of validation (b and e) and test (c and d) have considerably higher ranges than train data (a and d) for all the datasets we used. To better understand the effect of such high variance in the quality of generalisation in the representations of our network, we carry out K-nearest neighbor (KNN) experiments on the dataset without, and the dataset with distribution shift. As the in Figure 4 show, the performance degredation from validation (c) compared to test (d) in case of distribution mismatch is significantly higher compared to the performance drop from validation (a) to test (b) when the test data comes from a similar distribution. This observation is also aligned with what we observed in the visualisations from FIG2 that showed the data is more spread than validation data, when coming from a shifted distribution. To tackle this problem, we propose Deep Within-Class Covariance Analysis (DWCCA), a deep neural network layer that reformulates the conventional Within-Class Covariance Normalization (WCCN) BID12 as a DNN-compatible version. DWCCA is trained end-to-end using back-propagation, can be placed in any arbitrary position in a DNN, and is capable of significantly reducing the within-class covariance of the internal representation in a DNN.We empirically show that DWCCA significantly reduces the within-class covariance of the DNN's representations, in both cases. Further, we evaluate the generalization quality of the DNN's representations after applying DWCCA by performing nearest neighbor classification on its representations. Our show that DWCCA significantly improves the nearest neighbor classification in both cases, hence improving the generalization quality of the representations. And finally we report the end-to-end classification of the trained models on an acoustic scene classification task, using data from the annual IEEE Challenges on Detection and Classification of Acoustic Scenes and Events (DCASE). It turns out that the classification for the dataset with distribution shift are significantly improved by integrating the DWCCA layer, while the performance on the dataset without distribution mismatch stayed the same. The characteristics of the representations learned in CNNs can be influenced by many factors such as network architecture and the training data. The authors in BID19 investigated how architecture topologies affect the robustness and generalization of a representation and showed which representations from different architectures better transfer to other datasets and tasks. While the topology of the architecture influences the generality of its representations, several authors proposed methods that can improve the internal representation of a DNN. BID8 proposed a loss which learns linearly separable latent representations on top of a DNN, by maximising their smallest Linear Discriminant Analysis (LDA) BID10 eigenvalues. And in BID0, the authors propose creating a maximally correlated representation from different modalities. It has been shown that stand-alone CNNs are not very successful at generalizing to unseen data points from shifted distributions in tasks such as Acoustic Scene Classification (ASC), where such distributional shifts are common. ASC is defined as classifying environments from the sounds they produce BID1, and often these environments (e.g, home, park, city center) may sound very different in different cities or during different times of the year. Although in BID23 BID30 BID9 CNNs have shown promising in ASC when the unseen test data has a similar distribution to the training data, in BID21 BID33 similar CNNs that previously performed successfully in a matched distribution case, suffered significantly from the lack of generalization to a shifted distribution in the test set. To cope with this drawback of CNNs, BID27 investigated CNN manifold data augmentation using Generative Adversarial Networks BID11, while in BID15 BID31 BID27 the authors used an ensemble of CNNs as feature extractors and processed the deep features via Support Vector Machines (SVMs), followed by a late fusion of various models. They showed that although CNNs do not generalize well in the distribution shift case, they can learn useful features that can be incorporated for building new classifiers with better generalization properties. In this paper, we try to better understand these problems. We focus our efforts on investigating the reasons for the performance drop in CNNs when the data distribution is shifted. To this end, we analyze the internal representations in CNNs and propose DWCCA to tackle this issue. We start by introducing a common notation which will be used throughout the paper. We then first describe Within-Class Covariance Normalization (WCCN) -a classic machine learning method-, and then show how to cast it into a deep learning compatible version. Let W denote a set of N d-dimensional observations (feature vectors) belonging to C different classes c ∈ {1, ..., C}. The observations are in the present case either hand crafted features (e.g. i-vectors BID6) or any intermediate hidden representation of a deep neural network. WCCN is a linear projection that provides an effective compensation for the within-class variability and has proven to be effectively used in combination with different scoring functions BID5 BID22. The WCCN projection scales the feature space in the opposite direction of its within-class covariance matrix, which has the advantage that finding decision boundaries on the WCCN-projected data becomes easier BID13. The within-class covariance S w is estimated as: DISPLAYFORM0 where DISPLAYFORM1 Based on the definitions above we propose Deep Within-Class Covariance Analysis (DWCCA), a DNN-compatible formulation of WCCN. The parameters of our networks are optimized with Stochastic Gradient Descent (SGD), and trained with mini-batches. The deterministic version of WCCN described above is usually estimated on the entire training set, which by the definition of SGD is not available in the present case. In the following we propose a DWCCA Layer which helps to circumvent these problems. FIG1 shows a schematic sketch of how the DWCCA Layer can be incorporated in a neural network, and its gradient flow. Instead of computing the within-class covariance matrix S w on the entire training set W we provide an estimateŜ b on the observations W b of the respective mini-batches. Given this estimate we compute the corresponding mini-batch projection matrixB b and use it to maintain a moving average projection matrixB asB = (1 − α)B + αB b. This is done similarly when computing the mean and standard deviation for normalising the activations in batch normalization BID16. The hyper parameter α controls the influence of the data in the current batch b on the final DWCCA projection matrix. The output of this processing step is the DWCCA projected data W b = W bB of the respective mini-batch. The DWCCA Layer can be seen as a special dense layer with a predefined weight matrix, the projection matrixB, with the difference that the parameters are computed via the activations of the previous layer, and not learned via SGD. The proposed covariance normalization is applied directly during optimization. For training our network with back-propagation, we need to establish gradient flow. To this end, we implement the DWCCA Layer using the automatic differentiation framework Theano BID3 which already provides the derivatives of matrix inverses and the Cholesky decomposition. We refer the interested reader to BID29 for details on this derivative. In this section, we detail our experimental setup and present our empirical . To evaluate the performance of CNNs in situations with and without distribution mismatch, we use the TUT Acoustic Scenes 2016 (DCASE2016) BID26 and TUT Acoustic Scenes 2017 (DCASE2017) BID25 datasets. The DCASE2016 dataset consists of 15 different acoustic scenes: lakeside beach, bus, cafe/restaurant, car, city center, forest path, grocery store, home, library, metro station, office, urban park, residential area, train, and tram. All audio material was collected as 3-5 minutes length recordings and then was cut into segments of 30 seconds length. The DCASE2017 dataset consists of the same 15 classes and uses the same recordings in its development set (training and validation) as used for DCASE2016 development set. But these recordings were split into only 10s long segments instead of 30s, which makes the learning and classification task harder. A new evaluation set (unseen test) was recorded for DCASE2017. The reason for choosing these two datasets is that DCASE2016 and DCASE2017 are based on the same development (= training and validation) data, but have different evaluations (= independent test) sets. While the test data for DCASE2016 were collected jointly with the development data, new test recordings were collected for DCASE2017, at a later time, in different places. Hence, there is a distribution mismatch between development and test data in the case of DCASE2017, as the many environments sound different from those in the development set (which will also show, indirectly, in our experimental ). This is detailed in the report of DCASE2017 BID25.In the following, we will use DCASE2016 to represent a data scenario without distribution shift, and DCASE2017 as the distribution shift case. In all experiments, we use the official 4-fold cross validation splits provided with the datasets and report CV on all folds separately, as well as the performance of our method on the unseen test set. DISPLAYFORM0 2 × 2 Max-Pooling + Drop-Out(0.3) 3 × 3 Conv(pad-1, stride-1)-64-BN-ReLu 3 × 3 Conv(pad-1, stride-1)-64-BN-ReLu 2 × 2 Max-Pooling + Drop-Out(0.3) 3 × 3 Conv(pad-1, stride-1)-128-BN-ReLu 3 × 3 Conv(pad-1, stride-1)-128-BN-ReLu 3 × 3 Conv(pad-1, stride-1)-128-BN-ReLu 3 × 3 Conv(pad-1, stride-1)-128-BN-ReLu 2 × 2 Max-Pooling + Drop-Out(0.3) 3 × 3 Conv(pad-0, stride-Global-Average-Pooling DWCCA (if applied) 15-way Soft-Max As explained before, we use a VGG-Style CNN proposed in BID9 which uses audio spectrograms as inputs. We will refer to this model as vanilla. To evaluate the effect of our proposed method, we apply a DWCCA layer on the output of global average pooling, as can be seen in TAB0. We will refer to this model in our as dwcca. The internal representations we will analyze are computed by using the output of global average pooling layer in vanilla, and the output of the DWCCA layer in dwcca. We also provide additional baselines for comparison: We report the performance of the best end-toend CNN model BID23 in the DCASE2016 challenge BID24. This method uses an ensemble of various CNN models. For the DCASE2017 experiments, we also choose the best-performing end-to-end CNN from the DCASE2017 challenge BID25 as an additional baseline to vanilla: BID33 that uses an ensemble fusion of various ResNets BID14 using several channels of audio (left, right and difference).We report classification of our end-to-end CNNs on all folds, and on the unseen test set. For the unseen test, we average the probabilities produced by the 4 models trained on the individual training parts in CV folds from the development data. Additionally, we report calibrated where we use a linear logistic regression model in each fold to regress the prediction probabilities of each model to the true labels. This method is better known as late fusion and proved successful in BID9. The probabilistic averaging for unseen test is done similarly for late fusion, as explained above. The setup used in all of our experiments are as follows: The initial learning rate is set to 0.0001, and the ADAM optimizer is used BID18. We choose a similar α (0.1) to the one used in batchnorm. A model selection with max. patience of 20 is applied and the learning rate is halved in each step of maximum patience. The architecture of our models is specified in TAB0. All models are trained with stratified batch iterators and batch size of 75. In all our experiments, the spectrogram of the whole audio segment (30 sec in DCASE2016 and 10 sec in DCASE2017) was fed to our models. All spectrograms in our experiments are extracted with the madmom library BID4 using the mono signal, and the Lasagne library BID7 was used for building neural network architectures. The DWCCA Layer was implemented as a Lasagne-compatible layer in Theano BID2. We carried out various experiments to demonstrate the issue we encounter in ASC with CNNs. First, we train our models on DCASE2016 and DCASE2017. For a better understanding of the issues in the representation learned by the CNN, we provide a visualization of the VGG representations after being projected into 2D via Principal Component Analysis (PCA) BID17. The can be found in FIG2.For analysing the within-class covariance of the representations, we apply an eigenvalue decomposition on the covariance matrix of the representations from each class. This technique is also used in other studies such as BID20 to investigate the dynamics of learning in neural networks. These eigenvalues are a good indicator of the covariance of the representations in each class. If high, it means that the representations on that specific class have a high covariance, and if the difference between the highest eigenvalue and the lowest eigenvalue is high it also indicates that the variance of the representation in that class is not spread equally in all dimensions. These can be found in Figure 3. The shade in this plot represents the variance of eigenvalues over different classes. To indirectly assess the structure of the internal representation spaces learned, we carry out a knearest-neighbor classification experiment in these representation spaces; the K-nearest neighbor on both datasets will be shown in Figure 4.Finally, we report the end-to-end classification on both datasets, for various methods; these can be found in TAB2. For a deeper understanding of the proposed method, we additionally provided the class-wise f-measure of DWCCA and the baseline VGG (vanilla) in TAB3. In FIG2, the network's internal representations in each class are projected into 2D via PCA and each class is represented by a different colour. Looking at first (a-c) and second (d-f) row, it can be seen that for the dataset without mismatched distribution the embeddings of unseen data (validation and test) are spread less after applying DWCCA. Also comparing the unseen embeddings to the training embeddings (with lower opacity and in grey) it can be seen that the unseen embeddings projected closer to the training embeddings after applying DWCCA. Comparing third (g-i) and fourth (j-l) row, it can be seen that for the case of a distribution shift DWCCA also reduces the variance of the embeddings in each class, ing in them being embedded closer to the training embeddings (grey). This suggests that this property can improve the generalisation of the representations. We will empirically evaluate this hypothesis later in this section by applying KNN classification on the representations. Looking at Figure 3, we can see that in all plots from dataset with, and dataset without distribution shift, DWCCA significantly reduces the within-class variability. This can be observed by looking at the eigenvalues of the covariances of the representations. An interesting observation is the range of eigenvalues in vanilla: In both datasets, eigenvalues have significantly larger range on unseen data (validation and test) compared to the training data. The maximum eigenvalue in DCASE2016 is around 0.7, while the maximum eigenvalue for unseen is around 7, about 10 times more. Also the maximum eigenvalue of the train set of DCASE2017 is around 2, while the max. eigenvalue on unseen data is around 20 (10 times larger).By looking at the KNN in Fig. 4 it can be seen that in both cases (mismatch / no mismatch), the KNN classification accuracy increases by adding DWCCA. Also, while the KNN performance is in a reasonable range on the validation set of both datasets, the test accuraty in the mismatch case (DCASE2017) drops significantly compared to the validation set. Additionally it can be seen that applying DWCCA significantly improves the performance on the test set with shifted distribution, adding an improvement of about 6 percentage point, while the improvement on the test set without mismatch is around 2 percentage points. Looking at the of end-to-end classifications in TAB2, we see that the performance of vanilla on DCASE 2017 consistently and significantly improves when adding DWCCA, on all development folds as well as on the unseen test data. We observe around 6 percentage points improvement by adding DWCCA to VGG.Looking at the of the dataset without mismatch, we see that although the on all folds were improved by adding DWCCA, the on the unseen test set do not significantly change. This can be explained better by looking at FIG2: the embeddings of validation (b) and test (c) indicate that the test data is projected closer to the training set than the validation set. This observation suggests that the unseen test in DCASE2016 might be similar (even more similar than the validation data) to the training set. This can also be confirmed by looking at the of the best CNN baseline, as well as vanilla: the performances on the unseen test set are consistently higher than all the validation folds. Hence, DWCCA could not help as there was not a large generalisation gap between training and test. It is worth mentioning that both vanilla and DWCCA are single models, trained on mono single channel spectrograms and no ensemble or multi-channel features were used in these experiments. In other words, a single VGG model achieves comparable performances to an ensemble of multi-channel Resnets. We also provide class-wise f-measures on the unseen test set for both datasets in TAB3. While on the dataset without distribution shift, the average f1 stays the same by adding DWCCA in both calibrated and non calibrated models, we can observe that there is a boost of 13 percentage points on the "train" class which was the class with the lowest f1 (both calibrated and non calibrated). It seems that DWCCA does not have a significant impact on classes with high f1: "office" and "beach" which stay the highest correctly predicted classes and do not face significant changes by DWCCA.On the dataset with distribution shift, we can see a significant improvement of 4 and 7 percentage points on average f1 for non-calibrated and calibrated models, respectively. The worst class in DCASE2017 was "beach" with 32%, which was boosted by 24 and 37 percentage points for noncalibrated and calibrated models, respectively. On the other hand, the best performing class, "forest path", drops by only 2 and 3 percentage points for non-calibrated and calibrated models, respectively. From the experimental , we may thus conclude that overall, reducing the within-class covariance of representations using DWCCA in more robust performance and, in case of a large gap between training and test, DWCCA can improve the generalisation. Additionally, the networks tend to reach a more uniform performance across various classes by improving the performance on the worst classes while not significantly degrading the best performing classes. In this paper, we presented the DWCCA layer, a DNN compatible version of the classic WCCN which is used to normalize the within-class covariance of the DNN's representation and improve the performance of CNNs on data-points with shifted distributions. Using DWCCA, we improved the performance of the VGG network by around 6 percentage point when the test datapoints were from a shifted distribution. We analysed the embedding's generalisation by reporting KNN classification accuracies and showed that DWCCA also improves the generalisation of DNN representations both for with and without distribution mismatch. We also showed that large within-class covariance of representations can be a sign for bad generalisation and showed that DWCCA can significantly reduce WCC and improve generalisation of the representations. This work was supported by the Austrian Ministry for Transport, Innovation & Technology, the Ministry of Science, Research & Economy, and the Province of Upper Austria in the frame of the COMET center SCCH. The authors would like to thank Hasan Bahari of KU Leuven for feedback and fruitful discussions. We also gratefully acknowledge the support of the NVIDIA Corporation with the donation of a Titan X GPU used for this research.

We propose a novel deep neural network layer for normalising within-class covariance of an internal representation in a neural network that results in significantly improving the generalisation of the learned representations.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative models have proven to be an outstanding tool for representing high-dimensional probability distributions and generating realistic looking images. A fundamental characteristic of generative models is their ability to produce multi-modal outputs. However, while training, they are often susceptible to mode collapse, which means that the model is limited in mapping the input noise to only a few modes of the true data distribution. In this paper, we draw inspiration from Determinantal Point Process (DPP) to devise a generative model that alleviates mode collapse while producing higher quality samples. DPP is an elegant probabilistic measure used to model negative correlations within a subset and hence quantify its diversity. We use DPP kernel to model the diversity in real data as well as in synthetic data. Then, we devise a generation penalty term that encourages the generator to synthesize data with a similar diversity to real data. In contrast to previous state-of-the-art generative models that tend to use additional trainable parameters or complex training paradigms, our method does not change the original training scheme. Embedded in an adversarial training and variational autoencoder, our Generative DPP approach shows a consistent resistance to mode-collapse on a wide-variety of synthetic data and natural image datasets including MNIST, CIFAR10, and CelebA, while outperforming state-of-the-art methods for data-efficiency, convergence-time, and generation quality. Our code will be made publicly available. Deep generative models have gained enormous research interest in recent years as a powerful framework to learn high dimensional data in an unsupervised fashion. Generative Adversarial Networks (GANs) BID10 and Variational AutoEncoders (VAEs) are among the most dominant generative approaches. They consist of training two networks: a generator (decoder) and a discriminator (encoder), where the generator attempts to map random noise to fake data points that simulate the probability distribution of real data.. GANs are typically associated with higher quality images compared to VAEs. Nevertheless, in the process of learning multi-modal complex distributions, both models may converge to a trivial solution where the generator learns to produce few modes exclusively, as referred to by mode collapse problem. To address this, we propose utilizing Determinantal Point Processes (DPP) to model the diversity within data samples. DPP is a probabilistic model that has been mainly adopted for solving subset selection problems with diversity constraints BID21, such as video and document summarization. However, Sampling from a DPP requires quantifying the diversity of 2 N subsets, where N is the size of the ground set. This renders DPP sampling from true data to be computationally inefficient in the generation domain. The key idea of our work is to model the diversity within real and fake data throughout the training process, which does adds an insignificant computational cost. Then, We encourage producing samples of similar diversity distribution to the true-data by back-propagating the DPP metric through the generator. This way, generator explicitly learns to cover more modes of real distribution, and accordingly alleviates mode collapse. Recent approaches tackled mode-collapse in one of two different ways: improving the learning of the system to reach a better convergence point(e.g. BID28 ; BID0); or explicitly enforcing the models to capture diverse modes or map back to the true-data distribution (e.g. BID37 ; BID2). Here we focus on a relaxed version of the former, where we use the same learning paradigm of the standard GANs and only change the objective function. The advantage of such an approach is to avoid adding any extra trainable parameters to the trained system while maintaining the same back-propagation steps as the standard GANs. Thus, our model converges faster to a fair equilibrium point where the generator captures the diversity of the true-data distribution while preserving the quality of generations. Contribution. We introduce a new loss function, that we denote Generative Determinantal Point Processes (GDPP) loss. Our loss only assumes an access to a generator G, a feature extraction function φ(·), and sampler from true data distribution p d. The loss encourages the generator to diversify generated samples that match the diversity of real data. This criterion can be considered as a complement to the original adversarial loss which attempts to learn an indistinguishable distribution from the true-data distribution without being specific to diverse modes. We assess the performance of GDPP on three different synthetic data environments, while also verifying the superiority on three real-world images datasets. We compared our approach with state-of-the-art approaches of more complex architectures and learning paradigms. Experiments show that our method outperforms all competing methods in terms of alleviating modecollapse and generations quality. Among the tremendous amount of work that tackles the training challenges of Generative Adversarial Networks (GANs), a few methods stood out as significant contributions towards addressing the problem of mode collapse. Methods that map the data back to noise. BID4 are one of the earliest methods that proposed learning a reconstruction network besides learning the deep generative network. Adding this extra network to the system aims at reversing the action of the generator by mapping from data to noise. Likelihood-free variational inference (LFVI) BID38, merge this concept with learning implicit densities using hierarchical Bayesian modeling. Ultimately, VEEGAN BID37 used the same concept, but the authors did not base their reconstruction loss on the discriminator. This has the advantage of isolating the generation process from the discriminator's sensitivity to any of the modes. BID2 proposed several ways of regularizing the objective of adversarial learning including geometric metric regularizer, mode regularizer, and manifold-diffusion training. Mode regularization specifically has shown a potential into addressing the mode collapse problem and stabilizing the GANs training in general. Methods that provide a surrogate objective function. on the other hand propose with InfoGAN an information-theoretic extension of GANs that obtains disentangled representation of data by latent-code reconstitution through a penalty term in its objective function. InfoGAN includes an autoencoder over the latent codes; however, it was shown to have stability problems similar to the standard GAN and requires stabilization tricks. BID8 base the ModeGAN method on the assumption of the availability of sufficient samples of every mode on the training data. In particular, if a sample from the true data distribution belongs to a particular mode, then the generated fake sample is likely to belong to the same mode. The Unrolled-GAN of BID28 propose a novel objective to update the generator with respect to the unrolled optimization of the discriminator. This allows training to be adjusted between using the optimal discriminator in the generator's objective. It has been shown to improve the generator training process which in turn helps to reduce the mode collapse problem. Generalized LS-GAN of BID7 define a pullback operator to map generated samples to the data manifold. BID39, with a similar philosophy, additionally draws samples from a mixture of Gaussians instead of a single Gaussian. There is, however, no specific enforcement to diversify samples. Spectral normalization strategies have been recently proposed in the works of BID29 and SAGAN BID40 to further stabilize the training. We note that these strategies are orthogonal to our contribution and could be implemented in conjunction with ours to further improve the training stability of generator models. Finally, improving the Wasserstein GANs of, WGAN-GP BID11 introduce a gradient penalization employed in state-of-the-art systems BID18.Methods use multiple generators and discriminators. One of the popular methods to reduce mode collapse is using multiple generator networks to provide a better coverage of the true data distribution. BID24 propose using two generators with shared parameters to learn the joint distribution of the data. The two generators are trained independently on two domains to ensure a diverse generation. However, sharing the parameters guide both the generators to a similar subspace. Also, BID6 propose a similar idea of multiple discriminators that are being an ensemble, which was shown to produce better quality samples. Recently, BID8 proposed MAD-GAN which is a multi-agent GAN architecture incorporating multiple generators and one discriminator. Along with distinguishing real from fake samples, the discriminator also learns to identify the generator that generated the fake sample. The learning of such a system implies forcing different generators to learn unique modes, which helps in a better coverage of data modes. DualGAN of BID30 improves the diversity within GANs at the additional requirement of training two discriminators. In contrast to these approaches, our DPP-GAN does not require the training of extra networks which provides an easier and faster training as well as being less susceptible to overfitting. Finally, we also refer to BID23 as another approach addressing mode collapse. They do that by modifying the discriminator input with concatenated samples to better sample the diversity within real data. Nevertheless, such approach is subject to memory and computational constraints as a of the significant increase in batch size. DPP is a probabilistic measure that was introduced in quantum physics BID26 and has been studied extensively in random matrix theory BID15. It provides a tractable and efficient means to capture negative correlation with respect to a similarity measure, that in turn can be used to quantify the diversity within a subset. A key characteristic in DPP that the model is agnostic about the order of items as pointed out by BID9, and therefore can be used to model data that is randomly sampled from a certain distribution. A point process P on a ground set V is a probability measure on the power set of V (i.e., 2 N), where N = |V| is the size of the ground set. A point process P is called determinantal if, given a random subset Y drawn according to P, we have for every S ⊆ Y, DISPLAYFORM0 for some symmetric similarity kernel L ∈ R N ×N, where L S is the similarity kernel of subset S. L must be real, positive semidefinite matrix L I (all the eigenvalues of L is between 0 and 1); since it represents a probabilistic measure and all of its principal minors must be non-negative. L is often referred to as the marginal kernel because it contains all the information needed to compute the probability of any subset S being selected in V. L S denotes the sub-matrix of L indexed by S, specifically, L S ≡ [L ij]; i, j ∈ S. Hence, the marginal probability of including one element e i is p(e i ∈ Y) = L ii, and two elements DISPLAYFORM1 A large value of L ij reduces the likelihood of both elements to appear together. BID20 proposed decomposing the kernel L S as a Gram matrix: DISPLAYFORM2 Here q(e i) ≥ 0 may be seen as a quality score of an item e i in the ground set V, while φ i ∈ R D; D ≤ N and ||φ i || 2 = 1 is used as an 2 normalized feature vector of an item. In this manner, φ i φ j ∈ [−1, 1] is evaluated as a "normalized similarity" between items e i and e j of V, and the kernel L S is guaranteed to be real positive semidefinite matrix. Geometric interpretation: det(φ(S) φ(S)) = i λ i, where λ i is the i th eigen value of the matrix φ(S) φ(S). Hence, we may visualize that DPP models diverse representations of data because the determinant of φ(S) φ(S) corresponds to the volume in n-D represented by the multiplication of the variances of the data (i.e., the eigen values). DPP has proven to be a tremendously valuable tool when addressing the problem of diversity enforcing such as document summarization (e.g., BID21 ; Hong (a) Given a generator G, and feature extraction function φ, the diversity kernel is constructed as LS = φ · φ. We use the last feature map of the discriminator in GAN or the encoder in VAE as the feature representation φ of a fake/real batch.(b) Using φ obtained from generated samples, we model their diversity using LS B. We also model the diversity of a real batch by extracting its features and constructing its diversity kernel LD B. Adversarial loss aims at generating similar data points to the real, and diversity loss aims at matching fake data diversity kernelLS B to real data diversity kernel LD B. We draw inspiration from DPP to model a subset diversity using a kernel. During training, we extract the feature representation of real and fake batches φ real and φ f ake. Then, we construct their diversity kernels: LS B, LD B. Our loss encourages G to synthesize data of a diversity LS B similar to the real data diversity LD B.& Nenkova FORMULA0 ), pose estimation (e.g., BID12) and video summarization (e.g., BID9 ; BID27). For instance, BID41 proposed to learn the two parameters q, φ in eq. 2 to quantify the diversity of the kernel L S using MLPs based on spatio-temporal features of the video to perform summarization. Recently, BID16 proposed to use DPP to automatically create capsule wardrobes, i.e. assemble a minimal set of items that provide maximal mix-and-match outfits given an inventory of candidate garments. As illustrated in Fig. 1b, our GDPP loss encourages the generator to sample fake data of diversity similar to real data diversity. The key challenge is to model the diversity within real data and fake data. We discussed in Sec. 3 how DPP is used to model the negative correlation within a discrete data distribution, which is commonly employed as a measure of diversity. Thus, we construct a DPP kernel for both the real data and the generated samples at every iteration of the training process as shown in Fig. 1a. Then, we encourage the network to generate samples that have a similar diversity kernel to that of the training data. To simplify the optimization process, we choose to match the eigenvalues and eigenvectors of the fake data DPP kernel with their corresponding of the real data DPP kernel. Eigenvalues and vectors capture the manifold structure of both real and fake data, and hence renders the matching problem simpler. During training, a generative model G produces a batch of samples S B = {e 1, e 2, · · · e B}, where B is the batch size. Our aim is to produce S B that is probabilistically sampled following the DPP which satisfies: DISPLAYFORM0 such that Y is a random variable representing a subset drawn with a generative point process P, and L S B is the kernel matrix of the subset indexed by S, as detailed in Sec. 3. Connecting DPP to the data generation, we assume that G is the point process sampler that generates subset S B according to P. Let φ(S B) ∈ R d×B be a feature representation of the generated subset S B, where φ(·) is a feature extraction function. Therefore, the DPP kernel is constructed as follows: DISPLAYFORM1 where z B ∈ R dz×B is noise vector inputted to the generator G ing in the generated subset S B. Let us denote φ(S B) a feature representation of a generated batch and φ(D B) a feature representation of a true batch. Our aim is to match DISPLAYFORM2, where λ i real and λ i f ake are the i th eigenvalues of L D B and L S B respectively. Thus, our problem is reduced to learn a fake diversity kernel L S B close to the real diversity kernel L D B. We choose to match those kernels using their major characteristics: eigenvalues and eigenvectors. Our GDPP loss is composed of two components: diversity magnitude loss L m, and diversity structure loss L s as follows: Scaling the structure loss aims to induce noise invariance within the eigenvectors similarity learning. This can be seen as alleviating the effect of outlier structures that intrinsically exist within the real data on learning the diversity kernel of fake data. We note that all the eigenvalues of L S B and L S D will be real non-negative since both of the kernels are symmetric semi-positive definite. Therefore, the kernels represent a probability measure since none of the principal minors will be negative. DISPLAYFORM3 Integrating GDPP loss with GANs. For a primary benchmark, we integrate our GDPP loss with GANs. Since our aim is to avoid adding any extra trainable parameters, we utilize features extracted by the discriminator. We choose to use the hidden activations before the last layer as our feature extraction function φ. We apply 2 normalization on the obtained features that guarantees constructing a positive semi-definite matrix according to eq. 2. We finally integrate L DP P g into the GAN objective by only modifying the generator loss of the standard adversarial loss BID10 as follows: DISPLAYFORM4 Integrating GDPP loss with VAEs. A key property of our loss is its generality to any generative model. Beside incorporating our loss in GANs, we prove it can be also embedded within Variational Auto-Encoders (VAEs) proposed in BID19. We use the decoder network as our generator G and the final hidden activations within the encoder network as our feature extraction function φ.To compute L DP P at the training time, we feed an input training batch D B to the encoder constructing L D B. We also feed a random Gaussian noise to the decoder that generates a fake batch S B, which we then feed to the encoder to construct L S B. Finally, we compute L DP P as stated in eq. 2 using the 2 normalized features, then add it to the original VAE loss at the training time as follows: DISPLAYFORM5 5 EXPERIMENTSIn our experiments, we target evaluating the generation based on two criteria: mode collapse and generated samples quality. Due to the intractability of log-likelihood estimation, this problem tends to be non-trivial in real data. Therefore, we start by analyzing our method on synthetic data where we can accurately evaluate the performance. Then, we demonstrate the effectiveness of our method on real data using standard evaluation metrics. We use the same architecture and data on all the competing methods (See appendix A for details). Mode collapse and the quality of generations can be explicitly evaluated on synthetic data since the true distribution is well-defined. In this section, we evaluate the performance of the methods on mixtures of Gaussian of known mode locations and distribution (See appendix B for details). We use the same architecture for all the models, which is the same one used by BID28 and BID37. We note that the first four rows in TAB0 are obtained from BID37, since we are using the same architecture and training paradigm. Fig. 2 effect of each method on the 2D Ring and Grid data. As shown by the vanilla-GAN in the 2D Ring example (Fig. 2a), it can generate the highest quality samples however it only captures a single mode. On the other extreme, the WGAN-GP on the 2D grid (Fig. 2k) captures almost all modes in the true distribution, but this is only because it generates highly scattered samples that do not precisely depict the true distribution. GDPP-GAN (Fig. 2f,l) creates a precise representation of the true data distribution reflecting that the method learned an accurate structure manifold. Performance Evaluation: At every iteration, we sample fake points from the generator and real points from the given distribution. Mode collapse is quantified by the number of real modes recovered in fake data, and the generation quality is quantified by the % of High-Quality Samples. A generated sample is counted as high-quality if it was sampled within three standard deviations in case of 2D Ring or Grid, and ten standard deviations in case of the 1200D data. We train all models for 25K iterations, except for VEEGAN which is trained for 100K iterations to properly converge. At inference time, we generate 2500 samples from each of the trained models and measure both metrics. We report the numbers averaged over five runs with different random initialization in TAB0. GDPP-GAN clearly outperforms all other methods, for instance on the most challenging 1200D dataset that was designed to mimic a natural data distribution, bringing a 63% relative improvement in high-quality samples and 15% in mode detection over its best competitor WGAN-GP.Ablation Study: We run a study on the 2D Ring and Grid data to show the individual effects of each component in our loss. As shown in TAB2, optimizing the determinant det L S directly increases the diversity generating the highest quality samples. This works best on the 2D Ring since the true data distribution can be represented by a repulsion model. However, for more complex data such as the 2D Grid, optimizing the determinant fails because it does not well-represent the real manifold structure but aims at repelling the fake samples from each other. Learning the unnormalized structure is prone to outlier structures introduced by the noise in the data and in the learning process. However, when scaling the structure loss by the true-data eigenvalues seems to better disengage the model from noise that exists within the true-data features and only focus on learning the prominent structure. We evaluate the amount of training data needed by each method to reach the same local optima as evaluated by our two metrics on both the 2D Ring and Grid data. Since we are sampling the true-data from a mixture of Gaussians, we can generate an infinite size of training data. DISPLAYFORM0 Figure 2: Scatter plots of the true data(green dots) and generated data(blue dots) from different GAN methods trained on mixtures of Gaussians arranged in a ring (top) or a grid (bottom). Therefore, we can quantify the amount of the training data by using the batch-size while fixing the number of back-propagation steps. In this experiment FIG2, we run all the methods for the same number of iterations and vary the batch size. However, WGAN-GP tends to capture higher quality samples with fewer data. In the case of 2D Grid data, GDPP-GAN performs on par with other methods for small amounts of data, yet it tends to significantly outperform other methods on the quality of generated samples once trained on enough data. Time-Efficiency: Another property of interest is which method converges faster given the same amount of training data. In this experiment, we fix the batch size at 512 and train the models for a variable number of iterations FIG2. For the 2D Ring, Only VEE-GAN captures a higher number of modes before GDPP-GAN, however, they are of much lower quality than the ones generated by GDPP-GAN. In the 2D Grid data, GDPP-GAN performs on par with unrolled-GAN for the first 5,000 iterations while the others are falling behind. After that, our method significantly outperforms all the methods with respect to both the number of captured modes and the quality of generated samples. We also shows that the GDPP-GAN has an indistinguishable time cost over the DCGAN in Table 6, rendering it the fastest over other baselines. We use the experimental setting of state-of-the-art BID11 and BID28 for evaluating models on the Stacked MNIST and CIFAR10. On CelebA, we use the experimental setting of state-of-the-art BID17. Nonetheless, we investigated the robustness of our method by using a more challenging setting proposed by BID37 and we show its in Table 5 of Appendix C. In our evaluation, we focus on comparing with state-of-theart method that adopt a change in the original adversarial loss. Nevertheless, many of them can be deemed orthogonal to our contribution, and can enhance the generation if integrated with our approach. We also show that our method is robust to random initialization in Section C. Table 3: Performance of various methods on real datasets. Stacked-MNIST is evaluated using the number of captured modes (Mode Collapse) and KL-divergence between the generated class distribution and true class distribution (Quality of generations). CIFAR-10 is evaluated by Inference-via-Optimization (Mode-Collapse) and Inception-Score (Quality of generations). Stacked-MNIST A variant of MNIST designed to increase the number of discrete modes in the data. The data is synthesized by stacking three randomly sampled MNIST digits along the color channel ing in a 28x28x3 image. In this case, Stacked MNIST has 1000 discrete modes corresponding to the number of possible triplets of digits. Following BID11, we generate 50,000 images that are later used to train the networks. We train all the models for 15,000 iterations, except for DCGAN and unrolled-GAN that need 30,000 iterations to converge to a reasonable local-optima. We follow BID37 to evaluate methods on the number of recovered modes and the divergence between the true and fake distributions. We sample 26000 fake images for all the models. We identify the mode of each generated image by using the classifier mentioned in BID2 that is trained on the standard MNIST dataset to classify each channel of the fake sample. The quality of samples is evaluated by computing the KL-divergence between the generated label distribution and the training labels distribution. GDPP-GAN captures all the modes and generates a fake distribution that has the lowest KL-Divergence with the true-data. Moreover, when applied on the VAE, it doubles the number of modes captured (623 vs 341) and cuts the KL-Divergence to half (1.3 vs 2.4).We note that we run a separate experiment on MNIST in Section C.4 to assess the severity of mode collapse following BID34. We evaluate the methods on CIFAR-10 after training all the models for 100K iterations. Unlike Stacked-MNIST, the modes are intractable in this dataset. To assess the performance on this dataset, we use two metrics: Inception Score for the generation quality and Inference-viaOptimization for diversity. As shown in the Quantitative on CIFAR and Stacked MNIST (Table 3), GDPP-GAN consistently outperforms all other methods in both mode collapse and developing higher quality samples. When applying the GDPP on the VAE, it reduces the IvO by 63%, however, we note that both the inception-scores are considerably low which is also observed by BID36 when applying the VAE on CIFAR-10.Inference-via-optimization BID28, has been used to assess the severity of mode collapse in the generator by providing a metric to compare real images with the nearest generated image. In the case of mode collapse, there are some real images for which this distance is large. We measure this metric by sampling a real image x from the test set of real data. Then we optimize the 2 loss between x and generated image G(z) by modifying the noise vector z. If a method attains low MSE, then it can be assumed that this method captures more modes than ones that attain a higher MSE. FIG3 presents some real images with their nearest optimized generations. Randomly generated sample images can be seen in Appendix D. As demonstrated by BID37, this metric can be fooled by producing blurry images out of the optimization. That is why the inception score is necessary for this evaluation. Inception score BID35 is widely used as a metric for assessing the quality of images. It bases its validity from the premise that every realistic image should be recognizable by a standard architecture(e.g., Inception Network). Ideally, meaning that the score distribution for it must be dominated by one class. We also assess the stability of the training, by calculating the inception score at different stages while training on CIFAR-10 FIG4. Evidently, DCGAN has the least stable training with a high variation. However, by only adding GDPP penalty term to the generator loss, model generates high-quality images the earliest on training with a stable increase. CelebA Finally, to evaluate the performance of our loss on large-scale Adversarial training, we train Progressive-Growing GANs BID17 We show the effect of embedding our loss in adversarial training by adding it to the WGAN-GP this time instead of DCGAN loss, which is as well orthogonal to our loss. We train the model for 40K iterations corresponding to 4 scales up to 64 × 64 on CelebA dataset BID25. Unlike CIFAR-10, CelebA dataset does not simulate ImageNet because it only contains faces not natural scenes/objects. Therefore, using a model trained on ImageNet as a basis for evaluation (i.e., Inception Score), will cause inaccurate recognition. On the other hand, IvO operates by optimizing the noise vector to match real image. However, large scale datasets requires larger noise vector to cover all the synthetic manifold. This renders the optimization prone to divergence or convergence to poor local optimas; jeopardizing the metric effectiveness. We follow BID17 to evaluate the performance on CelebA using Sliced Wasserstein Distance (SWD) BID32. A small Wasserstein distance indicates that the distribution of the patches is similar, which entails that real and fake images appear similar in both appearance and variation at this spatial resolution. Accordingly, SWD metric can evaluate the quality of images as well as the severity of mode-collapse on large-scale datasets such as CelebA. TAB4 shows the average and minimum SWD metric across the last 10K training iterations. We chose this time frame because it shows a saturation in the training loss for all methods. For qualitative examples, refer to Fig. 11 in Appendix D. In this work, we introduce a novel criterion to train generative networks on capturing a similar diversity to one of the true data by utilizing Determinantal Point Process(DPP). We apply our criterion to Generative Adversarial training and the Variational Autoencoder by learning a kernel via features extracted from the discriminator/encoder. We train the generator on optimizing a loss between the fake and real, eigenvalues and eigenvectors of this kernel to simulate the diversity of the real data. Our GDPP framework accumulates many desirable properties: it does not require any extra trainable parameters, it operates in an unsupervised setting, yet it consistently outperforms stateof-the-art methods on a battery of synthetic data and real image datasets as measure by generation quality and invariance to mode collapse. Furthermore, GDPP-GANs exhibit a stabilized adversarial training and has been shown to be time and data efficient as compared to state-of-the-art approaches. Moreover, the GDPP criterion is architecture and model invariant, allowing it to be embedded with any variants of generative models such as adversarial feature learning or conditional GANs. The architectures of the generator and discriminator networks employed in our experiments are given in FIG5. Tan h In all of our experiments, we use Adam Optimizer with β 1 = 0.5 and = 1×10 −8. For the synthetic data experiments, we follow the configurations used by BID37 and BID28. We use 1 × 10 −4 for the discriminator learning rate, and 1 × 10 −3 for the generator learning rate. For synthetic data we use a batch size of 512. For Stacked-MNIST and CIFAR-10 we use a batch size of 64. For CIFAR-10, we use a batch size of 16.For the Stacked MNIST, CIFAR-10 and CelebA datasets, we use 2 × 10 −4 as the learning rate for both of the generator and the discriminator. To relatively stabilize the training of DCGAN, we follow the protocol in BID11 to train it by applying a learning rate scheduler. The decay is to happen with a ratio of 1/(#max − iters) at every iteration. The first data collection is introduced in BID28 as a mixture of eight 2D Gaussian distributions arranged in a ring. This distribution is the easiest to mimic since it only requires the generated data to have an equal repulsion from the center of the distribution, even if it is not targeted to the modes. The second and third collections were introduced by BID37. In the second collection, there is a mixture of twenty-five 2D Gaussian distributions arranged in a grid. Unlike the first collection, this one requires a more structured knowledge of the true data modes' locations. The last collection is a mixture of ten 700 dimensional Gaussian distributions embedded in a 1200 dimensional space. This mixture arrangement mimics the higher dimensional manifolds of natural images, and demonstrates the effectiveness of each method on manipulating sparse patterns. Since the weights of the generator are being initialized using a random number generator N, the of a generative model may be affected by poor initializations. In FIG6 we show qualitative examples on 2D Grid data, where we use high standard deviation for the random number generator (> 100) as an example of poor initializations. Evidently, GDPP-GAN respects the structure of the true data manifold even with poor initializations. On the other extreme, WGAN-GP tends to map the generated data to a disperse distribution covering all modes but with low quality generations. To further show the effectiveness of our approach, we examine it under a more challenging experimental setting. The experimental setting of BID37 entails an architecture and hyperparameters that produce relatively poor as compared with the setting of Table 3. For example, In BID37 setting, DCGAN produces 99 modes, while in our experimental setting, DCGAN produces 427 modes on Stacked MNIST dataset. We note that our main in Table 3 are computed using the same experimental setting suggested by BID11 and BID28 ) on a more realistic architecture. Our method remains to have a clear advantage when compared to the rest of the methods for both CIFAR-10 and Stacked-MNIST (e.g., covering 90.6% more modes on Stacked-MNIST from 150 to 286 and at a higher quality). We obtain the first four rows from BID37. CIFAR-10 #Modes (Max 1000) KL div. IvO DCGAN 99 3.4 0.00844 ALI 16 5.4 0.0067 Unrolled-GAN BID28 48.7 4.32 0.013 VEEGAN BID37 150 2.95 0.0068 GDPP-GAN (Ours) 286 2.12 0.0051 Table 5: Performance on real datasets using the challenging experimental setting of BID37. GDPP-GAN remains to outperform all baselines on both Stacked-MNIST and CIFAR-10 for all metrics. Eigendecomposition of an n×n matrix requires O(n 3 +n 2 log 2 nlogb) runtime within a relative error bound of 2 −b as shown in BID31. In our loss, we perform two eigendecompositions: L S B, L D B corresponding to the fake and true DPP kernels respectively. Therefore, the runtime analysis of our loss is O(n 3), where n is the batch size. Normally the batch size does not exceed 1024 for most training paradigms due to memory constraints. In our experiments, we used 512 for synthetic data and 64 or 16 for real data. Hence, the eigendecomposition does not account for a significant delay in the method. To further verify this claim, we measured the relative time that eigendecompositions take of each iteration time. We obtained 11.61% for Synthetic data, 9.36% for Stacked-MNIST data and 8.27% for CIFAR-10. We also show the average iteration running time of all baselines in Table 6. We computed the average of 1000 iterations across 5 different runs. Our method is the closest to the standard DCGAN running time, and faster than the rest of baselines by a large margin. Table 6: Average Iteration time for each of the baseline methods on CIFAR-10. GDPP-GAN obtains the closest time to the default DCGAN.C.4 NUMBER OF STATISTICALLY-DIFFERENT BINS (NDB) BID34 proposed to use a new evaluation metric to assess the severity mode collapse severity in a generative model. They based their metric on a simple observation: In two sets of samples that represent the same distribution, number of samples that fall into a given bin should be the same up to a sampling noise. In other words, if we clustered the true-data distribution and fake-data distribution to the same number of clusters/bins, then the number of samples from each distribution in every bin should be similar. We follow BID34 to compute this metric on MNIST dataset, and compare our method with their in Table 7. We note that we used their open-source implementation of the metric, and we obtained the first three rows from their paper. We use 20,000 samples from our model and the training data to compute the NDB/K. Model K=100 K=200 K=300 TRAIN 0.06 0.04 0.05MFA BID34 0.14 0.13 0.14 DCGAN 0.41 0.38 0.46 WGAN 0.16 0.20 0.21 GDPP-GAN 0.11 0.15 0.12 Table 7: NDB/K -numbers of statistically different bins, with significance level of 0.05, divided by the number of bins K (lower is better). (a) GDPP-GAN after 15K iterations.(b) GDPP-VAE after 45K iterations.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

The addition of a diversity criterion inspired from DPP in the GAN objective avoids mode collapse and leads to better generations.

Represent the short summary for finding the corresponding scientific article (long)

We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design, with theoretical guarantees.


Question: Represent the article for finding a single-sentence TLDR

Question: Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities ing in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks. Deep neural networks have enjoyed great success in learning across a wide variety of tasks. They played a crucial role in the seminal work of , starting an arms race of training larger networks with more hidden units, in pursuit of better test performance . In fact the networks used in practice are over-parametrized to the extent that they can easily fit random labels to the data . Even though they have such a high capacity, when trained with real labels they achieve smaller generalization error. Traditional wisdom in learning suggests that using models with increasing capacity will in overfitting to the training data. Hence capacity of the models is generally controlled either by limiting the size of the model (number of parameters) or by adding an explicit regularization, to prevent from overfitting to the training data. Surprisingly, in the case of neural networks we notice that increasing the model size only helps in improving the generalization error, even when the networks are trained without any explicit regularization -weight decay or early stopping (; ; c). In particular, Neyshabur et al. (2015c) observed that training on models with increasing number of hidden units lead to decrease in the test error for image classification on MNIST and CIFAR-10. Similar empirical observations have been made over a wide range of architectural and hyper-parameter choices (; ;). What explains this improvement in generalization with over-parametrization? What is the right measure of complexity of neural networks that captures this generalization phenomenon?Complexity measures that depend on the total number of parameters of the network, such as VC bounds, do not capture this behavior as they increase with the size of the network. Existing works suggested different norm, margin and sharpness based measures, to measure the capacity of neural networks, in an attempt to explain the generalization behavior observed in practice (b; ; ; ; ; We observe that even when after network is large enough to completely fit the training data(reference line), the test error continues to decrease for larger networks. Middle panel: Training fully connected feedforward network with single hidden layer on CIFAR-10. We observe the same phenomena as the one observed in ResNet18 architecture. Right panel: Unit capacity captures the complexity of a hidden unit and unit impact captures the impact of a hidden unit on the output of the network, and are important factors in our capacity bound (Theorem 1). We observe empirically that both average unit capacity and average unit impact shrink with a rate faster than 1/ √ h where h is the number of hidden units. Please see Supplementary Section A for experiments settings. BID0; BID0. In particular, showed a margin based generalization bound that depends on the spectral norm and 1,2 norm of the layers of a network. However, as shown in and in FIG6, these complexity measures fail to explain why over-parametrization helps, and in fact increase with the size of the network. numerically evaluated a generalization bound based on PAC-Bayes. Their reported numerical generalization bounds also increase with the increasing network size. These existing complexity measures increase with the size of the network, even for two layer networks, as they depend on the number of hidden units either explicitly, or the norms in their measures implicitly depend on the number of hidden units for the networks used in practice To study and analyze this phenomenon more carefully, we need to simplify the architecture making sure that the property of interest is preserved after the simplification. We therefore chose two layer ReLU networks since as shown in the left and middle panel of FIG0, it exhibits the same behavior with over-parametrization as the more complex pre-activation ResNet18 architecture. In this paper we prove a tighter generalization bound (Theorem 2) for two layer ReLU networks. Our capacity bound, unlike existing bounds, correlates with the test error and decreases with the increasing number of hidden units, in the experimental range considered. Our key insight is to characterize complexity at a unit level, and as we see in the right panel in FIG0 these unit level measures shrink at a rate faster than 1/ √ h for each hidden unit, decreasing the overall measure as the network size increases. When measured in terms of layer norms, our generalization bound depends on the Frobenius norm of the top layer and the Frobenius norm of the difference of the hidden layer weights with the initialization, which decreases with increasing network size (see FIG1).The closeness of learned weights to initialization in the over-parametrized setting can be understood by considering the limiting case as the number of hidden units go to infinity, as considered in and BID1. In this extreme setting, just training the top layer of the network, which is a convex optimization problem for convex losses, will in minimizing the training error, as the randomly initialized hidden layer has all possible features. Intuitively, the large number of hidden units here represent all possible features and hence the optimization problem involves just picking the right features that will minimize the training loss. This suggests that as we over-parametrize the networks, the optimization algorithms need to do less work in tuning the weights of the hidden units to find the right solution. indeed have numerically evaluated a PAC-Bayes measure from the initialization used by the algorithms and state that the Euclidean distance to the initialization is smaller than the Frobenius norm of the parameters. also make a similar empirical observation on the significant role of initialization, and in fact prove an initialization dependent generalization bound for linear networks. However they do not prove a similar generalization bound for neural networks. suggested a Fisher-Rao metric based complexity measure that correlates with generalization behavior in larger networks, but they also prove the capacity bound only for linear networks. Contributions: Our contributions in this paper are as follows.• We empirically investigate the role of over-parametrization in generalization of neural networks on 3 different datasets (MNIST, CIFAR10 and SVHN), and show that the existing complexity measures increase with the number of hidden units -hence do not explain the generalization behavior with over-parametrization.• We prove tighter generalization bounds (Theorem 2) for two layer ReLU networks, improving over previous . Our proposed complexity measure for neural networks decreases with the increasing number of hidden units, in the experimental range considered (see Section 2), and can potentially explain the effect of over-parametrization on generalization of neural networks.• We provide a matching lower bound for the Rademacher complexity of two layer ReLU networks with a scalar output. Our lower bound considerably improves over the best known bound given in , and to our knowledge is the first such lower bound that is bigger than the Lipschitz constant of the network class. We consider two layer fully connected ReLU networks with input dimension d, output dimension c, and the number of hidden units h. Output of a network is DISPLAYFORM0 h×d and V ∈ R c×h. We denote the incoming weights to the hidden unit i by u i and the outgoing weights from hidden unit i by v i. Therefore u i corresponds to row i of matrix U and v i corresponds to the column i of matrix V.We consider the c-class classification task where the label with maximum output score will be selected as the prediction. , we define the margin operator µ: R c × [c] → R as a function that given the scores f (x) ∈ R c for each label and the correct label y ∈ [c], it returns the difference between the score of the correct label and the maximum score among other labels, i.e. DISPLAYFORM1. We now define the ramp loss as follows: DISPLAYFORM2 For any distribution D and margin γ > 0, we define the expected margin loss of a predictor f as DISPLAYFORM3 The loss L γ defined this way is bounded between 0 and 1. We useL γ (f) to denote the empirical estimate of the above expected margin loss. As setting γ = 0 reduces the above to classification loss, we will use L 0 (f) andL 0 (f) to refer to the expected risk and the training error respectively. For any function class F, let γ • F denote the function class corresponding to the composition of the loss function and functions from class F. With probability 1 − δ over the choice of the training set of size m, the following generalization bound holds for any function f ∈ F (, Theorem 3.1): DISPLAYFORM0 where R S (H) is the Rademacher complexity of a class H of functions with respect to the training set S which is defined as: Rademacher complexity is a capacity measure that captures the ability of functions in a function class to fit random labels which increases with the complexity of the class. DISPLAYFORM1 We will bound the Rademacher complexity of neural networks to get a bound on the generalization error. Since the Rademacher complexity depends on the function class considered, we need to choose the right function class that only captures the real trained networks, which is potentially much smaller than networks with all possible weights, to get a complexity measure that explains the decrease in generalization error with increasing width. Choosing a bigger function class can in weaker capacity bounds that do not capture this phenomenon. Towards that we first investigate the behavior of different measures of network layers with increasing number of hidden units. The experiments discussed below are done on the CIFAR-10 dataset. Please see Section A for similar observations on SVHN and MNIST datasets. First layer: As we see in the second panel in FIG1 even though the spectral and Frobenius norms of the learned layer decrease initially, they eventually increase with h, with Frobenius norm increasing at a faster rate. However the distance Frobenius norm, measured w.r.t. initialization (U − U 0 F), decreases. This suggests that the increase in the Frobenius norm of the weights in larger networks is due to the increase in the Frobenius norm of the random initialization. To understand this behavior in more detail we also plot the distance to initialization per unit and the distribution of angles between learned weights and initial weights in the last two panels of FIG1. We indeed observe that per unit distance to initialization decreases with increasing h, and a significant shift in the distribution of angles to initial points, from being almost orthogonal in small networks to almost aligned in large networks. This per unit distance to initialization is a key quantity that appears in our capacity bounds and we refer to it as unit capacity in the remainder of the paper. Unit capacity. We define β i = u i − u 0 i 2 as the unit capacity of the hidden unit i. Second layer: Similar to first layer, we look at the behavior of different measures of the second layer of the trained networks with increasing h in the first panel of FIG1. Here, unlike the first layer, we notice that Frobenius norm and distance to initialization both decrease and are quite close suggesting a limited role of initialization for this layer. Moreover, as the size grows, since the Frobenius norm V F of the second layer slightly decreases, we can argue that the norm of outgoing weights v i from a hidden unit i decreases with a rate faster than 1/ √ h. If we think of each hidden unit as a linear separator and the top layer as an ensemble over classifiers, this means the impact of each classifier on the final decision is shrinking with a rate faster than 1/ √ h. This per unit measure again plays an important role and we define it as unit impact for the remainder of this paper. Unit impact. We define α i = v i 2 as the unit impact, which is the magnitude of the outgoing weights from the unit i. Motivated by our empirical observations we consider the following class of two layer neural networks that depend on the capacity and impact of the hidden units of a network. Let W be the following restricted set of parameters: DISPLAYFORM0 We now consider the hypothesis class of neural networks represented using parameters in the set W: DISPLAYFORM1 Our empirical observations indicate that networks we learn from real data have bounded unit capacity and unit impact and therefore studying the generalization behavior of the above function class can potentially provide us with a better understanding of these networks. Given the above function class, we will now study its generalization properties. In this section we prove a generalization bound for two layer ReLU networks. We first bound the Rademacher complexity of the class F W in terms of the sum over hidden units of the product of unit capacity and unit impact. Combining this with the equation FORMULA4 will give us the generalization bound. Theorem 1. Given a training set S = {x i} m i=1 and γ > 0, Rademacher complexity of the composition of loss function γ over the class F W defined in equations FORMULA6 and is bounded as follows: DISPLAYFORM0 The proof is given in the supplementary Section C. The main idea behind the proof is a new technique to decompose the complexity of the network into complexity of the hidden units. To our knowledge, all previous works decompose the complexity to that of layers and use Lipschitz property of the network to bound the generalization error. However, Lipschitzness of the layer is a rather weak property that ignores the linear structure of each individual layer. Instead, by decomposing the complexity across the hidden units, we get the above tighter bound on the Rademacher complexity of the two layer neural networks. The generalization bound in Theorem 1 is for any function in the function class defined by a specific choice of α and β fixed before the training procedure. To get a generalization bound that holds for all networks, it suffices to cover the space of possible values for α and β and take a union bound over it. The following theorem states the generalization bound for any two layer ReLU network 2.Theorem 2. For any h ≥ 2, γ > 0, δ ∈ and U 0 ∈ R h×d, with probability 1 − δ over the choice of the training set DISPLAYFORM1 and U ∈ R h×d, the generalization error is bounded as follows: DISPLAYFORM2 The above generalization bound empirically improves over the existing bounds, and decreases with increasing width for networks learned in practice (see Section 2.3). We also show an explicit lower bound for the Rademacher complexity (Theorem 3), matching the first term in the above generalization bound, thereby showing its tightness. The additive factorÕ(h/m) in the above bound is the of taking the union bound over the cover of α and β. As we see in FIG6, in the regimes of interest this additive term is small and does not dominate the first term, ing in an overall decrease in capacity with over-parametrization. In Appendix Section B, we further extend the generalization bound in Theorem 2 to p norms, presenting a finer tradeoff between the two terms. In TAB1 with respect to the size of the network trained on CIFAR-10. BID1 ): The first term in their bound U 2 V − V 0 1,2 is of smaller magnitude and behaves roughly similar to the first term in our bound U 0 2 V F (see FIG2 last two panels). The key complexity term in their bound is U − U 0 1,2 V 2, and in our bound is U − U 0 F V F, for the range of h considered. V 2 and V F differ by number of classes, a small constant, and hence behave similarly. However, U − U 0 1,2 can be as big as DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 when most hidden units have similar capacity. Infact their bound increases with h mainly because of this term U − U 0 1,2. As we see in the first and second panels of Experimental comparison. We train two layer ReLU networks of size h on CIFAR-10 and SVHN datasets with values of h ranging from 2 6 to 2 15. The training and test error for CIFAR-10 are shown in the first panel of FIG0, and for SVHN in the left panel of FIG4. We observe for both datasets that even though a network of size 128 is enough to get to zero training error, networks with sizes well beyond 128 can still get better generalization, even when trained without any regularization. We further measure the unit-wise properties introduce in the paper, namely unit capacity and unit impact. These quantities decrease with increasing h, and are reported in the right panel of FIG0 and second panel of FIG4. Also notice that the number of epochs required for each network size to get 0.01 cross-entropy loss decreases for larger networks as shown in the third panel of FIG4.For the same experimental setup, FIG6 compares the behavior of different capacity bounds over networks of increasing sizes. Generalization bounds typically scale as C/m where C is the effective capacity of the function class. The left panel reports the effective capacity C based on different measures calculated with all the terms and constants. We can see that our bound is the only that decreases with h and is consistently lower that other norm-based data-independent bounds. Our bound even improves over VC-dimension for networks with size larger than 1024. While the actual numerical values are very loose, we believe they are useful tools to understand the relative generalization behavior with respect to different complexity measures, and in many cases applying a set of data-dependent techniques, one can improve the numerical values of these bounds significantly (; BID0 each capacity bound normalized by its maximum in the range of the study for networks trained on CIFAR-10 and SVHN respectively. For both datasets, our capacity bound is the only one that decreases with the size even for networks with about 100 million parameters. All other existing norm-based bounds initially decrease for smaller networks but then increase significantly for larger networks. Our capacity bound therefore could potentially point to the right properties that allow the over-parametrized networks to generalize. Finally we check the behavior of our complexity measure under a different setting where we compare this measure between networks trained on real and random labels . We plot the distribution of margin normalized by our measure, computed on networks trained with true and random labels in the last panel of FIG4 -and as expected they correlate well with the generalization behavior. In this section we will prove a lower bound for the Rademacher complexity of neural networks, that matches the dominant term in the upper bound of Theorem 1. We will show our lower bound on a smaller function class than F W, with an additional constraint on spectral norm of the hidden layer. This allows for comparison with the existing , and also extends the lower bound to the bigger class F W. Theorem 3. Define the parameter set DISPLAYFORM0 and let F W be the function class defined on W by equation. Then, for any DISPLAYFORM1 Clearly, W ⊆ W, since it has an extra constraint. The complete proof is given in the supplementary Section C.3.Published as a conference paper at ICLR 2019The above complexity lower bound matches the first term, DISPLAYFORM2, in the upper bound of Theorem 1, upto 1 γ, which comes from the 1 γ -Lipschitz constant of the ramp loss l γ. To match the second term in the upper bound for Theorem 1, consider the setting with c = 1 and β = 0, ing in, DISPLAYFORM3 where DISPLAYFORM4 In other words, when β = 0, the function class DISPLAYFORM5, and therefore we have the above lower bound, showing that the upper bound provided in Theorem 1 is tight. It also indicates that even if we have more information, such as bounded spectral norm with respect to the reference matrix is small (which effectively bounds the Lipschitz of the network), we still cannot improve our upper bound. To our knowledge, all the previous capacity lower bounds for spectral norm bounded classes of neural networks with a scalar output and element-wise activation functions correspond to the Lipschitz constant of the network. Our lower bound strictly improves over this, and shows a gap between the Lipschitz constant of the network (which can be achieved by even linear models), and the capacity of neural networks. This lower bound is non-trivial, in the sense that the smaller function class excludes the neural networks with all rank-1 matrices as weights, and thus shows a Θ(√ h)-capacity gap between the neural networks with ReLU activations and linear networks. The lower bound therefore does not hold for linear networks. Finally, one can extend the construction in this bound to more layers by setting all the weight matrices in the intermediate layers to be the Identity matrix. for the function class defined by the parameter set: DISPLAYFORM6 Note that s 1 s 2 is the Lipschitz bound of the function class F Wspec. Given W spec with bounds s 1 and s 2, choosing α and β such that α 2 = s 1 and max i∈[h] β i = s 2 in W ⊂ W spec. Hence we get the following from Theorem 3, showing a stronger lower bound for this function class as well. DISPLAYFORM7 Hence our improves the lower bound in by a factor of √ h. Theorem 7 in also gives a Ω(s 1 s 2 √ c) lower bound, c is the number of outputs of the network, for the composition of 1-Lipschitz loss function and neural networks with bounded spectral norm, or ∞-Schatten norm. Our above even improves on this lower bound. In this paper we present a new capacity bound for neural networks that empirically decreases with the increasing number of hidden units, and could potentially explain the better generalization performance of larger networks. In particular, we focused on understanding the role of width in the generalization behavior of two layer networks. More generally, understanding the role of depth and the interplay between depth and width in controlling capacity of networks, remain interesting directions for future study. We also provided a matching lower bound for the capacity improving on the current lower bounds for neural networks. While these bounds are useful for relative comparison between networks of different size, their absolute values still remain larger than the number of training samples, and it is of interest to get bounds with numerically smaller values. In this paper we do not address the question of whether optimization algorithms converge to low complexity networks in the function class considered in this paper, or in general how does different hyper parameter choices affect the complexity of the recovered solutions. It is interesting to understand the implicit regularization effects of the optimization algorithms (a; ;) for neural networks, which we leave for future work. Below we describe the setting for each reported experiment. In this experiment, we trained a pre-activation ResNet18 architecture on CIFAR-10 dataset. The architecture consists of a convolution layer followed by 8 residual blocks (each of which consist of two convolution) and a linear layer on the top. Let k be the number of channels in the first convolution layer. The number of output channels and strides in residual blocks is then [k, k, 2k, 2k, 4k, 4k, 8k, 8k] and respectively. Finally, we use the kernel sizes 3 in all convolutional layers. We train 11 architectures where for architecture i we set k = 2 2+i/2. In each experiment we train using SGD with mini-batch size 64, momentum 0.9 and initial learning rate 0.1 where we reduce the learning rate to 0.01 when the cross-entropy loss reaches 0.01 and stop when the loss reaches 0.001 or if the number of epochs reaches 1000. We use the reference line in the plots to differentiate the architectures that achieved 0.001 loss. We do not use weight decay or dropout but perform data augmentation by random horizontal flip of the image and random crop of size 28 × 28 followed by zero padding. We trained fully connected feedforward networks on CIFAR-10, SVHN and MNIST datasets. For each data set, we trained 13 architectures with sizes from 2 3 to 2 15 each time increasing the number of hidden units by factor 2. For each experiment, we trained the network using SGD with mini-batch size 64, momentum 0.9 and fixed step size 0.01 for MNIST and 0.001 for CIFAR-10 and SVHN. We did not use weight decay, dropout or batch normalization. For experiment, we stopped the training when the cross-entropy reached 0.01 or when the number of epochs reached 1000. We use the reference line in the plots to differentiate the architectures that achieved 0.01 loss. Evaluations For each generalization bound, we have calculated the exact bound including the logterms and constants. We set the margin to 5th percentile of the margin of data points. Since bounds in BID2 and Neyshabur et al. (2015c) are given for binary classification, we multiplied BID2 by factor c and Neyshabur et al. (2015c) by factor √ c to make sure that the bound increases linearly with the number of classes (assuming that all output units have the same norm). Furthermore, since the reference matrices can be used in the bounds given in and BID0, we used random initialization as the reference matrix. When plotting distributions, we estimate the distribution using standard Gaussian kernel density estimation. Figures 6 and 7 show the behavior of several measures on networks with different sizes trained on SVHN and MNIST datasets respectively. The left panel of FIG10 shows the over-parametrization phenomenon in MNSIT dataset and the middle and right panels compare our generalization bound to others. In this section we generalize the Theorem 2 to p norm. The main new ingredient in the proof is the Lemma 11, in which we construct a cover for the p ball with entry-wise dominance. Theorem 5. For any h, p ≥ 2, γ > 0, δ ∈ and U 0 ∈ R h×d, with probability 1 − δ over the choice of the training set DISPLAYFORM0 and U ∈ R h×d, the generalization error is bounded as follows: DISPLAYFORM1 where. p,2 is the p norm of the row 2 norms. For p of order ln h, h e −p ≈ constant improves on the √ h additive term in Theorem 2 and DISPLAYFORM2 which is a tight upper bound for V F and is of the same order if all rows of V have the same norm -hence giving a tighter bound that decreases with h for larger values. In particular for p = ln h we get the following bound. Corollary 6. Under the settings of Theorem 5, with probability 1 − δ over the choice of the training set S = {x i} m i=1, for any function f (x) = V[Ux] +, the generalization error is bounded as follows: DISPLAYFORM3 We start by stating a simple lemma which is a vector-contraction inequality for Rademacher complexities and relates the norm of a vector to the expected magnitude of its inner product with a vector of Rademacher random variables. We use the following technical from in our proof. Lemma 7 (Propostion 6 of Maurer FORMULA2). Let ξ i be the Rademacher random variables. For any vector v ∈ R d, the following holds: DISPLAYFORM4 The above DISPLAYFORM5 Proof. DISPLAYFORM6 (i) follows from the Jensen's inequality. We next show that the Rademacher complexity of the class of networks defined in and FORMULA6 can be decomposed to that of hidden units. Lemma 9 (Rademacher Decomposition). Given a training set S = {x i} m i=1 and γ > 0, Rademacher complexity of the class F W defined in equations and is bounded as follows: DISPLAYFORM7 Proof. We prove the inequality in the lemma statement using induction on t in the following inequality: DISPLAYFORM8 where for simplicity of the notation, we let φ DISPLAYFORM9 The above statement holds trivially for the base case of t = 1 by the definition of the Rademacher complexity. We now assume that it is true for any t ≤ t and prove it is true for t = t + 1. DISPLAYFORM10 The last inequality follows from the √ 2 γ Lipschitzness of the ramp loss. The ramp loss is 1/γ Lipschitz with respect to each dimension but since the loss at each point only depends on score of the correct labels and the maximum score among other labels, it is √ 2 γ -Lipschitz. By Lemma 7, the right hand side of the above inequality can be bounded as follows: DISPLAYFORM11 This completes the induction proof. Lemma 10 (Ledoux-Talagrand contraction,). Let f: R + → R + be convex and increasing. Let φ i: R → R satisfy φ i = 0 and be L-Lipschitz. Let ξ i be independent Rademacher random variables. For any T ⊆ R n, DISPLAYFORM12 The above lemma will be used in the following proof of Theorem 1.Proof of Theorem 1. By Lemma 9, we have: DISPLAYFORM13 Now we can apply Lemma 10 with n = m × c, f DISPLAYFORM14, and we get DISPLAYFORM15 The proof is completed by taking sum of above inequality over j from 1 to h. We start by the following covering lemma which allows us to prove the generalization bound in Theorem 5 without assuming the knowledge of the norms of the network parameters. The following lemma shows how to cover an p ball with a set that dominates the elements entry-wise, and bounds the size of a one such cover. Lemma 11 (p covering lemma). Given any, D, β > 0, p ≥ 2, consider the set S DISPLAYFORM0 Proof. We prove the lemma by construction. DISPLAYFORM1 By Lemma 11, picking = ((1 + µ) 1/p − 1), we can find a set of vectors, DISPLAYFORM2 Lemma 13. For any h, p ≥ 2, c, d, γ, µ > 0, δ ∈ and U 0 ∈ R h×d, with probability 1 − δ over the choice of the training set DISPLAYFORM3 c×h and U ∈ R h×d, the generalization error is bounded as follows: DISPLAYFORM4 where DISPLAYFORM5 and. p,2 is the p norm of the column 2 norms. Proof. This lemma can be proved by directly applying union bound on Lemma 12 with for every C 1 ∈. For V p,2 ≤ 1 h 1/2−1/p, we can use the bound where C 1 = 1, and the additional constant 1 in Eq. 12 will cover that. The same is true for the case of U p,2 ≤ i h 1/2−1/p X F. When any of h 1/2−1/p V p,2 and h 1/2−1/p X F U p,2 is larger than DISPLAYFORM6, the second term in Eq. 12 is larger than 1 thus holds trivially. For the rest of the case, there exists (C 1, C 2) such that h 1/2−1/p C 1 ≤ h 1/2−1/p V p,2 + 1 and h 1/2−1/p C 2 ≤ h 1/2−1/p X F X F U p,2 + 1. Finally, we have γ √ m 4 ≥ 1 otherwise the second term in Eq. 12 is larger than 1. Therefore, DISPLAYFORM7 We next use the general in Lemma 13 to give specific for the case p = 2.Lemma 14. For any h ≥ 2, c, d, γ > 0, δ ∈ and U 0 ∈ R h×d, with probability 1 − δ over the choice of the training set S = {x i} m i=1 ⊂ R d, for any function f (x) = V[Ux] + such that V ∈ R c×h and U ∈ R h×d, the generalization error is bounded as follows: DISPLAYFORM8 Proof. To prove the lemma, we directly upper bound the generalization bound given in Lemma 13 for p = 2 and µ = 3 √ 2 4 − 1. For this choice of µ and p, we have 4(µ + 1) 2/p ≤ 3 √ 2 and ln N p,h is bounded as follows:ln N p,h = ln h/µ + h − 2 h − 1 ≤ ln e h/µ + h − 2 h − 1 h−1 = (h − 1) ln e + e h/µ − 1 h − 1 ≤ (h − 1) ln e + e h/µ h − 1 ≤ h ln(e + 2e/µ) ≤ 5hProof of Theorem 2. The proof directly follows from Lemma 14 and usingÕ notation to hide the constants and logarithmic factors. Next lemma states a generalization bound for any p ≥ 2, which is looser than 14 for p = 2 due to extra constants and logarithmic factors. Lemma 15. For any h, p ≥ 2, c, d, γ > 0, δ ∈ and U 0 ∈ R h×d, with probability 1 − δ over the choice of the training set S = {x i} m i=1 ⊂ R d, for any function f (x) = V[Ux] + such that V ∈ R c×h and U ∈ R h×d, the generalization error is bounded as follows: DISPLAYFORM9. p,2 is the p norm of the column 2 norms. Proof. To prove the lemma, we directly upper bound the generalization bound given in Lemma 13 for µ = e p − 1. For this choice of µ and p, we have (µ + 1) 2/p = e 2. Furthermore, if µ ≥ h, N p,h = 0, otherwise ln N p,h is bounded as follows: DISPLAYFORM10 = (h/(e p − 1) − 1) ln e + e h − 1 h/(e p − 1) − 1 ≤ (e 1−p h − 1) ln (eh)Since the right hand side of the above inequality is greater than zero for µ ≥ h, it is true for every µ > 0.Proof of Theorem 5. The proof directly follows from Lemma 15 and usingÕ notation to hide the constants and logarithmic factors. Proof of Theorem 3. We will start with the case h = d = 2 k, m = n2 k for some k, n ∈ N.We will pick V = α = [α 1 . . . α 2 k] for every ξ, and DISPLAYFORM0, where x i:= e i n. That is, the whole dataset are divides into 2 k groups, while each group has n copies of a different element in standard orthonormal basis. We further define j (ξ) = For any ξ ∈ {−1, 1} n, let Diag(β) be the square diagonal matrix with its diagonal equal to β and F(ξ) be the following: F(ξ):= [f 1, f 2, . . ., f 2 k] such that if i (ξ) ≥ 0, f i = f i, and if i (ξ) < 0, f i = 0, and we will choose U(ξ) as Diag(β) × F(ξ).Since F is orthonormal, by the definition of F(ξ), we have F(ξ) 2 ≤ 1 and the 2-norm of each row of F is upper bounded by 1. Therefore, we have U(ξ) 2 ≤ Diag(β) 2 F(ξ) 2 ≤ max i β i, and

Represent the TLDR for finding the corresponding scientific article.

We suggest a generalization bound that could partly explain the improvement in generalization with over-parametrization.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.