Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Normalising Flows (NFs) are a class of likelihood-based generative models that have recently gained popularity. They are based on the idea of transforming a simple density into that of the data. We seek to better understand this class of models, and how they compare to previously proposed techniques for generative modeling and unsupervised representation learning. For this purpose we reinterpret NFs in the framework of Variational Autoencoders (VAEs), and present a new form of VAE that generalises normalising flows. The new generalised model also reveals a close connection to denoising autoencoders, and we therefore call our model the Variational Denoising Autoencoder (VDAE). Using our unified model, we systematically examine the model space between flows, variational autoencoders, and denoising autoencoders, in a set of preliminary experiments on the MNIST handwritten digits. The experiments shed light on the modeling assumptions implicit in these models, and they suggest multiple new directions for future research in this space. Unsupervised learning offers the promise of leveraging unlabeled data to learn representations useful for downstream tasks when labeled data is scarce BID47, or even to generate novel data in domains where it is costly to obtain BID15. Generative models are particularly appealing for this as they provide a statistical model of the data, x, usually in the form of a joint probability density p (x). The model's density function, its samples and representations can then be leveraged in applications ranging from semi-supervised learning and speech and (conditional) image synthesis BID44 BID30 BID14 BID26 to gene expression analysis BID13 and molecule design BID10.In practice, data x is often high-dimensional and the optimization associated with learning p (x) can be challenging due to an abundance of local minima BID39 and difficulty in sampling from rich high-dimensional distributions BID34. Despite this, generative modelling has undergone a surge of advancements with recent developments in likelihood-based models BID25 BID8 BID44 and Generative Adversarial Networks (GANs; BID11). The former class is particularly attractive, as it offers (approximate) likelihood evaluation and the ability to train models using likelihood maximisation, as well as interpretable latent representations. Autoencoders have a rich history in the unsupervised learning literature owing to their intuitive and simple construction for learning complex latent representations of data. Through fitting a parameterised mapping from the data through a lower dimensional or otherwise constrained layer back to the same data, the model learns to summarise the data in a compact latent representation. Many variants of autoencoders have been proposed to encourage the model to better encode the underlying structure of the data though regularising or otherwise constraining the model (e.g., BID38 BID1 .Denoising Autoencoders (DAEs) are a variant of the autoencoder under which noise is added to the input data that the model must then output noise-free, i.e. x = f θ (x +) where is sampled from a, possibly structured BID48 BID49, noise distribution ∼ q. They are inspired by the idea that a good representation z would be robust to noise corrupting the data x and that adding noise would discourage the model from simply learning the identity mapping. Although DAEs have been cast as generative models, sampling and computing likelihoods under the model remains challenging. Variational Autoencoders (VAEs) instead assume a probabilistic latent variable model, in which n-dimensional data x correspond to m-dimensional latent representations z following some tractable prior distribution, i.e. x ∼ p φ (x|z) with z ∼ p (z) BID25. The task is then to learn parameters φ, which requires maximising the log marginal likelihood DISPLAYFORM0 In the majority of practical cases (e.g. p φ (x|z) taken to be a flexible neural network-conditional distribution) the above integral is intractable. A variational lower bound on the marginal likelihood is constructed using a variational approximation q θ (z|x) to the unknown posterior p (z|x): DISPLAYFORM1 The right-hand side of, denoted L (θ, φ), is known as the evidence lower bound (ELBO). It can be jointly optimised with stochastic optimisation w.r.t. parameters θ and φ in place of.Conditionals q θ (z|x) and p φ (x|z) can be viewed respectively as probabilistically encoding data x in the latent space, and reconstructing it from samples of this encoding. The first term of the ELBO encourages good reconstructions, whereas the second term encourages the model's latent variables to be distributed according to the prior p (z). Generating new data using this model is accomplished by reconstructing samples from the prior. Normalising Flows (NFs) suppose that the sought distribution p (x) can be obtained by warping a simple base density p (z), e.g. a normal distribution BID36. They make use of the change of variables formula to obtain p (x) through a learned invertible transformation z = f θ (x) as DISPLAYFORM2 Typically, f θ: R n → R n is obtained by stacking several simpler mappings, i.e. DISPLAYFORM3 and the log-determinant obtained as the sum of log-determinants of these mappings. This formulation allows for exact maximum likelihood learning, but requires f θ to be invertible and to have a tractable inverse and Jacobian determinant. This restricts the flexibility of known transformations that can be used in NFs BID8 BID3 and leads to large and computationally intensive models in practice BID26.NFs can also be thought of as VAEs with encoder and decoder modelled as Dirac deltas p θ (x|z) = δ (f θ (z)) and q θ (z|x) = δ f −1 θ (x), constructed using a restricted set of transformations. Furthermore, because NFs model continuous density, to prevent trivial solutions with infinite point densities discrete data must be dequantised by adding random noise BID42 BID40.The contribution of this work is two-fold. First, we shed new light on the relationship between DAEs, VAEs and NFs, and discuss the pros and cons of these model classes. Then, we also introduce several extensions of these models, which we collectively refer to as the Variational Denoising Autoencoders (VDAEs).In the most general form VDAEs generalise NFs and DAEs to discrete data and learned noise distributions. However, when the amount of injected noise is small, VDAE attains a form that allows for using non-invertible transformations (e.g. f θ : R n → R m, with m n). We demonstrate these theoretical advantages through preliminary experimental on the binary and continuous versions of the MNIST dataset. We model data x, a n dimensional vector that can have either continuous or discrete support. As is customary for VAEs, our model for x is hierarchical and assumes a set of latent variables z with tractable prior distribution p(z), and a flexible neural-network conditional distribution p(x|z). On top of this standard VAE setup, we specify the dimension of z to equal the dimension of the data x. In order to form the variational lower bound to train this model, we need an approximate inference model, or encoder, q θ (z|x). Here, we will use an encoder that samples the latents z as DISPLAYFORM0 where q is a tractable noise distribution and f θ (x) is a one-to-one transformation with tractable Jacobian-determinant. In order to use the encoder q θ (z|x) implied by this procedure, we not only need to sample from it, but we must also evaluate its entropy for the KL-term in. To do this we make use of the fact that z is a one-to-one transformation of the noise, given the training data x. Using the standard formulas for a change of variables, we thus get the following expression for the entropy of q θ (z|x): DISPLAYFORM1 where q (x|x) is a distribution whose sampling process is described in. Our variational lower bound on the data log marginal likelihood then becomes DISPLAYFORM2 where againx = x + and z = f θ (x).This is similar to a denoising autoencoder in that we try to reconstruct the original data x from the corrupted datax through the conditional model p(x|z). The difference with classical denoising autoencoders is that our objective has additional terms that regularise our latent representations z to be distributed according to a prior distribution p(z). In addition, the proposed setup allows us to learn the noise distribution q, where this is treated as a fixed hyperparameter in the literature on denoising autoencoders. This model is also a generalisation of normalising flows. Specifically, consider the special case where we take DISPLAYFORM3 then the lower bound in becomes the standard normalising flow log-likelihood. We provide a detailed derivation in Appendix A.The advantage of our generalised model over standard normalising flows is that our model allows for non-zero noise level σ 2. Interestingly, successful applications of normalising flows in the literature often already add a significant amount of noise in order to dequantise the data, and empirical suggest higher amounts of noise lead to models that produce better-looking samples (e.g. BID26 model only the top 5 bits of the data).In addition, our model does not require tying the parameters of the encoder and decoder. Although we are still using a flow-based encoder q θ (z|x), our decoder is not restricted to a specific functional form. The conditional distribution p(x|z) can e.g. have discrete support if the data x is discrete, making our model naturally applicable to data such as text or other highly-structured data, without requiring an explicit dequantisation step. When adding a significant amount of noise, a decoupled decoder will generally be able to achieve a higher variational lower bound compared to using the tied-parameter decoder. The VAE we proposed in Section 2 is more general than NFs, but it still requires an invertible one-to-one encoder with tractable Jacobian-determinant. This restricts our modeling choices since all transformations used in the encoder can only be chosen from a small set of transformations for which we know how to compute inverses and Jacobian-determinants. Additionally, the representation given by our encoder will be of the same dimension as our data x, which may not be optimal for all applications (e.g. model-based reinforcement learning BID15 BID17 or compression BID2). To relax these restrictions further we generalise our model to allow non-invertible encoders as well. We proceed by taking our model from Section 2, withx = x + and z = f θ (x), and performing a Taylor expansion of the ing latent variables z(x,) around = 0 (see Appendix B). This gives DISPLAYFORM0 where DISPLAYFORM1 is the Jacobian of f θ.For small noise levels, as used in Section 2, the O term becomes negligible. If the noise distribution is Gaussian, i.e. q = N 0, σ 2 I n, this means that for small σ we get DISPLAYFORM2 Using this form of encoder q θ (z|x), together with general prior p(z) and conditional distribution p(x|z), we get a VAE that still generalises NFs but now also allows us to choose non-invertible non-one-to-one transformations f θ. We refer to this even broader class of VAE as L-VDAE, for Linearised-VDAE. Evaluating the entropy H [q θ (z|x)] in this case requires computing the log-determinant of the covariance matrix C = JJ T for the data x: DISPLAYFORM0 where m is the dimensionality of z. When using transformations f θ without a tractable Jacobian (e.g. the general Residual Network (ResNet; BID20) blocks), we explicitly evaluate C and compute log det C = m i log λ i, where the eigenvalues λ i are obtained using the eigenvalue decomposition C = QΛQ T with Λ = diag (λ i |i = 1, . . ., m). The decomposition is further re-used in the backward pass when evaluating the derivative of the log-determinant using Jacobi's formula: DISPLAYFORM1 Evaluation of the Jacobian J can be done by performing reverse-mode automatic differentiation with respect to each element of z, thus incurs a factor of m additional computational cost. Covariance matrix C is obtained using a single matrix multiplication and takes O m 2 n operations with the eigenvalue decomposition taking another O m 3 operations. Taken together, evaluation of FORMULA11 takes O m 2 n operations, which is comparable to the O d 3 cost of Glow's 1x1 invertible convolutions in later layers (i.e. after repeating the use of the multi-scale architecture from BID9 that trades spatial dimensions for channels), where d refers to the number of channels used in the convolution. This computational cost is permissive for small latent space dimensionalities m. However, scaling up L-VDAE to larger latent spaces would require stochastic approximations of the log-determinant BID18. These approximations can be implemented efficiently through Jacobian-vector and vector-Jacobian products, without evaluating C or J explicitly, and can be optimised directly by backpropagating through the approximation. With this approach computational complexity will be linear in n subject to some regularity conditions. Sampling from the Gaussian variational posterior q φ is necessary for training and inference in L-VDAE. It can be accomplished using the standard reparameterisation trick BID25, where random normal noise ω ∼ N (0, I n) is transformed into the posterior sample as z = f θ (x) + Jω. We implement this as a Jacobian-vector product, which enables efficient sampling for cases when the Jacobian log-determinant of f θ is cheaper to evaluate than the Jacobian itself (e.g. when f θ is a flow). VDAE blends ideas from the VAE and NFs literature and is closely related to both model families. It is most similar to methods that combined variational inference and NFs by using the latter as part of the approximate variational posterior BID36 BID29 BID43. These methods use a strategy in which samples from the (Gaussian) posterior are further transformed with a NF, whereas in VDAE the posterior distribution is implicitly defined using a sampling procedure inspired by DAE, where posterior samples are obtained by transforming data with added noise using NFs. VDAE is a natural formulation of DAEs as probabilistic models. It is conceptually similar to the Denoising VAEs BID23, which propose an alternative probabilistic formulation of DAEs as VAEs. The method of Im et al., however, does not generalise NFs and, in contrast to VDAE, it requires explicitly choosing the type and amount of corruption noise. The idea of challenging the default choice of using uniform noise for dequantisation in NFs was also explored in Flow++ BID21, where the authors learned a flexible conditional noise model q (|x) as NF itself. Our sampling procedure is similar to dequantisation in Flow++, as it can be viewed as a of applying a NF to dequantisation given by an implicitly conditioned noise model. The main differences, however, are that in VDAE the decoder reconstructs the original (quantised) data, which is also what makes our model applicable to highly-structured data; and, in contrast to Flow++, VDAE can inject substancially more noise than a single dequantisation bin. In relation to VAEs, the linearised form of VDAE can be viewed as an extension of the vanilla VAE BID25 ) that replaces the diagonal Gaussian posterior with a Jacobian-based full covariance posterior. It is thus similar to methods that extend VAE with more flexible prior (e.g. autoregressive BID7 or mixture ) or variational posterior (e.g. full covariance Gaussian (a) or mixture BID35 BID32 ) distributions. Notably, unlike some of these methods, L-VDAE does not increase the number of parameters of the inference or generative networks. As a method that increases flexibility of transformations in NFs, L-VDAE with non-invertible encoders can be compared to Invertible Residual Networks (i-ResNets; BID3) and FFJORD BID12. These methods too depart from the requirement of restricting the form of the Jacobian of the ing transformation. Both, i-ResNets and FFJORD also drop the requirement of having an analytical inverse, which is similar to how VDAE seeks to learn an approximate inverse using its decoder network. However, unlike VDAE, these methods guarantee invertibility and provide ways of computing the exact inverse. Notably, the methods differ considerably in how they achieve the above generalisations. In i-ResNets Behrmann et al. make use of the ResNet network architecture BID20 and identify conditions on the eigenvalues of the residual blocks, under which they parameterise invertible mappings. They then make use of spectral normalisation BID33 to guarantee that the condition is satisfied throughout training; and employ fixed point iteration to invert the residual blocks for generation. i-ResNets further lift the restriction on the form of the Jacobian in a computationally tractable way by using Taylor series expansion in conjunction with stochastic trace estimation BID22.FFJORD BID12 ) is inspired by the re-interpretation of ResNets and NFs as discrete approximations of solutions to the initial value problem of some underlying ODE continuously transforming data x (from data-space to the latent z-space;). BID12; parameterise this ODE as a neural network f θ (z (t), t) to obtain Continuous-time Normalising Flows (CNFs), in which the change in log-density at time t is given by the instantaneous change of variables formula DISPLAYFORM0 The right-hand side of FORMULA13 is given by the trace of the Jacobian of transformation f θ instead of the log-determinant as in NFs. Combined with the use of stochastic trace estimation BID22, this difference alleviates the need to restrict transformations f θ to those with a tractable Jacobian log-determinant. However, the use of ODEs also necessitates employing an ODE solver to integrate for every evaluation of, and backpropagation through log p θ (z (t)). The number of function evaluations required for this increases with training and may become prohibitively large BID12.Finally, VDAE is loosely related to autoregressive generative models, as they both fall into the class of likelihood-based generative models. Autoregressive models factorise likelihood of highdimensional data as a product of simple per-dimension conditional distributions, i.e. p (x) = i p (x i |x 0, . . ., x i−1) (van den BID44 b). Factorised structure of these models necessitates sequential sampling, and a good choice of the ordering of dimensions of x. Overcoming these challenges in practice often requires highly engineered solutions, for example as in BID46 or BID31. Furthermore, data representations formed by hidden layers of autoregressive models appear to be more challenging to manipulate than in VAEs or NFs BID26. We performed empirical studies of the performance of VDAE on the image generation task on the MNIST dataset BID30, comparing it to a VAE implementation with a fully factorised Gaussian posterior and to the NICE BID8 normalising flow as baselines. For the VDAE encoder we used additive couplings to construct f θ from the implicit variational posterior; and, unless otherwise specified, fully-connected ResNet blocks followed by a sigmoid transformation to obtain the decoder parameters µ φ and p φ. A Gaussian distribution N (µ φ (z), λI n ) with a learned parameter λ was used for the continuous MNIST decoder; and Bernoulli (p φ (z)) for binary MNIST. Similarly, unless otherwise specified, ResNet blocks with linear projection layers to change dimensionality were used for the L-VDAE encoder and decoder. Details of the chosen architectures can be found in Appendix D. To model discrete 8-bit pixel values with continuous density models, we followed the procedure of BID42 to dequantise the data, and added noise u ∼ U to the pixel values prior to normalising them to. Note that for VDAE this was done prior and in addition to adding noise from the posterior sampling procedure to the inputs. Decoupled encoder and decoder We start by confirming for a range of noise levels and architectures that de-coupling the encoder and decoder networks in VDAE allows for achieving higher ELBOs. FIG0 compares ELBO attained by VDAE with Gaussian noise q = N 0, σ 2 I for a range of fixed σ. The show that any decoupling of the weights improves over the coupled network, in which the NICE flow is used in the encoder and its inverse -in the decoder. Specifically, we observe that for architectures with a sigmoid activation in the last layer of the decoder, the ELBO rapidly improves with decreasing noise levels. Based on these , in the following experiments we only consider the more general ResNet architecture in the VDAE decoder. We report average test set performance over 10 training runs; when sufficiently large, standard deviations are also given. Qualitative samples are drawn from models with the best test ELBO among the training runs. NLL was estimated via 5000 importance samples as in.Quantitative We now consider the cases when i) noise variance σ 2, in case of VDAE; or ii) in case of L-VDAE, the covariance scale σ from FORMULA10; are optimised together with the model. Results of these experiments are shown in TAB0. For ease of presentation, we also include evaluation for existing flow models (reproduced from BID3).We first note that for cases when the latent dimensionality is smaller than the input space (i.e. m < n), L-VDAE consistently outperforms the VAE baseline in terms of the achieved ELBO, albeit by a small margin. This is consistent with L-VDAE having a more powerful variational posterior. Moreover, for L-VDAE increasing the dimensionality of the latent space consistently improves the variational lower bound. Surprisingly, L-VDAE with n = m and VDAE break this trend and do not improve on the ELBOs obtained for m = 128. We also note that neither of our proposed extensions manage to achieve likelihoods comparable to NFs, including the NICE baseline. Both shortcomings could be explained by the difference in architectures between the methods. In contrast to the L-VDAE with m = n, which employs a NICE flow in the encoder, L-VDAE with m < n makes use of the more expressive ResNet blocks. Similarly, the flexibility of the NICE flow used in VDAE for the implicit posterior may be insufficient for a denoising VAE. We also observe that when using a NICE flow in the decoder, VDAE outperforms L-VDAE in terms of likelihood, signalling that the VDAE approach can further improve on the linearised models, if combined with a more powerful flow. Qualitative We found that without additional regularisation, such as fixing the decoder variance λ 2 or the noise variance σ 2 to values larger than what would have been learned by the model, or assigning a higher weight to the KL-term in the optimisation objective, our models would not produce high-quality samples for the continuous MNIST dataset. We thus omit continuous MNIST model samples from the main text, but explore the effect of fixing the noise variance on sample quality in Appendix E. To explore the applicability of VDAE to structured data, we applied it to the binarised version of the MNIST dataset. As is customary for dynamic MNIST, digits were binarised by sampling from a Bernoulli distribution given by the pixel intensities. Results in TAB1 mirror those we observed on the continuous MNIST, namely L-VDAE consistently achieves higher ELBO than the VAE baseline, which tends to improve as the latent dimensionality grows; and L-VDAE and VDAE, which make use of NICE in the decoder, attain significantly worse likelihood despite the increased dimensionality. Finally, VDAE also improves on L-VDAE with a NICE encoder. However, as shown in FIG1, and in contrast to the continuous MNIST , all our models produce plausible handwritten digit samples. We introduced Variational Denoising Autoencoders (VDAEs), a family of models the bridges the gap between VAEs, NFs and DAEs. Our model extends NFs to discrete data and non-invertible encoders that use lower-dimensional latent representations. Preliminary experiments on the MNIST handwritten digits demonstrate that our model can be successfully applied to data with discrete support, attaining competitive likelihoods and generating plausible digit samples. We also identified a failure mode of our models, in which their performance does not scale well to cases when latent and input dimensionalities are the same (i.e. when a flow-based encoder is used).Future work should address limitations of the method identified in our experiments. In particular, replacing additive coupling blocks with the more powerful invertible convolutions, affine coupling blocks and invertible residual blocks BID9 BID26 BID3 can significantly improve the variational posterior for high dimensions. It can also be interesting to explicitly condition the transformation f θ used for defining the posterior sampling procedure on the data x, for example by defining f θ (x,) ≡ f x,θ using a hyper-network BID16. For convenience we start by repeating the variational lower bound for our VDAE model, as presented in Section 2: DISPLAYFORM0 n×n we zero-out all elements of W L. The same scheme was employed for initialising additive coupling blocks, which can be viewed as residual blocks of a restricted form. Projection layers reduce dimensionality of their inputs using a linear map y = xW with x ∈ R 1×n, y ∈ R 1×m and W ∈ R n×m. This generally leads to loss of information and makes model training harder. To mitigate this effect we initialise the rows of W using a set of m random orthogonal vectors. The decoder projection layers, mapping data to higher dimensions, are then initialised to W T. All models were trained for 1000 epochs using the ADAM optimiser BID24 ) with a batch size of 1000 samples. To improve stability of the training, the learning rate was warmed up from 10 −5 to the chosen learning rate (see below) over the first 10 epochs. Further, the KL term was warmed up by linearly annealing its weight β from 0 to 1 over the first 100 epochs BID5.For each experiment, the learning rate schedule S ∈ {linear, none}, learning rate α ∈ 10 −5, 10 DISPLAYFORM0 and ADAM optimiser parameters β 2 ∈ {0.9, 0.99, 0.999, 0.9999} and ∈ 10 −4, 10 −5, 10 DISPLAYFORM1 were determined by using Bayesian optimisation of the ELBO on the validation set. NICE When implementing the model (standalone, or part of VDAE), we closely followed the architecture and hyper-parameters described in BID8. Namely, the network consisted of 4 additive coupling blocks, each with 5 fully-connected hidden layers of size 1024 with ReLU activations, followed by a linear layer (see Appendix C). Dimension partitioning was alternated between even and odd dimensions after every block. When used as a standalone model, a L 2 regularisation with weight λ = 0.01 was used to improve sample quality. L-VDAE and vanilla VAE When not used in conjunction with a NICE model in the encoder, the L-VDAE and VAE models employed a fully-connected ResNet architecture with B consecutive residual blocks followed by a linear projection layer to higher or lower dimensions. In the encoder, the last projection layer parameterised the means of the Gaussian variational posterior (and, in case of VAE, a parallel projection layer parameterised the log-variances). A sequence of 4 residual-projection "blocks" was used with the last block i = 4 projecting to m dimensions (dimensionality of the latents) and the blocks before it, respectively to min 2 i · m, 28 × 28 dimensions. Each residual block consisted of 2 hidden layers with ReLU activations followed by a linear layer (see Appendix C). The residual block hidden size H ∈ {32, 64, 128, 256, 1024} and the block multiplicity B ∈ {1, 2, 3} were chosen through Bayesian optimisation as described above. Unless otherwise specified, when used together with a NICE model, the VDAE and L-VDAE models employed a ResNet architecture in the decoder. In this case, the ResNet architecture was chosen to closely resemble that of the NICE model. Specifically, hyper-parameter values B = 1 and H = 1024 were used, and no projection layers were employed. Priors We employed a logistic prior with s = 1 and µ = 0 (as in BID8) for models that made use of the NICE flow (even if it was only used in the encoder network); and a factorised normal prior otherwise. Sample quality deteriorates at the extremes of the noise level spectrum: at high noise levels the model appears to be unable to learn the distribution, whereas at low noise levels the model appears to focus too much on the reconstruction error instead of organising the latent space. Just as in DAEs, the noise variance σ 2 in VDAEs can be used as a regulariser; and can be tuned for sample quality.

Represent the short summary for finding the corresponding scientific article (long)

We explore the relationship between Normalising Flows and Variational- and Denoising Autoencoders, and propose a novel model that generalises them.

represent the tldr for finding the corresponding scientific article
------
Examples:
Provided: "We perform massive experimental studies characterizing the relationships between Jacobian norms, linear regions, and generalization." Match: "In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models. In this work, we investigate this tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations. Our experiments survey thousands of models with different architectures, optimizers, and other hyper-parameters, as well as four different image classification datasets. We find that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the input-output Jacobian of the network, and that this correlates well with generalization. We further establish that factors associated with poor generalization -- such as full-batch training or using random labels -- correspond to higher sensitivity, while factors associated with good generalization -- such as data augmentation and ReLU non-linearities -- give rise to more robust functions. Finally, we demonstrate how the input-output Jacobian norm can be predictive of generalization at the level of individual test points. The empirical success of deep learning has thus far eluded interpretation through existing lenses of computational complexity BID2, numerical optimization BID4 BID8 BID5 and classical statistical learning theory : neural networks are highly non-convex models with extreme capacity that train fast and generalize well. In fact, not only do large networks demonstrate good test performance, but larger networks often generalize better, counter to what would be expected from classical measures, such as VC dimension. This phenomenon has been observed in targeted experiments BID29, historical trends of Deep Learning competitions BID3, and in the course of this work (Figure 1).This observation is at odds with Occam's razor, the principle of parsimony, as applied to the intuitive notion of function complexity (see §A.2 for extended discussion). One resolution of the apparent contradiction is to examine complexity of functions in conjunction with the input domain. f (x) = x 3 sin(x) may seem decisively more complex than g(x) = x. But restrained to a narrow input domain of [−0.01, 0 .01] they appear differently: g remains a linear function of the input, while f (x) = O x 4 resembles a constant 0. In this work we find that such intuition applies to neural networks, that behave very differently close to the data manifold than away from it (§4.1).We therefore analyze the complexity of models through their capacity to distinguish different inputs in the neighborhood of datapoints, or, in other words, their sensitivity. We study two simple metrics presented in §3 and find that one of them, the norm of the input-output Jacobian, correlates with generalization in a very wide variety of scenarios. Train loss Figure 1: 2160 networks trained to 100% training accuracy on CIFAR10 (see §A.5.5 for experimental details). Left: while increasing capacity of the model allows for overfitting (top), very few models do, and a model with the maximum parameter count yields the best generalization (bottom right). Right: train loss does not correlate well with generalization, and the best model (minimum along the y-axis) has training loss many orders of magnitude higher than models that generalize worse (left). This observation rules out underfitting as the reason for poor generalization in low-capacity models. See BID29 for similar findings in the case of achievable 0 training loss. This work considers sensitivity only in the context of image classification tasks. We interpret the observed correlation with generalization as an expression of a universal prior on (natural) image classification functions that favor robustness (see §A.2 for details). While we expect a similar prior to exist in many other perceptual settings, care should be taken when extrapolating our findings to tasks where such a prior may not be justified (e.g. weather forecasting). We first define sensitivity metrics for fully-connected neural networks in §3. We then relate them to generalization through a sequence of experiments of increasing level of nuance:• In §4.1 we begin by comparing the sensitivity of trained neural networks on and off the training data manifold, i.e. in the regions of best and typical (over the whole input space) generalization.• In §4.2 we compare sensitivity of identical trained networks that differ in a single hyperparameter which is important for generalization.• Further, §4.3 associates sensitivity and generalization in an unrestricted manner, i.e. comparing networks of a wide variety of hyper-parameters such as width, depth, non-linearity, weight initialization, optimizer, learning rate and batch size.• Finally, §4.4 explores how predictive sensitivity (as measured by the Jacobian norm) is for individual test points. The novelty of this work can be summarized as follows:• Study of the behavior of trained neural networks on and off the data manifold through sensitivity metrics (§4.1).• Evaluation of sensitivity metrics on trained neural networks in a very large-scale experimental setting and finding that they correlate with generalization (§4. 2, §4.3, §4.4). §2 puts our work in context of related research studying complexity, generalization, or sensitivity metrics similar to ours. We analyze complexity of fully-connected neural networks for the purpose of model comparison through the following sensitivity measures (see §3 for details):• estimating the number of linear regions a network splits the input space into;• measuring the norm of the input-output Jacobian within such regions. A few prior works have examined measures related to the ones we consider. In particular, BID33; BID24; have investigated the expressive power of fully-connected neural networks built out of piecewise-linear activation functions. Such functions are themselves piecewise-linear over their input domain, so that the number of linear regions into which input space is divided is one measure of how nonlinear the function is. A function with many linear regions has the capacity to build complex, flexible decision boundaries. It was argued in BID33 BID24 that an upper bound to the number of linear regions scales exponentially with depth but polynomially in width, and a specific construction was examined. derived a tight analytic bound and considered the number of linear regions for generic networks with random weights, as would be appropriate, for instance, at initialization. However, the evolution of this measure after training has not been investigated before. We examine a related measure, the number of hidden unit transitions along one-dimensional trajectories in input space, for trained networks. Further motivation for this measure is discussed in §3.Another perspective on function complexity can be gained by studying their robustness to perturbations to the input. Indeed, BID36 demonstrate on a toy problem how complexity as measured by the number of parameters may be of limited utility for model selection, while measuring the output variation allows the invocation of Occam's razor. In this work we apply related ideas to a large-scale practical context of neural networks with up to a billion free parameters (§4.2, §4.3) and discuss potential ways in which sensitivity permits the application of Occam's razor to neural networks (§A.2). provide theoretical support for the relevance of robustness, as measured by the input-output Jacobian, to generalization. They derive bounds for the generalization gap in terms of the Jacobian norm within the framework of algorithmic robustness . Our provide empirical support for their through an extensive number of experiments. Several other recent papers have also focused on deriving tight generalization bounds for neural networks BID1 BID6 BID31. We do not propose theoretical bounds in this paper but establish a correlation between our metrics and generalization in a substantially larger experimental setting than undertaken in prior works. In the context of regularization, increasing robustness to perturbations is a widely-used strategy: data augmentation, noise injection BID15, weight decay BID19, and max-pooling all indirectly reduce sensitivity of the model to perturbations, while BID37 explicitly penalize the Frobenius norm of the Jacobian in the training objective. In this work we relate several of the above mentioned regularizing techniques to sensitivity, demonstrating through extensive experiments that improved generalization is consistently coupled with better robustness as measured by a single metric, the input-output Jacobian norm (§4.2). While some of these findings confirm common-sense expectations (random labels increase sensitivity, Figure 4, top row), others challenge our intuition of what makes a neural network robust (ReLU-networks, with unbounded activations, tend to be more robust than saturating HardSigmoid-networks, Figure 4, third row). One of our findings demonstrates an inductive bias towards robustness in stochastic mini-batch optimization compared to full-batch training (Figure 4, bottom row). Interpreting this regularizing effect in terms of some measure of sensitivity, such as curvature, is not new BID13 BID16 ), yet we provide a novel perspective by relating it to reduced sensitivity to inputs instead of parameters. The inductive bias of SGD ("implicit regularization") has been previously studied in BID29, where it was shown through rigorous experiments how increasing the width of a singlehidden-layer network improves generalization, and an analogy with matrix factorization was drawn to motivate constraining the norm of the weights instead of their count. BID30 further explored several weight-norm based measures of complexity that do not scale with the size of the model. One of our measures, the Frobenius norm of the Jacobian is of similar nature (since the Jacobian matrix size is determined by the task and not by a particular network architecture). However, this particular metric was not considered, and, to the best of our knowledge, we are the first to evaluate it in a large-scale setting (e.g. our networks are up to 65 layers deep and up to 2 16 units wide). Sensitivity to inputs has attracted a lot of interest in the context of adversarial examples . Several attacks locate points of poor generalization in the directions of high sensitivity of the network BID32 BID25, while certain defences regularize the model by penalizing sensitivity BID10 or employing decaying (hence more robust) non-linearities BID20. However, work on adversarial examples relates highly specific perturbations to a similarly specific kind of generalization (i.e. performance on a very small, adversarial subset of the data manifold), while this paper analyzes average-case sensitivity (§3) and typical generalization. We propose two simple measures of sensitivity for a fully-connected neural network (without biases) f: R d → R n with respect to its input x ∈ R d (the output being unnormalized logits of the n classes). Assume f employs a piecewise-linear activation function, like ReLU. Then f itself, as a composition of linear and piecewise-linear functions, is a piecewise-linear map, splitting the input space R d into disjoint regions, implementing a single affine mapping on each. Then we can measure two aspects of sensitivity by answering 1. How does the output of the network change as the input is perturbed within the linear region?2. How likely is the linear region to change in response to change in the input?We quantify these qualities as follows:1. For a local sensitivity measure we adopt the Frobenius norm of the class probabilities Jacobian DISPLAYFORM0, where f σ = σ • f with σ being the softmax function BID39. Given points of interest x test, we estimate the sensitivity of the function in those regions with the average Jacobian norm: DISPLAYFORM1 that we will further refer to as simply "Jacobian norm". Note that this does not require the labels for x test.Interpretation. The Frobenius norm J(x) F = ij J ij (x) 2 estimates the averagecase sensitivity of f σ around x. Indeed, consider an infinitesimal Gaussian perturbation BID39 The norm of the Jacobian with respect to logits ∂f (x) /∂x T experimentally turned out less predictive of test performance (not shown). See §A.3 for discussion of why the softmax Jacobian is related to generalization. ∆x ∼ N (0, I): the expected magnitude of the output change is then DISPLAYFORM2 2. To detect a change in linear region (further called a "transition"), we need to be able to identify it. We do this analogously to. For a network with piecewiselinear activations, we can, given an input x, assign a code to each neuron in the network f, that identifies the linear region of the pre-activation of that neuron. E.g. each ReLU unit will have 0 or 1 assigned to it if the pre-activation value is less or greater than 0 respectively. Similarly, a ReLU6 unit (see definition in §A.4) will have a code of 0, 1, or 2 assigned, since it has 3 linear regions 2. Then, a concatenation of codes of all neurons in the network (denoted by c(x)) uniquely identifies the linear region of the input x (see §A.1.1 for discussion of edge cases). Given this encoding scheme, we can detect a transition by detecting a change in the code. We then sample k equidistant points z 0,..., z k−1 on a closed one-dimensional trajectory T (x) (generated from a data point x and lying close to the data manifold; see below for details) and count transitions t(x) along it to quantify the number of linear regions: DISPLAYFORM3 where the norm of the directional derivative ∂c(z)/∂ (dz) 1 amounts to a Dirac delta function at each transition (see §A.1.2 for further details).By sampling multiple such trajectories around different points, we estimate the sensitivity metric: DISPLAYFORM4 that we will further refer to as simply "transitions" or "number of transitions." To assure the sampling trajectory T (x test) is close to the data manifold (since this is the region of interest), we construct it through horizontal translations of the image x test in pixel space (Figure App.7, right). We similarly augment our training data with horizontal and vertical translations in the corresponding experiments (Figure 4, second row). As earlier, this metric does not require knowing the label of x test. Interpretation. We can draw a qualitative parallel between the number of transitions and curvature of the function. One measure of curvature of a function f is the total norm of the directional derivative of its first derivative f along a path: DISPLAYFORM5 A piecewise-linear function f has a constant first derivative f everywhere except for the transition boundaries. Therefore, for a sufficiently large k, curvature can be expressed as DISPLAYFORM6 where z 0,..., z k−1 are equidistant samples on T (x). This sum is similar to t(x) as defined in Equation 1, but quantifies the amount of change in between two linear regions in a nonbinary way. However, estimating it on a densely sampled trajectory is a computationallyintensive task, which is one reason we instead count transitions. As such, on a qualitative level, the two metrics (Jacobian norm and number of transitions) track the first and second order terms of the Taylor expansion of the function. Above we have defined two sensitivity metrics to describe the learned function around the data, on average. In §4.1 we analyze these measures on and off the data manifold by simply measuring them along circular trajectories in input space that intersect the data manifold at certain points, but generally lie away from it (Figure 2, left). In the following subsections (§4.2, §4.3) each study analyzes performance of a large number (usually thousands) of fully-connected neural networks having different hyper-parameters and optimization procedures. Except where specified, we include only models which achieve 100% training accuracy. This allows us to study generalization disentangled from properties like expressivity and trainability, which are outside the scope of this work. In order to efficiently evaluate the compute-intensive metrics (§3) in a very wide range of hyperparameters settings (see e.g. §A.5.5) we only consider fully-connected networks. Extending the investigation to more complex architectures is left for future work. We analyze the behavior of a trained neural network near and away from training data. We do this by comparing sensitivity of the function along 3 types of trajectories:1. A random ellipse. This trajectory is extremely unlikely to pass anywhere near the real data, and indicates how the function behaves in random locations of the input space that it never encountered during training.2. An ellipse passing through three training points of different class (Figure 2, left). This trajectory does pass through the three data points, but in between it traverses images that are linear combinations of different-class images, and are expected to lie outside of the natural image space. Sensitivity of the function along this trajectory allows comparison of its behavior on and off the data manifold, as it approaches and moves away from the three anchor points.3. An ellipse through three training points of the same class. This trajectory is similar to the previous one, but, given the dataset used in the experiment (MNIST), is expected to traverse overall closer to the data manifold, since linear combinations of the same digit are more likely to resemble a realistic image. Comparing transition density along this trajectory to the one through points of different classes allows further assessment of how sensitivity changes in response to approaching the data manifold. We find that, according to both the Jacobian norm and transitions metrics, functions exhibit much more robust behavior around the training data (Figure 2, center and right). We further visualize this effect in 2D in Figure 3, where we plot the transition boundaries of the last (pre-logit) layer of a neural network before and after training. After training we observe that training points lie in regions of low transition density. The observed contrast between the neural network behavior near and away from data further strengthens the empirical connection we draw between sensitivity and generalization in §4.2, §4.3 and §4.4; it also confirms that, as mentioned in §1, if a certain quality of a function is to be used for model comparison, input domain should always be accounted for. In §4.1 we established that neural networks implement more robust functions in the vicinity of the training data manifold than away from it. We now consider the more practical context of model selection. Given two perfectly trained neural networks, does the model with better generalization implement a less sensitive function? Figure 2: A 100%-accurate (on training data) MNIST network implements a function that is much more stable near training data than away from it. Left: depiction of a hypothetical circular trajectory in input space passing through three digits of different classes, highlighting the training point locations (π/3, π, 5π/3). Center: Jacobian norm as the input traverses an elliptical trajectory. Sensitivity drops significantly in the vicinity of training data while remaining uniform along random ellipses. Right: transition density behaves analogously. According to both metrics, as the input moves between points of different classes, the function becomes less stable than when it moves between points of the same class. This is consistent with the intuition that linear combinations of different digits lie further from the data manifold than those of same-class digits (which need not hold for more complex datasets). See §A.5.2 for experimental details. After Training Figure 3: Transition boundaries of the last (pre-logits) layer over a 2-dimensional slice through the input space defined by 3 training points (indicated by inset squares). Left: boundaries before training. Right: after training, transition boundaries become highly non-isotropic, with training points lying in regions of lower transition density. See §A.5.3 for experimental details. We study approaches in the machine learning community that are commonly believed to influence generalization (Figure 4, top to bottom):• random labels;• data augmentation;• ReLUs;• full-batch training. We find that in each case, the change in generalization is coupled with the respective change in sensitivity (i.e. lower sensitivity corresponds to smaller generalization gap) as measured by the Jacobian norm (and almost always for the transitions metric). Figure 4: Improvement in generalization (left column) due to using correct labels, data augmentation, ReLUs, mini-batch optimization (top to bottom) is consistently coupled with reduced sensitivity as measured by the Jacobian norm (center column). Transitions (right column) correlate with generalization in all considered scenarios except for comparing optimizers (bottom right). Each point on the plot corresponds to two neural networks that share all hyper-parameters and the same optimization procedure, but differ in a certain property as indicated by axes titles. The coordinates along each axis reflect the values of the quantity in the title of the plot in the respective setting (i.e. with true or random labels). All networks have reached 100% training accuracy on CIFAR10 in both settings (except for the data-augmentation study, second row; see §A.5.4 for details). See §A.5.5 for experimental details (§A.5.4 for the data-augmentation study) and §4.2.1 for plot interpretation. In Figure 4, for many possible hyper-parameter configurations, we train two models that share all parameters and optimization procedure, but differ in a single binary setting (i.e. trained on true or random labels; with or without data augmentation; etc). Out of all such network pairs, we select only those where each network reached 100% training accuracy on the whole training set (apart from the data augmentation study). The two generalization or sensitivity values are then used as the x and y coordinates of a point corresponding to this pair of networks (with the plot axes labels denoting the respective value of the binary parameter considered). The position of the point with respect to the diagonal y = x visually demonstrates which configuration has smaller generalization gap / lower sensitivity. We now perform a large-scale experiment to establish direct relationships between sensitivity and generalization in a realistic setting. In contrast to §4.1, where we selected locations in the input space, and §4.2, where we varied a single binary parameter impacting generalization, we now sweep simultaneously over many different architectural and optimization choices (§A.5.5).Our main is presented in FIG3, indicating a strong relationship between the Jacobian norm and generalization. In contrast, Figure App.8 demonstrates that the number of transitions is not alone sufficient to compare networks of different sizes, as the number of neurons in the networks has a strong influence on transition count. In §4.3 we established a correlation between sensitivity (as measured by the Jacobian norm) and generalization averaged over a large test set (10 4 points). We now investigate whether the Jacobian norm can be predictive of generalization at individual points. As demonstrated in FIG4 (top), Jacobian norm at a point is predictive of the cross-entropy loss, but the relationship is not a linear one, and not even bijective (see §A.3 for analytic expressions explaining it). In particular, certain misclassified points (right sides of the plots) have a Jacobian norm many orders of magnitude smaller than that of the correctly classified points (left sides). However, we do remark a consistent tendency for points having the highest values of the Jacobian norm to be mostly misclassified. A similar yet noisier trend is observed in networks trained using 2 -loss as depicted in FIG4 (bottom). These observations make the Jacobian norm a promising quantity to consider in the contexts of active learning and confidence estimation in future research. We have investigated sensitivity of trained neural networks through the input-output Jacobian norm and linear regions counting in the context of image classification tasks. We have presented extensive experimental evidence indicating that the local geometry of the trained function as captured by the input-output Jacobian can be predictive of generalization in many different contexts, and that it varies drastically depending on how close to the training data manifold the function is evaluated. We further established a connection between the cross-entropy loss and the Jacobian norm, indicating that it can remain informative of generalization even at the level of individual test points. Interesting directions for future work include extending our investigation to more complex architectures and other machine learning tasks. The way of encoding a linear region c (z) of a point z described in §3 guarantees that different regions obtain different codes, but different codes might be assigned to the same region if all the neurons in any layer of the network are saturated (or if weights leading from the transitioning unit to active units are exactly zero, or exactly cancel). However, the probability of such an arrangement drops exponentially with width and hence is ignored in this work. The equality between the discrete and continuous versions of t (x) in Equation 1 becomes exact with a high-enough sampling density k such that there are no narrow linear regions missed in between consecutive points (precisely, the encoding c (z) has to only change at most once on the line between two consecutive points z i and z i+1 ).For computational efficiency we also assume that no two neurons transitions simultaneously, which is extremely unlikely in the context of random initialization and stochastic optimization. Figure App.7: Depiction of a trajectory in input space used to count transitions as defined in §3. An interpolation between 28 horizontal translations of a single digit in a complex trajectory that constrains all points to lie close to the translation-augmented data, and allows for a tractable estimate of transition density around the data manifold. This metric is used to compare models in §4.2 and §4.3. Straight lines indicate boundaries between different linear regions (straight-line boundaries between linear regions is accurate for the case of a single-layer piecewise-linear network. The partition into linear regions is more complex for deeper networks). FIG4, each plot shows 5 random networks that fit the respective training set to a 100% with each network having a unique color. See §A.5.6 for experimental details. Here we briefly discuss the motivation of this work in the context of Occam's razor. Occam's razor is a heuristic for model comparison based on their complexity. Given a dataset D, Occam's razor gives preference to simpler models H. In the Bayesian interpretation of the heuristic BID14 simplicity is defined as evidence P [D|H] and is often computed using the Laplace approximation. Under further assumptions BID22, this evidence can be shown to be inversely proportional to the number of parameters in the model. Therefore, given a uniform prior P [H] on two competing hypothesis classes, the class posterior P [H|D] ∼ P [D|H] P [H] is higher for a model with fewer parameters. An alternative, qualitative justification of the heuristic is through considering the evidence as a normalized probability distribution over the whole dataset space: DISPLAYFORM0 and remarking that models with more parameters have to spread the probability mass more evenly across all the datasets by virtue of being able to fit more of them (Figure App.10, left). This similarly suggests (under a uniform prior on competing hypothesis classes) preferring models with fewer parameters, assuming that evidence is unimodal and peaks close to the dataset of interest. Occam's razor for neural networks. As seen in Figure 1, the above reasoning does not apply to neural networks: the best achieved generalization is obtained by a model that has around 10 4 times as many parameters as the simplest model capable of fitting the dataset (within the evaluated search space).On one hand, BID26; demonstrate on concrete examples that a high number of free parameters in the model doesn't necessarily entail high complexity. On the other hand, a large body of work on the expressivity of neural networks BID33 BID24 shows that their ability to compute complex functions increases rapidly with size, while validates that they also easily fit complex (even random) functions with stochastic optimization. Classical metrics like VC dimension or Rademacher complexity increase with size of the network as well. This indicates that weights of a neural network may actually correspond to its usable capacity, and hence "smear" the evidence P [D|H] along a very large space of datasets D, making the dataset of interest D less likely. Potential issues. We conjecture the Laplace approximation of the evidence P [D|H] and the simplified estimation of the "Occam's factor" in terms of the accessible volume of the parameter space might not hold for neural networks in the context of stochastic optimization, and, in particular, do not account for the combinatorial growth of the accessible volume of parameter space as width increases BID23. Similarly, when comparing evidence as probability distributions over datasets, the difference between two neural networks may not be as drastic as in Figure App.10 (left), but more nuanced as depicted in Figure App.10 (right), with the evidence ratio being highly dependent on the particular dataset. We interpret our work as defining hypothesis classes based on sensitivity of the hypothesis (which yielded promising in BID36 on a toy task) and observing a strongly non-uniform prior on these classes that enables model comparison. Indeed, at least in the context of natural images classification, putting a prior on the number of parameters or Kolmogorov complexity of the hypothesis is extremely difficult. However, a statement that the true classification function is robust to small perturbations in the input is much easier to justify. As such, a prior P [H] in favor of robustness over sensitivity might fare better than a prior on specific network hyper-parameters. Above is one way to interpret the correlation between sensitivity and generalization that we observe in this work. It does not explain why large networks tend to converge to less sensitive functions. We conjecture large networks to have access to a larger space of robust solutions due to solving a highly-underdetermined system when fitting a dataset, while small models converge to more extreme weight values due to being overconstrained by the data. However, further investigation is needed to confirm this hypothesis. Reality? might nonetheless concentrate the majority of probability mass on simple functions and the evidence curves might intersect at a small angle. In this case, while a dataset D lying close to the intersection can be fit by both models, the Bayesian evidence ratio depends on its exact position with respect to the intersection. DISPLAYFORM0 Here we analyze the relationship between the Jacobian norm and the cross-entropy loss at individual test points as studied in §4.4.Target class Jacobian. We begin by relating the derivative of the target class probability J y(x) to per-point cross-entropy loss l(x) = − log [f σ (x)] y(x) (where y(x) is the correct integer class).We will denote f σ (x) by σ and drop the x argument to de-clutter notation (i.e. write f instead of f (x)). Then the Jacobian can be expressed as DISPLAYFORM0 where is the Hadamard element-wise product. Then indexing both sides of the equation at the correct class y yields DISPLAYFORM1 where e y is a vector of zeros everywhere except for e y = 1. Taking the norm of both sides in DISPLAYFORM2 We now assume that magnitudes of the individual logit derivatives vary little in between logits and over the input space 3: DISPLAYFORM3 /n. Since σ lies on the (n − 1)-simplex ∆ n−1, under these assumptions we can bound:(DISPLAYFORM4 and finally DISPLAYFORM5 or, in terms of the cross-entropy loss l = − log σ y : DISPLAYFORM6 We validate these approximate bounds in Figure App .11 (top).Full Jacobian. Equation 5 establishes a close relationship between J y and loss l = − log σ y, but of course, at test time we do not know the target class y. This allows us to only bound the full Jacobian norm from below: DISPLAYFORM7 For the upper bound, we assume the maximum-entropy case of σ y: σ i ≈ (1 − σ y)/(n − 1), for i = y. The Jacobian norm is DISPLAYFORM8 where the first summand becomes: All reported values, when applicable, were evaluated on the whole training and test sets of sizes 50000 and 10000 respectively. E.g. "generalization gap" is defined as the difference between train and test accuracies evaluated on the whole train and test sets. DISPLAYFORM9 When applicable, all trajectories/surfaces in input space were sampled with 2 20 points. All figures except for 6 and App.11 are plotted with (pale) error bars (when applicable). The reported quantity was usually evaluated 8 times with random seeds from 1 to 8 4, unless specified otherwise. E.g. if a network is said to be 100%-accurate on the training set, it means that each of the 8 randomlyinitialized networks is 100%-accurate after training. The error bar is centered at the mean value of the quantity and spans the standard error of the mean in each direction. If the bar appears to not be visible, it may be smaller than the mean value marker. Weight initialization, training set shuffling, data augmentation, picking anchor points of data-fitted trajectories, selecting axes of a zero-centered elliptic trajectory depend on the random seed. A random zero-centered ellipse was obtained by generating two axis vectors with normallydistributed entries of zero mean and unit variance (as such making points on the trajectory have an expected norm equal to that of training data) and sampling points on the ellipse with given axes. A random data-fitted ellipse was generated by projecting three arbitrary input points onto a plane where they fall into vertices of an equilateral triangle, and then projecting their circumcircle back into the input space. Relevant figure 3.A 15-layer ReLU6-network of width 300 was trained on MNIST for 2 18 steps using SGD with momentum BID38; images were randomly translated with wrapping by up to 4 pixels in each direction, horizontally and vertically, as well as randomly flipped along each axis, and randomly rotated by 90 degrees clockwise and counter-clockwise. The sampling grid in input space was obtain by projecting three arbitrary input points into a plane as described in §A.5.2 such that the ing triangle was centered at 0 and it's vertices were at a distance 0.8 form the origin. Then, a sampling grid of points in the [−1 ; 1] ×2 square was projected back into the input space. Relevant figures: 4 (second row) and 5 (bottom).All networks were trained for 2 18 steps of batch size of 256 using SGD with momentum. Learning rate was set to 0.005 and momentum term coefficient to 0.9.Data augmentation consisted of random translation of the input by up to 4 pixels in each direction with wrapping, horizontally and vertically. The input was also flipped horizontally with probability 0.5. When applying data augmentation (second row of Figure 4), the network is unlikely to encounter the canonical training data, hence few configurations achieved 100% training accuracy. However, we verified that all networks trained with data augmentation reached a higher test accuracy than their analogues without, ensuring that the generalization gap shrinks not simply because of lower training accuracy. For each dataset, networks of width {100, 200, 500, 1000, 2000, 3000}, depth {2, 3, 5, 10, 15, 20} and activation function {ReLU, ReLU6, HardTanh, HardSigmoid} were evaluated on 8 random seeds from 1 to 8. Relevant figures: 1, 4 (except for the second row), 5 (top), App.8. 335671 networks were trained for 2 19 steps with random hyper-parameters; if training did not complete, a checkpoint at step 2 18 was used instead, if available. When using L-BFGS, the maximum number of iterations was set to 2684. The space of available hyper-parameters included 5:"

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.