Question:
represent the text to find a very short summary (1-2 sentences max):

For instance you may be given 'Deep neural networks, in particular convolutional neural networks, have become highly effective tools for compressing images and solving inverse problems including denoising, inpainting, and reconstruction from few and noisy measurements. This success can be attributed in part to their ability to represent and generate natural images well. Contrary to classical tools such as wavelets, image-generating deep neural networks have a large number of parameters---typically a multiple of their output dimension---and need to be trained on large datasets. In this paper, we propose an untrained simple image model, called the deep decoder, which is a deep neural network that can generate natural images from very few weight parameters. The deep decoder has a simple architecture with no convolutions and fewer weight parameters than the output dimensionality. This underparameterization enables the deep decoder to compress images into a concise set of network weights, which we show is on par with wavelet-based thresholding. Further, underparameterization provides a barrier to overfitting, allowing the deep decoder to have state-of-the-art performance for denoising. The deep decoder is simple in the sense that each layer has an identical structure that consists of only one upsampling unit, pixel-wise linear combination of channels, ReLU activation, and channelwise normalization. This simplicity makes the network amenable to theoretical analysis, and it sheds light on the aspects of neural networks that enable them to form effective signal representations. Data models are central for signal and image processing and play a key role in compression and inverse problems such as denoising, super-resolution, and compressive sensing. These data models impose structural assumptions on the signal or image, which are traditionally based on expert knowledge. For example, imposing the assumption that an image can be represented with few non-zero wavelet coefficients enables modern (lossy) image compression BID1 and efficient denoising BID6.In recent years, it has been demonstrated that for a wide range of imaging problems, from compression to denoising, deep neural networks trained on large datasets can often outperform methods based on traditional image models BID19 BID0 BID18 BID4 BID22. This success can largely be attributed to the ability of deep networks to represent realistic images when trained on large datasets. Examples include learned representations via autoencoders BID12 and generative adversarial models BID8. Almost exclusively, three common features of the recent success stories of using deep neural network for imaging related tasks are i) that the corresponding networks are over-parameterized (i.e., they have much more parameters than the dimension of the image that they represent or generate), ii) that the networks have a convolutional structure, and perhaps most importantly, iii) that the networks are trained on large datasets. An important exception that breaks with the latter feature is a recent work by Ulyanov et al. BID20, which provides an algorithm, called the deep image prior (DIP), based on deep neural networks, that can solve inverse problems well without any training. Specifically, Ulyanov et al. demonstrated that fitting the weights of an over-parameterized deep convolutional network to a single image, together with strong regularization by early stopping of the optimization, performs competitively on a variety of image restoration problems. This is surprising because it does not involve a training dataset, which means that the notion of what makes an image'natural' is contained in a combination of the network structure and the regularization. However, without regularization the proposed network has sufficient capacity to overfit to noise, preventing meaningful image denoising. These prior works demonstrating the effectiveness of deep neural networks for image generation beg the question whether there may be a deep neural network model of natural images that is underparameterized and whose architecture alone, without algorithmic assistance, forms an efficient model for natural images. In this paper, we propose a simple image model in the form of a deep neural network that can represent natural images well while using very few parameters. This model thus enables image compression, denoising, and solving a variety of inverse problems with close to or state of the art performance. We call the network the deep decoder, due to its resemblance to the decoder part of an autoencoder. The network does not require training, and contrary to previous approaches, the network itself incorporates all assumptions on the data, is under-parameterized, does not involve convolutions, and has a simplicity that makes it amenable to theoretical analysis. The key contributions of this paper are as follows:• The network is under-parameterized. Thus, the network maps a lower-dimensional space to a higher-dimensional space, similar to classical image representations such as sparse wavelet representations. This feature enables image compression by storing the coefficients of the network after its weights are optimized to fit a single image. In Section 2, we demonstrate that the compression is on-par with wavelet thresholding BID1, a strong baseline that underlies JPEG-2000. An additional benefit of underparameterization is that it provides a barrier to overfitting, which enables regularization of inverse problems.• The network itself acts as a natural data model. Not only does the network require no training (just as the DIP BID20); it also does not critically rely on regularization, for example by early stopping (in contrast to the DIP). The property of not involving learning has at least two benefits: The same network and code is usable for a number of applications, and the method is not sensitive to a potential misfit of training and test data.• The network does not use convolutions. Instead, the network does have pixelwise linear combinations of channels, and, just like in a convolutional neural network, the weights are shared among spatial positions. Nonetheless, these are not convolutions because they provide no spatial coupling between pixels, despite how pixelwise linear combinations are sometimes called'1x1 convolutions.' In contrast, the majority of the networks for image compression, restoration, and recovery have convolutional layers with filters of nontrivial spatial extent BID19; BID0; BID18; BID4 BID22. This work shows that relationships characteristic of nearby pixels of natural images can be imposed directly by upsampling layers.• The network only consists of a simple combination of few building blocks, which makes it amenable to analysis and theory. For example, we prove that the deep decoder can only fit a small proportion of noise, which, combined with the empirical observation that it can represent natural images well, explains its denoising performance. The remainder of the paper is organized as follows. In Section 2, we first demonstrate that the deep decoder enables concise image representations. We formally introduce the deep decoder in Section 3. In Section 4, we show the performance of the deep decoder on a number of inverse problems such as denoising. In Section 5 we discuss related work, and finally, in Section 6 we provide theory and explanations on what makes the deep decoder work. Intuitively, a model describes a class of signals well if it is able to represent or approximate a member of the class with few parameters. In this section, we demonstrate that the deep decoder, an untrained, non-convolutional neural network, defined in the next section, enables concise representation of an image-on par with state of the art wavelet thresholding. The deep decoder is a deep image model G: R N → R n, where N is the number of parameters of the model, and n is the output dimension, which is (much) larger than the number of parameters (n N). The parameters of the model, which we denote by C, are the weights of the network, and not the input of the network, which we will keep fixed. To demonstrate that the deep decoder enables concise image representations, we choose the number of parameters of the deep decoder, N, such that it is a small fraction of the output dimension of the deep decoder, i.e., the dimension of the images 1.We draw 100 images from the ImageNet validation set uniformly at random and crop the center to obtain a 512x512 pixel color image. For each image x *, we fit a deep decoder model G(C) by minimizing the loss DISPLAYFORM0 with respect to the network parameters C using the Adam optimizer. We then compute for each image the corresponding peak-signal-to-noise ratio, defined as 10 log 10 (1/MSE), where DISPLAYFORM1 is the image generated by the network, and x * is the original image. We compare the compression performance to wavelet compression BID1 by representing each image with the N -largest wavelet coefficients. Wavelets-which underly JPEG 2000, a standard for image compression-are one of the best methods to approximate images with few coefficients. In Fig. 1 we depict the . It can be seen that for large compression factors (3 · 512 2 /N = 32.3), the representation by the deep decoder is slightly better for most images (i.e., is above the red line), while for smalle compression factors (3 · 512 2 /N = 8), the wavelet representation is slightly better. This experiment shows that deep neural networks can represent natural images well with very few parameters and without any learning. The observation that, for small compression factors, wavelets enable more concise representations than the deep decoder is intuitive because any image can be represented exactly with sufficiently many wavelet coefficients. In contrast, there is no reason to believe a priori that the deep decoder has zero representation error because it is underparameterized. The main point of this experiment is to demonstrate that the deep decoder is a good image model, which enables applications like solving inverse problems, as in Section 4. However, it also suggest that the deep decoder can be used for lossy image compression, by quantizing the coefficients C and saving the quantized coefficients. In the appendix, we show that image representations of the deep decoder are not sensitive to perturbations of its coefficients, thus quantization does not have a detrimental effect on the image quality. Deep networks were used successfully before for the compression of images BID19 BID0 BID18. In contrast to our work, which is capable of compressing images without any learning, the aforementioned works learn an encoder and decoder using convolutional recurrent neural networks BID19 and convolutional autoencoders BID18 based on training data. We consider a decoder architecture that transforms a randomly chosen and fixed tensor B 0 ∈ R n0×k0 consisting of k 0 many n 0 -dimensional channels to an n d × k out dimensional image, where k out = 1 for a grayscale image, and k out = 3 for an RGB image with three color channels. Throughout, n i has two dimensions; for example our default configuration has n 0 = 16 × 16 and n d = 512 × 512. The network transforms the tensor B 0 to an image by pixel-wise linearly combining the channels, upsampling operations, applying rectified linear units (ReLUs), and normalizing DISPLAYFORM0... lin. comb., upsampling, ReLU, CN lin. comb., sigmoid Figure 1: The deep decoder (depicted on the right) enables concise image representations, onpar with state-of-the-art wavelet based compression. The crosses on the left depict the PSNRs for 100 randomly chosen ImageNet-images represented with few wavelet coefficients and with a deep decoder with an equal number of parameters. A cross above the red line means the corresponding image has a smaller representation error when represented with the deep decoder. The deep decoder is particularly simple, as each layer has the same structure, consisting of a pixel-wise linear combination of channels, upsampling, ReLU nonlinearities, and channelwise normalization (CN). the channels. Specifically, the channels in the (i + 1)-th layer are given by DISPLAYFORM1 Here, the coefficient matrices C i ∈ R ki×ki+1 contain the weights of the network. Each column of the tensor B i C i ∈ R ni×ki+1 is formed by taking linear combinations of the channels of the tensor B i in a way that is consistent across all pixels. Then, cn(·) performs a channel normalization operation which is equivalent to normalizing each channel individually, and can be viewed as a special case of the popular batch normalization proposed in BID13. Specifically, let Z i = relu(U i B i C i) be the channels in the i-th layer, and let z ij be the j-th channel in the i-th layer. Then channel normalization performs the following transformation: DISPLAYFORM2, where mean and var compute the empirical mean and variance, and γ ij and β ij are parameters, learned independently for each channel, and is a fixed small constant. Learning the parameter γ and β helps the optimization but is not critical. This is a special case of batch normalization with batch size one proposed in BID13, and significantly improves the fitting of the model, just like how batch norm alleviates problems encountered when training deep neural networks. The operator U i ∈ R ni+1×ni is an upsampling tensor, which we choose throughout so that it performs bi-linear upsampling. For example, if the channels in the input have dimensions n 0 = 16×16, then the upsampling operator U 0 upsamples each channel to dimensions 32 × 32. In the last layer, we do not upsample, which is to say that we choose the corresponding upsampling operator as the identity. Finally, the output of the d-layer network is formed as DISPLAYFORM3 where Fig. 1 for an illustration. Throughout, our default architecture is a d = 6 layer network with k i = k for all i, and we focus on output images of dimensions n d = 512 × 512 and number of channels k out = 3. Recall that the parameters of the network are given by C = {C 0, C 1, . . ., C d}, and the output of the network is only a function of C, since we choose the tensor B 0 at random and fix it. Therefore, we write x = G(C). Note that the number of parameters is given by DISPLAYFORM4 DISPLAYFORM5 where the term 2k i corresponds to the two free parameters associated with the channel normalization. Thus, the number of parameters is N = dk 2 + 2dk + 3k. In the default architectures with d = 6 and k = 64 or k = 128, we have that N = 25,536 (for k = 64) and N =100,224 (k = 128) out of an RGB image space of dimensionality 512 × 512 × 3 = 786,432 parameters. We finally note that naturally variations of the deep decoder are possible; for example in a previous version of this manuscript, we applied upsampling after applying the relu-nonlinearity, but found that applying it before yields slightly better . While the deep decoder does not use convolutions, its structure is closely related to that of a convolutional neural network. Specifically, the network does have pixelwise linear combinations of channels, and just like in a convolutional neural network, the weights are shared among spatial positions. Nonetheless, pixelwise linear combinations are not proper convolutions because they provide no spatial coupling of pixels, despite how they are sometimes called 1 × 1 convolutions. In the deep decoder, the source of spatial coupling is only from upsampling operations. In contrast, a large number of networks for image compression, restoration, and recovery have convolutional layers with filters of nontrivial spatial extent BID19; BID0; BID18; BID4; BID22. Thus, it is natural to ask whether using linear combinations as we do, instead of actual convolutions yields better . Our simulations indicate that, indeed, linear combinations yield more concise representations of natural images than p × p convolutions, albeit not by a huge factor. Recall that the number of parameters of the deep decoder with d layers, k channels at each layer, and 1 × 1 convolutions is N (d, k; 1) = dk 2 + 3k + 2dk. If we consider a deep decoder with convolutional layers with filters of size p × p, then the number of parameters is: DISPLAYFORM0 If we fix the number of channels, k, but increase p to 3, the representation error only decreases since we increase the number of parameters (by a factor of approximately 32). We consider image reconstruction as described in Section 2. For a meaningful comparison, we keep the number of parameters fixed, and compare the representation error of a deep decoder with p = 1 and k = 64 (the default architecture in our paper) to a variant of the deep decoder with p = 3 and k = 22, so that the number of parameters is essentially the same in both configurations. We find that the representation of the deep decoder with p = 1 is better (by about 1dB, depending on the image), and thus for concise image representations, linear combinations (1 × 1 convolutions) appear to be more effective than convolutions of larger spatial extent. In this section, we use the deep decoder as a structure-enforcing model or regularizers for solving standard inverse problems: denoising, super-resolution, and inpainting. In all of those inverse problems, the goal is to recover an image x from a noisy observation y = f (x) + η. Here, f is a known forward operator (possibly equal to identity), and η is structured or unstructured noise. We recover the image x with the deep decoder as follows. Motivated by the finding from the previous section that a natural image x can (approximately) be represented with the deep decoder as G(C), we estimate the unknown image from the noisy observation y by minimizing the loss DISPLAYFORM0 with respect to the model parameters C. LetĈ be the of the optimization procedure. We estimate the image asx = G(Ĉ).We use the Adam optimizer for minimizing the loss, but have obtained comparable with gradient descent. Note that this optimization problem is non-convex and we might not reach a global minimum. Throughout, we consider the least-squares loss (i.e., we take · 2 to be the 2 norm), but the loss function can be adapted to account for structure of the noise. We remark that fitting an image model to observations in order to solve an inverse problem is a standard approach and is not specific to the deep decoder or deep-network-based models in general. Specifically, a number of classical signal recovery approaches fit into this framework; for example solving a compressive sensing problem with 1 -norm minimization amounts to choosing the forward operator as f (x) = Ax and minimizing over x in a 1 -norm ball. We start with the perhaps most basic inverse problem, denoising. The motivation to study denoising is at least threefold: First, denoising is an important problem in practice, second, many inverse problem can be solved as a chain of denoising steps BID15 Figure 2: An application of the deep decoder for denoising the astronaut test image. The deep decoder has performance on-par with state of the art untrained denoising methods, such as the DIP method BID20 and the BM3D algorithm BID5.problem is simple to model mathematically, and thus a common entry point for gaining intuition on a new method. Given a noisy observation y = x + η, where η is additive noise, we estimate an image with the deep decoder by minimizing the least squares loss G(C) − y 2 2, as described above. The in Fig. 2 and Table 1 demonstrate that the deep decoder has denoising performance onpar with state of the art untrained denoising methods, such as the related Deep Image Prior (DIP) method BID20 (discussed in more detail later) and the BM3D algorithm BID5. Since the deep decoder is an untrained method, we only compared to other state-of-the-art untrained methods (as opposed to learned methods such as BID22).Why does the deep decoder denoise well? In a nutshell, from Section 2 we know that the deep decoder can represent natural images well even when highly underparametrized. In addition, as a consequence of being under-parameterized, the deep decoder can only represent a small proportion of the noise, as we show analytically in Section 6, and as demonstrated experimentally in FIG2. Thus, the deep decoder "filters out" a significant proportion of the noise, and retains most of the signal. How to choose the parameters of the deep decoder? The larger k, the larger the number of latent parameters and thus the smaller the representation error, i.e., the error that the deep decoder makes when representing a noise-free image. On the other hand, the smaller k, the fewer parameters, and the smaller the range space of the deep decoder G(C), and thus the more noise the method will remove. The optimal k trades off those two errors; larger noise levels require smaller values of k (or some other form of regularization). If the noise is significantly larger, then the method requires either choosing k smaller, or it requires another means of regularization, for example early stopping of the optimization. For example k = 64 or 128 performs best out of {32, 64, 128}, for a PSNR of around 20dB, while for a PSNR of about 14dB, k = 32 performs best. We next super-resolve images with the deep denoiser. We define a forward model f that performs downsampling with the Lanczos filter by a factor of four. We then downsample a given image by a factor of four, and then reconstruct it with the deep decoder (with k = 128, as before). We compare performance to bi-cubic interpolation and to the deep image prior, and find that the deep decoder outperforms bicubic interpolation, and is on-par with the deep image prior (see Table 1 in the appendix). Finally, we use the deep decoder for inpainting, where we are given an inpainted image y, and a forward model f mapping a clean image to an inpainted image. The forward model f is defined by a mask that describes the inpainted region, and simply maps that part of the image to zero. FIG1 and Table 1 demonstrate that the deep decoder performs well on the inpainting problems; however, the deep image prior performs slightly better on average over the examples considered. For the impainting problem we choose a significantly more expressive prior, specifically k = 320. Image compression, restoration, and recovery algorithms are either trained or untrained. Conceptually, the deep decoder image model is most related to untrained methods, such as sparse representations in overcomplete dictionaries (for example wavelets BID6 and curvelets BID17). A number of highly successful image restoration and recovery schemes are not directly based on generative image models, but rely on structural assumptions about the image, such as exploiting self-similarity in images for denoising BID5 and super-resolution BID7 ).Since the deep decoder is an image-generating deep network, it is also related to methods that rely on trained deep image models. Deep learning based methods are either trained end-to-end for tasks ranging from compression BID19 BID0 BID18 BID4 BID22 to denoising BID4 BID22, or are based on learning a generative image model (by training an autoencoder or GAN BID12 BID8) and then using the ing model to solve inverse problems such as compressed sensing BID3, denoising BID11, phase retrieval, and blind deconvolution BID2, by minimizing an associated loss. In contrast to the deep decoder, where the optimization is over the weights of the network, in all the aforementioned methods, the weights are adjusted only during training and then are fixed upon solving the inverse problem. Most related to our work is the Deep Image Prior (DIP), recently proposed by Ulyanov et al. BID20. The deep image prior is an untrained method that uses a network with an hourglass or encoder-decoder architecture, similar to the U-net and related architectures that work well as autoencoders. The key differences to the deep decoder are threefold: i) the DIP is over-parameterized, whereas the deep decoder is under-parameterized. ii) Since the DIP is highly over-parameterized, it critically relies on regularization through early stopping and adding noise to its input, whereas the deep decoder does not need to be regularized (however, regularization can enhance performance). iii) The DIP is a convolutional neural network, whereas the deep decoder is not. We further illustrate point ii) comparing the DIP and deep decoder by denoising the astronaut image from Fig. 2. In FIG2 we plot the Mean Squared Error (MSE) over the number of iterations of the optimizer for fitting the noisy astronaut image x + η. Note that to fit the model, we minimize the error G(C) − (x + η) 2 2, because we are only given the noisy image, but we plot the MSE between the representation and the actual, true image G(C t) − x 2 2 at iteration t. Here, C t are the parameters of the deep decoder after t iterations of the optimizer. In FIG2 and (c), we plot the loss or MSE associated with fitting the noiseless astronaut image, x (G(C t) − x 2 2 ) and the noise itself, η, (G(C t) − η 2 2 ). Models are fitted independently for the noisy image, the noiseless image, and the noise. The plots in FIG2 show that with sufficiently many iterations, both the DIP and the DD can fit the image well. However, even with a large number of iterations, the deep decoder can not fit the noise well, whereas the DIP can. This is not surprising, given that the DIP is over-parameterized and the deep decoder is under-parameterized. In fact, in Section 6 we formally show that due to the The third panel shows the MSE of the output of DD or DIP for an image consisting purely of noise, as computed relative to that noise. Due to under-parameterization, the deep decoder can only fit a small proportion of the noise, and thus enables image denoising. Early stopping can mildly enhance the performance of DD; to see this note that in panel (a), the minimum is obtained at around 5000 iterations and not at 50,000. The deep image prior can fit noise very well, but fits an image faster than noise, thus early stopping is critical for denoising performance.underparameterization, the deep decoder can only fit a small proportion of the noise, no matter how and how long we optimize. As a consequence, it filters out much of the noise when applied to a natural image. In contrast, the DIP relies on the empirical observation that the DIP fits a structured image faster than it fits noise, and thus critically relies on early stopping. In the previous sections we empirically showed that the deep decoder can represent images well and at the same time cannot fit noise well. In this section, we formally show that the deep decoder can only fit a small proportion of the noise, relative to the degree of underparameterization. In addition, we provide insights into how the components of the deep decoder contribute to representing natural images well, and we provide empirical observations on the sensitivity of the parameters and their distribution. We start by showing that an under-parameterized deep decoder can only fit a proportion of the noise relative to the degree of underparameterization. At the heart of our argument is the intuition that a method mapping from a low-to a high-dimensional space can only fit a proportion of the noise relative to the number of free parameters. For simplicity, we consider a one-layer network, and ignore the batch normalization operation. Then, the networks output is given by DISPLAYFORM0 Here, we take C = (C 0, c 1), where C 0 is a k × k matrix and c 1 is a k-dimensional vector, assuming that the number of output channels is 1. While for the performance of the deep decoder the choice of upsampling matrix is important, it is not relevant for showing that the deep decoder cannot represent noise well. Therefore, the following statement makes no assumptions about the upsampling matrix U 0. Proposition 1. Consider a deep decoder with one layer and arbitrary upsampling and input matrices. That is, let B 0 ∈ R n0×k and U 0 ∈ R n×n0. Let η ∈ R n be zero-mean Gaussian noise with covariance matrix σI, σ > 0. Assume that k 2 log(n 0)/n ≤ 1/32. Then, with probability at least DISPLAYFORM1 The proposition asserts that the deep decoder can only fit a small portion of the noise energy, precisely a proportion determined by its number of parameters relative to the output dimension, n. Our The blue curves show a one-dimensional piecewise smooth signal, and the red crosses show estimates of this signal by a one-dimensional deep decoder with either linear or convex upsampling. We see that linear upsampling acts as an indirect signal prior that promotes piecewise smoothness.simulations and preliminary analytic suggest that this statement extends to multiple layers in that the lower bound becomes 1 − c DISPLAYFORM2, where c is a numerical constant. Note that the lower bound does not directly depend on the noise variance σ since both sides of the inequality scale with σ 2. Upsampling is a vital part of the deep decoder because it is the only way that the notion of locality explicitly enters the signal model. In contrast, most convolutional neural networks have spatial coupling between pixels both by unlearned upsampling, but also by learned convolutional filters of nontrivial spatial extent. The choice of the upsampling method in the deep decoder strongly affects the'character' of the ing signal estimates. We now discuss the impacts of a few choices of upsampling matrices U i, and their impact on the images the model can fit. No upsampling: If there is no upsampling, or, equivalently, if U i = I, then there is no notion of locality in the ing image. All pixels become decoupled, and there is then no notion of which pixels are near to each other. Specifically, a permutation of the input pixels (the rows of B 0) simply induces the identical permutation of the output pixels. Thus, if a deep decoder without upsampling could fit a given image, it would also be able to fit random permutations of the image equally well, which is practically equivalent to fitting random noise. Nearest neighbor upsampling: If the upsampling operations perform nearest neighbor upsampling, then the output of the deep decoder consists of piecewise constant patches. If the upsampling doubles the image dimensions at each layer, this would in patches of 2 d × 2 d pixels that are constant. While this upsampling method does induce a notion of locality, it does so too strongly in the sense that squares of nearby pixels become identical and incapable of fitting local variation within natural images. Linear and convex, non-linear upsampling: The specific choice of upsampling matrix affects the multiscale'character' of the signal estimates. To illustrate this, FIG3 shows the signal estimate from a 1-dimensional deep decoder with upsampling operations given by linear upsampling (x 0, x 1, x 2, . . .) → (x 0, 0.5x 0 + 0.5x 1, x 1, 0.5x 1 + 0.5x 2, x 2, . . .) and convex nonlinear upsampling given by (x 0, x 1, x 2, . . .) → (x 0, 0.75x 0 + 0.25x 1, x 1, 0.75x 1 + 0.25x 2, x 2, . . .). Note that while both models are able to capture the coarse signal structure, the convex upsampling in a multiscale fractal-like structure that impedes signal representation. In contrast, linear upsampling is better able to represent smoothly varying portions of the signal. Linear upsampling in a deep decoder indirectly encodes the prior that natural signals are piecewise smooth and in some sense have approximately linear behavior at multiple scales 6.3 NETWORK INPUT Throughout, the network input is fixed. We choose the network input B 1 by choosing its entries uniformly at random. The particular choice of the input is not very important; it is however desirable that the rows are incoherent. To see this, as an extreme case, if any two rows of B 1 are equal and if the upsampling operation preserves the values of those pixels exactly (for example, as with the linear upsampling from the previous section), then the corresponding pixels of the output image is Figure 6: The left panel shows an image reconstruction after training a deep decoder on the MRI phantom image (PSNR is 51dB). The right panel shows how the deep decoder builds up an image starting from a random input. From top to bottom are the input to the network and the activation maps (i.e., relu(B i C i)) for eight out of the 64 channels in layers one to six. Table 1: Performance comparison of the deep decoder for denoising (DN), superresolution (SR), and inpainting (IP), in peak signal to noise ratio (PSNR). Note that identity corresponds to the PSNR of the noise and corruption in the DN and IP experiments, respectively. also exactly the same, which restricts the range space of the deep decoder unrealistically, since for any pair of pixels, the majority of natural images does not have exactly the same value at this pair of pixels. The deep decoder is tasked with coverting multiple noise channels into a structured signal primarily using pixelwise linear combinations, ReLU activation funcions, and upsampling. Using these tools, the deep decoder builds up an image through a series of successive approximations that gradually morph between random noise and signal. To illustrate that, we plot the activation maps (i.e., relu(B i C i)) of a deep decoder fitted to the phantom MRI test image (see Fig. 6). We choose a deep decoder with d = 5 layers and k = 64 channels. This image reconstruction approach is in contrast to being a semantically meaningful hierarchical representation (i.e., where edges get combined into corners, that get combined into simple sample, and then into more complicated shapes), similar to what is common in discriminative networks. RH is partially supported by NSF award IIS-1816986, an NVIDIA Academic GPU Grant, and would like to thank Ludwig Schmidt for helpful discussions on the deep decoder in general, and in particular for suggestions on the experiments in Section 2.Code to reproduce the is available at https://github.com/reinhardh/ supplement_deep_decoder APPENDIX A PROOF OF PROPOSITION 1Suppose that the network has one layer, i.e., G(C) = relu(U 0 B 0 C 0)c 1. We start by re-writing B 1 = relu(B 0 C 0) in a convenient form. For a given vector x ∈ R n, denote by diag(x > 0) the matrix that contains one on its diagonal if the respective entry of x is positive and zero otherwise. Let c jci denote the i-th column of C j, and denote by W ji ∈ {0, 1} k×k the corresponding diagonal matrix W ji = diag(U j B j c jci > 0). With this notation, we can write DISPLAYFORM0 where [c 1] i denotes the i-th entry of c 1. Thus, G(C) lies in the union of at-most-k 2 -dimensional subspaces of R n, where each subspace is determined by the matrices {W 0j} k j=1. The number of those subspaces is bounded by n k 2. This follows from the fact that for the matrix A:= U 0 B 0, by Lemma 1 below, the number of different matrices W 0j is bounded by n k. Since there are k matrices, the number of different sets of matrices is bounded by n k 2. Lemma 1. For any A ∈ R n×k and k ≥ 5, |{diag(Av > 0)A|v ∈ R k }| ≤ n k.Next, fix the matrixes {W 0j} j. As G(C) lies in an at-most-k 2 -dimensional subspace, let S be a k 2 -dimensional subspace that contains the range of G for these fixed {W 0j} j. It follows that Now, we make use of the following bound on the projection of the noise η onto a subspace. Lemma 2. Let S ⊂ R n be a subspace with dimension. Let η ∼ N (0, I n) and β ≥ 1. Then, P P S c η n, then P X − n ≥ 2 √ nx + 2x ≤ e −x, DISPLAYFORM1 With these, we obtain P [X ≥ 5βn] ≤ e −βn if β ≥ 1,P [X ≤ n/2] ≤ e −n/16.We have noiseless C 1 noisy C 2 noisy C 3 noisy C 4 noisy C 5 noisy C 6 noisy c 7 noisy Figure 7: Sensitivity to parameter perturbations of the weights in each layer, and images generated by perturbing the weights in different layers, and keeping the weights in the other layers constant. A.1 PROOF OF LEMMA 1Our goal is to count the number of sign patterns (Av > 0) ∈ {0, 1}. Note that this number is equal to the maximum number of partitions one can get when cutting a k-dimensional space with n many hyperplanes that all pass through the origin, and are perpendicular to the rows of A. This number if well known (see for example BID21) and is upper bounded by DISPLAYFORM2 Thus, DISPLAYFORM3 where the last inequality holds for k ≥ 5. The deep decoder is not overly sensitive to perturbations of its coefficients. To demonstrate this, fit the standard test image Barbara with a deep decoder with 6 layers and k = 128, as before. We then perturb the weights in a given layer i (i.e., the matrix C i) with Gaussian noise of a certain signal-tonoise ratio relative to C i and leave the other weights and the input untouched. We then measure the peak signal-to-noise ratio in the image domain, and plot the corresponding curve for each layer (see Fig. 7). It can be seen that the representation provided by the deep decoder is relatively stable with respect to perturbations of its coefficients, and that it is more sensitive to perturbations in higher levels. Finally, in FIG7 we depict the distribution of the weights of the network after fitted to the Barbara test image, and note that the weights are approximately Gaussian distributed. The distribution of the weighs is approximately Gaussian.' and it should match with 'We introduce an underparameterized, nonconvolutional, and simple deep neural network that can, without training, effectively represent natural images and solve image processing tasks like compression and denoising competitively.' but not with 'We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.'.

Question: Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the r\^{o}le played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling. In particular, DN layers constructed from these operations can be interpreted as {\em max-affine spline operators} (MASOs) that have an elegant link to vector quantization (VQ) and $K$-means. While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax. {\em This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs).} We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural ``hard'' VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding ``soft'' VQ inference problems. We further extend the framework by hybridizing the hard and soft VQ optimizations to create a $\beta$-VQ inference that interpolates between hard, soft, and linear VQ inference. A prime example of a $\beta$-VQ DN nonlinearity is the {\em swish} nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation. Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters. Deep (neural) networks (DNs) have recently come to the fore in a wide range of machine learning tasks, from regression to classification and beyond. A DN is typically constructed by composing a large number of linear/affine transformations interspersed with up/down-sampling operations and simple scalar nonlinearities such as the ReLU, absolute value, sigmoid, hyperbolic tangent, etc. BID13. Scalar nonlinearities are crucial to a DN's performance. Indeed, without nonlinearity, the entire network would collapse to a simple affine transformation. But to date there has been little progress understanding and unifying the menagerie of nonlinearities, with few reasons to choose one over another other than intuition or experimentation. Recently, progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU, leaky ReLU, and absolute value activations and downsampling operations like max-, average-, and channel-pooling BID1. In particular, these operations can be interpreted as max-affine spline operators (MASOs) BID16; BID14 that enable a DN to find a locally optimized piecewise affine approximation to the prediction operator given training data. A spline-based prediction is made in two steps. First, given an input signal x, we determine which region of the spline's partition of the domain (the input signal space) it falls into. Second, we apply to x the fixed (in this case affine) function that is assigned to that partition region to obtain the prediction y = f (x).The key of BID1 is any DN layer constructed from a combination of linear and piecewise affine and convex is a MASO, and hence the entire DN is merely a composition of MASOs. MASOs have the attractive property that their partition of the signal space (the collection of multidimensional "knots") is completely determined by their affine parameters (slopes and offsets). This provides an elegant link to vector quantization (VQ) and K-means clustering. That is, during learning, a DN implicitly constructs a hierarchical VQ of the training data that is then used for splinebased prediction. This is good progress for DNs based on ReLU, absolute value, and max-pooling, but what about DNs based on classical, high-performing nonlinearities that are neither piecewise affine nor convex like the sigmoid, hyperbolic tangent, and softmax or fresh nonlinearities like the swish BID20 that has been shown to outperform others on a range of tasks?Contributions. In this paper, we address this gap in the DN theory by developing a new framework that unifies a wide range of DN nonlinearities and inspires and supports the development of new ones. The key idea is to leverage the yinyang relationship between deterministic VQ/K-means and probabilistic Gaussian Mixture Models (GMMs) BID3. Under a GMM, piecewise affine, convex nonlinearities like ReLU and absolute value can be interpreted as solutions to certain natural hard inference problems, while sigmoid and hyperbolic tangent can be interpreted as solutions to corresponding soft inference problems. We summarize our primary contributions as follows:Contribution 1: We leverage the well-understood relationship between VQ, K-means, and GMMs to propose the Soft MASO (SMASO) model, a probabilistic GMM that extends the concept of a deterministic MASO DN layer. Under the SMASO model, hard maximum a posteriori (MAP) inference of the VQ parameters corresponds to conventional deterministic MASO DN operations that involve piecewise affine and convex functions, such as fully connected and convolution matrix multiplication; ReLU, leaky-ReLU, and absolute value activation; and max-, average-, and channelpooling. These operations assign the layer's input signal (feature map) to the VQ partition region corresponding to the closest centroid in terms of the Euclidean distance, Contribution 2: A hard VQ inference contains no information regarding the confidence of the VQ region selection, which is related to the distance from the input signal to the region boundary. In response, we develop a method for soft MAP inference of the VQ parameters based on the probability that the layer input belongs to a given VQ region. Switching from hard to soft VQ inference recovers several classical and powerful nonlinearities and provides an avenue to derive completely new ones. We illustrate by showing that the soft versions of ReLU and max-pooling are the sigmoid gated linear unit and softmax pooling, respectively. We also find a home for the sigmoid, hyperbolic tangent, and softmax in the framework as a new kind of DN layer where the MASO output is the VQ probability. Contribution 3: We generalize hard and soft VQ to what we call β-VQ inference, where β ∈ is a free and learnable parameter. This parameter interpolates the VQ from linear (β → 0), to probabilistic SMASO (β = 0.5), to deterministic MASO (β → 1). We show that the β-VQ version of the hard ReLU activation is the swish nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc through experimentation BID20.Contribution 4: Seen through the MASO lens, current DNs solve a simplistic per-unit (per-neuron), independent VQ optimization problem at each layer. In response, we extend the SMASO GMM to a factorial GMM that that supports jointly optimal VQ across all units in a layer. Since the factorial aspect of the new model would make naïve VQ inference exponentially computationally complex, we develop a simple sufficient condition under which a we can achieve efficient, tractable, jointly optimal VQ inference. The condition is that the linear "filters" feeding into any nonlinearity should be orthogonal. We propose two simple strategies to learn approximately and truly orthogonal weights and show on three different datasets that both offer significant improvements in classification per-formance. Since orthogonalization can be applied to an arbitrary DN, this and our theoretical understanding are of independent interest. This paper is organized as follows. After reviewing the theory of MASOs and VQ for DNs in Section 2, we formulate the GMM-based extension to SMASOs in Section 3. Section 4 develops the hybrid β-VQ inference with a special case study on the swish nonlinearity. Section 5 extends the SMASO to a factorial GMM and shows the power of DN orthogonalization. We wrap up in Section 6 with directions for future research. Proofs of the various appear in several appendices in the Supplementary Material. We first briefly review max-affine spline operators (MASOs) in the context of understanding the inner workings of DNs BID1 BID14, with each spline formed from R piecewise affine and convex mappings. The MASO parameters consist of the "slopes" A ∈ R K×R×D and the "offsets/biases" B ∈ R K×R. See Appendix A for the precise definition. Given the input x ∈ R D and parameters A, B, a MASO produces the output z ∈ R K via DISPLAYFORM0 where [z] k denotes the k th dimension of z. The three subscripts of the slopes tensor [A] k,r,d correspond to output k, partition region r, and input signal index d. The two subscripts of the offsets/biases tensor [B] k,r correspond to output k and partition region r. An important consequence of FORMULA0 is that a MASO is completely determined by its slope and offset parameters without needing to specify the partition of the input space (the "knots" when D = 1). Indeed, solving automatically computes an optimized partition of the input space R D that is equivalent to a vector quantization (VQ) BID19; BID9. We can make the VQ aspect explicit by rewriting in terms of the Hard-VQ (HVQ) matrix T H ∈ R K×R. that contains K stacked one-hot row vectors, each with the one-hot position at index [t] k ∈ {1, . . ., R} corresponding to the arg max over r = 1,..., R of. Given the HVQ matrix, (or equivalently, a region of the input space), the input-output mapping is affine and fully determined by DISPLAYFORM1 We retrieve from by noting that DISPLAYFORM2 The key for this paper is that the layers of a very large class of DN are MASOs. Hence, such a DN is a composition of MASOs, where each layer MASO has as input the feature map DISPLAYFORM3 and produces DISPLAYFORM4, with corresponding to the layer. Each MASO has thus specific parameters A , B .Theorem 1. Any DN layer comprising a linear operator (e.g., fully connected or convolution) composed with a convex and piecewise affine operator (such as a ReLU, leaky-ReLU, or absolute value activation; max/average/channel-pooling; maxout; all with or without skip connections) is a MASO Balestriero & Baraniuk (2018a; .Appendix A provides the parameters A , B for the MASO corresponding to the th layer of any DN constructed from linear plus piecewise affine and convex components. Given this connection, we will identify z (−1) above as the input (feature map) to the MASO DN layer and z as the output (feature map). We also identify [z ] k in FORMULA0 and FORMULA1 as the output of the k th unit (aka neuron) of the th layer. MASOs for higher-dimensional tensor inputs/outputs are easily developed by flattening. The MASO/HVQ connection provides deep insights into how a DN clusters and organizes signals layer by layer in a hierarchical fashion BID1. However, the entire ap-proach requires that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax. The goal of this paper is to extend the MASO analysis framework of Section 2 to these and an infinitely large class of other nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs). For now, we focus on a single unit k from layer of a MASO DN, which contains both linear and nonlinear operators; we generalize below in Section 5. The key to the MASO mechanism lies in the VQ variables [t ] k ∀k, since they fully determine the output via. For a special choice of bias, the VQ variable computation is equivalent to the K-means algorithm BID1. DISPLAYFORM0, the MASO VQ partition corresponds to a K- DISPLAYFORM1 For example, consider a layer using a ReLU activation function. Unit k of that layer partitions its input space using a K-means model with R = 2 centroids: the origin of the input space and the unit layer parameter [A ] k,1,·. The input is mapped to the partition region corresponding to the closest centroid in terms of the Euclidean distance, and the corresponding affine mapping for that region is used to project the input and produce the layer output as in FORMULA1.We now leverage the well-known relationship between K-means and Gaussian Mixture Models (GMMs) BID4 to GMM-ize the deterministic VQ process of max-affine splines. As we will see, the constraint on the value of B k,rin Proposition 1 will be relaxed thanks to the GMM's ability to work with a nonuniform prior over the regions (in contrast to K-means).To move from a deterministic MASO model to a probabilistic GMM, we reformulate the HVQ selection variable [t DISPLAYFORM2 Armed with this, we define the following generative model for the layer input z ( −1) as a mixture of R Gaussians with mean DISPLAYFORM3 and identical isotropic covariance with parameter σ DISPLAYFORM4 with ∼ N(0, Iσ 2). Note that this GMM generates an independent vector input z (−1) for every unit k = 1,..., D in layer. For reasons that will become clear below in Section 3.3, we will refer to the GMM model as the Soft MASO (SMASO) model. We develop a joint, factorial model for the entire MASO layer (and not just one unit) in Section 5. Given the GMM and an input z (−1), we can compute a hard inference of the optimal VQ selection variable [t ] k via the maximum a posteriori (MAP) principle DISPLAYFORM0 The following is proved in Appendix E.1.Theorem 2. Given a GMM with parameters σ 2 = 1 and DISPLAYFORM1, t = 1,..., R , the MAP inference of the latent selection variable [t ] k given in can be computed via the MASO HVQ DISPLAYFORM2 Note in Theorem 2 that the bias constraint of Proposition 1 (which can be interpreted as imposing a uniform prior [π ] k,· ) is completely relaxed. HVQ inference of the selection matrix sheds light on some of the drawbacks that affect any DN employing piecewise affine, convex activation functions. First, during gradient-based learning, the gradient will propagate back only through the activated VQ regions that correspond to the few 1-hot entries in T H. The parameters of other regions will not be updated; this is known as the "dying neurons phenomenon" BID22; BID0. Second, the overall MASO mapping is continuous but not differentiable, which leads to unexpected gradient jumps during learning. Third, the HVQ inference contains no information regarding the confidence of the VQ region selection, which is related to the distance of the query point to the region boundary. As we will now see, this extra information can be very useful and gives rise to a range of classical and new activation functions. We can overcome many of the limitations of HVQ inference in DNs by replacing the 1-hot entries of the HVQ selection matrix with the probability that the layer input belongs to a given VQ region DISPLAYFORM0 which follows from the simple structure of the GMM. This corresponds to a soft inference of the categorical variable DISPLAYFORM1 H as the noise variance in → 0. Given the SVQ selection matrix, the MASO output is still computed via. The SVQ matrix can be computed indirectly from an entropy-penalized MASO optimization; the following is reproved in Appendix E.2 for completeness. DISPLAYFORM2 Proposition 2, which was first established in BID17; BID18, unifies HVQ and SVQ in a single optimization problem. The transition from HVQ to SVQ FORMULA15 is obtained simply by adding the entropy regularization H(t). Notice that removing the Entropy regularization from leads to the same VQ as. We summarize this finding in Table. 1. Remarkably, switching from HVQ to SVQ MASO inference recovers several classical and powerful nonlinearities and provides an avenue to derive completely new ones. Given a set of MASO parameters A , B for calculating the layer-output of a DN via, we can derive two distinctly different DNs: one based on the HVQ inference of and one based on the SVQ inference of. The following are proved in Appendix E.5. Appendix C discusses how the GMM and SVQ formulations shed new light on the impact of parameter initialization in DC learning plus how these formulations can be extended further. Value for DISPLAYFORM0 DISPLAYFORM1 Changing viewpoint slightly, we can also derive classical nonlinearities like the sigmoid, tanh, and softmax BID13 from the soft inference perspective. Consider a new soft DN layer whose unit output [z ] k is not the piecewise affine spline of but rather the probability [z DISPLAYFORM0) that the input z falls into each VQ region. The following propositions are proved in Appendix E.6. Combining and yields a hybrid optimization for a new β-VQ that recovers hard, soft, and linear VQ inference as special cases DISPLAYFORM0 with the new hyper-parameter [β ] k ∈. The β-VQ obtained from the above optimization problem utilizes [β ] k to balance the impact of the regularization term (introduced in the SVQ derivation), allowing to recover and interpolate the VQ between linear, soft and hard (see Table. 1). The following is proved in Appendix E.3. Theorem 3. The unique global optimum of FORMULA19 is given by DISPLAYFORM1 The β-VQ covers all of the theory developed above as special cases: β = 1 yields HVQ, β = TAB1 summarizes some of the many nonlinearities that are within reach of the β-VQ. DISPLAYFORM2 The GMM models the impact of only a single layer unit on the layer-input z (−1). We can easily extend this model to a factorial model for z (−1) that enables all D units at layer to combine their syntheses: DISPLAYFORM0 with ∼ N(0, Iσ 2). This new model is a mixture of R Gaussians with means DISPLAYFORM1 and identical isotropic covariances with variance σ 2. The factorial aspect of the model means that the number of possible combinations of the t values grow exponentially with the number of units. Hence, inferring the latent variables t quickly becomes intractable. However, we can break this combinatorial barrier and achieve efficient, tractable VQ inference by constraining the MASO slope parameters A to be orthogonal DISPLAYFORM2 Orthogonality is achieved in a fully connected layer (multiplication by the dense matrix W composed with activation or pooling) when the rows of W are orthogonal. Orthogonality is achieved in a convolution layer (multiplication by the convolution matrix C composed with activation or pooling) when the rows of C are either non-overlapping or properly apodized; see Appendix E.4 for the details plus the proof of the following . Theorem 4. If the slope parameters A of a MASO are orthogonal in the sense of FORMULA0 DISPLAYFORM3 In an orthogonal, factorial MASO, optimal inference can be performed independently per factor, as opposed to jointly over all of the factors. Orthogonality renders the joint MAP inference of the factorial model's VQs tractable. The following is proved in Appendix E.4.Practically, this not only lowers the computational complexity tremendously but also imparts the benefit of "uncorrelated unit firing," which has been shown to be advantageous in DNs BID21. Beyond the scope of this paper, such an orthogonalization strategy can also be applied to more general factorial models such as factorial; BID10 and factorial HMMs BID11. Table 2: Classification experiment to demonstrate the utility of orthogonal DN layers. For three datasets and the same largeCNN architecture (detailed in Appendix D), we tabulate the classification accuracy (larger is better) and its standard deviation averaged over 5 runs with different Adam learning rates. In each case, orthogonal fully-connected and convolution matrices improve the classification accuracy over the baseline. Corollary 1. When the conditions of Theorem 4 are fulfilled, the joint MAP estimate for the VQs of the factorial model DISPLAYFORM4 and thus can be computed with linear complexity in the number of units. The advantages of orthogonal or near-orthogonal filters have been explored empirically in various settings, from to , typically demonstrating improved performance. Table 2 tabulates the of a simple confirmation experiment with the largeCNN architecture described in Appendix D. We added to the standard cross-entropy loss a term DISPLAYFORM5 2 that penalizes non-orthogonality (recall). We did not cross-validate the penalty coefficient λ but instead set it equal to 1. The tabulated show clearly that favoring orthogonal filters improves accuracy across both different datasets and different learning settings. Since the orthogonality penalty does not guarantee true orthogonality but simply favors it, we performed one additional experiment where we reparametrized the fully-connected and convolution matrices using the Gram-Schmidt (GS) process BID7 so that they were truly orthogonal. Thanks to the differentiability of all of the operations involved in the GS process, we can backpropagate the loss to the orthogonalized filters in order to update them in learning. We also used the swish activation, which we showed to be a β-VQ nonlinearity in Section 4. Since the GS process adds significant computational overhead to the learning algorithm, we conducted only one experiment on the largest dataset (CIFAR100). The exactly orthogonalized largeCNN achieved a classification accuracy of 61.2%, which is a major improvement over all of the in the bottom (CIFAR100) cell of Table 2. This indicates that there are good reasons to try to improve on the simple orthogonality-penalty-based approach. Our development of the SMASO model opens the door to several new research questions. First, we have merely scratched the surface in the exploration of new nonlinear activation functions and pooling operators based on the SVQ and β-VQ. For example, the soft-or β-VQ versions of leakyReLU, absolute value, and other piecewise affine and convex nonlinearities could outperform the new swish nonlinearity. Second, replacing the entropy penalty in the and with a different penalty will create entirely new classes of nonlinearities that inherit the rich analytical properties of MASO DNs. Third, orthogonal DN filters will enable new analysis techniques and DN probing methods, since from a signal processing point of view problems such as denoising, reconstruction, compression have been extensively studied in terms of orthogonal filters. This work was partially supported by NSF grants IIS-17-30574 and IIS-18-38177, AFOSR grant FA9550-18-1-0478, ARO grant W911NF-15-1-0316, ONR grants N00014-17-1-2551 and N00014-18-12571, DARPA grant G001534-7500, and a DOD Vannevar Bush Faculty Fellowship (NSSEFF) grant N00014-18-1-2047. propose a brief approximation . Due to the specificity of the convolution operator we are able to provide a tractable inference coupled with an apodization scheme. To demonstrate this, we first highlight that any input can be represented as a direct sum of its apodized patches. Then, we see that filtering apodized patches with a filter is equivalent to convolving the input with apodized filters. We first need to introduce the patch notation. We define a patch P[z ( −1) ](pi, pj) ∈ {1, . . ., I } × {1, . . ., J } as the slice of the input with indices c = 1,..., K , i = (all channels) and (i, j) ∈ {pi,..., pi + I The above highlights the ability to treat an input via its collection of patches with the condition to apply the defined apodization function. With the above, we can demonstrate how minimizing the per patch reconstruction loss leads to minimizing the overall input modeling DISPLAYFORM0 which represents the internal modeling of the factorial model applied across filters and patches. As a , when performing the per position minimization one minimizes an upper bound which ultimately reaches the global minimum as We first present the topologies used in the experiments except for the notation ResNetD-W which is the standard wide ResNet based topology with depth D and width W. We thus have the following network architectures for smallCNN and largeCNN: largeCNN

Represent the TLDR for finding the corresponding scientific article

Reformulate deep networks nonlinearities from a vector quantization scope and bridge most known nonlinearities together.

Represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR

Question: Engineered proteins offer the potential to solve many problems in biomedicine, energy, and materials science, but creating designs that succeed is difficult in practice. A significant aspect of this challenge is the complex coupling between protein sequence and 3D structure, and the task of finding a viable design is often referred to as the inverse protein folding problem. We develop generative models for protein sequences conditioned on a graph-structured specification of the design target. Our approach efficiently captures the complex dependencies in proteins by focusing on those that are long-range in sequence but local in 3D space. Our framework significantly improves upon prior parametric models of protein sequences given structure, and takes a step toward rapid and targeted biomolecular design with the aid of deep generative models. A central goal for computational protein design is to automate the invention of protein molecules with defined structural and functional properties. This field has seen tremendous progess in the past two decades BID14, including the design of novel 3D folds BID20, enzymes BID30, and complexes BID4. However, the current practice often requires multiple rounds of trial-and-error, with first designs frequently failing BID19 BID28. Several of the challenges stem from the bottom-up nature of contemporary approaches that rely on both the accuracy of energy functions to describe protein physics as well as on the efficiency of sampling algorithms to explore the protein sequence and structure space. Here, we explore an alternative, top-down framework for protein design that directly learns a conditional generative model for protein sequences given a specification of the target structure, which is represented as a graph over the sequence elements. Specifically, we augment the autoregressive self-attention of recent sequence models BID34 with graph-based descriptions of the 3D structure. By composing multiple layers of structured self-attention, our model can effectively capture higher-order, interaction-based dependencies between sequence and structure, in contrast to previous parameteric approaches BID24 BID36 that are limited to only the first-order effects. The graph-structured conditioning of a sequence model affords several benefits, including favorable computational efficiency, inductive bias, and representational flexibility. We accomplish the first two by leveraging a well-evidenced finding in protein science, namely that long-range dependencies in sequence are generally short-range in 3D space BID23 BID3. By making the graph and self-attention similarly sparse and localized in 3D space, we achieve computational scaling that is linear in sequence length. Additionally, graph structured inputs offer representational flexibility, as they accomodate both coarse,'flexible backbone' (connectivity and topology) as well as fine-grained (precise atom locations) descriptions of structure. We demonstrate the merits of our approach via a detailed empirical study. Specifically, we evaluate our model at structural generalization to sequences of protein folds that were outside of the training set. Our model achieves considerably improved generalization performance over the recent deep models of protein sequence given structure as well as structure-naïve language models. Generative models for proteins A number of works have explored the use of generative models for protein engineering and design. and BID36 proposed neural models for sequences given 3D structure, where the amino acids at different positions in the sequence are predicted independently of one another. BID11 introduced a generative model for protein sequences conditioned on a 1D, context-free grammar based specification of the fold topology. BID8 and BID37 used deep neural networks to model the conditional distribution of letters in a specific position given the structure and sequence of all surrounding residues. In contrast to these works, our model captures the joint distribution of the full protein sequence while grounding these dependencies in terms of long-range interactions arising from the structure. In parallel to the development of structure-based models, there has been considerable work on deep generative models for protein sequences in individual protein families with directed BID26 BID31 and undirected BID33 ) latent variable models. These methods have proven useful for protein engineering, but presume the availability of a large number of sequences from a particular family. More recently, several groups have obtained promising using unconditional protein language models BID6 BID0 BID12 BID27 to learn protein sequence representations that can transfer well to supervised tasks. While serving different purposes, we emphasize that one advantage of conditional generative modeling is to facilitate adaptation to specific (and potentially novel) parts of structure space. Language models trained on hundreds of millions of evolutionary sequences are unfortunately still'semantically' bottlenecked by the much smaller number of evolutionary 3D folds (perhaps thousands) that the sequences design. We propose evaluating protein language models with structure-based splitting of sequence data (Section 3, albeit on much smaller sequence data), and begin to see how unconditional language models may struggle to assign high likelihoods to sequences from out-of-training folds. In a complementary line of research, deep models of protein structure BID2 BID16 BID1 have been proposed recently that could be used to craft 3D structures for input to sequence design. Protein design For classical approaches to computational protein design, which are based on joint modeling of structure and sequence, we refer the reader to a review of both methods and accomplishments in BID14. More recently, proposed a non-parametric approach to protein design in which a target design is decomposed into substructural motifs that are then queried against a protein database. In this work we will focus on comparisons with direct parametric models of the sequence-structure relationship. Self-Attention Our model extends the Transformer BID7 to additionally capture sparse, pairwise relational information between sequence elements. The dense variation of this problem was explored in BID29 and. As noted in those works, incorporating general pairwise information incurs O(N 2) memory (and computational) cost for sequences of length N, which can be highly limiting for training on GPUs. We circumvent this cost by instead restricting the self-attention to the sparsity of the input graph. Given this graph-structured self-attention, our model may also be reasonably cast in the framework of message-passing or graph neural networks BID10 BID5. Our approach is similar to Graph Attention Networks BID35, but augmented with edge features and an autoregressive decoder. We represent protein structure in terms of an attributed graph G = (V, E) with node features V = {v 1, . . ., v N} and edge features E = {e ij} i =j over the sequence residues (amino acids). This formulation can accommodate different variations on the macromolecular design problem, including 3D considerations For a rigid-body design problem, the structure for conditioning is a fixed set of backbone coordinates X = {x i ∈ R 3 : 1 ≤ i ≤ N}, where N is the number of positions 1. We desire a graph representation of the coordinates G(X) that has two properties:• Invariance. The features are invariant to rotations and translations.• Locally informative. The edge features incident to v i due to its neighbors N(i),i.e. {e ij} j∈N(i), contain sufficient information to reconstruct all adjacent coordinates {x j} j∈N(i) up to rigid-body motion. While invariance is motivated by standard symmetry considerations, the second property is motivated by limitations of current graph neural networks BID10. In these networks, updates to node features v i depend only on the edge and node features adjacent to v i. However, typically, these features are insufficient to reconstruct the relative neighborhood positions {x j} j∈N(i), so individual updates cannot fully depend on the'local environment'. For example, pairwise distances D ij and D il are insufficient to determine if x j and x l are on the same or opposite sides of x i. We develop invariant and locally informative features by first augmenting the points x i with'orientations' O i that define a local coordinate system at each point. We define these in terms of the backbone geometry as DISPLAYFORM0 where b i is the negative bisector of angle between the rays (x i−1 − x i) and (x i+1 − x i), and n i is a unit vector normal to that plane. Formally, we have DISPLAYFORM1 Finally, we derive the spatial edge features e (s) ij from the rigid body transformation that relates reference frame (x i, O i) to reference frame (x j, O j). While this transformation has 6 degrees of freedom, we decompose it into features for distance, direction, and orientation as DISPLAYFORM2 Here r(·) is a function that lifts the distances into a radial basis 2, the term in the middle corresponds to the relative direction of x j in the reference frame of (x i, O i), and q(·) converts the 3 × 3 relative rotation matrix to a quaternion representation. Quaternions represent rotations as four-element vectors that can be efficiently and reasonably compared by inner products. Positional encodings Taking a cue from the original Transformer model, we obtain positional embeddings e (p) ij that encode the role of local structure around node i. Specifically, we need to model the positioning of each neighbor j relative to the node under consideration i. Therefore, we obtain the position embedding as a sinusoidal function of the gap i − j. Note that this is in contrast to the absolute positional encodings of the original Transformer, and instead matches the relative encodings in BID29.Node and edge features Finally, we obtain an aggregate edge encoding vector e ij by concatenating the structural encodings e (s) ij and the positional encodings e (p) ij and then linearly transforming them to have the same dimension as the model. We only include edges in the k-nearest neighbors graph of X, with k = 30 for all experiments. For node features, we compute the three dihedral angles of the protein backbone (φ i, ψ i, ω i) and embed these on the 3-torus as {sin, cos} × (φ i, ψ i, ω i). We also consider'flexible backbone' descriptions of 3D structure based solely on topological binary edge features. We combine the relative positional encodings with two binary edge features: contacts that indicate when the distance between C α residues at i and j are less than 8 Angstroms and hydrogen bonds which are directed and defined by the electrostatic model of DSSP BID17. These features implicitly integrate over different 3D backbone configurations that are compatible with the specified topology. In this work, we introduce a Structured Transformer model that draws inspiration from the selfattention based Transformer model BID34 and is augmented for scalable incorporation of relational information. While general relational attention incurs quadratic memory and computation costs, we avert these by restricting the attention for each node i to the set N(i, k) of its k-nearest neighbors in 3D space. Since our architecture is multilayered, iterated local attention can derive progressively more global estimates of context for each node i. Second, unlike the standard Transformer, we also include edge features to embed the spatial and positional dependencies in deriving the attention. Thus, our model generalizes Transformer to spatially structured settings. Autoregressive decomposition We decompose the joint distribution of the sequence given structure p(s|x) autoregressively as DISPLAYFORM0 where the conditional probability p(s i |x, s <i) of amino acid s i at position i is conditioned on both the input structure x and the preceding amino acids s <i = {s 1, . . . s i−1} 4. These conditionals are parameterized in terms of two sub-networks: an encoder that computes refined node embeddings from structure-based node features V(x) and edge features E(x) and a decoder that autoregressively predicts letter s i given the preceding sequence and structural embeddings from the encoder. Encoder Our encoder module is designed as follows. A transformation W h: DISPLAYFORM1 Each layer of the encoder implements a multi-head self-attention component, where head ∈ [L] can attend to a separate subspace of the embeddings via learned query, key and value transformations BID34. The queries are derived from the current embedding at node i while the keys and values from the relational information r ij = (h j, e ij) at adjacent nodes j ∈ N (i, k). Specifically, W DISPLAYFORM2 where m DISPLAYFORM3 The of each attention head l are collected as the weighted sum h DISPLAYFORM4 ij and then concatenated and transformed to give the update DISPLAYFORM5 We update the embeddings with this residual and alternate between these self-attention layers and position-wise feedforward layers as in the original Transformer BID34. We stack multiple layers atop each other, and thereby obtain continually refined embeddings as we traverse the layers bottom up. The encoder yields the embeddings produced by the topmost layer as its output. Decoder Our decoder module has the same structure as the encoder but with augmented relational information r ij that allows access to the preceding sequence elements s <i in a causally consistent manner. Whereas the keys and values of the encoder are based on the relational information r ij = (h j, e ij), the decoder can additionally access sequence elements s j as DISPLAYFORM6 Here h DISPLAYFORM7 is the embedding of node j in the current layer of the decoder, h (enc) j is the embedding of node j in the final layer of the encoder, and g(s j) is a sequence embedding of amino acid s j at node j. This concatenation and masking structure ensures that sequence information only flows to position i from positions j < i, but still allows position i to attend to subsequent structural information. We stack three layers of self-attention and position-wise feedforward modules for the encoder and decoder with a hidden dimension of 128 throughout the experiments 5. Dataset To evaluate the ability of the models to generalize across different protein folds, we collected a dataset based on the CATH hierarchical classification of protein structure BID25. For all domains in the CATH 4.2 40% non-redundant set of proteins, we obtained full chains up to length 500 (which may contain more than one domain) and then cluster-split these at the CATH topology level (i.e. fold level) into training, validation, and test sets at an 80/10/10 split. Chains containing multiple CATH tpologies were purged with precedence for test over validation over train. Our splitting procedure ensured that no two domains from different sets would share the same topologies (folds). The final splits contained 18025 chains in the training set, 1637 chains in the validation set, and 1911 chains in the test set. Optimization We trained models using the learning rate schedule and initialization of BID34, a dropout BID32 rate of 10%, and early stopping based on validation perplexity.5 except for the decoder-only language model experiment which used a hidden dimension of 256 Many protein sequences may reasonably design the same 3D structure BID21, and so we focus on likelihood-based evaluations of model performance. Specifically, we evaluate the perplexity per letter of test protein folds (topologies) that were held out from the training and validation sets. Protein perplexities What kind of perplexities might be useful? To provide context, we first present perplexities for some simple models of protein sequences in TAB0. The amino acid alphabet and its natural frequencies upper-bound perplexity at 20 and ∼17.8, respectively. Random protein sequences under these null models are unlikely to be functional without further selection BID18. First order profiles of protein sequences such as those from the Pfam database BID9, however, are widely used for protein engineering. We found the average perplexity per letter of profiles in Pfam 32 (ignoring alignment uncertainty) to be ∼11.6. This suggests that even models with high perplexities of this order have the potential to be useful models for the space of functional protein sequences. The importance of structure We found that there was a significant gap between unconditional language models of protein sequences and models conditioned on structure. Remarkably, for a range of structure-independent language models, the typical test perplexities turned out to be ∼16-17 TAB1, which were barely better than null letter frequencies TAB0. We emphasize that the RNNs were not broken and could still learn the training set in these capacity ranges. It would seem that protein language models trained on one subset of 3D folds (in our cluster-splitting procedure) generalize poorly to predict the sequences of unseen folds, which is important to consider when training protein language models for protein engineering and design. All structure-based models had (unsurprisingly) considerably lower perplexities. In particular, our Structured Transformer model attained a perplexity of ∼7 on the full test set. When we compared different graph features of protein structure TAB2, we indeed found that using local orientation information was important. We also compared to a recent method SPIN2 that predicts, using deep neural networks, protein sequence profiles given protein structures BID24. Since SPIN2 is computationally intensive (minutes per protein for small proteins) and was trained on complete proteins rather than chains, we evaluated it on two subsets of the full test set: a'Small' subset of the test set containing chains up to length 100 and a'Single chain' subset containing only those models where the single chain accounted for the entire protein record in the Protein Data Bank. Both subsets discarded any chains with structural gaps. We found that our Structured Transformer model considerably improved upon the perplexities of SPIN2 TAB1. We presented a new deep generative model to'design' protein sequences given a graph specification of their structure. Our model augments the traditional sequence-level self-attention of Transformers BID34 with relational 3D structural encodings and is able to leverage the spatial locality of dependencies in molecular structures for efficient computation. When evaluated on unseen folds, the model achieves significantly improved perplexities over the state-of-the-art parametric generative models. Our framework suggests the possibility of being able to efficiently design and engineer protein sequences with structurally-guided deep generative models, and underscores the central role of modeling sparse long-range dependencies in biological sequences. We thank members of the MIT MLPDS consortium for helpful feedback and discussions.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We learn to conditionally generate protein sequences given structures with a model that captures sparse, long-range dependencies.

Represent the TLDR for finding the corresponding scientific article

We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.