Question: represent the scientific article to find a short tldr.

Question: Unsupervised representation learning holds the promise of exploiting large amount of available unlabeled data to learn general representations. A promising technique for unsupervised learning is the framework of Variational Auto-encoders (VAEs). However, unsupervised representations learned by VAEs are significantly outperformed by those learned by supervising for recognition. Our hypothesis is that to learn useful representations for recognition the model needs to be encouraged to learn about repeating and consistent patterns in data. Drawing inspiration from the mid-level representation discovery work, we propose PatchVAE, that reasons about images at patch level. Our key contribution is a bottleneck formulation in a VAE framework that encourages mid-level style representations. Our experiments demonstrate that representations learned by our method perform much better on the recognition tasks compared to those learned by vanilla VAEs. Due to the availability of large labeled visual datasets, supervised learning has become the dominant paradigm for visual recognition. That is, to learn about any new concept, the modus operandi is to collect thousands of labeled examples for that concept and train a powerful classifier, such as a deep neural network. This is necessary because the current generation of models based on deep neural networks require large amounts of labeled data . This is in stark contrast to the insights that we have from developmental psychology on how infants develop perception and cognition without any explicit supervision . Moreover, the supervised learning paradigm is ill-suited for applications, such as health care and robotics, where annotated data is hard to obtain either due to privacy concerns or high cost of expert human annotators. In such cases, learning from very few labeled images or discovering underlying natural patterns in large amounts of unlabeled data can have a large number of potential applications. Discovering such patterns from unlabeled data is the standard setup of unsupervised learning. Over the past few years, the field of unsupervised learning in computer vision has followed two seemingly different tracks with different goals: generative modeling and self-supervised learning. The goal of generative modeling is to learn the probability distribution from which data was generated, given some training data. A learned model can draw samples from the same distribution or evaluate the likelihoods of new data. Generative models are also useful for learning compact representation of images. However, we argue that these representations are not as useful for visual recognition. This is not surprising since the task of reconstructing images does not require the bottleneck representation to sort out meaningful data useful for recognition and discard the rest; on the contrary, it encourages preserving as much information as possible for reconstruction. In comparison, the goal in selfsupervised learning is to learn representations that are useful for recognition. The standard paradigm is to establish proxy tasks that don't require human-supervision but can provide signals useful for recognition. Due to the mismatch in goals of unsupervised learning for visual recognition and the representations learned from generative modeling, self-supervised learning is a more popular way of learning representations from unlabeled data. However, fundamental limitation of this self-supervised paradigm is that we need to define a proxy-task that can mimic the desired recognition. It is not always possible to establish such a task, nor are these tasks generalizable across recognition tasks. In this paper, we take the first steps towards enabling the unsupervised generative modeling approach of VAEs to learn representations useful for recognition. Our key hypothesis is that for a representation to be useful, it should capture just the interesting parts of the images, as opposed to everything in the images. What constitutes an interesting image part has been defined and studied in earlier works that pre-date the end-to-end trained deep network methods; ). Taking inspiration from these works, we propose a novel representation that only encodes such few parts of an image that are repetitive across the dataset, i.e., the patches that occur often in images. By avoiding reconstruction of the entire image our method can focus on regions that are repeating and consistent across many images. In an encoder-decoder based generative model, we constrain the encoder architecture to learn such repetitive parts -both in terms of representations for appearance of these parts (or patches in an image) and where these parts occur. We formulate this using variational auto-encoder (β-VAEs) , where we impose novel structure on the latent representations. We use discrete latents to model part presence or absence and continuous latents to model their appearance. We present this approach, PatchVAE, in Section 3 and demonstrate that it learns representations that are much better for recognition as compared to those learned by the standard β-VAEs . In addition, we propose in Section 3.4 that losses that favor foreground, which is more likely to contain repetitive patterns, in representations that are much better at recognition. In Section 4, we present on CIFAR100 , MIT Indoor Scene Recognition , Places , and ImageNet datasets. Our contributions are as follows: • We propose a novel patch-based bottleneck in the VAE framework that learns representations that can encode repetitive parts across images. • We demonstrate that our method, PatchVAE, learns unsupervised representations that are better suited for recognition in comparison to traditional VAEs. • We show that losses that favor foreground are better for unsupervised learning of representations for recognition. • We perform extensive ablation analysis to understand the importance of different aspects of the proposed PatchVAE architecture. Due to its potential impact, unsupervised learning (particularly for deep networks) is one of the most researched topics in visual recognition over the past few years. Generative models such as VAEs (; ; ;), PixelRNN (van den), PixelCNN , and their variants have proven effective when it comes to learning compressed representation of images while being able to faithfully reconstruct them as well as draw samples from the data distribution. GANs (; ; ;) on the other hand, while don't model the probability density explicitly, can still produce high quality image samples from noise. There has been work combining VAEs and GANs to be able to simultaneously learn image data distribution while being able to generate high quality samples from it . Convolution sparse coding is an alternative approach for reconstruction or image in-painting problems. Our work complements existing generative frameworks in that we provide a structured approach for VAEs that can learn beyond low-level representations. We show the effectiveness of the representations learned by our model by using them for standard visual recognition tasks. There has been a lot of work in interpreting or disentangling representations learned using generative models such as VAEs (; ;). However, there is little evidence of effectiveness of disentangled representations in visual recognition tasks. In our work, we focus on incorporating inductive biases in these generative models (e.g., VAEs) such that they can learn representations better suited for visual recognition tasks. A related, but orthogonal, line of work is self-supervised learning where a proxy task is designed to learn representation useful for recognition. These proxy tasks vary from simple tasks like arranging patches in an image in the correct spatial order (; and arranging frames from a video in correct temporal order , to more involved tasks like in-painting and context prediction . We follow the best practices from this line of work for evaluating the learned representations. Our encoder network computes a set of feature maps f using φ(x). This is followed by 2 independent single layer networks -bottom network generates part visibility parameters Q V. We combine Q V with output of top network to generate part appearance parameters Q A. We sample zvis and zapp to constructẑ as described in Section 3.2 which is input to the decoder network. We also visualize the corresponding priors for latents zapp and zvis in the dashed gray boxes. Our work builds upon VAE framework proposed by. We briefly review relevant aspects of the VAE framework and then present our approach. Standard VAE framework assumes a generative model for data where first a latent z is sampled from a prior p(z) and then the data is generated from a conditional distribution G(x|z). A variational approximation Q(z|x) to the true intractable posterior is introduced and the model is learned by minimizing the following negative variational lower bound (ELBO). Q(z|x) is often referred to as an encoder as it can be viewed as mapping data to the the latent space, while G(x|z) is referred to as a decoder (or generator) that can be viewed as mapping latents to the data space. Both Q and G are commonly paramterized as neural networks. Fig. 1a shows the commonly used VAE architecture. If the conditional G(x|z) takes a gaussian form, negative log likelihood in the first term of RHS of Eq. 1 becomes mean squared error between generator output x = G(x|z) and input data x. In the second term, prior p(z) is assumed to be a multi-variate normal distribution with zero-mean and diagonal covariance N (0, I) and the loss simplifies to When G and Q are differentiable, entire model can be trained with SGD using reparameterization trick . propose an extension for learning disentangled representation by incorporating a weight factor β for the KL Divergence term yielding VAE framework aims to learn a generative model for the images where the latents z represent the corresponding low dimensional generating factors. The latents z can therefore be treated as image representations that capture the necessary details about images. However, we postulate that representations produced by the standard VAE framework are not ideal for recognition as they are learned to capture all details, rather than capturing'interesting' aspects of the data and dropping the rest. This is not surprising since there formulation does not encourage learning semantic information. For learning semantic representations, in the absence of any relevant supervision (as is available in self-supervised approaches), inductive biases have to be introduced. Therefore, taking inspiration from works on unsupervised mid-level pattern discovery; ), we propose a formulation that encourages the encoder to only encode such few parts of an image that are repetitive across the dataset, i.e., the patches that occur often in images. Since the VAE framework provides a principled way of learning a mapping from image to latent space, we consider it ideal for our proposed extension. We chose β-VAEs for their simplicity and widespread use. In Section 3.2, we describe our approach in detail and in Section 3.4 propose a modification in the reconstruction error computation to bias the error term towards foreground high-energy regions (similar to the biased initial sampling of patterns in). Given an image x, let f = φ(x) be a deterministic mapping that produces a 3D representation f of size h × w × d e, with a total of L = h × w locations (grid-cells). We aim to encourage the encoder network to only encode parts of an image that correspond to highly repetitive patches. For example, a random patch of noise is unlikely to occur frequently, whereas patterns like faces, wheels, windows, etc. repeat across multiple images. In order capture this intuition, we force the representation f to be useful for predicting frequently occurring parts in an image, and use just these predicted parts to reconstruct the image. We achieve this by transforming f toẑ which encodes a set of parts at a small subset of L locations on the grid cells. We refer toẑ as "patch latent codes" for an image. Next we describe how we re-tool the β-VAE framework to learn these local latent codes. We first describe our setup for a single part and follow it up with a generalization to multiple parts (Section 3.3). Image Encoding. Given the image representation f = φ(x), we would like to learn part representations at each grid location l (where l ∈ {1, . . ., L}). A part is parameterized by its appearance z app and its visibility z l vis (i.e., presence or absence of the part at grid location l). We use two networks, vis | f ) of the part parameters z app and z l vis respectively. Since the mapping f = φ(x) is deterministic, we can re-write these distribu-. Therefore, given an image x the encoder networks estimate the posterior Note that f is a deterministic feature map, whereas z app and z l vis are stochastic. Image Decoding. We utilize a generator or decoder network G, that given z vis and z app, reconstructs the image. First, we sample a part appearanceẑ app (d p dimensional, continuous) and then sample part visibilitiesẑ l vis (L dimensional, binary) one for each location l from the posteriorŝ Next, we construct a 3D representationẑ by placingẑ app at every location l where the part is present (i.e.,ẑ l vis = 1). This can be implemented by a broadcasted product ofẑ app andẑ l vis. We refer toẑ as patch latent code. Again note that f is deterministic andẑ is stochastic. Finally, a deconvolutional network takesẑ as input and generates an imagex. This image generation process can be written aŝ Since all latent variables (z l vis for all l and z app) are independent of each other, they can be stacked as This enables us to use a simplified the notation (refer to and): Note that despite the additional structure, our model still resembles the setup of variational autoencoders. The primary difference arises from: use of discrete latents for part visibility, patchbased bottleneck imposing additional structure on latents, and feature assembly for generator. Training. We use the training setup of β-VAE and use the maximization of variational lower bound to train the encoder and decoder jointly (described in Section 3.1). The posterior Q A, which captures the appearance of a part, is assumed to be a zero-mean Normal distribution with diagonal covariance N (0, I). The posterior Q V, which captures the presence or absence a part, is assumed to be a Bernoulli distribution Bern z prior vis with prior z prior vis. Therefore, the ELBO for our approach can written as (refer to): where, the D KL term can be expanded as: Implementation details. As discussed in Section 3.1, the first and second terms of the RHS of can be trained using L2 reconstruction loss and reparameterization trick . In addition, we also need to compute KL Divergence loss for part visibility. Learning discrete probability distribution is a challenging task since there is no gradient defined to backpropagate reconstruction loss through the stochastic layer at decoder even when using the reparameterization trick. Therefore, we use the relaxed-bernoulli approximation for training part visibility distributions z, where (h, w) are spatial dimensions and d e is the number of channels. Therefore, the number of locations vis > 1, the part occurs at multiple locations in an image. Since all these locations correspond to same part, their appearance should be the same. To incorporate this, we take the weighted average of the part appearance feature at each location, weighted by the probability that the part is present. Since we use the probability values for averaging the is deterministic. This operation is encapsulated by the Q A encoder (refer to Figure 1b). During image generation, we sampleẑ app once and replicate it at each location whereẑ l vis = 1. During training, this forces the model to: only predictẑ l vis = 1 where similar looking parts occur, and learn a common representation for the part that occurs at these locations. Note that z app can be modeled as a mixture of distributions (e.g., mixture of gaussians) to capture complicated appearances. However, in this work we assume that the convolutional neural network based encoders are powerful enough to map variable appearance of semantic concepts to similar feature representations. Therefore, we restrict ourselves to a single gaussian distribution. Next we extend the framework described above to use multiple parts. To use N parts, we use N × 2 encoder networks Q vis parameterize the i th part. Again, this can be implemented efficiently as 2 networks by concatenating the outputs together. The image generator samplesẑ vis from the outputs of these encoder networks and constructsẑ (i). We obtain the final patch latent codeẑ by concatenating allẑ For this multiple part case, can be written as: Similarly, and can be written as: The training details and assumptions of posteriors follow the previous section. Figure 2: Concepts captured by parts: We visualize a few representative examples for several parts to qualitatively demonstrate the visual concepts captured by parts. For each part, we crop image patches centered on the part location where it is predicted to be present. Selected patches are sorted by part visibility probability as score. We have manually selected a diverse set from the top 50 occurrences from the training images. As visible, a single part may capture diverse set of concepts that are similar in shape or texture or occur in similar context, but belong to different categories. We show which categories the patches come from. The L2 reconstruction loss used for training β-VAEs (and other reconstruction based approaches) gives equal importance to each region of an image. This might be reasonable for tasks like image compression and image de-noising. However, for the purposes of learning semantic representations, not all regions are equally important. For example, "sky" and "walls" occupy large portions of an image, whereas concepts like "windows," "wheels,", "faces" are comparatively smaller, but arguably more important. To incorporate this intuition, we use a simple and intuitive strategy to weigh the regions in an image in proportion to the gradient energy in the region. More concretely, we compute laplacian of an image to get the intensity of gradients per-pixel and average the gradient magnitudes in 8 × 8 local patches. The weight multiplier for the reconstruction loss of each 8 × 8 patch in the image is proportional to the average magnitude of the patch. All weights are normalized to sum to one. We refer to this as weighted loss (L w). Note that this is similar to the gradient-energy biased sampling of mid-level patches used in;. In Appendix 6.1, we show examples of weight masks for some of the images. In addition, we also consider an adversarial training strategy from GANs to train VAEs as proposed by , where the discriminator network from GAN implicitly learns to compare images and gives a more abstract reconstruction error for the VAE. We refer to this variant by using'GAN' suffix in experiments. In Section 4, we demonstrate that the proposed weighted loss (L w) is complementary to the discriminator loss from adversarial training, and these losses in better recognition capabilities for both β-VAE and PatchVAE. Datasets. We evaluate our proposed model on CIFAR100 , MIT Indoor Scene Recognition , and Places datasets. Details of these datasets can be found in Appendix 6.2. Learning paradigm. In order to evaluate the utility of features learned for recognition, we setup the learning paradigm as follows: we will first train the model in an unsupervised manner on all the images other than test set images. After that, we discard the generator network and use only part of the encoder network φ(x) to train a supervised model on the classification task of the respective dataset. We study different training strategies for the classification stage as discussed later. Training details. In all experiments, we use the following architectures. For CIFAR100, Indoor67, and Place205, φ(x) has a conv layer followed by two residual blocks . For ImageNet, φ(x) is a ResNet18 model (a conv layer followed by four residual blocks). For all datasets, Q A and Q V have a single conv layer each. For classification, we start from φ(x), and add a fully-connected layer with 512 hidden units and a final fully-connected layer as classifier. More details can be found in Appendix 6.2 and 6.3. During the unsupervised learning part of training, all methods are trained for 90 epochs for CIFAR100 and Indoor67, 2 epochs for Places205, and 30 epochs for ImageNet dataset. All methods use ADAM optimizer for training, with initial learning rate of 1 × 10 −4 and a minibatch size of 128. For relaxed bernoulli in Q V, we start with the temperature of 1.0 with an annealing rate of 3 × 10 −5 (details in ). For training the classifier, all methods use stochastic gradient descent (SGD) with momentum with a minibatch size of 128. Initial learning rate is 1 × 10 −2 and we reduce it by a factor of 10 every 30 epochs. All experiments are trained for 90 epochs for CIFAR100 and Indoor67, 5 epochs for Places205, and 30 epochs for ImageNet datasets. Table 1: Classification on CIFAR100, Indoor67, and Places205. We initialize the classification model with the representations φ(x) learned from unsupervised learning task. The model φ(x) comprises of a conv layer followed by two residual blocks (each having 2 conv layers). First column (called 'Conv1') corresponds to Top-1 classification accuracy with pre-trained model with the first conv layer frozen, second and third columns correspond to with 3 conv layers and 5 conv layers frozen respectively. Details in Section 4.1. Baselines. We use the β-VAE model (Section 3.1) as our primary baseline. In addition, we use weighted loss and discriminator loss ing in the β-VAE-* family of baselines. We also compare against a BiGAN model from. We use similar backbone architectures for encoder/decoder (and discriminator if present) across all methods, and tried to keep the number of parameters in different approaches comparable to the best of our ability. Exact architecture details can be found in Appendix 6.3. In Table 1, we report the top-1 classification on CIFAR100, Indoor67, and Places205 datasets for all methods with different training strategies for classification. First, we keep all the pre-trained weights in φ(x) from the unsupervised task frozen and only train the two newly added conv layers in the classification network (reported under column 'Conv5'). We notice that our method (with different losses) generally outperforms the β-VAE counterpart by a healthy margin. This shows that the representations learned by PatchVAE framework are better for recognition compared to β-VAEs. Moreover, better reconstruction losses ('GAN' and L w) generally improve both β-VAE and PatchVAE, and are complementary to each other. Next, we fine-tune the last residual block along with the two conv layers ('Conv3' column). We observe that PatchVAE performs better than VAE under all settings except the for CIFAR100 with just L2 loss. However, when using better reconstruction losses, the performance of PatchVAE improves over β-VAE. Similarly, we fine-tune all but the first conv layer and report the in'Conv1' column. Again, we notice similar trends, where our method generally performs better than β-VAE on Indoor67 and Places205 dataset, but β-VAE performs better CIFAR100 by a small margin. When compared to BiGAN, PatchVAE representations are better on all datasets ('Conv5') by a huge margin. However, when fine-tuning the pre-trained weights, BiGAN performs better on two out of four datasets. We also report using pre-trained weights in φ(x) using supervised ImageNet ImageNet Results. Finally, we report on the large-scale ImageNet benchmark in Table 2. For these experiments, we use ResNet18 architecture for all methods. All weights are first learned using the unsupervised tasks. Then, we fine-tune the last two residual blocks and train the two newly added conv layers in the classification network (therefore, first conv layer and the following two residual blocks are frozen). We notice that PatchVAE framework outperforms β-VAE under all settings, and the proposed weighted loss helps both approaches. Finally, the last row in Table 2 reports classification of same architecture randomly initialized and trained end-to-end on ImageNet using supervised training for comparison. We study the impact of various hyper-parameters used in our experiments. For the purpose of this evaluation, we follow a similar approach as in the'Conv5' column of Table 1 and all hyperparameters from the previous section. We use CIFAR100 and Indoor67 datasets for ablation analysis. Maximum number of patches. Maximum number of parts N used in our framework. Depending on the dataset, higher value of N can provide wider pool of patches to pick from. However, it can also make the unsupervised learning task harder, since in a minibatch of images, we might not get too many repeat patches. Table 3 (left) shows the effect of N on CIFAR100 and Indoor67 datasets. We observe that while increasing number of patches improves the discriminative power in case of CIFAR100, it has little or negative effect in case of Indoor67. A possible reason for this decline in performance for Indoor67 can be smaller size of the dataset (i.e., fewer images to learn). Number of hidden units for a patch appearanceẑ app. Next, we study the impact of the number of channels in the appearance featureẑ app for each patch (d p). This parameter reflects the capacity of individual patch's latent representation. While this parameter impacts the reconstruction quality of images. We observed that it has little or no effect on the classification performance of the base features. Results are summarized in Table 3 (right) for both CIFAR100 and Indoor67 datasets. Prior probability for patch visibility z prior vis. In all our experiments, prior probability for a patch is fixed to 1/N, i.e., inverse of maximum number of patches. The intuition is to encourage each location on visibility maps to fire for at most one patch. Increasing this patch visibility prior will allow all patches to fire at the same location. While this would make the reconstruction task easier, it will become harder for individual patches to capture anything meaningful. Table 4 shows the deterioration of classification performance on increasing z prior vis. Patch visibility loss weight β vis. The weight for patch visibility KL Divergence has to be chosen carefully. If β vis is too low, more patches can fire at same location and this harms the the learning capability of patches; and if β vis is too high, decoder will not receive any patches to reconstruct from and both reconstruction and classification will suffer. Table 5 summarizes the impact of varying β vis. We presented a patch-based bottleneck in a VAE framework that encourages learning useful representations for recognition. Our method, PatchVAE, constrains the encoder architecture to only learn patches that are repetitive and consistent in images as opposed to learning everything, and therefore in representations that perform much better for recognition tasks compared to vanilla VAEs. We also demonstrate that losses that favor high-energy foreground regions of an image are better for unsupervised learning of representations for recognition. 6 APPENDIX 6.1 VISUALIZATION OF WEIGHTED LOSS Figure 3 shows an illustration of the reconstruction loss L w proposed in Section 3.4. Notice that in first column, guitar has more weight that rest of the image. Similarly in second, fourth and sixth columns that train, painting, and people are respectively weighed more heavily by L w than rest of the image; thus favoring capturing the foreground regions. Image Laplacian Figure 3: Masks used for weighted reconstruction loss Lw. First row contains images randomly samples from MIT Indoor datatset. Second and third rows have the corresponding image laplacians and final reconstruction weight masks respectively. In the last row, we take the product of first and third row to highlight which parts of image are getting more attention while reconstruction. 6.2 DATASET AND TRAINING DETAILS CIFAR100 consists of 60000 32 × 32 color images in 100 classes, with 600 images per class. There are 50000 training images and 10000 test images. Indoor dataset contains 67 categories, and a total of 15620 images ('Indoor67'). Train and test subsets consist of 80 and 20 images per class respectively. Places dataset has 2.5 millions of images with 205 categories ('Places205'). Finally, we report on the large-scale ImageNet dataset, which has ∼1.28M training and 50k validation images spanning 1000 categories. The generator network has two deconv layers with batchnorm and a final deconv layer with tanh activation. When training with'GAN' loss, the additional discriminator has four conv layers, two of with have batchnorm. In this section, we share the exact architectures used in various experiments. As discussed in Section 4, we evaluated our proposed model on CIFAR100, Indoor67, and Places205 datasets. We resize and center-crop the images such that input image size for CIFAR100 datasets is 32 × 32 × 3 while for Indoor67 and Places205 datasets input image size is 64 × 64 × 3. PatchVAE can treat images of various input sizes in exactly same way allowing us to keep the architecture same for different datasets. In case of VAE and BiGAN however, we have to go through a fixed size bottleneck layer and hence architectures need to be a little different for different input image sizes. Wherever possible, we have tried to keep the number of parameters in different architectures comparable. Tables 6 and 7 show the architectures for encoders used in different models. In the unsupervised learning task, encoder comprises of a fixed neural network backbone φ(x), that given an image of size h × w × 3 generated feature maps of size This backbone architecture is common to different models discussed in the paper and consists of a single conv layer followed by 2 residual blocks. We refer to this φ(x) as Resnet-9 and it is described as Conv1-5 layers in Table 10. Rest of the encoder architecture varies depending on the model in consideration and is described in the tables below. Tables 8 and 9 show the architectures for decoders used in different models. We use a pyramid like network for decoder where feature map size is doubled in consecutive layers, while number of channels is halved. Final non-linearity used in each decoder is tanh.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A patch-based bottleneck formulation in a VAE framework that learns unsupervised representations better suited for visual recognition.

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.


Question: Represent the article for finding a single-sentence TLDR

Question: Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks (RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that arise in its training. Specifically, we parameterize the transition matrix by its singular value decomposition (SVD), which allows us to explicitly track and control its singular values. We attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. By explicitly controlling the singular values, our proposed svdRNN method allows us to easily solve the exploding gradient problem and we observe that it empirically solves the vanishing gradient issue to a large extent. We note that the SVD parameterization can be used for any rectangular weight matrix, hence it can be easily extended to any deep neural network, such as a multi-layer perceptron. Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it potentially makes the optimization process easier. Our extensive experimental also demonstrate that the proposed framework converges faster, and has good generalization, especially when the depth is large. Deep neural networks have achieved great success in various fields, including computer vision, speech recognition, natural language processing, etc. Despite their tremendous capacity to fit complex functions, optimizing deep neural networks remains a contemporary challenge. Two main obstacles are vanishing and exploding gradients, that become particularly problematic in Recurrent Neural Networks (RNNs) since the transition matrix is identical at each layer, and any slight change to it is amplified through recurrent layers BID3 ).Several methods have been proposed to solve the issue, for example, Long Short Term Memory (LSTM) BID8 ) and residual networks BID7 ). Another recently proposed class of methods is designed to enforce orthogonality of the square transition matrices, such as unitary and orthogonal RNNs (oRNN) BID1; BID13 ). However, while these methods solve the exploding gradient problem, they limit the expressivity of the network. In this paper, we present an efficient parametrization of weight matrices that arise in a deep neural network, thus allowing us to stabilize the gradients that arise in its training, while retaining the desired expressive power of the network. In more detail we make the following contributions:• We propose a method to parameterize weight matrices through their singular value decomposition (SVD). Inspired by BID13 ), we attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. The SVD parametrization allows us to retain the desired expressive power of the network, while enabling us to explicitly track and control singular values.• We apply our SVD parameterization to recurrent neural networks to exert spectral constraints on the RNN transition matrix. Our proposed svdRNN method enjoys similar space and time complexity as the vanilla RNN. We empirically verify the superiority of svdRNN over RNN/oRNN, in some case even LSTMs, over an exhaustive collection of time series classification tasks and the synthetic addition and copying tasks, especially when the network depth is large.• Theoretically, we show how our proposed SVD parametrization can make the optimization process easier. Specifically, under a simple setting, we show that there are no spurious local minimum for the linear svdRNN in the population risk.• Our parameterization is general enough to eliminate the gradient vanishing/exploding problem not only in RNNs, but also in various deep networks. We illustrate this by applying SVD parametrization to problems with non-square weight matrices, specifically multi-layer perceptrons (MLPs) and residual networks. We now present the outline of our paper. In Section 2, we discuss related work, while in Section 3 we introduce our SVD parametrization and demonstrate how it spans the whole parameter space and does not limit expressivity. In Section 4 we propose the svdRNN model that is able to efficiently control and track the singular values of the transition matrices, and we extend our parameterization to non-square weight matrices and apply it to MLPs in Section 5. Section 6 provides the optimization landscape of svdRNN by showing that linear svdRNN has no spurious local minimum. Experimental on MNIST and a popular time series archive are present in Section 7. Finally, we present our and future work in Section 8. Numerous approaches have been proposed to address the vanishing and exploding gradient problem. Long short-term memory (LSTM) BID8 ) attempts to address the vanishing gradient problem by adding additional memory gates. Residual networks BID7 ) pass the original input directly to the next layer in addition to the original layer output. BID14 performs gradient clipping, while BID15 applies spectral regularization to the weight matrices. Other approaches include introducing L 1 or L 2 penalization on successive gradient norm pairs in back propagation BID15 ).Recently the idea of restricting transition matrices to be orthogonal has drawn some attention. BID12 proposed initializing recurrent transition matrices to be identity or orthogonal (IRNN). This strategy shows better performance when compared to vanilla RNN and LSTM. However, there is no guarantee that the transition matrix is close to orthogonal after a few iterations. The unitary RNN (uRNN) algorithm proposed in BID1 parameterizes the transition matrix with reflection, diagonal and Fourier transform matrices. By construction, uRNN ensures that the transition matrix is unitary at all times. Although this algorithm performs well on several small tasks, BID19 showed that uRNN only covers a subset of possible unitary matrices and thus detracts from the expressive power of RNN. An improvement over uRNN, the orthogonal RNN (oRNN), was proposed by BID13. oRNN uses products of Householder reflectors to represent an orthogonal transition matrix, which is rich enough to span the entire space of orthogonal matrices. Meanwhile, BID18 empirically demonstrate that the strong constraint of orthogonality limits the model's expressivity, thereby hindering its performance. Therefore, they parameterize the transition matrix by its SVD, W = U ΣV (factorized RNN) and restrict Σ to be in a range close to 1; however, the orthogonal matrices U and V are updated by geodesic gradient descent using the Cayley transform, thereby ing in time complexity cubic in the number of hidden nodes which is prohibitive for large scale problems. Motivated by the shortcomings of the above methods, our work in this paper attempts to answer the following questions: Is there an efficient way to solve the gradient vanishing/exploding problem without hurting expressive power?As brought to wide notice in BID7, deep neural networks should be able to preserve features that are already good. BID6 consolidate this point by showing that deep linear residual networks have no spurious local optima. In our work, we broaden this concept and bring it to the area of recurrent neural networks, showing that each layer is not necessarily near identity, but being close to orthogonality suffices to get a similar . Generalization is a major concern in training deep neural networks. BID2 provide a generalization bound for neural networks by a spectral Lipschitz constant, namely the product of spectral norm of each layer. Thus, our scheme of restricting the spectral norm of weight matrices reduces generalization error in the setting of BID2. As supported by the analysis in BID5, since our SVD parametrization allows us to develop an efficient way to constrain the weight matrix to be a tight frame BID17 ), we consequently are able to reduce the sensitivity of the network to adversarial examples. The SVD of the transition matrix W ∈ R n×n of an RNN is given by W = U ΣV T, where Σ is the diagonal matrix of singular values, and U, V ∈ R n×n are orthogonal matrices, i.e., BID16 ). During the training of an RNN, our proposal is to maintain the transition matrix in its SVD form. However, in order to do so efficiently, we need to maintain the orthogonal matrices U and V in compact form, so that they can be easily updated by forward and backward propagation. In order to do so, as in BID13, we use a tool that is commonly used in numerical linear algebra, namely Householder reflectors (which, for example, are used in computing the QR decomposition of a matrix). DISPLAYFORM0 Given a vector u ∈ R k, k ≤ n, the n × n Householder reflector H n k (u) is defined as: DISPLAYFORM1The Householder reflector is clearly a symmetric matrix, and it can be shown that it is orthogonal, i.e., H 2 = I . Further, when u = 0, it has n−1 eigenvalues that are 1, and one eigenvalue which is −1 (hence the name that it is a reflector). In practice, to store a Householder reflector, we only need to store u ∈ R k rather than the full matrix. Given a series of vectors {u i} n i=k where u k ∈ R k, we define the map: DISPLAYFORM2 where the right hand side is a product of Householder reflectors, yielding an orthogonal matrix (to make the notation less cumbersome, we remove the superscript from H n k for the rest of this section). Theorem 1. The image of M 1 is the set of all n × n orthogonal matrices. The proof of Theorem 1 is an easy extension of the Householder QR factorization Theorem, and is presented in Appendix A. Although we cannot express all n × n matrices with M k, any W ∈ R n×n can be expressed as the product of two orthogonal matrices U, V and a diagonal matrix Σ, i.e. by its SVD: DISPLAYFORM3, we finally define our proposed SVD parametrization: DISPLAYFORM4 Theorem 2. The image of M 1,1 is the set of n × n real matrices. i.e. DISPLAYFORM5 The proof of Theorem 2 is based on the singular value decomposition and Theorem 1, and is presented in Appendix A. The astute reader might note that M 1,1 seemingly maps an input space of n 2 + 2n dimensions to a space of n 2 dimensions; however, since H n k (u k) is invariant to the norm of u k, the input space also has exactly n 2 dimensions. Although Theorems 1 and 2 are simple extensions of well known linear algebra , they ensure that our parameterization has the ability to represent any matrix and so the full expressive power of the RNN is retained. Theorem 3. The image of M k1,k2 includes the set of all orthogonal n×n matrices if k 1 +k 2 ≤ n+2.Theorem 3 indicates that if the total number of reflectors is greater than n: (n − k 1 + 1) + (n − k 2 + 1) ≥ n, then the parameterization covers all orthogonal matrices. Note that when fixing σ = 1, DISPLAYFORM6 In this section, we apply our SVD parameterization to RNNs and describe the ing svdRNN algorithm in detail. Given a hidden state vector from the previous step h (t−1) ∈ R n and input x (t−1) ∈ R ni, RNN computes the next hidden state h (t) and output vector o (t) ∈ R no as: DISPLAYFORM0 DISPLAYFORM1 In svdRNN we parametrize the transition matrix W ∈ R n×n using m 1 + m 2 Householder reflectors as: DISPLAYFORM2 This parameterization gives us several advantages over the regular RNN. First, we can select the number of reflectors m 1 and m 2 to balance expressive power versus time and space complexity. By Theorem 2, the choice m 1 = m 2 = n gives us the same expressive power as vanilla RNN. Notice oRNN could be considered a special case of our parametrization, since when we set m 1 + m 2 ≥ n and σ = 1, we can represent all orthogonal matrices, as proven by Theorem 3. Most importantly, we are able to explicitly control the singular values of the transition matrix. In most cases, we want to constrain the singular values to be within a small interval near 1. The most intuitive method is to clip the singular values that are out of range. Another approach would be to initialize all singular values to 1, and add a penalty term σ − 1 2 to the objective function. Here, we have applied another parameterization of σ proposed in BID18: DISPLAYFORM3 where f is the sigmoid function andσ i is updated from u i, v i via stochastic gradient descent. The above allows us to constrain σ i to be within [σ * − r, σ * + r]. In practice, σ * is usually set to 1 and r 1. Note that we are not incurring more computation cost or memory for the parameterization. For regular RNN, the number of parameters is (n o + n i + n + 1)n, while for svdRNN it is (n o + DISPLAYFORM4 . In the extreme case where m 1 = m 2 = n, it becomes (n o + n i + n + 3)n. Later we will show that the computational cost of svdRNN is also of the same order as RNN in the worst case. In forward propagation, we need to iteratively evaluate h (t) from t = 0 to L using. The only different aspect from a regular RNN in the forward propagation is the computation of W h (t−1). Note that in svdRNN, W is expressed as product of m 1 + m 2 Householder matrices and a diagonal matrix. Thus W h (t−1) can be computed iteratively using (m 1 + m 2) inner products and vector additions. Denotingû k = 0 n−k u k, we have: DISPLAYFORM0 Thus, the total cost of computing W h (t−1) is O((m 1 + m 2)n) floating point operations (flops). Detailed analysis can be found in Section 4.2. Let L({u i}, {v i}, σ, M, Y, b) be the loss or objective function, DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 Back propagation for svdRNN requires DISPLAYFORM4 ∂Σ (t) and DISPLAYFORM5 ∂h (t−1). These partial gradients can also be computed iteratively by computing the gradient of each Householder matrix at a time. We drop the superscript (t) now for ease of exposition. Givenĥ = H k (u k)h and g = ∂L ∂ĥ, we have DISPLAYFORM6 Details of forward and backward propagation can be found in Appendix (B). One thing worth noticing is that the oRNN method in BID13 actually omitted the last term in by assuming that u k are fixed. Although the scaling of u k in the Householder transform does not affect the transform itself, it does produce different gradient update for u k even if it is scaled to norm 1 afterwards. TAB0 gives the time complexity of various algorithms. Hprod and Hgrad are defined in Algorithm 2 3 (see Appendix (B)). Algorithm 2 needs 6k flops, while Algorithm 3 uses (3n + 10k) flops. Since u k 2 only needs to be computed once per iteration, we can further decrease the flops to 4k and (3n + 8k). Also, in back propagation we can reuse α in forward propagation to save 2k flops. flops DISPLAYFORM0 In this section, we extend the parameterization to non-square matrices and use Multi-Layer Perceptrons(MLP) as an example to illustrate its application to general deep networks. For any weight matrix W ∈ R m×n (without loss of generality m ≤ n), its reduced SVD can be written as: DISPLAYFORM0 DISPLAYFORM1. Thus we can extend the SVD parameterization for any non-square matrix: DISPLAYFORM2 whereΣ = (diag(σ)|0) if m < n and (diag(σ)|0) otherwise. Next we show that we only need 2 min(m, n) reflectors (rather than m + n) to parametrize any m × n matrix. By the definition of H n k, we have the following lemma: DISPLAYFORM3 Here V *,i indicates the ith column of matrix V. According to Lemma 1, we only need at most first m Householder vectors to express V L, which in the following Theorem: Theorem 4. If m ≤ n, the image of M m,n 1,n−m+1 is the set of all m × n matrices; else the image of M m,n n−m+1,1 is the set of all m × n matrices. Similarly if we constrain u i, v i to have unit length, the input space dimensions of M m,n 1,n−m+1 and M m,n m−n+1,1 are both mn, which matches the output dimension. Thus we extend Theorem 2 to the non-square case, which enables us to apply SVD parameterization to not only the RNN transition matrix, but also to general weight matrices in various deep learning models. For example, the Multilayer perceptron (MLP) model is a class of feedforward neural network with fully connected layers: DISPLAYFORM4 say n t < n t−1, we have: DISPLAYFORM5 nt−1 (v nt−1). We can use the same forward/backward propagation algorithm as described in Algorithm 1. Besides RNN and MLP, SVD parameterization method also applies to more advanced frameworks, such as Residual networks and LSTM, which we will not describe in detail here. Since we can control and upper bound the singular values of the transition matrix in svdRNN, we can clearly eliminate the exploding gradient problem. In this section, we now analytically illustrate the advantages of svdRNN with lower-bounded singular values from the optimization perspective. For the theoretical analysis in this section, we will limit ourselves to a linear recurrent neural network, i.e., an RNN without any activation. Linear recurrent neural network. For simplicity, we follow a setting similar to BID6. For compact presentation, we stack the input data as X ∈ R n×t, where X = x |x | · · · |x (t−1), and transition weights as W ∈ R n×nt where W = W |W 2 | · · · |W t. Then we can simplify the output as: DISPLAYFORM0 By absorbing M and b in each data x (t) and assuming h = 0, we further simplify the output as: DISPLAYFORM1 Suppose the input data X ∼ D, and assume its underlying relation to the output is y = Avec(X)+η, where A ∈ R n×nt and residue η ∈ R n satisfies E X ∼D [η|X] = 0. We consider the individual loss: DISPLAYFORM2 2. Claim 1. With linear recurrent neural networks, the population risk DISPLAYFORM3 DISPLAYFORM4 Therefore when ∇ W R[W] = 0 suffices R(W) = R *, meaning W reaches the global minimum. Theorem 5 potentially explains why our system is easier to optimize, since with our scheme of SVD parametrization, we have the following corollary. Corollary 1. With the update rule in, linear svdRNNs have no spurious local minimum. While the above analysis lends further credence to our observed experimental , we leave it to future work to perform a similar analysis in the presence of non-linear activation functions. In this section, we provide empirical evidence that shows the advantages of SVD parameterization in both RNNs and MLPs. For RNN models, we compare our svdRNN algorithm with (vanilla) RNN, IRNN(Le et al. FORMULA2), oRNN BID13 ) and LSTM BID8 ). The transition matrix in IRNN is initialized to be orthogonal while other matrices are initialized by sampling from a Gaussian distribution. For MLP models, we implemented vanilla MLP, Residual Network (ResNet) BID7 ) and used SVD parameterization for both of them. We used a residual block of two layers in ResNet. In most cases leaky Relu is used as activation function, except for LSTM, where leaky Relu will drastically harm the performance. To train these models, we applied Adam optimizer with stochastic gradient descent BID11 ). These models are implemented with Theano (Al-Rfou et al. FORMULA2). In this experiment, we focus on the time series classification problem, where time series are fed into RNN sequentially, which then tries to predict the right class upon receiving the sequence end BID10 ). The dataset we choose is the largest public collection of classlabeled time-series with widely varying length, namely, the UCR time-series collection from BID4 2. We present the test accuracy on 20 datasets with RNN, LSTM, oRNN and svdRNN in TAB3 (Appendix C) and Figure 1. In all experiments, we used hidden dimension n h = 32, and chose total number of reflectors for oRNN and svdRNN to be m = 16 (for svdRNN m 1 = m 2 = 8).We choose proper depth t as well as input size n i. Given sequence length L, since tn i = L, we choose n i to be the maximum divisor of L that satisfies depth ≤ √ L. To have a fair comparison (a) (b) (c) Figure 1: Performance comparisons of the RNN based models on three UCR datasets.of how the proposed principle itself influences the training procedure, we did not use dropout in any of these models. As illustrated in the optimization process in Figure 1, this ed in some overfitting (see (a) CBF), but on the other hand it shows that svdRNN is able to prevent overfitting. This supports our claim that since generalization is bounded by the spectral norm of the weights BID2, svdRNN will potentially generalize better than other schemes. This phenomenon is more drastic when the depth is large (e.g. ArrowHead(251 layers) and FaceAll(131 layers)), since regular RNN, and even LSTM, have no control over the spectral norms. Also note that there are substantially fewer parameters in oRNN and svdRNN as compared to LSTM. In this experiment, we compare different models on the MNIST image dataset. The dataset was split into a training set of 60000 instances and a test set of 10000 instances. The 28 × 28 MNIST pixels are flattened into a vector and then traversed by the RNN models. Table 2 shows accuracy scores across multiple We tested different models with different network depth as well as width. Figure 2(a)(b) shows the test accuracy on networks with 28 and 112 layers (20 and 128 hidden dimensions) respectively. It can be seen that the svdRNN algorithms have the best performance and the choice of r (the amount that singular values are allowed to deviate from 1) does not have much influence on the final precision. Also we explored the effect of different spectral constraints and explicitly tracked the spectral margin (max i |σ i − 1|) of the transition matrix. Intuitively, the influence of large spectral margin should increase as the network becomes deeper. Figure 2(d) shows the spectral margin of different RNN models. Although IRNN has small spectral margin at first few iterations, it quickly deviates from orthogonal and cannot match the performance of oRNN and svdRNN. Figure 2(e) shows the magnitude of first layer gradient ∂L ∂h 2. RNN suffers from vanishing gradient at first 50k iterations while oRNN and svdRNN are much more stable. Note that LSTM can perform relatively well even though it has exploding gradient in the first layer. We also tested RNN and svdRNN with different amount of non-linearity, as shown in Figure 2 (c). This is achieved by adjusting the leak parameter in leaky Relu: f (x) = max(leak · x, x). With leak = 1.0, it reduces to the identity map and when leak = 0 we are at the original Relu function. From the figures, we show that svdRNN is resistant to different amount of non-linearity, namely converge faster and achieve higher accuracy invariant to the amount of the leak factor. To explore the reason underneath, we illustrate the gradient in Figure 2 (f), and find out svdRNN could eliminate the gradient vanishing problem on all circumstances, while RNN suffers from gradient vanishing when non-linearity is higher. FORMULA2 256(m = 32) ≈ 11k 97.2 RNN BID18 128 ≈ 35k 94.1 uRNN BID1 ) 512 ≈ 16k 95.1 RC uRNN BID19 ) 512 ≈ 16k 97.5 FC uRNN BID19 ) 116 ≈ 16k 92.8 factorized RNN BID18 ) 128 ≈ 32k 94.6 LSTM BID18 ) 128 ≈ 64k 97.3 Table 2: Results for the pixel MNIST dataset across multiple algorithms. For the MLP models, each instance is flattened to a vector of length 784 and fed to the input layer. After the input layer there are 40 layers with hidden dimension 32 (Figure 3(a) ) or 30 to 100 layers with hidden dimension 128 (Figure 3(b) ). On a 40-layer network, svdMLP and svdResNet achieve similar performance as ResNet while MLP's convergence is slower. However, when the network is deeper, both MLP and ResNet start to fail. With n h = 128, MLP is not able to function with L > 35 and ResNet with L > 70. On the other hand, the SVD based methods are resilient to increasing depth and thus achieve higher precision.(a) (b) Figure 3: MLP models on MNIST with L layers n h hidden dimension In this paper, we have proposed an efficient SVD parametrization of various weight matrices in deep neural networks, which allows us to explicitly track and control their singular values. This parameterization does not restrict the network's expressive power, while simultaneously allowing fast forward as well as backward propagation. The method is easy to implement and has the same time and space complexity as compared to original methods like RNN and MLP. The ability to control singular values helps in avoiding the gradient vanishing and exploding problems, and as we have empirically shown, gives good performance. Although we only showed examples in the RNN and MLP framework, our method is applicable to many more deep networks, such as Convolutional Networks etc. However, further experimentation is required to fully understand the influence of using different number of reflectors in our SVD parameterization. Also, the underlying structures of the image of M k1,k2 when k 1, k 2 = 1 is a subject worth investigating. DISPLAYFORM0 Proof of Proposition 1. For n = 1, note that H 1 1 (u 1) = ±1. By setting u 1 = 0 if B 1,1 > 0 and u 1 = 0 otherwise, we have the factorization desired. Assume that the holds for n = k, then for n = k + 1 set u k+1 = B 1 − B 1 e 1. Here B 1 is the first column of B and e 1 = (1, 0, ..., 0). Thus we have DISPLAYFORM1, whereB ∈ R k×k. Note that H k+1 k+1 (u k+1) = I k+1 when u k+1 = 0 and the above still holds. By DISPLAYFORM2 is an upper triangular matrix with positive diagonal elements. Thus the holds for any n by the theory of mathematical induction. A.2 PROOF OF THEOREM 1 Proof. Observe that the image of M 1 is a subset of O(n), and we now show that the converse is also true. Given A ∈ O(n), by Proposition 1, there exists an upper triangular matrix R with positive diagonal elements, and an orthogonal matrix Q expressed as DISPLAYFORM3, such that A = QR. Since A is orthogonal, we have A A = AA = I n, thus:A A = R Q QR = R R = I n; Q AA Q = Q QRR Q Q = RR = I n Thus R is orthogonal and upper triangular matrix with positive diagonal elements. So R = I n and DISPLAYFORM4 Proof. It is easy to see that the image of M 1,1 is a subset of R n×n. For any W ∈ R n×n, we have its SVD, W = U ΣV, where Σ = diag(σ). By Theorem 1, for any orthogonal matrix U, V ∈ R n×n, there exists DISPLAYFORM0 Proof. Let A ∈ R n×n be an orthogonal matrix. By Theorem 1, there exist DISPLAYFORM1, such that A = M 1 (a 1, ..., a n). Since A is also orthogonal, for the same reason, there exist DISPLAYFORM2 v t = 0, t = k 2 + k 1 − 2,..., n, and then we have: DISPLAYFORM3 Else, assign: DISPLAYFORM4.., n, and then we have: DISPLAYFORM5 A.5 PROOF OF THEOREM 4Proof. It is easy to see that the image of M m,n *, * is a subset of R m×n. For any W ∈ R m×n, we have its SVD, W = U ΣV, where Σ is an m × n diagonal matrix. By Theorem 1, for any orthogonal DISPLAYFORM6 Similarly, for n < m, we have: DISPLAYFORM7 Remark: here when W and ∆W are not commutative, each W i ∆W should instead be written as DISPLAYFORM8 Since the change of order doesn't impact the analysis, we informally simplify the expressions here. DISPLAYFORM9 C.1 DETAILS ON THE TIME SERIES CLASSIFICATION TASK For the time series classification task, we use the training and testing sets directly from the UCR time series archive http://www.cs.ucr.edu/˜eamonn/time_series_data/, and randomly choose 20% of the training set as validation data. We provide the statistical descriptions of the datasets and experimental in TAB3 BID1. The Adding task requires the network to remember two marked numbers in a long sequence and add them. Each input data includes two sequences: top sequence whose values are sampled uniformly from and bottom sequence which is a binary sequence with only two 1's. The network is asked to output the sum of the two values. From the empirical in Figure 4, we can see that when the network is not deep (number of layers L=30 in (a)(d)), every model outperforms the baseline of 0.167 (always output 1 regardless of the input). Also, the first layer gradients do not vanish for all models. However, on longer sequences (L=100 in (b)(e)), IRNN failed and LSTM converges much slower than svdRNN and oRNN. If we further increase the sequence length (L=300 in (c)(f)), only svdRNN and oRNN are able to beat the baseline within reasonable number of iterations. We can also observe that the first layer gradient of oRNN/svdRNN does not vanish regardless of the depth, while IRNN/LSTM's gradient vanish as L becomes lager.(a) (b) (c) (d) (e) (f) Figure 4: RNN models on the adding task with L layers and n h hidden dimension. The top plots show the test MSE, while the bottom plots show the magnitude of the gradient at the first layer. Let A = {a i} 9 i=0 be the alphabet. The input data sequence x ∈ A T +20 where T is the time lag. x 1:10 are sampled uniformly from i{a i} 7 i=0 and x T +10 is set to a 9. Rest of x i is set to a 8. The network is asked to output x 1:10 after seeing a 9. That is to copy x 1:10 from the beginning to the end with time lag T.A baseline strategy is to predict a 8 for T +10 entrees and randomly sample from {a i} 7 i=1 for the last 10 digits. From the empirical in FIG3, svdRNN consistently outperforms all other models. IRNN and LSTM models are not able to beat the baseline with large time lag. In fact, the loss of RNN/LSTM is very close to the baseline (memoryless strategy) indicates that they do not memorize any useful information throughout the time lag.

Represent the TLDR for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent image style transferring methods achieved arbitrary stylization with input content and style images. To transfer the style of an arbitrary image to a content image, these methods used a feed-forward network with a lowest-scaled feature transformer or a cascade of the networks with a feature transformer of a corresponding scale. However, their approaches did not consider either multi-scaled style in their single-scale feature transformer or dependency between the transformed feature statistics across the cascade networks. This shortcoming ed in generating partially and inexactly transferred style in the generated images. To overcome this limitation of partial style transfer, we propose a total style transferring method which transfers multi-scaled feature statistics through a single feed-forward process. First, our method transforms multi-scaled feature maps of a content image into those of a target style image by considering both inter-channel correlations in each single scaled feature map and inter-scale correlations between multi-scaled feature maps. Second, each transformed feature map is inserted into the decoder layer of the corresponding scale using skip-connection. Finally, the skip-connected multi-scaled feature maps are decoded into a stylized image through our trained decoder network. Recent image style transferring methodsJohnson et al.; BID18 improved image generating speed up to sub-realtime processing by learning a feed-forward network of a single style or several fixed stylesDumoulin et al.. proposed an adaptive instance normalization layer (AdaIN) that adaptively transforms the statistics of an encoded content feature into that of a target style feature and they achieved style transferring into arbitrary input target style. However, they did not consider multi-scaled style characteristics of an imageGatys et al. but only a single scale feature in differentiating styles inside AdaIN layer. Li et al. Li et al. (2017b) proposed to use cascade networks that cumulatively transfer the multi-scaled style characteristics by using a network per scale as shown in FIG0 (a). They also transformed correlation between channels of feature map by using their whitening and coloring transformer (WCT). However, their cascade scheme requires multiple feed-forward passes to produce a stylized image and it is not guaranteed that the transferred style through a network is preserved after going through the subsequent networks because of inter-scale dependency in the multi-scaled styles of an image. Therefore, transferring multi-scaled style without interference between scales is still remained to study. In this paper, we propose an improved feed-forward network structure (FIG0) and a multi-scaled style transferring method, called total style transfer, to efficiently perform style transfer in all scales of feature maps through a single feed-forward pass. Our work has the following contributions.• Transforming both intra-scale and inter-scale statistics of multi-scaled feature map: There exist both of inter and intra-correlations in the encoded multi-scaled feature map as shown in fig.2 (b). Therefore, we match the second-order statistics, i.e., mean and covariance, of the encoded multi-scaled feature map considering the correlations not only between channels in each scale (intra-scale correlation) but also between scales (inter-scale correlation). Our feature transformer makes the transformed feature map closer to the target style feature map and this in an output style closer to the target style. Figure 2: Correlation between channels in the multi-scaled feature map of the input image (a) extracted from the pre-trained VGG16 BID16. The area corresponding to each scale of feature map is divided into red lines. In case of intra-scale feature transform, the diagonal rectangles on the correlation matrix are used. In case of inter-scale feature transform, entire region of the correlation matrix is considered.• Decoder learning with multi-scaled style loss: we use a multi-scaled style loss consistent to the feature transformer, i.e., mean and covariance loss between the concatenated feature map (FIG1). Using our multi-scaled style loss allows the decoder network to generate an output image of co-occurring multi-scale patterns which is better style expression than independently occurring scale patterns on the image that the existing methods generated.• Multi-scaled style transfer with a single feed-forward network: we use skip-connections for each decoder layer as shown in FIG0 (b) to consider the transformed feature map as well as the decoded feature map. By doing this, the style of scale corresponding to the layer and the transferred multi-scaled style so far are optimally merged into the next layer. Therefore, our method transfers multi-scaled style through a feed-forward pass in a single network instead of multiple feed-forward passes of cascade networks (FIG0) without considering inter-scale correlation. In the remained of this paper, we review previous works closely related to this work in Sec. 2, our multi-scaled style transforming method is described in Sec. 3, the effectiveness of our method is tested and proven by a bundle of experiments in Sec. 4, and this work is concluded in Sec. 5. Gatys et al. BID2 represented content and style features of an image using a deep feature map, i.e., the filtered responses of a learned convolutional neural network (CNN). To stylize an input image, they performed pixel-wise optimization of the image to reduce the feature losses of BID9 interpreted the process of generating a stylized image by matching Gram matrix BID2 as a problem of maximum mean discrepancy (MMD) specifically with a second-order polynomial kernel. Using a feed-forward neural network BID7; BID18 moved the time consuming online optimization process into an offline feed-forward network learning to speed up the image generating speed of the previous method BID2. The generated style quality was also improved by using instance normalization (IN) BID19. Dumoulin et al. extended the previous single style network to transfer multiple styles. They used conditional instance normalization (CIN) layers in a single network. As selecting learnable affine parameters corresponding the specific style in the CIN layers, the feed-forward network transfers the selected style. This method achieved generation of pre-trained multiple styles with a single network. To generalize a single network for arbitrary style transfer, Huang et al. BID5 proposed to use a feature transformer called adaptive instance normalization (AdaIN) layer between encoder and decoder networks. Once feature maps of content and target style images are encoded, AdaIN directly adjusts the mean and standard deviation of a content feature map into those of a target style feature map, and then the adjusted feature map is decoded into an output image of the target style. Li et al. BID10 further improved the arbitrary stylization by using covariance, instead of standard deviation, which considers the correlation between feature channels. To transfer multi-scaled style, Li et al. BID10 3.1 MULTI-SCALE FEATURE TRANSFORM As described in BID2, each scaled feature of CNN represents different style characteristics of an image. So, we utilize multiple feature transformers for each scale feature to transfer total style characteristics of an image. In this section, we explain two schemes of our total style transfer, i.e, intra-scale and inter-scale feature transform, with a single feed-forward network.3.1.1 INTRA-SCALE FEATURE TRANSFORM Our intra-scale feature transform is a set of independent single-scale style transforms as an extended multi-scale version of the single-scale correlation alignment of CORAL BID17 or WCT Li et al. (2017b of style image, where C i, H c,i (or H s,i), and W c,i (or W s,i) represent the number of channels, spatial height, and width of i-th scale features respectively. For a single-scale style transform with these features, CORAL or WCT performs style normalization and stylization sequentally. In the style normalization step, first zero-centered featureF c,i ∈ R Ci×(Hc,i×Wc,i) of the content feature F c,i is calculated and then the content feature F c,i is normalized intoF c,i by using its own covariance matrix cov(F c,i) ∈ R Ci×Ci as eq.1. DISPLAYFORM0 In the stylization step, the normalized content featureF c,i is stylized into F cs,i by using the square root of covariance matrix cov(F s,i) ∈ R Ci×Ci of zero-centered style featureF s,i and spatial mean µ s,i ∈ R Ci×1 of the style feature F s,i as eq.2. DISPLAYFORM1 Our intra-scale transform method applies the above single-scale transform independently to each scaled feature for i = 1..3 corresponding to {relu_1_2, relu_2_2, relu_3_3} layers. Then, those transformed features F cs,i, i = 1..3 are inserted into the subsequent decoder through skip-connenction. More detail about skip-connection will be described in Sec.3.1.3. As shown in fig.2 (b), there exists not only inter-channel correlation in a certain scale feature but also inter-scale correlation between multi-scale features. These correlations should be considered in order to transfer total style characteristics of an image. CORAL BID17 or WCT Li et al. (2017b) did not consider inter-scale correlation but only inter-channel correlation. Therefore, we propose inter-scale feature transformer which considers more style characteristics of image for style transfer. To perform feature transform considering both inter-channel and inter-scale correlations, we apply the intra-scale feature transform of Sec.3.1.1 to the concatenated feature F c ∈ R i Ci×(Hc,1×Wc,1) of content image and F s ∈ R i Ci×(Hs,1×Ws,1) of style image (eq.3) instead of independently applying to each scaled features F c,i and F s,i of Sec.3.1.1. DISPLAYFORM0 As shown in FIG1, the content features F c,i and style features F s,i for i = 1..3 are spatially upsampled into U (F c,i) and U (F s,i) of a common size (we use the largest size of F c,1 or F s,1 corresponding to {relu_1_2}) and concatenated into F c and F s respectively along the channel axis. After going through a transformer, the transformed feature F cs is split and downsampled into F cs,i ∈ R Ci× (Hc,i×Wc,i) of the original feature size as shown FIG1 (b) and eq.4, DISPLAYFORM1 where, D i (f) is a function which spatially downsamples f into H c,i × W c,i. These features are inserted into the subsequent decoder through skip-connenction of Sec.3.1.3. To utilize the transformed multi-scale features in generating output stylized image, decoding architecture of previous decoder network should be modified because each decoder layer has two input feature maps as FIG0, one is a decoded feature map from the previous decoder layer, the other is a (intra-scale or inter-scale) transformed feature from the transformer. We adopt skip-connection, which has been applied to several applications of computer vision field BID14; BID13; BID6 BID0 but not to image style transfer yet, to merge the two feature maps in decoding process as shown in FIG1. Skip-connected two scale features are optimally merged by learnable convolution layer and this improves the quality of the decoded image by considering multi-scale filter responses. Our method is different from the previous cascade scheme of BID10 because we use a single encoder/decoder network, parallel transformers for each scale feature, and merges multi-scaled styles optimally while the cascade scheme needs several encoder/decoder networks (one network per scale feature) and sequentially transfers scaled styles from large to small scale at the risk of degradation in previously transferred scale of style. also used a single decoder like ours but it sequentially applied feature transformers from large to small scale without considering possible degradation of the previously transferred scale. We need an appropriate objective function for decoder network to generate a stylized image from the transformed feature map. Among the existing losses such as Gram BID2, , and reconstruction error BID10, we adopt Mean-Std loss BID5 with some modification because of its consistency with AdaIN transformer. Instead of using Mean-Std loss as it is, we use Mean-Covariance loss to additionally consider interchannel and inter-scale correlations, which is consistent with our feature transformers described in Sec.3.1.In case of using intra-scale feature transform (Sec.3.1.1), our style loss (eq.5) is calculated as the summation of mean loss BID5 and covariance loss, i.e., square root of Frobenius distance between covariance matrices of feature maps of output and target style images. In case of using inter-scale feature transform (Sec.3.1.1), the summation of mean and covariance losses of the concatenated features are used as the style loss (eq.6). DISPLAYFORM0 DISPLAYFORM1 where subscript o represents of output stylized image. We used VGG16 feature extractor BID16 as the encoder and a mirror-structured network as the decoder of our style transfer network. Our decoder network has 2 times larger number of channels in the corresponding layer of skipconnections than the previous methods; BID5. {relu_1_2, relu_2_2, relu_3_3, relu_4_3} layers were used in calculating style loss and {relu_3_3} layer in calculating content loss. Here, we used the same content loss of BID2 and our multi-scaled style loss in Sec.3.2. For training data set, MS-COCO train2014 BID11 and Painter By Numbers BID8 were used as content image set and large style image set respectively. Each dataset consists of about 80,000 images. And we used an additional small style image set of 77 style images to verify the effect of our proposed method as the number of training style increases. Each image was resized into 256 pixels in short side maintaining the original aspect ratio in both training and test phases, and, only for training phase, randomly cropped into pixels to avoid boundary artifact. We trained networks with 4 batches of random (content, style) image pairs, 4 epochs, and learning rate of 10. And all experiments were performed on Pytorch 0.3.1 framework with NVIDIA GTX 1080 Ti GPU card, CUDA 9.0, and CuDNN 7.0. In order to verify the effect of our multi-scale feature transform for varying number of training style images, we trained two networks, one with the small style image set of 77 images and the other with the large style image set of about 80,000 images. Then we compared the output stylized images of the networks. Fig.4 shows an example of the output stylized images using our intra-scale or inter-scale feature transform method. With the network trained by a small style image set, the images generated by our intra-scale transform (fig.4 (c) ) show very similar texture style to the target style images (fig.4 (b) ). And those by our inter-scale transform (fig.4 (d) ) show even better style of texture. With the network trained by a large style image set (fig.4 (e,d) ), the images also show the same tendency that inter-scale is better in expressing the texture of target style. Because of the existing correlations between scales as shown in fig.2 (b), inter-scale feature transform which considers interscale correlations shows the better quality of style than intra-scale transform. To verity the effect of skip-connections in our style transfer network, we trained three different networks. The first one has the conventional single layer encoder/transformer/decoder architecture Huang & Belongie FORMULA0; BID10 which a sinlge feature transformer on {relu_3_3} without skip-connection. The second one has muti-scale feature transformers on {relu_3_3,relu_2_2} and one skip-connection on {relu_2_2}. The last one has multiscale feature transformers on {relu_3_3,relu_2_2,relu_1_2} and two skip-connections on {relu_2_2,relu_1_2}. Fig.5 shows an example of the output stylized image from the three different networks. As the number of skip-connections increases from fig.5 (c) to (e), the style loss decreases from 0.535 to 0.497 and, accordingly, the color tone of the stylized image is getting better matched to the target style (fig.5(b) ) and small patterns appear. To clarify the contributions of the skip-connected feature from the transformer and the decoded feature from the previous scale of decoder layer, we observed the absolute values of loss gradients with respect to the convolution weights of skip-connected decoder layers during the network learning process. As shown in fig.6(a), the gradient values for the skip-connected feature (channel indices from 129 to 256) on {relu_2_2} layer of decoder network are much larger than those for the decoded feature (channel indices from 1 to 128) at the beginning of training. This means that the skip-connected feature which already has target style through transformer dominantly affected to the decoder learning at the start of training phase. This happens because the previous decoder layer {relu_3_3} has random initial weights and outputs noisy feature at the start of training phase. As iteration goes, the gradient values for both features became similar to each other. This means that both skip-connected feature and decoded feature were equally utilized to generate an image of multi-scaled style. FIG0 shows that the gradient values for the skip-connected feature (channel indices from 65 to 128) are smaller than those for the decoded feature (channel indices from 1 to 64) at the latter decoder layer. This means that the decoded feature of the latter decoder layer already has accumulated multi-scaled styles by the previous skip-connection and this ed in the less impact of the skip-connected feature. However, using the skip-connection with the stylized feature of smaller scale has a certain effect on the image in color tone matching as shown in fig.5(d,e). We compared the image quality of our method with those of the existing methods BID2; BID5; BID10. We took the output stylized images for BID2 after 700 iterations of Adam optimization with learning rate 10 DISPLAYFORM0, for BID5 with the same setting of our method except transformer and loss, and for BID10 with style strength α = 0.6 (as mentioned in their work) and 3 cascade networks of VGG16 structure. It e ra ti o n s(× 5 0 0) C h a n n e l s (b) Learning Gradient of relu_1_2 C h a n n e l s It e ra ti o n s(× 5 0 0) Figure 6: Amplitude of loss gradients with resprct to the convolution weights in the skip-connected decoder layers during the learning process: The gradients are drawn every 500 iterations. The former half of the channels are for decoded feature from the previous scale and the latter are for skip-connected feature from transformer. Based on the gradients of 1st skip-connected layer (a) and 2nd skip-connected layer (b), the skip-connected (transformed) feature highly seems to affect to the decoder in initial interations but both decoded and transformed features samely affect as iteration goes. And the latter decoder layer (b) is less affected by the skip-connected feature than the former layer (a). FIG6 shows the generated images from the existing and our intra/inter-scale feature transform methods. Compared to the online optimization method BID2 FIG6 ), the other methods based on feed-forward network generated images of somewhat degraded style quality (FIG6). However, thanks to our muti-scaled style transfer which considering inter-channel or inter-scale correlation, texture detail and color tone of the generated image of our method with inter-scale (FIG6) or intra-scale feature transform (FIG6) are more similar to the target style than those of single-scale style transfer without considering inter-channel correlation BID5 (FIG6) are. BID2, (b) is, (c) is BID5 and (d) is BID10.Compared to BID10, the generated images of our method present the styles closer to the target styles because our methods trained a decoder network to minimize style loss between target style and output images while BID10 trained its decoder network to minimize reconstruction loss of content images. We also compared our method to that performs multi-scale feature transform with a single decoder network. For a fair comparison, as the structure of , we used VGG19 network BID16 up to {relu_4_1} layer as the encoder and its mirrored structure as the decoder, and we also used additional style loss (eq.5 or eq.6) on image-level which is corresponding to the image reconstruction loss of Avatar-Net. As shown in FIG8, Both our intra (e) and inter (f) methods generated the stylized images with both detailed shapes of content images (a) and multi-scaled strokes of target style images (b). In contrast, the generated images of Avatar-Net (c, d) show somewhat deemed content shapes and blurred or burnt color patterns without detailed strokes. And selecting appropriate patch size corresponding to the scale of style pattern was necessary in Avatar-Net (in second row of FIG8, {patch size=1} (c) did not show large square patterns but {patch size=3} (d) did.) while our method did not require any scale matching parameter due to multi-scaled skip-connections in the decoder network. For a quantitative comparison, we compared the content and style losses of the generated images of the existing methods and ours. Table. 1 shows the measured average (standard deviation) losses across 847 tests of style transfer. Online optimization method BID2 achieved the smallest style loss with a low content loss. Among the feed-forward networks trained with a small style image set, achieved the lowest style loss (red colored numbers). This is because it has a learnable transformer and this ed in a most optimized transformer. However, it is not extendable to arbitrary style transfer. Among the arbitrary style transferring methods, our method achieved the lowest style loss (blue colored numbers) with inter-scale feature transformer, and the second lowest style loss (green colored numbers) with intra-scale feature transformer. Among the feed-forward networks trained with a large style image set, our method shows the lowest style loss with inter-scale feature transformer and the second lowest style loss with intra-scale feature transformer in the same manner of the with a small style image set. For the content loss with large style image set, the best method in style loss (our-inter) shows the highest content loss and the second best method (our-intra) shows the lowest content loss. This interesting can be interpreted that inter-scale correlation has not only style of an image but also content of the image. Tasks of transferring style and preserving content are a trade-off in feedforward network methods. The of with small style image set also shows the same content/style trade-off. Therefore, we can select either inter-scale or intra-scale feature transformer according to user preference or purpose of application. Our method achieved 31% less encoder/decoder feed-forward time (4.4 ms in average of 1000 trials with images of 240 by 240 pixels) and 4% less number of parameters (3,655,296 parameters) than the existing cascade network scheme Li et al. (2017b) (6.4 ms, 3,769,856 parameters). In this paper, we proposed a total style transfer network that generates an image through a single feed-forward network by utilizing multi-scale features of content and style images. Our intra-scale feature transformer transfers multi-scale style characteristics of the target style image and our inter-scale feature transformer transfers even more style characteristics of inter-scale correlation into the content image. By using our intra/inter scale feature transform, our total style transfer network achieved the lowest style loss among the existing feed-forward network methods. In addition, we modified the feed-forward network structure by using skip-connections which make our decoder network to utilize all transformed multi-scale features. This modification allowed a single feed-forward network to generate image of multi-scaled style without using multiple feedforward networks of cascade scheme, and ed in the reduced test time by 31% and memory consumption by 4% compared to cascade network scheme.

A paper suggesting a method to transform the style of images using deep neural networks.

Represent the short summary for finding the corresponding scientific article (long)

We propose a new compression method, Inter-Layer Weight Prediction (ILWP) and quantization method which quantize the predicted residuals between the weights in convolution layers.