Question: Represent the article for finding a single-sentence TLDR

Question: Previous work has found difficulty developing generative models based on variational autoencoders (VAEs) for text. To address the problem of the decoder ignoring information from the encoder (posterior collapse), these previous models weaken the capacity of the decoder to force the model to use information from latent variables. However, this strategy is not ideal as it degrades the quality of generated text and increases hyper-parameters. In this paper, we propose a new VAE for text utilizing a multimodal prior distribution, a modified encoder, and multi-task learning. We show our model can generate well-conditioned sentences without weakening the capacity of the decoder. Also, the multimodal prior distribution improves the interpretability of acquired representations. Research into generative models for text is an important field in natural language processing (NLP) and various models have been historically proposed. Although supervised learning with recurrent neural networks is the predominant way to construct generative language models BID22 BID28 BID26, auto-regressive word-by-word sequence generation is not good at capturing interpretable representations of text or controlling text generation with global features BID1. In order to generate sentences conditioned on probabilistic latent variables, BID1 proposed Variational Autoencoders (VAEs) BID11 for sentences. However, some serious problems that prevent training of the model have been reported. The problem that has been mainly discussed in previous papers is called "posterior collapse" BID25. Because decoders for textual VAEs are trained with "teacher forcing" BID27, they can be trained to some extent without relying on latent variables. As a , the KL term of the optimization function (Equation 1) converges to zero and encoder input is ignored BID1. Successful textual VAEs have solved this problem by handicapping the decoder so the model is forced to utilize latent variables BID1 BID30. However, we believe that weakening the capacity of the decoder may lower the quality of generated texts and requires careful hyper-parameter turning to find the proper capacity. Therefore, we take a different approach. We focus on two overlooked problems. First, previous research fails to address the problem inherent to the structure of VAEs. The fundamental cause of posterior collapse (apart from teacher forcing) is the existence of a suboptimal local minimum for the KL term. Second, although existing models use a LSTM as the encoder, it is known that this simple model is not sufficient for text generation tasks (; BID14 BID26 . In this work, we propose a new architecture for textual VAEs with two modifications to solve these problems. First, we use a multimodal prior distribution and an unimodal posterior distribution to eliminate the explicit minima of ignoring the encoder (Chapter 3.2). Multimodal prior distributions for VAEs have been proposed recently for image and video tasks BID7 BID3. Specifically, our model uses a Gaussian Mixture distribution as prior distribution which is trained with the method proposed by BID23.(a) The overall architecture of existing models.(b) The overall architecture of our model. In the encoder, hidden states of the self-attention Encoder and BoW are concatenated. The decoder estimates BoW of the input text from the latent variables as a sub-task in addition to generating text. In our model, the prior distribution of the latent variables is a Gaussian mixture model. Second, we modify the encoder (Chapter 3.3). We empirically compare a number of existing encoders and adopt a combination of two. The first is the recently proposed method of embedding text into fixed-size variables using the attention mechanism BID12. Although this method was originally proposed for classification tasks, we show this encoder is also effective at text generation tasks. The second is a a Bag-of-Words encoding of input text to help the encoder. It has been reported that a simple Bag-of-Words encoding is effective at embedding the semantic content of a sentence BID18. Our experiments show that the modified encoder produces improved only when other parts of the model are modifed as well to stabilize training. Additionally, our imply that the self-attention encoder captures grammatical structure and Bag-of-Words captures semantic content. Finally, to help the model acquire meaningful latent variables without weakening the decoder, we add multi-task learning (Chapter 3.4). We find that a simple sub-task of predicting words included in the text significantly improves the quality of output text. It should be noted that this task does not cause posterior collapse as it does not require teacher forcing. With these modifications, our model outperforms baselines on BLEU score, showing that generated texts are well conditioned on information from the encoder (Chapter 4.3). Additionally, we show that each component of the multimodal prior distribution captures grammatical or contextual features and improves interpretability of the global features (Chapter 4.5). BID1 is the first work to apply VAEs to language modeling. They identify the problem of posterior collapse for textual VAEs and propose the usage of word dropout and KL annealing. BID16 models text as Bag-of-Words with VAEs. This is part of the motivation behind the usage of Bag-of-Words for textual VAEs. BID30 hypothesize that posterior collapse can be prevented by controlling the capacity of the decoder and propose a model with a dilated CNN decoder which allows changing the effective filter size. BID21 use a deconvolutional layer without teacher forcing to force the model into using information from the encoder. Our use of a multimodal prior distribution is inspired by previous works which try to modify prior distributions of VAEs. BID7 and BID3 apply a VAE with Gaussian Mixture prior distribution to video and clustering, respectively. BID23 propose the construction of a prior distribution from a mixture of posterior distributions of some trainable pseudo-inputs. Another recent proposal to restrict the latent variables is to use discrete latent variables BID20 BID25. Some discrete autoencoder models for text modeling has been proposed. While some show promise, discretization such as Gumbel-Softmax BID6 and Vector Quantization BID25 ) is required to train discrete autoencoders with gradient descent as the gradient of discrete hidden state cannot be calculated directly. A multimodal prior distribution can be regarded as a smoothed autoencoder model with discrete latent variables BID3 ) without a requirement for discretization. 3.1 VARIATIONAL AUTOENCODER FOR TEXT GENERATION 3.1.1 VARIATIONAL AUTOENCODER A RNN language model is trained to learn a probability distribution of the next word x t conditioned on all previous words x 1, x 2,..., x t−1 BID17. A language model conditioned on a deterministic latent vector z (such as input text representation) has been proposed as well BID22: DISPLAYFORM0 Although these models can be regarded as a generative model with auto-regressive sampling, they cannot capture interpretable probabilistic structures of global features. BID1 propose a new language model which explicitly captures probabilistic latent variables of global features with Variational Autoencoders BID11.Variational Autoencoders (VAEs) are one way to construct a generative model based on neural networks, which learns Variational Bayes through gradient decent. A VAE has an encoder q φ (z|x) and a decoder p θ (x|z) each parameterized by a neural network. In many cases, a standard Gaussian distribution is used for the prior distribution of the latent vector p(z) and a Gaussian distribution is used for q φ (z|x). Instead of directly maximizing the intractable marginal probability p(x) = p(z)p θ (x|z)dz, we maximize the evidence lower bound: DISPLAYFORM1 = L ELBO As the model samples from q φ (z|x), the reparameterization trick BID11 can be used to train the model with gradient descent. Previous work on textual VAEs BID1 BID30 simply applied this model to sequence-to-sequence text generation models FIG0 ). Recent works BID1 BID30 have identified several obstacles for training VAEs for text generation. One of the largest problems, referred to as "posterior collapse" BID25, is that training textual VAEs often drives the second term of Equation 1 (KL term) close to zero BID1. When the KL term becomes zero, no information from the input text is reflected on latent variables since q φ (z|x) and p(z) are identical. This is an undesirable outcome since latent variables are expected to capture a meaningful representation of input to generate conditional output. However, to aid stabilization, the previous ground truth word is given to the decoder each time during training (teacher forcing BID27). As this technique is applied to textual VAEs as well, a simple language model based on LSTM can be trained without information from the decoder and cause posterior collapse. In order to solve this problem, previous methods try to weaken the decoder to force the model to use information from the encoder. However, weakening the capacity of the decoder is not an ideal strategy since it can lower the quality of generated text and requires additional hyper-parameters specifying decoder capacity. In this paper, we propose three modifications to the model and successfully improve upon textual VAEs without restricting the capacity of the decoder. These modifications are explained in the following chapters: 3.2, 3.3, and 3.4. In typical VAEs, a standard normal distribution N is used as the prior distribution p(z) and a normal distribution N (µ, σ 2) is used as the posterior distribution q φ (z|x). Although this model is also used for previous textual VAE models BID1 BID30, there is a trivial local minimum p(z) = q φ (z|x) which makes KL(q φ (z|x)|p(z)) in Equation 1 zero, manifesting in what is referred to as posterior collapse. Roughly speaking, we can avoid this if q φ (z|x) cannot be identical to p(z). One simple way to achieve this is to use a multimodal distribution as the prior distribution p(z) and an unimodal distribution as the posterior distribution q φ (z|x). This idea is motivated by recently proposed VAE models with a multimodal prior distribution for image and video generation BID7 BID3. We provide further explanation in Appendix A and discuss that modification for the decoder is not necessary if the problems in prior distribution is fixed. The problem with using a multimodal distribution as a prior for VAEs is deciding on what kind of distribution to use. Models which learn a multimodal prior distribution along with other parts of the VAE have been recently proposed BID7 BID3 BID23. One successful model uses a multimodal prior distribution of a variational mixture of posteriors prior (VampPrior) BID23. VampPrior VAEs have multiple trainable pseudo-inputs u k and regard the mixture of the posterior distributions of the pseudo-inputs DISPLAYFORM0 as the prior distribution (K is a pre-defined number of pseudo-inputs). Pseudoinputs are trained at the same time as the other components of the VAE. Although pseudo-inputs have the same size as the input image for the VAE in the original work BID23, we use pseudo-inputs which are projected onto µ and σ directly FIG1 ).In our experiments, we find a multimodal prior distribution performs unsupervised clustering and each component of multimodal prior distribution captures specific features of a sentence. Moreover, the components themselves also form clusters, creating a hierarchical structure within the representation space (Chapter 4.5). Existing models of textual VAEs use a simple LSTM as an encoder BID1 BID30. However, recent research into text generation has found that simple LSTMs do not have enough capacity to encode information from the whole text. Motivated by the of our experiments (Chapter 3.3), we propose concatenating the representation from the self-attention encoder and Bag-of-Words information. Ideally, self-attention encodes grammatical structure and Bag-of-Words encodes overall meaning. Our experiments imply our model is successful in this kind of division of roles (Chapter 4.4). The attention mechanism (; BID14) is a popular model to encode text with LSTMs. Since VAEs are models with fixed size probabilistic latent variables, this mechanism with variable size representation cannot be applied directly. Therefore, we use a recently proposed method called self-attention BID12 (Figure 3), an effective model to embed text into a fixed Figure 3: A self-attention encoder. This model encodes variable length input into a fixed length representation using an attention mechanism. The fixed length representation is acquired by summing up the hidden states of the bi-directional LSTM based on attention weights. Attention weights a s1,..., a sn are calculated by (a s1, . . ., a sn) = softmax(w s2 tanh(W 1 H T)).size vector representation for classification tasks using an attention mechanism. Our experiments show that embedded representations from self-attention are useful for text generation. The self-attention model uses hidden states of bi-directional LSTM h 1,..., h n with variable length. To acquire a fixed sized representation m s, hidden states are summarized with attention weights m s = n i=1 a si h i. Attention weights are calculated by using a weight matrix W 1 with shape d-by-2u (u is the size of a hidden state of bi-directional LSTM and d is a hyper-parameter) and a vector w s2 with size d: DISPLAYFORM0 Here H is a n-by-2u matrix of the hidden states H = (h 1, . . ., h n). To get richer information, r different weights (r is a hyper-parameter) are calculated with a r-by-d weight matrix W 2 = (w 12, . . ., w r2) in the model: DISPLAYFORM1 Here the softmax is performed along the second dimension. Finally, a fixed sized representation is acquired by M = AH. We simply flatten the matrix M into a representation vector. All parameters are trained with gradient descent. Previous research shows the effectiveness of Bag-of-Words in NLP tasks such as text classification BID5. Because the difficulty of encoding the content of the input sentence with LSTM is known, we propose using a simple Bag-of-Words input to encode the content of the sentence for text generation tasks. Also, since VAEs are trained in a stochastic manner, it is difficult to train the encoder. Since Bag-of-Words input is much easier to train compared to LSTMs and self-attention encoders, it will help stabilize training. We simply summarize word representation of all words in the input text and project this vector with a linear layer. In NLP deep learning tasks, some methods to improve the performance of the main task with multitask learning has been reported. For example, multi-lingual training even improves the of each language in translation task BID4 and sub-task of phone recognition improves the of speech recognition BID24. One of the effects of multi-task learning is said that it enables to acquire better intermediate representations BID13. Also, a recently proposed model to encode chemical structure with VAEs show that multi-task learning improves the quality of embedded representation BID19.To address the largest problem of VAEs for text, the difficulty in learning meaningful latent variables, we propose using multi-task learning in our model. However, using additional information such as grammatical properties or labels is not desirable for language modeling with textual VAEs. We find that the simple task of predicting words in output text can help the model improve the quality of output text. Additionally, this sub-task will alleviate the problem of posterior collapse since it does not contain auto-regressive structure which in turn requires training with teacher forcing. We compare our model with two models proposed by Bowman et al. FORMULA5 and BID30. Basically, we use the same configurations for these models. For the model of BID30, we use a SCMM-VAE model in the original paper and pretrain the encoder. For the multimodal prior distribution model, we report the score of a prior distribution with 500 components and analyze the acquired representation space with one with 100 components for ease of analysis. We use 100,000 sentences from a scale document dataset "Yahoo! Answers Comprehensive Questions and Answers version 1.0" for training to acquire the . For details of the dataset and model parameters, see Appendix B. We compare a self-attention encoder BID12, a LSTM encoder, and a Bag-of-Words encoder with tasks to embed a text into 128 sized vector and show the in Table 1. First, we compare the models on a sequence-to-sequence autoencoder model. We show that the self-attention encoder works best in terms of BLEU score. However, we find that the self-attention encoder has a higher false negative rate compared to even a simple LSTM at the task of predicting the words in an input text. From this , we hypothesize that the self-attention encoder is good at acquiring the structure of a sentence or focusing on specific information but is not good at embedding all the information in a sentence. From these , we decided to use self-attention and Bag-of-Words for our encoder. The for language modeling are shown in TAB2. We report the reconstruction loss (negative log likelihood) of text, KL divergence and BLEU of textual VAEs. The show that multi-task learning and a multimodal prior distribution in isolation both improve the model. On the other hand, changing the encoder in isolation has no influence on . Note that this is not the case for non-VAE models. However, when multi-task learning is also used, incorporating Bag-of-Words input (the first modification of the encoder) improves the score. Moreover, when we use a multimodal prior distribution, the self-attention encoder, the second modification of the encoder, outperforms the LSTM encoder. This implies that it is difficult to train the encoder (especially the self-attention encoder) of VAEs unlesss the overall model is improved as well. Therefore, when other parts of the model are improved in tandem and training becomes more stable, the improved ability of the encoder is utilized. Finally, our model with all modifications (the last line) outperforms baselines by a significant margin. Our model uses self-attention and Bag-of-Words as the encoder. We show the which imply that self-attention acquires grammatical structure and Bag-of-Words provides semantic content. is it possible to death penalty in the world to death penalty? Table 3: Sampling from the posterior distribution of our model when different input is given to the self-attention and Bag-of-Words encoders. "SA" is a sentence given to self-attention encoder and "BoW" is a sentence given to the Bag-of-Words encoder. For details, see Chapter 4.4. For more samples, see Table 7 in Appendix D.First, to see the relationship between these two encoders, we analyze generated sentences when different sentences are provided to self-attention and Bag-of-Words encoder. We show examples of the in Table 3. Generated sentences in Table 3 have similar grammatical structure to the input of the self-attention encoder and nouns in the sentences are strongly affected by the Bag-of-Words encoder. Moreover, by looking into the attention weights of the self-attention encoder, we can see which parts of a sentence the encoder focuses on as shown by BID12. We show the maximum attention weight for each word in FIG2. We can see that the self-attention encoder assigns a larger weight to words which determine the structure of a sentence such as interrogatives and prepositions rather than nouns. In addition, attention weights are similar between sentences which share grammatical structure even when nouns or word lengths differ. We show our model properly acquires a representation of sentences and a multimodal prior distribution helps us interpret acquired representation with unsupervised clustering. By sampling from each component, we can see that our model successfully performs clustering. We find that sentences allocated to to components respectively have one of at least two things in common: grammatical structure or topic. For sentences sampled from components, please see Table 8 in Appendix D. Table 4.We show a new method to interpret the global structure of the acquired representation space. We analyze the representation space by visualizing the means of 100 components in the multimodal prior distribution of our model with t-SNE BID15 and show the in FIG3. In addition to the fact that each component clusters together, we now see the clusters themselves form into larger clusters, creating a hierarchical relationship. We take a further look into two clear clusters indicated in FIG3. First, we sample from component 38, 56, and 94 in cluster 1 and show the in Table 4. From the sampled sentences, we can see that components in cluster 1 share grammatical structure "[interrogative] can I [verb]" and each component has its own topics (computer, politics, culture). On the other hand, components in cluster 2 share the topics (politics or human relationship) and each component has its own grammatical structure. Also, from FIG3, components 52, 31, and 37 seem to be on the circle in this order and we can see the continuous changes of grammatical structure in this order. Thus, we can observe that our model acquires a hierarchical structure of sentences and the structure can be easily interpreted through analysis of components in the multimodal prior distribution. As models with multimodal distributions are relatively new, we hope methods to control multimodal prior distribution are investigated further in future works. However, we emphasize that our is already impressive since without a multiomodal prior, extensive search with sampling or additional labels is required to interpret the structure of acquired text representation. A multimodal prior distribution makes it much easier to understand the structure of the representation space though analysis of components of the distribution. how do u get a group of african american to join? how do i increase a mortgage loan in indiana? how do you get married in canada? Table 4: Samples from components of the prior distribution from cluster 1 (above) and 2 (below) in FIG3. Components in cluster 1 share grammatical structure and components in cluster 2 share topics. Please see Chapter 4.5 for more details. and increases hyper-parameters. We show (i) multimodal prior distribution, (ii) improvement of the encoder and (iii) multi-task learning can improve the model with a simple LSTM decoder. We show theoretical justification for a multimodal prior distribution as a solution for posterior collapse. We use the equivalent objective for ELBO (Equation 1) by BID31: DISPLAYFORM0 where DISPLAYFORM1, which does not depend on any parameters. This objective can be minimized to zero without utilizing latent variables under the assumption that (i) the decoder is sufficiently flexible and (ii) the posterior distribution can be trained so p(z) = q φ (z|x) BID31. This nature of ELBO causes posterior collapse in VAEs. There are two simple ways to break these assumptions. First, if the capacity of the decoder is restricted, assumption (i) cannot be satisfied. This is the theoretical underpinning for previous approaches used in textual VAEs BID1 BID30 which restrict the capacity of the decoder. However, as previously discussed, weakening the decoder is undesirable. Additionally, hyper-parameter search is required to strike a balance between the two terms if the KL term is not modified as well. Therefore, we propose to break the assumption (ii) with a multimodal prior distribution. When the prior distribution p(z) is a multimodal distribution and the posterior distribution q φ (z|x) is an unimodal distribution, there is no way to satisfy p(z) = q φ (z|x). Moreover, there will be multiple minima for KL(q φ (z|x)|p(z)). When Kullback-Leibler divergence KL(q|p) between the Gaussian mixture distribution p and the normal distribution q is minimized (here we assume that q is trainable), q will be allocated to one component of p since KL(q|p) = q(z) log q(z) p(z) dz becomes larger when p is assigned a low probability in an area where q is assigned a high probability FIG4 ). In such a formulation, there is no clear global minima for the KL term and the posterior distribution is not forced to ignore information from the encoder. We propose a hypothesis that the modification of the decoder is not necessary if multimodal prior distribution is used. In practice, it is natural to assume that training the decoder so p D (x) = p θ (x|z) for all z is much harder than to make KL(q φ (z|x)|p(z)) = 0. Under the assumption, the model will be trained so KL(q φ (z|x)|p(z)) = 0 as the first step and this condition force the decoder to be trained so p D (x) = p θ (x|z) for all z when there is no modification of the model. Although this is the opposite way from the explanation by BID31, this process is more natural in practice since it is easy to train prior distribution. Therefore, if we modify the model to avoid KL(q φ (z|x)|p(z)) = 0, the decoder will not try to satisfy p D (x) = p θ (x|z) for all z but learn the conditioned distribution for each z. This analysis motivates us to modify textual VAE without weakening the capacity of the decoder. The of our experiments are consistent with this hypothesis. B. As test and validation dataset, we use 10,000 sentences each. We set the maximum length of a sentence to 60 words (ignore the rest of the sentence, the average length of the original sentences is 38.12 words) and use the most common 40,000 words for this experiment. Our model uses self-attention and Bag-of-Words in the encoder and a LSTM for the decoder. The size of the hidden state of LSTM is 256 for both for LSTM and self-attention. The size of the word embedding is 256 and the size of the latent variables is 128. For the self-attention encoder, we use d = 350 and r = 30. In accordance with BID1 BID30, we feed the latent variables on every step of the decoder LSTM by concatenating it with the word embedding. We applied 0.4 word dropout for input text to the decoder for our model and the model from BID1. In this paper, we modify the model without restricting the capacity of the decoder. However, the method used by BID1 called word dropout, which was originally proposed to weaken the decoder, is now seen as a method of smoothing BID29. As this method is also effective and harmless for non-VAE text generation task, we use word dropout for our model. In addition, we pretrain the encoder and the decoder with sequence-to-sequence text generation for our multi-prior distribution model. Note that it was impossible to pretain decoders for previous models since it can in posterior collapse. For multi-prior distribution, we compare 4 numbers of components and found that performance is not sensitive to this hyperparameter, although a larger number of components in a slightly better score TAB5. As using a prior distribution with many components leads to overfitting, over-regularization, and high computational complexity BID23, we report the score of a prior distribution with 500 components and analyze the acquired representation space with 100 components for ease of analysis. We compare our model with two models proposed by BID1 and BID30. Basically, we use the same configurations for these models. For the model of BID30, we use the SCMM-VAE model in the original paper and pretrain the encoder. We use Adam BID10 for the optimizer. According to our experiments, setting the learning rate to 5 × 10 −4 and β 1 to 0.5 performs the best. For KL weight annealing, we set the initial weight for the KL term to be 0 and increase it linearly to 1 until epoch 30. After KL weight annealing, we train for 80 epochs with learning rate decay (0.95 for every epoch). Table 6: Semi-supervised learning. LM-LSTM and SA-LSTM come from BID2, they denotes the LSTM initialized with an autoencoder and a language model. The methods of semisupervised learning with VAEs use the same scheme as BID30. LSTM is a simple supervised model. The structure of semi-supervised models using VAEs is taken from BID30. We use the topic of a sentence from the dataset as a label and feed the encoded representation from the encoder to the discriminator. We report the of semi-supervised learning in Table 6. Our models do not differ from semi-supervised learning baselines. This can be understood because this semi-supervised learning assumes that label information is helpful or necessary to generate proper sentences. Our experiments show that our model both is conditioned by the encoder and also generates proper sentences without labels. This is consistent with the reasoning from BID30 that the best models for language modeling and semi-supervised learning are different. We show additional samples for Table 3 in Table 7. Please see what happens to the death penalty for a day? SA is it true that australia likes war to update their new improve weapons? BoW definition of traditional education the death penalty and violence in a community is it true that a good alternative to get a degree of education?is it possible to death penalty in the world to death penalty? is it true that the age of people to change lots of traditional?is there a place to get a peaceful death penalty in the world? is it a good idea of having a 100 % of education?are there any place in the city to make a death penalty? Table 7: Sampling from posterior distribution of our model when different texts are input to selfattention and Bag-of-Words of the encoder. "SA" is a sentence given to self-attention encoder and "BoW" is a sentence to Bag-of-Words encoder. For detail, see Chapter 4.4. 1 is it true that the only way to provide the holy rabbit through the same answer? is it possible to go to a police officer? is it possible to have to pay for her home do n't you think the president is the worst president of the us is the liberal, the jewish religion will be cut the food to do? who is the president of the united states? Table 8: Samples from components of prior distribution. Component 1, 22, and 76 generate sentences with common structure. On the other hand, component 60, 68, and 83 generate structurally diverse sentences on the same topics (computer, sports). <UNK>is a word not in the dictionary. For detail, see Chapter 4.5. We report text from 6 components of a multimodal prior distribution from our model in Table 8. We found two types of features allocated for components. The first one is grammatical structure. Components 1, 22, and 77 in Table 8 each generate similarly structured sentences: sentences from component 1 begin with "it is true that" or "it is possible to", sentences from component 22 begin with "does anyone (anybody)", and sentences from component 77 begin with "what is the best way to". This is straightforward to interpret as properly acquiring grammatical structure will lower reconstruction loss. More interestingly, sentences generated the next type of components, namely components 60, 68, and 83 are each on the same topic. Sentences generated from component 60 are about sports, those from component 68 are about computer (music), and those from component 83 are about politics. However, these sentences do not share grammatical structure and generate sentences with diverse structures.

We propose a model of variational autoencoders for text modeling without weakening the decoder, which improves the quality of text generation and interpretability of acquired representations.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy. Over the past few years, deep neural networks have achieved state-of-the-art performance on several challenging tasks in the domains of computer vision, speech recognition, and natural language processing. Driven by increasing amounts of data and computational power, deep learning models have become bigger and deeper to better learn from data. While these models are typically deployed in a datacenter back-end, preserving user privacy and reducing user-perceived query times mandate the migration of the intelligence offered by these deep neural networks towards edge computing devices. Deploying large, accurate deep learning models to resource-constrained computing environments such as mobile phones, smart cameras etc. for on-device inference poses a few key challenges. Firstly, state-of-the-art deep learning models routinely have millions of parameters requiring~MBs of storage, whereas on-device memory is limited. Furthermore, it is not uncommon for even a single model inference to invoke~billions of memory accesses and arithmetic operations, all of which consume power and dissipate heat which may drain the limited battery capacity and/or test the device's thermal limits. Confronting these challenges, a growing body of work has emerged that intends to discover methods for compressing neural network models while limiting any potential loss in model quality. Latencysensitive workloads relying on energy-efficient on-device neural network inference are often memory bandwidth-bound, and model compression offers the two-fold benefit of reducing the total number of energy-intensive memory accesses as well as improving the inference time due to an effectively higher memory bandwidth for fetching compressed model parameters. Within the realm of model compression techniques, pruning away (forcing to zero) the less salient connections (parameters) in the neural network has been shown to reduce the number of nonzero parameters in the model with little to no loss in the final model quality. Model pruning enables trading off a small degradation in model quality for a reduction in model size, potentially reaping improvements in inference time and energy-efficiency. The ing pruned model typically has sparse connection matrices, so efficient inference using these sparse models requires purpose-built hardware capable of loading sparse matrices and/or performing sparse matrix-vector operations BID30 BID23. Also, representing sparse matrices carries with it an additional storage overhead increasing the model's net memory footprint which must also be taken into consideration. In this work, we perform a closer examination of the effectiveness of model pruning as a means for model compression. From the perspective of on-device neural network inference, given a bound on the model's memory footprint, how can we arrive at the most accurate model? We aim to answer this question by comparing the quality of the models obtained through two distinct methods: training a large model, but pruned to obtain a sparse model with a small number of nonzero parameters (large-sparse); and training a small-dense model with size comparable to the large-sparse model. Both of these methods expose a model accuracy and size tradeoff, but differ remarkably in terms of their implications on the design of the underlying hardware architecture. For this comparative study, we pick models across a diverse set of application domains: InceptionV3 BID26 and MobileNets BID13 for image recognitions tasks, stacked LSTMs for language modeling, and seq2seq models used in Google's Neural Machine Translation BID28 system. In the process of this investigation, we also develop a simple gradual pruning approach that requires minimal tuning and can be seamlessly incorporated within the training process and demonstrate its applicability and performance on an assortment of neural network architectures. Early works in the 1990s BID18 BID12 performed pruning using a second-order Taylor approximation of the increase in the loss function of the network when a weight is set to zero. In Optimal Brain Damage BID18, the saliency for each weight was computed using a diagonal Hessian approximation, and the low-saliency weights were pruned from the network and the network was retrained. In Optimal Brain Surgeon BID12, the saliency for each weight was computed using the inverse Hessian matrix, and the low-saliency weights were pruned and all other weights in the network were updated using the Hessian matrix. More recently, magnitude-based weight pruning methods have become popular techniques for network pruning BID10 a; BID25 BID22. Magnitude-based weight pruning techniques are computationally efficient, scaling to large networks and datasets. Our automated gradual pruning algorithm prunes the smallest magnitude weights to achieve a preset level of network sparsity. In contrast with the works listed above, our paper focuses on comparing the model accuracy and size tradeoff of large-sparse versus small-dense models. A work similar to ours is the work by BID22 on pruning a RNN and GRU model for speech recognition and showing that a sparse RNN that was pruned outperformed a dense RNN trained normally of comparable size. While they provide one data point comparing the performance of a sparse vs dense model, our work does an extensive comparison of sparse vs dense models across a wide range of models in different domains (vision and NLP). Narang et al. also introduce a gradual pruning scheme based on pruning all the weights in a layer less than some threshold (manually chosen) which is linear with some slope in phase 1 and linear with some slope in phase 2 followed by normal training. Compared to their approach, we do not have two phases and do not have to choose two slopes, and we do not need to choose weight thresholds for each layer (we rely on a sparsity schedule which determines the weight thresholds). Thus, our technique is simpler, doesn't require much hyperparameter tuning, and is shown to perform well across different models. Within the context of reducing model size by removing redundant connections, several recent works BID2 BID16 BID3 propose techniques to prune and induce sparsity in a structured way, motivated primarily by the desire to speedup computations on existing hardware architectures optimized for dense linear algebra. Such techniques perform coarse-grain pruning and depend critically on the structure of the convolutional layers, and may not be directly extensible to other neural network architectures that lack such structural properties (LSTMs for instance). On the contrary, our method does not make any assumptions about the structure of the network or its constituent layers and is therefore more generally applicable. While pruning focuses on reducing the number of non-zero parameters, in principle, model pruning can be used in conjunction with other techniques to further reduce model size. Quantization tech- niques aim to reduce the number of bits required to represent each parameter from 32-bit floats to 8 bits or fewer. Different quantization techniques such as fixed-point quantization BID27 or vector quantization BID8 achieve different compression ratios and accuracies but also require different software or hardware to support inference at runtime. Pruning can be combined with quantization to achieve maximal compression BID9. In addition, an emerging area of research is low precision networks where the parameters and/or activations are quantized to 4 bits or fewer BID20 BID14 BID24 BID31. Besides quantization, other potentially complementary approaches to reducing model size include low-rank matrix factorization BID5; BID15 BID17 and group sparsity regularization to arrive at an optimal layer size BID1. We extend the TensorFlow BID0 framework to prune the network's connections during training. For every layer chosen to be pruned, we add a binary mask variable which is of the same size and shape as the layer's weight tensor and determines which of the weights participate in the forward execution of the graph. We inject ops into the TensorFlow training graph to sort the weights in that layer by their absolute values and mask to zero the smallest magnitude weights until some desired sparsity level s is reached. The back-propagated gradients flow through the binary masks, and the weights that were masked in the forward execution do not get updated in the backpropagation step. We introduce a new automated gradual pruning algorithm in which the sparsity is increased from an initial sparsity value s i (usually 0) to a final sparsity value s f over a span of n pruning steps, starting at training step t 0 and with pruning frequency ∆t: DISPLAYFORM0 The binary weight masks are updated every ∆t steps as the network is trained to gradually increase the sparsity of the network while allowing the network training steps to recover from any pruninginduced loss in accuracy. In our experience, varying the pruning frequency ∆t between 100 and 1000 training steps had a negligible impact on the final model quality. Once the model achieves the target sparsity s f, the weight masks are no longer updated. The intuition behind this sparsity function in equation is to prune the network rapidly in the initial phase when the redundant connections are abundant and gradually reduce the number of weights being pruned each time as there are fewer and fewer weights remaining in the network, as illustrated in FIG0. In the experimental presented in this paper, pruning is initiated after the model has been trained for a few epochs or from a pre-trained model. This determines the value for the hyperparameter t 0. A suitable choice for n is largely dependent on the learning rate schedule. Stochastic gradient descent (and its many variants) typically decay the learning rate during training, and we have observed that pruning in the presence of an exceedingly small learning rate makes it difficult for the subsequent training steps to recover Figure 2a shows the learning rate and the pruning schedule used for training sparse-InceptionV3 BID26 models. All the convolutional layers in this model are pruned using the same sparsity function, and pruning occurs in the regime where the learning rate is still reasonably high to allow the network to heal from the pruning-induced damage. Figure 2b offers more insight into how this pruning scheme interacts with the training procedure. For the 87.5% sparse model, with the gradual increase in sparsity, there comes a point when the model suffers a near-catastrophic degradation, but recovers nearly just as quickly with continued training. This behavior is more pronounced in the models trained to have higher sparsity. TAB0 compares the performance of sparse-InceptionV3 models pruned to varying extents. As expected, there is a gradual degradation in the model quality as the sparsity increases. However, a 50% sparse model performs just as well as the baseline (0% sparsity), and there is only a 2% decrease in top-5 classification accuracy for the 87.5% sparse model which offers an 8x reduction in number of nonzero (NNZ) model parameters. Also note that since the weights are initialized randomly, the sparsity in the weight tensors does not exhibit any specific structure. Furthermore, the pruning method described here does not depend on any specific property of the network or the constituent layers, and can be extended directly to a wide-range of neural network architectures.4 COMPARING large-sparse AND small-dense MODELS MobileNets are a class of efficient convolutional neural networks designed specifically for mobile vision applications BID13. Instead of using standard convolutions, MobileNets are based on a form of factorized convolutions called depthwise separable convolution. Depthwise separable convolutions consist of a depthwise convolution followed by a 1x1 convolution called a pointwise convolution. This factorization significantly reduces the number of parameters in the model by filtering and combining input channels in two separate steps instead of together as in the standard convolution. The MobileNet architecture consists of one standard convolution layer acting on the input image, a stack of depthwise separable convolutions, and finally averaging pooling and fully connected layers. For the dense baseline model with width multiplier 1.0, there are a total of 4.21M parameters, 99% of which are in the 1x1 pointwise convolution layers (74.6%) and fully connected layers (24.3%). We do not prune the parameters in the one standard convolution layer and in the depthwise convolution layers since there are very few parameters in those layers (1.1%).The width multiplier is a parameter of the MobileNet network that allows trading off the accuracy of the model with the number of parameters and computational cost. The width multiplier of the baseline model is 1.0. For a given width multiplier α ∈, the number of input channels and the number of output channels in each layer is scaled by α relative to the baseline 1.0 model. We compare the performance of dense MobileNets trained with width multipliers 0.75, 0.5, and 0.25 with the performance of sparse MobileNets pruned from dense 1.0 MobileNet in Figure 3 and TAB1 on the ImageNet dataset. We see that for a given number of non-zero parameters, sparse MobileNets are able to outperform dense MobileNets. For example, the 75% sparse model (which has 1.09 million parameters and a top-1 accuracy of 67.7%) outperforms the dense 0.5 MobileNet (which has 1.32 million parameters and a top-1 accuracy of 63.7%) by 4% in top-1 accuracy while being smaller. Similarly, the 90% sparse model (which has 0.46 million parameters and a top-1 accuracy of 61.8%) outperforms the dense 0.25 MobileNet (which has 0.46 million parameters and a top-1 accuracy of 50.6%) by 10.2% in top-1 accuracy while having the same number of non-zero parameters. Overall, pruning is a promising approach for model compression even for an architecture that was designed to be compact and efficient by using depthwise separable convolutions instead of standard convolutions as a factorization-like technique to reduce the number of parameters. The sparsity parameter is shown to be an effective way to trade off the accuracy of a model with its memory usage and compares favorably with the width multiplier in MobileNet. Training a sparse MobileNet using our gradual pruning algorithm is also easy. For pruning a dense MobileNet, we used the same learning rate schedule as for training a dense MobileNet but with an initial learning rate 10 times smaller than for training a dense MobileNet, and all other hyperparameters were kept the same. We train an LSTM language model on the Penn Tree Bank dataset using the models and training procedure described in BID29. At each time step, the LSTM language model outputs the probability of the next word in the sentence given the history of previous words. The loss func- tion is the average negative log probability of the target words, and the perplexity is the exponential of the loss function. The language model is composed of an embedding layer, 2 LSTM layers, and a softmax layer. The vocabulary size is 10,000, and the LSTM hidden layer size is 200 for the small model, 650 for the medium model, and 1,500 for the large model. In the case of the large model, there are 15M parameters in the embedding layer, 18M parameters in each of the two LSTM layers, and 15M parameters in the softmax layer for a total of 66M parameters. Different hyperparameters are used to train the different-sized models. When pruning a model of a certain size, we use the same hyperparameters that were used for training the dense model of that size. We compare the performance of the dense models with sparse models pruned from medium and large to 80%, 85%, 90%, 95%, and 97.5% sparsity in Figure 4 and TAB2. In this case, we see that sparse models are able to outperform dense models which have significantly more parameters (note the log scale for the number of parameters). The 90% sparse large model (which has 6.6 million parameters and a perplexity of 80.24) is able to outperform the dense medium model (which has 19.8 million parameters and a perplexity of 83.37), a model which has 3 times more parameters. Compared with MobileNet, pruning PTB model likely gives better because the PTB model is larger with significantly more parameters. Our show that pruning works very well not only on the dense LSTM weights and dense softmax layer but also the dense embedding matrix. This suggests that during the optimization procedure the neural network can find a good sparse embedding for the words in the vocabulary that works well together with the sparse connectivity structure of the LSTM weights and softmax layer. From Figure 4 and TAB2, we also see that the 85% sparse medium model (which has 3 million parameters and a perplexity of 85.17) outperforms the 95% sparse large model (which has 3.3 million parameters and a perplexity of 87.83). The accuracy of the 95% sparse large model is comparable to the accuracy of the 90% sparse medium model (which has 2 million parameters and a perplexity of 87.86). Together, these suggest that there is an optimal compression range when pruning. In the case of PTB, pruning to 95% sparsity for a compression ratio of 20x significantly degrades the performance of the sparse model compared to pruning to 90% sparsity for a compression ratio of 10x, as seen in Figure 4 from the curve of perplexity vs. number of parameters traced by either of the sparse models. These suggest that in order to get the best-performing sparse model of a certain size, we should train a dense model that is 5x-10x larger and then prune to the desired number of parameters rather than taking the largest and best-performing dense model and pruning this model by 20x or more to the desired number of parameters, assuming that the difference in performance of the two dense baseline models is not that large. We note that it may be possible to obtain slightly better for pruning to 95% sparsity or higher with more hyperparameter tuning, and the we obtained for pruning a model of a certain size were from using exactly the same hyperparameter configuration as for training the dense model of that size. The Google Neural Machine Translation (NMT) architecture is a seq2seq model with attention BID28. We use the open-source TensorFlow implementation available at BID21. The model is based on an encoder-decoder architecture. The encoder has an embedding layer which maps the source vocabulary of 36,548 words into a k-dimensional space, 1 bidirectional LSTM layer, and 3 standard LSTM layers. The decoder has an embedding layer which maps the target vocabulary of 36,548 words into a k-dimensional space, 4 LSTM layers with attention, and finally a softmax layer. For the dense baseline model with number of units k = 1024, there are 37.4M parameters in each of the encoder embedding, decoder embedding, and softmax layers and 98.6M parameters in all of the LSTM layers for a total of 211M parameters. We apply pruning to all of the LSTM layers, embedding layers, and softmax layers, but we do not prune the attention parameters of which there are relatively few. The other dense models were obtained by varying the number of units k. We use the WMT16 German and English dataset with news-test2013 as the dev set and news-test2015 as the test set. The BLEU score is reported as a measure of the translation quality. The learning rate schedule used for training the dense models is 170K iterations with initial learning rate 1.0 and 170K iterations with learning rate decay of 0.5 every 17K iterations. For pruning a dense model, the learning rate schedule we use is 70K iterations with initial learning rate 0.5 and 170K iterations with learning rate decay of 0.5 every 17K iterations, and all other hyperparameters were kept the same. Since we noticed that the NMT training procedure had high variance, we tested several pruning schemes applied to NMT. Our standard implementation of gradual pruning increases the sparsity of every layer to the same sparsity level at each pruning step. We tested a variant which we call "layerwise constant" sparsity: instead of simultaneously increasing the sparsity of all layers to some sparsity level at each pruning step, we subdivide the pruning interval and increase the sparsity of one layer at a time to that sparsity level. This potentially has the effect of reducing the impact of pruning and allowing the network to recover better with training. Finally, we compared with "global" pruning: we prune the smallest magnitude weights across the entire network, regardless of which layer they are in. Global pruning produces a different sparsity level for each layer and was shown to perform well on NMT in the work of BID25. Overall, the layerwise constant pruning scheme performed best on average, so we report the with the layerwise constant pruning scheme in FIG2 and TAB3. We note that there is high variance in the due to the stochasticity of the training process, as illustrated by the error bar in FIG2 which is the standard deviation of the BLEU score of 10 randomly initialized and independently trained NMT models. The in TAB3 show that for 80% sparsity (5x compression), the pruned model actually achieves a slightly higher BLEU score than the baseline model (though we note the error bar). For 85% sparsity, the BLEU score drops by around 0.25, and for 90% sparsity, the BLEU score drops by around 0.6. When we compare the performance of dense and sparse models in FIG2 and TAB3, we again see that sparse models outperform even larger-sized dense models. The BLEU score of the dense model falls off quickly after 2x reduction in model size while the BLEU score of the sparse model starts to fall off only after 5x reduction in NNZ parameters. The net memory footprint of a sparse model includes the storage for the nonzero parameters and any auxiliary data structures needed for indexing these elements. Pruning models helps reduce the number of nonzero-valued connections in the network; however the overhead in sparse matrix storage inevitably diminishes the achievable compression ratio. The bit-mask sparse matrix representation requires 1 bit per matrix element indicating whether the element is nonzero, and a vector containing all the nonzero matrix elements. This representation incurs a constant overhead regardless of the model sparsity. In the compressed sparse row (column) storage (CSR(C)) adopted in BID23, each nonzero parameter in the sparse matrix is associated with a count (usually stored as a 4 or 5 bit integer) of the number of zeros preceding it. The overhead in this case is proportional to the NNZ in the model. TAB5 compares these two representations for sparse-MobileNets. The CSR(C) representation can enable higher compression ratio for networks with high sparsity. Note, however, that the bit-mask representation offers marginally lower overhead at smaller sparsity levels. In spite of this overhead, large-sparse models appear to achieve higher accuracy than small-dense models with comparable memory footprint. For instance, MobileNet with width multiplier 1 and sparsity 50% has similar footprint as MobileNet with width multiplier 0.75, but obtains higher accuracy. TAB6 further highlights the trade-off between model size and accuracy for dense and sparse models. The performance gap between large-sparse and small-dense models widens for larger models such as as the PTB language models and NMT (see TAB2 and TAB3). It is worth noting that the presented in this work were obtained by training neural networks using 32-bit floating point representation. For neural networks trained to perform inference using reduced precision (8-bit integer, for instance) arithmetic, the memory overhead of sparse matrix storage represents a bigger fraction of the total memory footprint. Quantization of the parameters to a reduced precision number representation is also an effective method for model compression, and the interplay between model quantization and pruning and their collective impact on model accuracy merits a closer examination. We defer that investigation to a future extension to this work. This work sheds light on the model size and accuracy trade-off encountered in pruned deep neural networks. We demonstrate that large-sparse models outperform comparably-sized small-dense models across a diverse set of neural network architectures. We also present a gradual pruning technique that can be applied with ease across these different architectures. We believe these will encourage the adoption of model pruning as a tool for compressing neural networks for deployment in resource-constrained environments. At the same time, we hold the opinion that our will provide further impetus to the hardware architecture community to customize the next generation of deep learning accelerator architectures to efficiently handle sparse matrix storage and computations.

Represent the short summary for finding the corresponding scientific article (long)

We demonstrate that large, but pruned models (large-sparse) outperform their smaller, but dense (small-dense) counterparts with identical memory footprint.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.