Question: Represent the scientific article to find a short TLDR

Question: Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice. Advances in techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks BID1 BID12 BID8 BID18, with the most computationally-hungry models obtaining the highest scores BID13 BID7 BID14 BID16. As a , training a state-of-the-art model now requires substantial computational resources which demand considerable energy, along with the associated financial and environmental costs. Research and development of new models multiplies these costs by thousands of times by requiring retraining to experiment with model architectures and hyperparameters. Whereas a decade ago most NLP models could be trained and developed on a commodity laptop or server, many now require multiple instances of specialized hardware such as GPUs or TPUs, therefore limiting access to these highly accurate models on the basis of finances. Even when these expensive computational resources are available, model training also incurs a substantial cost to the environment due to the energy required to power this hardware for weeks or months at a time. Though some of this energy may come from renewable or carbon credit-offset resources, the high energy demands of these models are still a concern since energy is not currently derived from carbon-neural sources in many locations, and when renewable energy is available, it is still limited to the equipment we have to produce and store it, and energy spent training a neural network might better be allocated to heating a family's home. It is estimated that we must cut carbon emissions by half over the next decade to deter escalating rates of natural disaster, and based on the estimated CO 2 emissions listed in TAB1, model training and development likely make up a substantial portion of the greenhouse gas emissions attributed to many NLP researchers. To heighten the awareness of the NLP community to this issue and promote mindful practice and policy, we characterize the dollar cost and carbon emissions that from training the neural networks at the core of many state-of-the-art NLP models. We do this by estimating the kilowatts of energy required to train a variety of popular off-the-shelf NLP models, which can be converted to approximate carbon emissions and electricity costs. To estimate the even greater resources required to transfer an existing model to a new task or develop new models, we perform a case study of the full computational resources required for the development and tuning of a recent state-of-the-art NLP pipeline BID17. We conclude with recommendations to the community based on our findings, namely: Time to retrain and sensitivity to hyperparameters should be reported for NLP machine learning models; academic researchers need equitable access to computational resources; and researchers should prioritize developing efficient models and hardware. To quantify the computational and environmental cost of training deep neural network models for NLP, we perform an analysis of the energy required to train a variety of popular offthe-shelf NLP models, as well as a case study of the complete sum of resources required to develop LISA BID17, a state-of-the-art NLP model from EMNLP 2018, including all tuning and experimentation. We measure energy use as follows. We train the models described in §2.1 using the default settings provided, and sample GPU and CPU power consumption during training. Each model was trained for a maximum of 1 day. We train all models on a single NVIDIA Titan X GPU, with the exception of ELMo which was trained on 3 NVIDIA GTX 1080 Ti GPUs. While training, we repeatedly query the NVIDIA System Management Interface 2 to sample the GPU power consumption and report the average over all samples. We estimate the total time expected for models to train to completion using training times and hardware reported in the original papers. We then calculate the power consumption in kilowatt-hours (kWh) as follows. Let p c be the average power draw (in watts) from all CPU sockets during training, let p r be the average power draw from all DRAM (main memory) sockets, let p g be the average power draw of a GPU during training, and let g be the number of GPUs used to train. We estimate total power consumption as combined GPU, CPU and DRAM consumption, then multiply this by Power Usage Effectiveness (PUE), which accounts for the additional energy required to support the compute infrastructure (mainly cooling). We use a PUE coefficient of 1.58, the 2018 global average for data centers BID0. Then the total power p t required at a given instance during training is given by: DISPLAYFORM0 The U.S. Environmental Protection Agency (EPA) provides average CO 2 produced (in pounds per kilowatt-hour) for power consumed in the U.S. , which we use to convert power to estimated CO 2 emissions: DISPLAYFORM1 This conversion takes into account the relative proportions of different energy sources (primarily natural gas, coal, nuclear and renewable) consumed to produce energy in the United States. TAB3 lists the relative energy sources for China, Germany and the United States compared to the top three cloud service providers. The U.S. breakdown of energy is comparable to that of the most popular cloud compute service, Amazon Web Services, so we believe this conversion to provide a reasonable estimate of CO 2 emissions per kilowatt hour of compute energy used. We analyze four models, the computational requirements of which we describe below. All models have code freely available online, which we used out-of-the-box. For more details on the models themselves, please refer to the original papers. Transformer. The Transformer model BID18 is an encoder-decoder architecture primarily recognized for efficient and accurate machine translation. The encoder and decoder each consist of 6 stacked layers of multi-head selfattention. BID18 report that the Transformer base model (65M parameters) was trained on 8 NVIDIA P100 GPUs for 12 hours, and the Transformer big model (213M parameters) was trained for 3.5 days (84 hours; 300k steps). This model is also the basis for recent work on neural architecture search (NAS) for machine translation and language modeling BID16, and the NLP pipeline that we study in more detail in §4.2 BID17. BID16 report that their full architecture search ran for a total of 979M training steps, and that their base model requires 10 hours to train for 300k steps on one TPUv2 core. This equates to 32,623 hours of TPU or 274,120 hours on 8 P100 GPUs. ELMo. The ELMo model BID13 is based on stacked LSTMs and provides rich word representations in context by pre-training on a large amount of data using a language modeling objective. Replacing context-independent pretrained word embeddings with ELMo has been shown to increase performance on downstream tasks such as named entity recognition, semantic role labeling, and coreference. BID13 report that ELMo was trained on 3 NVIDIA GTX 1080 GPUs for 2 weeks (336 hours).BERT. The BERT model BID7 provides a Transformer-based architecture for building contextual representations similar to ELMo, but trained with a different language modeling objective. BERT substantially improves accuracy on tasks requiring sentence-level representations such as question answering and natural language inference. BID7 report that the BERT base model (110M parameters) was trained on 16 TPU chips for 4 days (96 hours). NVIDIA reports that they can train a BERT model in 3.3 days (79.2 hours) using 4 DGX-2H servers, totaling 64 Tesla V100 GPUs BID10. GPT-2. This model is the latest edition of OpenAI's GPT general-purpose token encoder, also based on Transformer-style self-attention and trained with a language modeling objective . By training a very large model on massive data, BID14 show high zero-shot performance on question answering and language modeling benchmarks. The large model described in BID14 has 1542M parameters and is reported to require 1 week (168 hours) of training on 32 TPU v3 chips. 6 There is some precedent for work characterizing the computational requirements of training and inference in modern neural network architectures in the computer vision community. BID11 present a detailed study of the energy use required for training and inference in popular convolutional models for image classification in computer vision, including fine-grained analysis comparing different neural network layer types. BID5 assess image classification model accuracy as a function of model size and gigaflops required during inference. They also measure average power draw required during inference on GPUs as a function of batch size. Neither work analyzes the recurrent and self-attention models that have become commonplace in NLP, nor do they extrapolate power to estimates of carbon and dollar cost of training. Analysis of hyperparameter tuning has been performed in the context of improved algorithms for hyperparameter search BID3 BID2 BID15. To our knowledge there exists to date no analysis of the computation required for R&D and hyperparameter tuning of neural network models in NLP. 6 Via the authors on Reddit. 7 GPU lower bound computed using pre-emptible P100/V100 U.S. resources priced at $0.43-$0.74/hr, upper bound uses on-demand U.S. resources priced at $1.46-$2.48/hr. We similarly use pre-emptible ($1.46/hr-$2.40/hr) and on-demand ($4.50/hr-$8/hr) pricing as lower and upper bounds for TPU v2/3; cheaper bulk contracts are available. Table 3: Estimated cost of training a model in terms of CO 2 emissions (lbs) and cloud compute cost (USD). 7 Power and carbon footprint are omitted for TPUs due to lack of public information on power draw for this hardware. Table 3 lists CO 2 emissions and estimated cost of training the models described in §2.1. Of note is that TPUs are more cost-efficient than GPUs on workloads that make sense for that hardware (e.g. BERT). We also see that models emit substantial carbon emissions; training BERT on GPU is roughly equivalent to a trans-American flight. BID16 report that NAS achieves a new stateof-the-art BLEU score of 29.7 for English to German machine translation, an increase of just 0.1 BLEU at the cost of at least $150k in on-demand compute time and non-trivial carbon emissions. To quantify the computational requirements of R&D for a new model we study the logs of all training required to develop LinguisticallyInformed Self-Attention BID17 ), a multi-task model that performs part-of-speech tagging, labeled dependency parsing, predicate detection and semantic role labeling. This model makes for an interesting case study as a representative NLP pipeline and as a Best Long Paper at EMNLP.Model training associated with the project spanned a period of 172 days (approx. 6 months). During that time 123 small hyperparameter grid searches were performed, ing in 4789 jobs in total. Jobs varied in length ranging from a minimum of 3 minutes, indicating a crash, to a maximum of 9 days, with an average job length of 52 hours. All training was done on a combination of NVIDIA Titan X (72%) and M40 (28%) GPUs. 8 The sum GPU time required for the project totaled 9998 days (27 years). This averages to 8 We approximate cloud compute cost using P100 pricing. about 60 GPUs running constantly throughout the 6 month duration of the project. TAB6 lists upper and lower bounds of the estimated cost in terms of Google Cloud compute and raw electricity required to develop and deploy this model. 9 We see that while training a single model is relatively inexpensive, the cost of tuning a model for a new dataset, which we estimate here to require 24 jobs, or performing the full R&D required to develop this model, quickly becomes extremely expensive.are compatible with their setting. More explicit characterization of tuning time could also reveal inconsistencies in time spent tuning baseline models compared to proposed contributions. Realizing this will require: a standard, hardwareindependent measurement of training time, such as gigaflops required to convergence, and a standard measurement of model sensitivity to data and hyperparameters, such as variance with respect to hyperparameters searched. Academic researchers need equitable access to computation resources. Recent advances in available compute come at a high price not attainable to all who desire access. Most of the models studied in this paper were developed outside academia; recent improvements in state-of-the-art accuracy are possible thanks to industry access to large-scale compute. Limiting this style of research to industry labs hurts the NLP research community in many ways. First, it stifles creativity. Researchers with good ideas but without access to large-scale compute will simply not be able to execute their ideas, instead constrained to focus on different problems. Second, it prohibits certain types of research on the basis of access to financial resources. This even more deeply promotes the already problematic "rich get richer" cycle of research funding, where groups that are already successful and thus well-funded tend to receive more funding due to their existing accomplishments. Third, the prohibitive start-up cost of building in-house resources forces resource-poor groups to rely on cloud compute services such as AWS, Google Cloud and Microsoft Azure. While these services provide valuable, flexible, and often relatively environmentally friendly compute resources, it is more cost effective for academic researchers, who often work for nonprofit educational institutions and whose research is funded by government entities, to pool resources to build shared compute centers at the level of funding agencies, such as the U.S. National Science Foundation. For example, an off-the-shelf GPU server containing 8 NVIDIA 1080 Ti GPUs and supporting hardware can be purchased for approximately $20,000 USD. At that cost, the hardware required to develop the model in our case study (approximately 58 GPUs for 172 days) would cost $145,000 USD plus electricity, about half the estimated cost to use on-demand cloud GPUs. Unlike money spent on cloud compute, however, that invested in centralized resources would continue to pay off as resources are shared across many projects. A government-funded academic compute cloud would provide equitable access to all researchers. Researchers should prioritize computationally efficient hardware and algorithms. We recommend a concerted effort by industry and academia to promote research of more computationally efficient algorithms, as well as hardware that requires less energy. An effort can also be made in terms of software. There is already a precedent for NLP software packages prioritizing efficient models. An additional avenue through which NLP and machine learning software developers could aid in reducing the energy associated with model tuning is by providing easyto-use APIs implementing more efficient alternatives to brute-force grid search for hyperparameter tuning, e.g. random or Bayesian hyperparameter search techniques BID3 BID2 BID15. While software packages implementing these techniques do exist, 10 they are rarely employed in practice for tuning NLP models. This is likely because their interoperability with popular deep learning frameworks such as PyTorch and TensorFlow is not optimized, i.e. there are not simple examples of how to tune TensorFlow Estimators using Bayesian search. Integrating these tools into the workflows with which NLP researchers and practitioners are already familiar could have notable impact on the cost of developing and tuning in NLP.

We quantify the energy cost in terms of money (cloud credits) and carbon footprint of training recently successful neural network models for NLP. Costs are high.

Represent the short summary for finding the corresponding scientific article (long):

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.


Question: Represent the scientific article to find a short TLDR

Question: Many models based on the Variational Autoencoder are proposed to achieve disentangled latent variables in inference. However, most current work is focusing on designing powerful disentangling regularizers, while the given number of dimensions for the latent representation at initialization could severely inﬂuence the disentanglement. Thus, a pruning mechanism is introduced, aiming at automatically seeking for the intrinsic dimension of the data while promoting disentangled representations. The proposed method is validated on MPI3D and MNIST to be advancing state-of-the-art methods in disentanglement, reconstruction, and robustness. The code is provided on the https://github.com/WeyShi/FYP-of-Disentanglement. To advance disentanglement, models based on the Variational Autoencoder (VAE) are proposed in terms of additional disentangling regularizers. However, in this paper, we introduce an orthogonal mechanism that is applicable to most state-of-theart models, ing in higher disentanglement and robustness for model configurationsespecially the choice of dimensionality for the latent representation. Intuitively, both excessive and deficient latent dimensions can be detrimental to achieving the best disentangled latent representations. For excessive dimensions, powerful disentangling regularizers, like the β-VAE , can force information to be split across dimensions, ing in capturing incomplete features. On the other hand, having too few dimensions inevitably leads to an entangled representation, such that each dimension could capture enough information for the subsequent reconstruction. In this paper, we introduce an approximated L 0 regularization to prune the dimension of the latent representation vector. Consequently, our Pruning Variational Autoencoders (PVAE) framework is applicable to most state-of-the-art VAE-based models due to its orthogonality with current approaches. But in this challenge, we choose to put the pruning mechanism onto the DIP-VAE (for Disentangled Inferred Prior VAE) due to its decent performance on MPI3D . In the context of pruning, the aim of L 0 is to compress the network, while here the goal is seeking for the intrinsic dimension for the latent representation, which is achieved by the balance between several terms. 2.1. The Masked Base Model: Masked DIP-VAE Basically, we desire to achieve binary masks m, depending on some learnable parameters α, to control each dimension. Thus, the DIP-VAE loss term with masks can be formulated as follow: where x, p(x), z, and p(z) are the input images, the data distribution, the latent variables (the output of the encoder), and their prior, respectively, and µ φ (·), and p θ (·), q φ (z|·) denote the function of the encoder's mean path, the decoder, and the encoder. Meanwhile, ] denotes the covariance matrix of the pruned mean representations. There are two points to note about the Kullback-Leibler (KL) divergence terms. Firstly, they decompose across dimensions (z i) because we assumed factorized prior and variational posterior distributions. Secondly, the KL term for each dimension is multiplied by the mask for consistency when that dimension is forced to zero, which can be understood in terms of inference with spike-and-slab distributions (see , Appendix A). With a second term denoting L 0 regularization over e, the samples drawn from the q φ (z|x), the total loss can be formulated as where, which is the sum of the probability of m j being positive. where γ < 0 and ζ > 1 are the lower and upper bounds for the stretched range, and β is the temperature coefficient of the masks generation process introduced in Section 2.3. The given formulation is slightly different from for clarity. The mask vector m is clamped such that m j ∈. The binary masks m are modelled as following Bernoulli distributions with parameters α: proposed to obtain these masks in a differentiable fashion, feeding uniform random variables through a sigmoid-like function whose location depends on α. Furthermore, to ensure that masks are likely to be exactly 0 or 1, they stretch the value range of the sigmoid-like function to be [ζ, γ] and then clamp it to be. This process can be formulated as below, and is illustrated in Appendix A: To align it with VAE, we need the encoder to output means µ φ (x) and variances σ 2 φ (x) of q(z|x) instead of means µ φ (x) and log σ 2 φ (x) such that after pruning we have a N rather than N for a specific dimension. In detail, a mask is multiplied with each pair of mean and variance and the KL divergence for the corresponding dimension, such that dimensions can effectively be'switched off' and not affect training. To avoid numerical instability in the KL divergence, we add a small positive constant to σ 2 φ (x). Given the outputs of the last layer of the original encoder, the L0Pair layer can be expressed as In terms of the structure of the encoder and the decoder, we adopt the default settings given in the starter kit 1, which is based on row 3 of Table 1 on page 13 of. We list our choices of hyperparameters in Appendix B. The L 0 regularization in the pruning mechanism facilitates the performance and the robustness of vanilla DIP-VAE on MPI3D by approaching the intrinsic dimension during training. In Appendix B, we additionally present on MNIST with a JointVAE extension of the proposed PVAE (PJVAE), which further validates the disentanglement benefits of pruning. A pruning mechanism that is complementary to most current state-of-the-art VAE-based disentangling models is introduced and validated on MPI3D and MNIST. The approximated L 0 regularization facilitates the model to capture better-disentangled representations with optimal size and increases the robustness to initialization. Moreover, with the same hyperparameters, the model approaches the intrinsic dimension for several datasets including MNIST and MPI3D, even with an extra-large number of dimensions at initialization. Even given the intrinsic dimension, the PVAE still outperforms other SOTA methods in terms of disentanglement and reconstruction. The default parameters are given in Table 1. As for the optimizer and its Learning rate, we select Adam optimizer with 10 −4. Moreover, the τ = 0.1 generalizes well on both MNIST and CelebA 2. To capture the discrete features like Digits , we adopt one additional discrete variable and the model becomes Pruning Joint VAE (PJVAE). Since there is only one discrete variable, it is unnecessary to impose further disentanglement on it (the disentanglement on discrete variables is beyond the scope of this report). In Figure 3, we can see the advantage of pruning on MNIST, especially when the initialization far deviates from the intrinsic dimension (which is still unknown for MNIST, but is estimated to be around 10 by several methods). However, the PJVAE is robust to the initialization as long as it is given enough latent space at initialization. Surprisingly, with appropriate initialization, its reconstruction occasionally becomes better than the VAE, with consistent higher disentanglement performance. Furthermore, on this dataset PJVAE outperforms DIP-VAE in both metrics. Inspecting the variation between different initialization, we can validate the robustness of PJVAE versus the other two methods. In general, in terms of TC, PJVAE possesses obvious advantages. And reconstruction performance is the same, PJVAE also showing a consistent lower error. Note that both VAE and DIP-VAE are initialized with one additional 10-value categorical (discrete) variable for a fair comparison. The only difference between this DIP-VAE (actually, DIP-JointVAE) and PJVAE, is the approximated L 0. MNIST. The number denotes the total dimensionality of the latent variables at initialization. TC stands for Total Correlation.

The Pruning VAE is proposed to search for disentangled variables with intrinsic dimension.

Represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the scientific article to find a short tldr

Question: We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment. Representation learning BID1 ) plays a critical role in many modern machine learning systems. Representations map raw data to more useful forms and the choice of representation is an important component of any application. Broadly speaking, there are two areas of research emphasizing different details of how to learn useful representations. The supervised training of high-capacity models on large labeled datasets is critical to the recent success of deep learning techniques for a wide range of applications such as image classification BID25 ), speech recognition ), and machine translation ). Analysis of the task specific representations learned by these models reveals many fascinating properties BID58 ). Image classifiers learn a broadly useful hierarchy of feature detectors re-representing raw pixels as edges, textures, and objects BID55 ). In the field of computer vision, it is now commonplace to reuse these representations on a broad suite of related tasks -one of the most successful examples of transfer learning to date BID42 ).There is also a long history of unsupervised representation learning BID41 ). Much of the early research into modern deep learning was developed and validated via this approach BID14; BID17; BID49; BID4; BID27 ). Unsupervised learning is promising due to its ability to scale beyond the small subsets and subdomains of data that can be cleaned and labeled given resource, privacy, or other constraints. This advantage is also its difficulty. While supervised approaches have clear objectives that can be directly optimized, unsupervised approaches rely on proxy tasks such as reconstruction, density estimation, or generation, which do not directly encourage useful representations for specific tasks. As a , much work has gone into designing objectives, priors, and architectures meant to encourage the learning of useful representations. We refer readers to for a detailed review. Despite these difficulties, there are notable applications of unsupervised learning. Pre-trained word vectors are a vital part of many modern NLP systems BID5 ). These representations, learned by modeling word co-occurrences, increase the data efficiency and generalization capability of NLP systems BID45; BID3 ). Topic modelling can also discover factors within a corpus of text which align to human interpretable concepts such as "art" or "education" BID2 ).How to learn representations of phrases, sentences, and documents is an open area of research. Inspired by the success of word vectors, BID23 propose skip-thought vectors, a method of training a sentence encoder by predicting the preceding and following sentence. The representation learned by this objective performs competitively on a broad suite of evaluated tasks. More advanced training techniques such as layer normalization BID0 ) further improve . However, skip-thought vectors are still outperformed by supervised models which directly optimize the desired performance metric on a specific dataset. This is the case for both text classification tasks, which measure whether a specific concept is well encoded in a representation, and more general semantic similarity tasks. This occurs even when the datasets are relatively small by modern standards, often consisting of only a few thousand labeled examples. In contrast to learning a generic representation on one large dataset and then evaluating on other tasks/datasets, BID6 proposed using similar unsupervised objectives such as sequence autoencoding and language modeling to first pretrain a model on a dataset and then finetune it for a given task. This approach outperformed training the same model from random initialization and achieved state of the art on several text classification datasets. Combining word-level language modelling of a dataset with topic modelling and fitting a small neural network feature extractor on top has also achieved strong on document level sentiment analysis BID7 ).Considering this, we hypothesize two effects may be combining to in the weaker performance of purely unsupervised approaches. Skip-thought vectors were trained on a corpus of books. But some of the classification tasks they are evaluated on, such as sentiment analysis of reviews of consumer goods, do not have much overlap with the text of novels. We propose this distributional issue, combined with the limited capacity of current models, in representational underfitting. Current generic distributed sentence representations may be very lossy -good at capturing the gist, but poor with the precise semantic or syntactic details which are critical for applications. The experimental and evaluation protocols may be underestimating the quality of unsupervised representation learning for sentences and documents due to certain seemingly insignificant design decisions. BID12 also raises concern about current evaluation tasks in their recent work which provides a thorough survey of architectures and objectives for learning unsupervised sentence representations -including the above mentioned skip-thoughts. In this work, we test whether this is the case. We focus in on the task of sentiment analysis and attempt to learn an unsupervised representation that accurately contains this concept. BID37 showed that word-level recurrent language modelling supports the learning of useful word vectors. We are interested in pushing this line of work to learn representations of not just words but arbitrary scales of text with no distinction between sub-word, word, phrase, sentence, or document-level structure. Recent work has shown that traditional NLP task such as Named Entity Recognition and Part-of-Speech tagging can be performed this way by processing text as a byte sequence BID10 ). Byte level language modelling is a natural choice due to its simplicity and generality. We are also interested in evaluating this approach as it is not immediately clear whether such a low-level training objective supports the learning of high-level representations. We train on a very large corpus picked to have a similar distribution as our task of interest. We also benchmark on a wider range of tasks to quantify the sensitivity of the learned representation to various degrees of out-of-domain data and tasks. Much previous work on language modeling has evaluated on relatively small but competitive datasets such as Penn Treebank BID33 ) and Hutter Prize Wikipedia BID18 ). As discussed in BID19 performance on these datasets is primarily dominated by regularization. Since we are interested in high-quality sentiment representations, we chose the Amazon product review dataset introduced in BID35 as a training corpus. In deduplicated form, this dataset contains over 82 million product reviews from May 1996 to July 2014 amounting to over 38 billion training bytes. Due to the size of the dataset, we first split it into 1000 shards containing equal numbers of reviews and set aside 1 shard for validation and 1 shard for test. Many potential recurrent architectures and hyperparameter settings were considered in preliminary experiments on the dataset. Given the size of the dataset, searching the wide space of possible configurations is quite costly. To help alleviate this, we evaluated the generative performance in terms of log-likelihood of smaller candidate models after a single pass through the dataset and selected the best performing architecture according to this metric for large scale experiments. The model chosen is a single layer multiplicative LSTM BID24 ) with 4096 units. We observed multiplicative LSTMs to converge faster than normal LSTMs for the hyperparameter settings that were explored both in terms of data and wall-clock time. The model was trained for a single epoch on mini-batches of 128 subsequences of length 256 for a total of 1 million weight updates. States were initialized to zero at the beginning of each shard and persisted across updates to simulate full-backpropagation and allow for the forward propagation of information outside of a given subsequence. Adam BID22 ) was used to accelerate learning with an initial 5e-4 learning rate that was decayed linearly to zero over the course of training. Weight normalization BID46 ) was applied to the LSTM parameters. Data-parallelism was used across 4 Pascal Titan X gpus to speed up training and increase effective memory size. Training took approximately one month. The model is compact, containing approximately as many parameters as there are reviews in the training dataset. It also has a high ratio of compute to total parameters due to operating at a byte level. The model reaches 1.12 bits per byte. Our model processes text as a sequence of UTF-8 encoded bytes BID54 ). For each byte, the model updates its hidden state and predicts a probability distribution over the next possible byte. The hidden state of the model serves as an online summary of the sequence which encodes all information the model has learned to preserve that is relevant to predicting the future bytes of the sequence. We are interested in understanding the properties of the learned encoding. The process of extracting a feature representation is outlined as follows:• Since newlines are used as review delimiters in the training dataset, all newline characters are replaced with spaces to avoid the model resetting state.• Any leading whitespace is removed and replaced with a newline+space to simulate a start token. Any trailing whitespace is removed and replaced with a space to simulate an end token. HTML is unescaped and HTML tags are removed to mirror preprocessing applied to the training dataset. The text is encoded as a UTF-8 byte sequence.• Model states are initialized to zeros. The model processes the sequence and the final cell states of the mLSTM are used as a feature representation. We follow the methodology established in BID23 by training a logistic regression classifier on top of our model's representation on datasets for tasks including semantic relatedness, text classification, and paraphrase detection. For the details on these comparison experiments, we refer the reader to their work. One exception is that we use an L1 penalty for text classification instead of L2 as we found this performed better in the very low data regime. BID47 ) CNN BID21 ) DMN BID26 ) P-SL999 LSTM BID52 ) NSE BID39 ) CT-LSTM BID30 4.1 TEXT CLASSIFICATION AND SENTIMENT ANALYSIS Table 1 shows the of our model on 4 standard text classification datasets. The performance of our model is noticeably lopsided. On the MR BID44 ) and CR BID15 ) sentiment analysis datasets we improve the state of the art by a significant margin. The MR and CR datasets are sentences extracted from Rotten Tomatoes, a movie review website, and Amazon product reviews (which almost certainly overlaps with our training corpus). This suggests that our model has learned a rich representation of text from a similar domain. On the other two datasets, SUBJ's subjectivity/objectivity detection BID43 ) and MPQA's opinion polarity BID51 ) our model has no noticeable advantage over other unsupervised representation learning approaches and is still outperformed by a supervised approach. To better quantify the learned representation, we also test on a wider set of sentiment analysis datasets with different properties. The Stanford Sentiment Treebank (SST) BID47 ) was created specifically to evaluate more complex compositional models of language. It is derived from the same base dataset as MR but was relabeled via Amazon Mechanical Turk and includes dense labeling of the phrases of parse trees computed for all sentences. For the binary subtask, this amounts to ST BID23 77.3 81.8 92.6 87.9 STLN BID0 79.5 83.1 93.7 89.3 SDAE BID12 74.6 78.0 90.8 86.9 CNN BID21 81.5 85.0 93.4 89.6 Adasent 83.1 86.3 95.5 93.3 byte mLSTM 86.9 91.4 94.6 88.5 Table 1: Small dataset classification accuracies FullBoW BID31 11.1 NBSVM BID36 8.1 Sentiment unit (ours) 7.7 SA-LSTM BID6 7.2 byte mLSTM (ours) 7.1 TopicRNN BID7 6.2 Virt Adv BID38 5.9 Table 2: IMDB sentiment analysis 76961 total labels compared to the 6920 sentence level labels. As a demonstration of the capability of unsupervised representation learning to simplify data collection and remove steps from a traditional NLP pipeline, our reported ignore these dense labels and computed parse trees, using only the raw text and sentence level labels. The representation learned by our model achieves 91.8% significantly outperforming the state of the art of 90.2% by a 30 model ensemble BID30 ). As visualized in FIG1, our model is very data efficient. It matches the performance of baselines using as few as a dozen labeled examples and outperforms all previous with only a few hundred labeled examples. This is under 10% of the total sentences in the dataset. Confusingly, despite a 16% relative error reduction on the binary subtask, it does not reach the state of the art of 53.6% on the fine-grained subtask, achieving 52.9%. Figure 3: Histogram of cell values for the sentiment unit on IMDB reviews. We conducted further analysis to understand what representations our model learned and how they achieve the observed data efficiency. The benefit of an L1 penalty in the low data regime (see FIG1) is a clue. L1 regularization is known to reduce sample complexity when there are many irrelevant features BID40 ). This is likely to be the case for our model since it is trained as a language model and not as a supervised feature extractor. By inspecting the relative contributions of features on various datasets, we discovered a single unit within the mLSTM that directly corresponds to sentiment. In Figure 3 we show the histogram of the final activations of this unit after processing IMDB reviews BID31 ) which shows a bimodal distribution with a clear separation between positive and negative reviews. In FIG3 we visualize the activations of this unit on 6 randomly selected reviews from a set of 100 high contrast reviews which shows it acts as an online estimate of the local sentiment of the review. Fitting a threshold to this single unit achieves a test accuracy of 92.30% which outperforms a strong supervised on the dataset, the 91.87% of NB-SVM trigram BID36 ), but is still below the semi-supervised state of the art of 94.09% BID38 ). Using the full 4096 unit representation achieves 92.88%. This is an improvement of only 0.58% over the sentiment unit suggesting that almost all information the model retains that is relevant to sentiment analysis is represented in the very compact form of a single scalar. Table 2 has a full list of on the IMDB dataset. Method r ρ MSE ST BID23 0.858 0.792 0.269 STLN BID0 0.858 0.788 0.270Tree-LSTM BID48 ST BID23 73.0 82.0 SDAE BID12 76.4 83.4MTMETRICS BID32 Encouraged by these , we were curious how well the model's representation scales to larger datasets. We try our approach on the binary version of the Yelp Dataset Challenge in 2015 as introduced in BID56. This dataset contains 598,000 examples which is an order of magnitude larger than any other datasets we tested on. When visualizing performance as a function of number of training examples in FIG2, we observe a "capacity ceiling" where the test accuracy of our approach only improves by a little over 1% across a four order of magnitude increase in training data. Using the full dataset, we achieve 95.22% test accuracy. This better than a BoW TFIDF baseline at 93.66% but slightly worse than the 95.64% of a linear classifier on top of the 500,000 most frequent n-grams up to length 5.The observed capacity ceiling is an interesting phenomena and stumbling point for scaling our unsupervised representations. We think a variety of factors are contributing to cause this. Since our model is trained only on Amazon reviews, it is does not appear to be sensitive to concepts specific to other domains. For instance, Yelp reviews are of businesses, where details like hospitality, location, and atmosphere are important. But these ideas are not present in reviews of products. Additionally, there is a notable drop in the relative performance of our approach transitioning from sentence to document datasets. This is likely due to our model working on the byte level which leads to it focusing on the content of the last few sentences instead of the whole document. Finally, as the amount of labeled data increases, the performance of a simple linear model trained on top of a static representation will eventually saturate. Complex models explicitly trained for a task can continue to improve and eventually outperform our approach when provided with enough labeled data. With this context, the observed reveal a natural ordering. On a small sentence level dataset from a very similar domain (the movie reviews of Stanford Sentiment Treebank) our model sets a new state of the art. But on a large, document level dataset from a slightly different domain (the Yelp reviews) it is only competitive with standard baselines. The features of our model are learned via an unsupervised objective. A potential concern with this training methodology is that it is unreliable. There is no guarantee that the representations learned will be useful for a desired task because the model is not optimizing directly for this. This could lead to high variance of performance on desired tasks. This concern is further amplified for the properties of individual features and not just the representation as whole. While the work of has shown that networks differing in only random initialization can learn the same features, this was This product does what it is supposed to. I always keep three of these in my kitchen just in case ever I need a replacement cord. Great little item. Hard to put on the crib without some kind of embellishment. My guess is just like the screw kind of attachment I had. Best hammock ever! Stays in place and holds it's shape. Comfy (I love the deep neon pictures on it), and looks so cute. tested for CNNs trained for supervised image classification. To check whether this is the case for our approach, we trained a second model using the same hyperparameters differing only in weight parameterization and compared to the model discussed in this paper. The best single unit in this model (selected via validation set) achieves 92.42% test accuracy on IMDB compared to the 92.30% of the unit visualized in this paper. This suggests that the sentiment unit is an example of a convergent representation. Besides classification, we also evaluate on two other standard tasks: semantic relatedness and paraphrase detection. While our model performs competitively on Microsoft Research Paraphrase Corpus BID8 ) in TAB2, it performs poorly on the SICK semantic relatedness task BID34 ) in Table 4. It is likely that the form and content of the semantic relatedness task, which is built on top of descriptions of images and videos and contains sentences such as "A sea turtle is hunting for fish" is effectively out-of-domain for our model which has only been trained on the text of product reviews. Although the focus of our analysis has been on the properties of our model's representation, it is trained as a generative model and we are also interested in its generative capabilities. BID16 and BID9 both designed conditional generative models to disentangle the content of text from various attributes like sentiment or tense. We were curious whether a similar could be achieved using the sentiment unit. In TAB5 we show that by simply setting the sentiment unit to be positive or negative, the model generates corresponding positive or negative reviews. While all sampled negative reviews contain sentences with negative sentiment, they sometimes contain sentences with positive sentiment as well. This might be reflective of the bias of the training corpus which contains over 5x as many five star reviews as one star reviews. Nevertheless, it is interesting to see that such a simple manipulation of the model's representation has a noticeable effect on its behavior. The samples are also high quality for a byte level language model and often include valid sentences. It is an open question why our model recovers the concept of sentiment in such a precise, disentangled, interpretable, and manipulable way. It is possible that sentiment as a conditioning feature has strong predictive capability for language modelling. This is likely since sentiment is such an important component of a review. Previous work analyzing LSTM language models showed the existence of interpretable units that indicate position within a line or presence inside a quotation BID20 ). In many ways, the sentiment unit in this model is just a scaled up example of the same phenomena. The update equation of an LSTM could play a role. The element-wise operation of its gates may encourage axis-aligned representations. Models such as word2vec have also been observed to have small subsets of dimensions strongly associated with specific tasks BID28 ).Our work highlights the sensitivity of learned representations to the data distribution they are trained on. The make clear that it is unrealistic to expect a model trained on a corpus of books, where the two most common genres are Romance and Fantasy, to learn an encoding which preserves the exact sentiment of a review. Likewise, it is unrealistic to expect a model trained on Amazon product reviews to represent the precise semantic content of a caption of an image or a video. There are several promising directions for future work highlighted by our . The observed performance plateau, even on relatively similar domains, suggests improving the representation model both in terms of architecture and size. Since our model operates at the byte-level, hierarchical/multitimescale extensions could improve the quality of representations for longer documents. The sensitivity of learned representations to their training domain could be addressed by training on a wider mix of datasets with better coverage of target tasks. Finally, our work encourages further research into language modelling as it demonstrates that the standard language modelling objective with no modifications is sufficient to learn high-quality representations.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Byte-level recurrent language models learn high-quality domain specific representations of text.

represent the short summary for finding the corresponding scientific article (long)

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Discrete latent-variable models, while applicable in a variety of settings, can often be difficult to learn. Sampling discrete latent variables can in high-variance gradient estimators for two primary reasons: 1) branching on the samples within the model, and 2) the lack of a pathwise derivative for the samples. While current state-of-the-art methods employ control-variate schemes for the former and continuous-relaxation methods for the latter, their utility is limited by the complexities of implementing and training effective control-variate schemes and the necessity of evaluating (potentially exponentially) many branch paths in the model. Here, we revisit the Reweighted Wake Sleep (RWS;) algorithm, and through extensive evaluations, show that it circumvents both these issues, outperforming current state-of-the-art methods in learning discrete latent-variable models. Moreover, we observe that, unlike the Importance-weighted Autoencoder, RWS learns better models and inference networks with increasing numbers of particles, and that its benefits extend to continuous latent-variable models as well. Our suggest that RWS is a competitive, often preferable, alternative for learning deep generative models. Learning deep generative models with discrete latent variables opens up an avenue for solving a wide range of tasks including tracking and prediction BID28, clustering BID31, model structure learning BID0, speech modeling BID16, topic modeling BID1, language modeling BID4, and concept learning BID17 BID20. Furthermore, recent deep-learning approaches addressing counting BID6, attention BID37, adaptive computation time BID9, and differentiable data structures BID9 BID11, underscore the importance of models with conditional branching induced by discrete latent variables. Current state-of-the-art methods optimize the evidence lower bound (ELBO) based on the importance weighted autoencoder (IWAE) BID3 by using either reparameterization BID19 BID32, continuous relaxations of the discrete latents BID15 or the REINFORCE method BID36 with control variates BID24 BID25 BID12 BID35 BID8.Despite the effective large-scale learning made possible by these methods, several challenges remain. First, with increasing number of particles, the IWAE ELBO estimator adversely impacts inference-network quality, consequently impeding learning of the generative model. Second, using continuous relaxations in a biased gradient estimator, and in models with stochastic branching, forces evaluation of potentially exponential number of branching paths. For example, a continuous relaxation of the cluster identity in a Gaussian mixture model (GMM) (Section 4.3) forces the evaluation of a weighted average of likelihood parameters over all clusters instead of selecting the parameters based on just one. Finally, while control-variate methods may be employed to reduce variance, their practical efficacy can be somewhat limited as in some cases they involve designing and jointly optimizing a separate neural network which can be difficult to tune (Section 4.3).To address these challenges, we revisit the reweighted wake-sleep (RWS) algorithm BID2, comparing it extensively with state-of-the-art methods for learning discrete latent-variable models, and demonstrate its efficacy in learning better generative models and inference networks, and improving the variance of the gradient estimators, over a range of particle budgets. Going forward, we review the current state-of-the-art methods for learning deep generative models with discrete latent variables (Section 2), revisit RWS (Section 3), and present an extensive evaluation of these methods (Section 4) on (i) the Attend, Infer, Repeat (AIR) model BID6 to perceive and localise multiple MNIST digits, (ii) a continuous latent-variable model on MNIST, and (iii) a pedagogical GMM example, exposing a shortcoming of RWS that we fix using defensive importance sampling BID13. Our experiments confirm that RWS is a competitive, often preferable, alternative that unlike IWAE, learns better models and inference networks with increasing particle budgets. Consider data (x (n) ) N n=1 sampled from a true (unknown) generative model p(x), a family of generative models p θ (z, x) of latent variable z and observation x parameterized by θ and a family of inference networks q φ (z|x) parameterized by φ. We would like to learn the generative model by maximizing the marginal likelihood over data: DISPLAYFORM0. We would simultaneously also like to learn an inference network q φ (z|x) that amortizes inference given observation x; i.e., q φ (z|x) maps an observation x to an approximation of p θ * (z|x). Amortization ensures that evaluation of this function is cheaper than performing approximate inference of p θ * (z|x) from scratch. Our focus here is on such joint learning of the generative model and the inference network, here referred to as "learning a deep generative model", although we note that other approaches exist that learn just the generative model BID7 BID26 or the inference network BID29 BID21 in isolation. We begin with a review of IWAEs BID3 as a general approach for learning deep generative models using stochastic gradient descent (SGD) methods, focusing on generative-model families with discrete latent variables, for which the high variance of the naïve gradient estimator impedes learning. We will also review control-variate and continuous-relaxation methods for gradient-variance reduction. The IWAE used alongside such gradient-variance reduction methods is currently the dominant approach for learning deep generative models with discrete latent variables. BID3 introduced the IWAE which maximizes an average of ELBOs over data,, where, given number of particles K, DISPLAYFORM0 When K = 1, this reduces to the variational autoencoder (VAE) BID19 BID32. BID3 show that ELBO K IS (θ, φ, x) is a lower bound on log p θ (x) and that increasing K leads to a tighter lower bound. Further, tighter lower bounds arising from increasing K improve learning of the generative model, but worsen learning of the inference network, as the signal-to-noise ratio of θ's gradient estimator is O(DISPLAYFORM1 Moreover, poor learning of the inference network, beyond a certain point (large K), can actually worsen learning of the generative model as well; a finding we explore in Section 4.3.Optimizing the IWAE objective using SGD methods requires unbiased gradient estimators of ELBO K IS (θ, φ, x) with respect to θ and φ BID33.The θ gradient ∇ θ ELBO K IS (θ, φ, x) is estimated by sampling z 1:K ∼ Q φ (·|x) and evaluating DISPLAYFORM2 is estimated similarly for models with reparameterizable latents, discrete (and other non-reparameterizable) latents require the REINFORCE gradient estimator BID36 DISPLAYFORM3 Since the gradient estimator in Eq. can often suffer from high variance, mainly due to the effect of 1, a number of approaches have been developed to ameliorate the issue. When employing continuous relaxation methods, all discrete latent variables in the model can be replaced by the Concrete or Gumbel-Softmax BID15 distribution, whose gradients can be approximated using the reparameterization trick BID19 BID32. The main practical difficulty here is tuning the temperature parameter: low temperatures reduce the gradient estimator bias, but rapidly increase its variance. Moreover, since such relaxations are defined on a simplex, applying them to models that exhibit branching requires the computationally expensive evaluation of all (potentially exponentially many) paths in the generative model. Variational inference for Monte Carlo objectives (VIMCO) BID25 ) is a method that doesn't require designing an explicit control variate, as it exploits the particle set obtained in IWAE.It replaces 1 with DISPLAYFORM0 where term B is independent of z and highly correlated with term A.Finally, assuming z k is a discrete random variable with C categories 1, REBAR BID35 ) and RELAX BID8 improve on the methods of BID24 and BID12, and replaces 1 with DISPLAYFORM1 where g k is a C-dimensional vector of reparameterized Gumbel random variates, z k is a one-hot argmax function of g k, andg k is a vector of reparameterized conditional Gumbel random variates conditioned on z k. The conditional Gumbel random variates are a form of Rao-Blackwellization used to reduce variance. The control variate c ρ, parameterized by ρ, is optimized to minimize the gradient variance estimates concurrently with the main ELBO optimization, leading to state-of-theart performance on, for example, sigmoid belief networks BID27. The main practical difficulty in using this method is choosing a suitable family of c ρ, as some choices lead to higher variance despite the concurrent gradient variance minimization. Moreover, the objective for the concurrent optimization requires evaluating a Jacobian-vector product that induces an overhead of O(D φ D ρ) where D φ, D ρ are number of inference network and control-variate parameters respectively. Reweighted wake-sleep (RWS) BID2 comes from another family of algorithms for learning deep generative models, eschewing a single objective over parameters θ and φ in favour of individual objectives for each. We review the RWS algorithm and discuss its advantages and disadvantages. Reweighted wake-sleep (RWS) BID2 is an extension of the wake-sleep algorithm both of which, like IWAE, jointly learn a generative model and an inference network given data. While IWAE targets a single objective, RWS alternates between objectives, updating the generative model parameters θ using a wake-phase θ update and the inference network parameters φ using either a sleep-or a wake-phase φ update (or both).Wake-phase θ update. Given a current value of φ, θ is updated using an unbiased estimate of DISPLAYFORM0, which can be obtained directly without needing reparameterization or control variates as the sampling distribution Q φ (·|x) is independent of θ.2 Sleep-phase φ update. Here, φ is updated to maximize the negative Kullback-Leibler (KL) divergence between the posteriors under the generative model and the inference network, averaged over the data distribution of the current generative model DISPLAYFORM1 The gradient of this objective is E p θ (z,x) [∇ φ log q φ (z|x)] and can be estimated by sampling z, x from the generative model p θ (z, x) and evaluating ∇ φ log q φ (z|x). The variance of such an estimator can be reduced at a standard Monte Carlo rate by increasing the number of samples of z, x. Wake-phase φ update. Here, φ is updated to maximize the negative KL divergence between the posteriors under the generative model and the inference network, averaged over the true data distribution DISPLAYFORM2 The outer expectation of the gradient DISPLAYFORM3 can be estimated using a single sample x from the true data distribution p(x), given which, the inner expectation can be estimated using a self-normalized importance sampler with K particles using q φ (z|x) as the proposal distribution. This in the following estimator DISPLAYFORM4 where DISPLAYFORM5, in a similar fashion to Eq.. Note that equation 5 is negative of the second term of the REINFORCE estimator of the IWAE ELBO in equation 2. The crucial difference of the wake-phase φ update to the sleep-phase φ update is that the expectation in Eq. FORMULA10 is over the true data distribution p(x) unlike the expectation in Eq. FORMULA9 which is under the current model distribution p θ (x). The former is desirable from the perspective of amortizing inference over data from p(x). Although the estimator in Eq. FORMULA12 is biased, this bias decreases as K increases. While BID2 refer to the use of the sleep-phase φ update followed by an equally-weighted mixture of the wake-phase and sleep-phase updates of φ, we here use RWS to refer to the variant that employs just the wake-phase φ update, and not the mixture. The rationale for our preference will be made clear from the empirical evaluations (Section 4). While the gradient update of θ targets the same objective as IWAE, the gradient update of φ targets the objective in Eq. in the sleep case and Eq. in the wake case. This leads to two advantages for RWS over IWAE. First, since we don't need to use REINFORCE, using RWS leads to much lower variance of gradient estimators of φ. Second, the φ updates in RWS directly target minimization of the expected KL divergences from true to approximate posteriors. Increasing the computational budget (using more Monte Carlo samples in the sleep-phase φ update case and higher number of particles K in the wake-phase φ update) in a better estimator of these expected KL divergences. This is different to IWAE, where optimizing ELBO K IS targets a KL divergence on an extended sampling space which for K > 1 doesn't correspond to a KL divergence between true and approximate posteriors (in any order). Consequently, increasing number of particles in IWAE leads to worse learning of inference networks. The objective of the sleep-phase φ update in equation 3 is an expectation of KL under the current model distribution p θ (x) rather than the true one p(x). This makes the sleep-phase φ update suboptimal since the inference network must always follow a "doubly moving" target (both p θ (x) and p θ (z|x) change during the optimization). The IWAE and RWS algorithms have primarily been applied to problems with continuous latent variables and/or discrete latent variables that do not actually induce branching such as sigmoid belief networks BID27. The purpose of the following experiments is to compare RWS to IWAE used alongside the control variate and continuous relaxation methods described in Section 3 on models with conditional branching, where, as we will show, the various control-variate schemes underperform in relation to RWS. In several ways, including ELBOs achieved and average distance between true and amortized posteriors, we empirically demonstrate that increasing the number of particles K hurts learning in IWAE but improves learning in RWS.The first experiment, using the deep generative model from Attend, Infer, Repeat (AIR) BID6, demonstrates better learning of the generative model in a model containing both discrete latent variables used for branching as well as continuous latent variables in a complex visual data domain (Section 4.1). The next experiment on MNIST (Section 4.2) does so in a model with continuous latent variables. Finally, a GMM experiment (Section 4.3) serves as a pedagogical example to understand sources of advantage for RWS in more detail. Notationally, the different variants of RWS will be referred to as wake-sleep (WS) and wake-wake (WW). The wake-phase θ update is always used. We refer to using it in conjunction with the sleep-phase φ update as WS and using it in conjunction with the wake-phase φ update as WW. We tried using both wake-and sleep-phase φ updates however, in addition to doubling the amount of sampling, found that doing so only improves performance on the continuous latent variable model. The number of particles K used for the wake-phase θ and φ updates will always be specified. The sleep phase φ update will also use K samples from p θ (z, x). First, we evaluate WW and VIMCO on AIR , a structured deep generative model with both discrete and continuous latent variables. AIR uses the discrete variable to decide how many continuous variables are necessary to explain an image (see supplementary material for details). The sequential inference procedure of AIR poses a difficult problem, since it implies a sequential decision process with possible branching. See BID6 for the details of the model (see supplementary material for the model in our notation).We set the maximum number of inference steps in AIR to three and we train it on images of size 50× 50 with zero, one or two MNIST digits. The training and testing data sets consist of 60000 and 10000 images, respectively, which are generated from the respective MNIST train/test datasets. Unlike AIR, which used Gaussian likelihood with fixed standard deviation and continuous inputs (i.e., input x ∈ 50×50 ), we use a Bernoulli likelihood and binarized data; the stochastic binarization is the same as in BID3. This choice is motivated by initial experiments, which have shown that the original setup is detrimental to the sleep phase of WS -samples from the generative model did not look similar to true data even after the training has converged. Training is performed over two million iterations by RmsProp BID34 with the learning rate of 10 −5, which is divided by three after 400k and 1000k training iterations. We set the glimpse size to 20 × 20.We first evaluate the generative model via the average test log marginal where each log marginal is estimated by a one-sample, 5000-particle IWAE estimate. The inference network is then evaluated via the average test KL from the inference network to the posterior under the current model where each D KL (q φ (z|x), p θ (z|x)) is estimated as a difference between the log marginal estimate above and a 5000-sample, one-particle IWAE estimate. Note that this KL estimate is merely a proxy to the desired KL from the inference network to the posterior under the true model. This experiment confirms FIG1 ) that increasing number of particles hurts learning in VIMCO but improves learning in WW. Increasing K improves WW monotonically but VIMCO only up to a point. WW also in significantly lower variance and better inference networks than VIMCO. RWS has typically only been considered for learning models with discrete latent variables. In order to gauge its applicability to a wider class of problems, we evaluate it on a variational autoencoder with normally distributed latent variables for MNIST. To do this, we use the training procedure and the model with a single stochastic layer of BID3 and their stochastic binarization of data. with the batch size of 32. We evaluate performance in the same way as we evaluate the AIR model. Additionally, we evaluate the number of active latent units BID3. This experiment confirms (FIG2 and TAB0) that increasing number of particles hurts learning in IWAE but improves learning in WW. Increasing K does improve the marginal likelihood attained by WW and IWAE with reparameterization. However, the latter learns a better model only up to a point (K = 128) -further increase in the number of particles has diminishing returns. WW also in better inference networks than IWAE as showed by the KL plot on the right of FIG2 To see what is going on, we study a GMM which branches on a discrete latent variable to select cluster assignments. The generative model and inference network are defined as DISPLAYFORM0 where z ∈ {0, . . ., C −1}, C is the number of clusters and µ c, σ Note that KL is between the inference network and the current generative model. Quality of the generative model: WS and WW improve with larger particle budget thanks to lower variance and lower bias estimators of the gradient respectively. IWAE methods suffer with a larger particle budget. WS performs the worst as a consequence of computing the expected KL under model distribution p θ (x) equation 3 instead of the true data distribution p(x) as with WW equation 4. WW suffers from zero-forcing (described in text) in low-particle regimes, but learns the best model fastest in the many-particle regime; δ-WW additionally learns well in the low-particle regime. (Middle) The quality of the inference network develops identically to that of the generative model. (Bottom) WW and WS have lower-variance gradient estimators of φ than IWAE, as they don't include the high-variance term 1 in equation 2. This is a necessary, but not sufficient, condition for efficient learning with other important factors being gradient direction and the ability to escape local optima. under the true model. The true model is set to p θtrue (x) where softmax(θ true) c = (c+5)/ C i=1 (i+5) (c = 0, . . ., C − 1), i.e. the mixture probabilities are linearly increasing with the z FIG0. We fix the mixture parameters in order to study the important features of the problem at hand in isolation. DISPLAYFORM1 IWAE was trained with REINFORCE, RELAX, VIMCO and the Concrete distribution. We also train using WS and WW. We fix C = 20 and increase number of particles from K = 2 to 20. We use the Adam optimizer with the learning rate 10 −3 and default β parameters. At each iteration, a batch of 100 data points is generated from the true model to train. Having searched over several temperature schedules for the Concrete distribution, we use the one with the lowest trainable terminal temperature (linearly annealing from 3 to 0.5). We found that using the control variate c ρ (g 1:K) =(1 + C)-16-16-1 (with tanh nonlinearity) led to most stable training (see supplementary material for more details).We evaluate the generative model, inference network and the variance of the gradient estimator of φ. The generative model is evaluated via the L2 distance between the probability mass functions (PMFs) of its prior and true prior as softmax(θ) − softmax(θ true). The inference network is evaluated via the L2 distance between PMFs of the current and true posteriors, averaged over a fixed set (M = 100) of observations (x (m) test ) M m=1 from the true model: DISPLAYFORM2 is the dth element of one of φ's gradient estimators (e.g. equation 2 for REINFORCE) and std(·) is estimated using 10 samples. Here, we demonstrate that using WS and WW with larger particle budgets leads to a better inference networks whereas this is not the case for IWAE methods FIG3. Recall that the former is because using more samples to estimate the gradient of the sleep φ objective equation 3 for WS reduces variance at a standard Monte Carlo rate and that using more particles in equation 5 to estimate the gradient of the wake φ objective in a lower bias. The latter is because using more particles in the signal-to-noise of IWAE's φ gradient estimator to drop at the rate O(1/ √ K).Learning of the generative model, as a consequence of inference-network learning, is also better for WS and WW, but worse for IWAE methods when the particle budget is increased. This is because the the θ gradient estimator (common to all methods), ∇ θ ELBO K IS (θ, φ, x) can be seen as an importance sampling estimator whose quality is tied to the proposal distribution (inference network).WW and WS have lower variance gradient estimators than IWAE, even if used with control-variate and continuous-relaxation methods. This is because φ's gradient estimators for WW and WS don't include the high-variance term 1 in equation 2. This is a necessary but not sufficient condition for efficient learning with other important factors being gradient direction and the ability to escape local optima (explored below). Employing the Concrete distribution gives low-variance gradients for φ to begin with, but the model learns poorly due to the high gradients bias (due to high temperature hyperparameter). We now describe a failure mode, that affects WS, WW, VIMCO and REINFORCE, which we will refer to as zero-forcing. It is best illustrated by inspecting the generative model and the inference network as the training progresses, focusing on the low-particle (K = 2) regime FIG4 ). For WS, the generative model p θ (z) peaks at z = 9 and puts zero mass for z > 9; the inference network q φ (z|x) becomes the posterior for this model which, in this model, also has support at most {0, . . ., 9} for all x. This is a local optimum for WS because (i) the inference network already approximates the posterior of the model p θ (z, x) well, and (ii) the generative model p θ (z), being trained using samples from q φ (z|x), has no samples outside of its current support ({0, . . ., 9}). Similar failure mode occurs for WW and VIMCO/REINFORCE although the support of the locally optimal p θ (z) is larger ({0, . . ., 14} and {0, . . ., 17} respectively). DISPLAYFORM3 While this failure mode is a particular feature of the GMM, we hypothesize that WS and WW suffer from it more, as they alternate between two different objectives for optimizing θ and φ. WS attempts to amortize inference for the current model distribution p θ (x) which reinforces the coupling between the generative model and the inference network, making it easier to get stuck in a local optimum. WW with few particles (say K = 1) on the other hand, in a highly-biased gradient estimator equation 5 that samples z from q φ (·|x) and evaluates ∇ φ log q φ (z|x); this encourages the inference network to concentrate mass. This behavior is not seen in WW with many particles where it is the best algorithm at learning both a good generative model and inference network (FIG3, right).We propose a simple extension of WW, denoted δ-WW, that mitigates this shortcoming by changing the proposal of the self-normalized importance sampling estimator in Eq. to q φ,δ (z|x) = (1 − δ)q φ (z|x) + δUniform(z). We use δ = 0.2, noting that the method is robust to a range of values. Using a different proposal than the inference network q φ (z|x) means that using the lowparticle estimator in Eq. no longer leads to zero-forcing. This is known as defensive importance sampling BID13, and is used to better estimate integrands that have long tails using short-tailed proposals. Using δ-WW outperforms all other algorithms in learning both the generative model and the inference network. Our experiments suggest that RWS learns both better generative models and inference networks in models that involve discrete latent variables, while performing just as well as state-of-the-art on continuous-variable models as well. The AIR experiment (Section 4.1) shows that the trained inference networks are unusable when trained with high number of particles. Moreover, the MNIST experiment (Section 4.2) suggests that RWS is competitive even on models with continuous latent variables, especially for high number of particles where IWAE ELBO starts suffering from worse inference networks. The GMM experiment (Section 4.3) illustrates that this is at least at least in part due to a lower variance gradient estimator for the inference network and the fact that for RWSunlike the case of optimizing IWAE ELBO )-increasing number of particles actually improves the inference network. In the low-particle regime, the GMM suffers from zeroforcing of the generative model and the inference network, which is ameliorated using defensive RWS. Finally, all experiments show that, beyond a certain point, increasing the particle budget starts to affect the quality of the generative model for IWAE ELBO whereas this is not the case for RWS. As a consequence of our findings, we recommend reconsidering using RWS for learning deep generative models, especially those containing discrete latent variables that induce branching.

Empirical analysis and explanation of particle-based gradient estimators for approximate inference with deep generative models.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.