Question: Represent the scientific article to find a short TLDR.

Question: The recent development of Natural Language Processing (NLP) has achieved great success using large pre-trained models with hundreds of millions of parameters. However, these models suffer from the heavy model size and high latency such that we cannot directly deploy them to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like BERT, MobileBERT is task-agnostic; that is, it can be universally applied to various downstream NLP tasks via fine-tuning. MobileBERT is a slimmed version of BERT-LARGE augmented with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we use a bottom-to-top progressive scheme to transfer the intrinsic knowledge of a specially designed Inverted Bottleneck BERT-LARGE teacher to it. Empirical studies show that MobileBERT is 4.3x smaller and 4.0x faster than original BERT-BASE while achieving competitive on well-known NLP benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves 0.6 GLUE score performance degradation, and 367 ms latency on a Pixel 3 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a 90.0/79.2 dev F1 score, which is 1.5/2.1 higher than BERT-BASE. The NLP community has witnessed a revolution of pre-training self-supervised models. These models usually have hundreds of millions of parameters. They are trained on huge unannotated corpus and then fine-tuned for different small-data tasks (; ; ; ;). Among these models, BERT , which stands for Bidirectional Encoder Representations from Transformers , shows substantial accuracy improvements compared to training from scratch using annotated data only. However, as one of the largest models ever in NLP, BERT suffers from the heavy model size and high latency, making it impractical for resource-limited mobile devices to deploy the power of BERT in mobile-based machine translation, dialogue modeling, and the like. There have been some works that task-specifically distill BERT into compact models (; ; ;). To the best of our knowledge, there is not yet any work for building a task-agnostic lightweight pre-trained model, that is, a model that can be fine-tuned on downstream NLP tasks just like what the original BERT does. In this paper, we propose MobileBERT to fill this gap. In practice, task-agnostic compression of BERT is desirable. Task-specific compression needs to first fine-tune the original large BERT model into task-specific teachers and then distill. Such a process is way more complicated and costly than directly fine-tuning a task-agnostic compact model. At first glance, it may seem straightforward to obtain a task-agnostic compact version of BERT. For example, one may just take a narrower or shallower architecture of BERT, and then train it with a prediction loss together with a distillation loss . Unfortunately, empirical show that such a straightforward approach in significant accuracy loss . This may not be that surprising. It aligns with a well-known observation that shallow networks usually do not have enough representation power while narrow and deep networks are difficult to train. Our MobileBERT is designed to be as deep as BERT LARGE while each layer is made much narrower via adopting bottleneck structures and balancing between self-attentions and feed- MobileBERT is trained by progressively transferring knowledge from IB-BERT. forward networks (Figure 1). To train MobileBERT, we use a bottom-to-top progressive scheme to transfer the intrinsic knowledge of a specially designed Inverted Bottleneck BERT LARGE (IB-BERT) teacher to it. As a pre-trained NLP model, MobileBERT is both storage efficient (w.r.t model size) and computationally efficient (w.r.t latency) for mobile and resource-constrained environments. Experimental on several NLP tasks show that while being 4.3× smaller and 4.0× faster, MobileBERT can still achieve competitive compared to BERT BASE. On the natural language inference tasks of GLUE, MobileBERT can have only 0.6 GLUE score performance degradation with 367 ms latency on a Pixel 3 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT obtains 90.3/80.2 dev F1 score which is 1.5/2.1 higher than BERT BASE. 2 RELATED WORK 2.1 BERT BERT takes the embedding of source tokens as input. Each building block of BERT contains one Multi-Head self-Attention (MHA) module and one Feed-Forward Network (FFN) module, which are connected by skip connections. The MHA module allows the model to jointly attend to information from different subspaces, while the position-wise FFN consists of a two-layer linear transformation with gelu activation , which increase the representational power of the model. Figure 1 (a) illustrates the original BERT architecture. In the pre-training stage, BERT is required to predict the masked tokens in sentences (mask language modeling task), as well as whether one sentence is the next sentence of the other (next sentence prediction task). In the fine-tuning stage, BERT is further trained on task-specific annotated data. Exploiting knowledge transfer to compress model size was first proposed by. The idea was then adopted in knowledge distillation , which requires the smaller student network to mimic the class distribution output of the larger teacher network. Fitnets make the student mimic the intermediate hidden layers of the teacher to train narrow and deep networks. show that the knowledge of the teacher can also be obtained from the neurons in the top hidden layer. Similar to our proposed progressive knowledge transfer scheme, proposed a sequential knowledge transfer scheme to distill knowledge from a deep teacher into a shallow student in a sequential way. proposed to transfer the attention maps of the teacher on images. proposed to transfer the similarity of hidden states and word alignment from an autoregressive Transformer teacher to a non-autoregressive student. Recently, knowledge transfer for BERT has attracted much attention. Researchers have distilled BERT into smaller pre-trained BERT models , an extremely small bi-directional , and smaller models on sequence labeling tasks . distill BERT into shallower students through knowledge distillation and an additional knowledge transfer of hidden states on multiple intermediate layers. In contrast to these works, we only use knowledge transfer in the pre-training stage and do not require a fine-tuned teacher for task-specific knowledge in the down-stream tasks. Moreover, compared to patient knowledge distillation which transfers knowledge for all intermediate layers simultaneously to alleviate over-fitting in down-stream task fine-tuning, we design a novel progressive knowledge transfer which eases the pre-training of our compact MobileBERT. The pre-training of BERT is challenging. This problem becomes more severe when we pre-train a compact BERT model from scratch . To tackle this problem, we propose a bottom-to-top progressive knowledge transfer scheme. Specifically, we first train a wider teacher network that is easier to optimize, and then progressively train the student network from bottom to top, requiring it to mimic the teacher network layer by layer. In our algorithm, the student and the teacher can be any multi-head attention encoder such as Transformer , BERT or XLNet. We take BERT as an example in the following description. The progressive knowledge transfer is divided into L stages, where L is the number of layers. Figure 2 illustrates the diagram and algorithm of progressive knowledge transfer. The idea of progressive transfer is that when training the (+1) th layer of the student, the th layer is already well-optimized. As there are no soft target distributions that can be used for the intermediate states of BERT, we propose the following two knowledge transfer objectives, i.e., feature map transfer and attention transfer, to train the student network. Particularly, we assume that the teacher and the student have the same 1) feature map size, 2) the number of layers, and 3) the number of attention heads. Since each layer in BERT merely takes the output of the previous layer as input, the most important thing in progressively training the student network is that the feature maps of each layer should be as close as possible to those of the teacher, i.e., well-optimized. In particular, the mean squared error between the normalized feature maps of the student and the teacher is used as the objective: where is the index of layers, T is the sequence length, and N is the feature map size. The layer normalization is added to stabilize the layer-wise training loss. We also minimize two statistics discrepancies on mean and variance in feature map transfer: where µ and σ 2 represents mean and variance, respectively. Our empirical studies show that minimizing the statistics discrepancy is helpful when layer normalization is removed from BERT to reduce inference latency (see more discussions in Section 4.3). The attention mechanism greatly boosts the performance of NLP and becomes a crucial building block in Transformer and BERT. Many papers words. This motivates us to use self-attention maps from the well-optimized teacher to help the training of the student in augmentation to the feature map transfer. In particular, we minimize the KL-divergence between the per-head self-attention distributions of the teacher and the student: where A is the number of attention heads. Our final progressive knowledge transfer loss L P KT for the th stage is a linear combination of the objectives stated above. As shown in the right panel of Figure 2, we progressively train each layer of the student by minimizing the knowledge transfer loss. In other words, when we train the th layer, we freeze all the trainable parameters in the layers below. We can somewhat soften the training process as follows. When training a layer, we further tune the lower layers with a small learning rate rather than entirely freezing them. Freezing the lower layers can be regarded as a special case of this softened process with the learning rate being zero. There is no knowledge transfer for the beginning embedding layer and the final classifier. They are are the same for the student and teacher. After the progressive knowledge transfer, we further pre-train MobileBERT until convergence. We use a linear combination of the original masked language modeling (MLM) loss, next sentence prediction (NSP) loss, and the new knowledge distillation loss as our pre-training distillation loss: where [N] is the set of masked tokens, P tr (i) and P st (i) are two predicted distributions respectively from the teacher and student model on the masked tokens, and α is a hyperparameter in. We do not perform knowledge distillation on the next sentence prediction (NSP) task as it has been shown to be unimportant. In this section, we present the MobileBERT architecture and the underlining design principle, i.e., how to exploit the benefits of the proposed progressive knowledge transfer. MobileBERT is a much slimmed version of BERT LARGE. As illustrated in Figure 1 (c), to align its feature maps with the teacher's, it is augmented with the bottleneck modules , which have additional shortcut connections outside the original non-linear modules. Through the bottleneck modules, MobileBERT can increase the dimension of its block outputs by a linear transformation, while decreasing the dimension of its block inputs by another linear transformation. So the intra-block hidden size (hidden size of the original non-linear modules) stays unchanged. Symmetrically, to align with the student's feature maps, we can also place the inverted bottleneck modules in the BERT LARGE teacher (Figure 1b). We refer this variant of BERT LARGE as IB-BERT. Through the inverted bottleneck modules, we can effectively reduce the feature map size of the teacher without losing its representational power. We may either only use bottleneck for the student or only the inverted bottleneck for the teacher to align their feature maps. However, when using both of them, we have a chance to search for a better feature map size for the teacher and student to obtain a more compact student model while not hurting the performance of the teacher. A problem introduced by the bottleneck structure of MobileBERT is that the balance between selfattentions and feed-forward networks is broken. In original BERT, the ratio of the parameter numbers in self-attentions and feed-forward networks is always 1:2. But in the bottleneck structure, the inputs to the self-attentions are from wider feature maps (of inter-block size), while the inputs to the feed-forward networks are from narrower bottlenecks (of intra-block size). This in that the self-attentions in MobileBERT will contain more parameters than normally. Therefore, we propose to use stacked feed-forward networks in MobileBERT to re-balance it. As illustrated in 1(c), each MobileBERT layer contains one self-attention but several stacked feed-forward networks. By model latency analysis 1, we find that layer normalization and gelu activation accounted for a considerable proportion of total latency. Therefore, we replace them with new operations in our MobileBERT. Remove layer normalization We replace the layer normalization of a n-channel hidden state h with an element-wise linear transformation: where γ, β ∈ R n and • denotes the Hadamard product. Please note that NoNorm has different properties from LayerNorm even in test mode since the original layer normalization is not a linear operation for a batch of vectors. Use relu activation We replace the gelu activation with simpler relu activation. We conduct extensive experiments to search good model settings for the IB-BERT teacher and the MobileBERT student. We replace the original embedding table by a 3-convolution from a smaller embedding table with embedding size 128 to keep the number of embedding parameters in different model settings the same. We start with SQuAD v1.1 dev F1 score as the metric to measure the performance of different model settings. Since BERT pre-training is time and resource consuming, in the architecture search stage, we only train each model for 125k steps with 2048 batch size, which halves the training schedule of original BERT . Architecture Search of the Teacher As shrinking the inter-block size can effectively compress the model while maintaining its representational power , our design philosophy for the teacher model is to use as small inter-block hidden size (feature map size) as possible as long as there is no accuracy loss. Under this guideline, we design experiments to manipulate the inter-block size of a BERT LARGE -sized IB-BERT, and the are shown in the left panel of Table 1 with labels (a)-(e). As can be seen, decreasing the inter-block hidden size doesn't damage the performance of BERT until the inter-block size is smaller than 512. As a , we choose the IB-BERT LARGE with its inter-block hidden size being 512 as the teacher model. One may wonder whether we can also shrink the intra-block hidden size of the teacher, as this may bridge the gap between the student and teacher . We conduct experiments and the are shown in the left panel of Table 1 with labels (f)-(i). We can see that when the intra-block hidden size is reduced, the model performance is dramatically worse. This means that the intra-block hidden size, which represents the representation power of non-linear modules, plays a crucial role in BERT. Therefore, unlike the inter-block hidden size, we do not shrink the intrablock hidden size of our teacher model. Besides, by comparing (a) and (f) in Table 1, we can see that reducing the number of heads from 16 to 4 does not harm the performance of BERT. This is in line with the observation in the recent literature . Architecture Search of the Student We seek a compression ratio of 4× for BERT BASE, so we design a set of MobileBERT models all with approximately 25M parameters but different ratios of the parameter numbers in MHA and FFN to select a good student model. The right part of Table 1 shows our experimental . They have different balances between self-attentions and feedforward networks. From the table, we can see that the model performance reaches the peak when the ratio of parameters in MHA and FFN is 0.4 ∼ 0.6. This may justify why the original Transformer chooses the parameter ratio of self-attention and feed-forward networks to 0.5. We choose the architecture with 128 intra-block hidden size and 4 stacked FFNs as the student model in consideration of model accuracy and training efficiency. We also accordingly set the number of attention heads in the teacher model to 4 in preparation for the progressive knowledge transfer. Table 2 demonstrates the model settings of our IB-BERT LARGE teacher and MobileBERT student. Following BERT , we use the BooksCorpus and English Wikipedia as our pre-training data. To make the IB-BERT LARGE teacher reach the same accuracy as original BERT LARGE, we train IB-BERT LARGE on 256 TPU v3 chips for 500k steps with a batch size of 4096 and LAMB optimizer . For MobileBERT, we also use the same training schedule. Besides, progressive knowledge transfer of MobileBERT over 24 layers takes 240k steps, so that each layer of MobileBERT is trained for 10k steps. For the downstream tasks, all reported are obtained by simply fine-tuning MobileBERT just like what the original BERT does. To fine-tune the pre-trained models, we search the optimization hyperparameters in a search space including different batch sizes (16/32/48), learning rates ( * e-5), and the number of epochs. The search space is different from the original BERT because we find that MobileBERT usually needs a larger learning rate and more training epochs in fine-tuning. We select the model for testing according to their performance on the development (dev) set. The General Language Understanding Evaluation (GLUE) benchmark is a collection of 9 natural language understanding tasks. We briefly describe these tasks in Appendix F. Following BERT , we use the final hidden vector corresponding to the first input token as model output, and introduced a new linear classification layer for the final predictions. We submit the predictions of MobileBERT and MobileBERT without operational optimizations to the online test evaluation system 2 of GLUE to get the test . We compare MobileBERT with BERT BASE and a few other state-of-the-art pre-BERT models on the GLUE leaderboard: OpenAI GPT and ELMo . We also compare with a recent work on compressing BERT: BERT-PKD . The are listed in Table 3. 3 We can see that our MobileBERT is quite competitive with the original BERT BASE. It outperforms BERT BASE a bit on QNLI and RTE tasks, while the overall GLUE score performance gap is only 0.6. Moreover, It outperform the strong OpenAI GPT baseline by 0.8 GLUE score with 4.3× smaller model size. We also find that the introduced operational optimizations hurt the model performance a bit. Without these optimizations, MobileBERT can even outperform BERT BASE by 0.2 GLUE score. SQuAD is a large-scale reading comprehension datasets. SQuAD1.1 only contains questions that always have an answer in the given context, while SQuAD2.0 contains unanswerable questions. Following BERT , we treat questions that do not have an answer as having an answer span with start and end at the sentence classification token to fine-tune a MobileBERT on SQuAD2.0. We evaluate MobileBERT only on the SQuAD dev datasets, as there is nearly no single model submission on SQuAD test leaderboard 4. We compare our MobileBERT with BERT BASE and a strong baseline DocQA . As shown in Table 4, MobileBERT outperforms a large margin over BERT BASE and DocQA. We notice that MobileBERT also outperforms BERT BASE on QNLI, a question-answering GLUE task. This may be due to that since we search the model settings on SQuAD, MobileBERT may be over-fitted to question answering tasks. We perform an ablation study to investigate how each component of MobileBERT contributes to its performance on the dev data of a few GLUE tasks with diverse characteristics. To accelerate the experiment process, we halve the original pre-training schedule in the ablation study. We conduct a set of ablation experiments with regard to Attention Transfer (AT), Feature Map Transfer (FMT) and Pre-training Distillation (PD). The operational OPTimizations (OPT) are removed in these experiments. Moreover, to investigate the effectiveness of the proposed novel architecture of MobileBERT, we compare MobileBERT with two compact BERT models from. For a fair comparison, we also design our own BERT baseline BERT SMALL*, which is the best model setting we can find with roughly 25M parameters under the original BERT architecture. The detailed model setting of BERT SMALL* can be found in Table 2. Besides these experiments, to verify the performance of MobileBERT on real-world mobile devices, we export the models with Tensorflow Lite 5 APIs and measure the inference latencies on a single large core of a Pixel 3 phone with a fixed sequence length of 128. The are listed in Table 5. We first can see that the propose Feature Map Transfer contributes most to the performance improvement of MobileBERT, while Attention Transfer and Pre-training Distillation also play positive roles. As expected, the proposed operational OPTimizations hurt the model performance a bit, but it brings a crucial speedup of 1.68×. In architecture comparison, we find that although specifically designed for progressive knowledge transfer, our MobileBERT architecture alone is still quite competitive. It outperforms BERT SMALL * and BERT SMALL on all compared tasks, while outperforming the 1.7× sized BERT MEDIUM on the SST-2 task. Finally, we can L1 H1 L1 H2 L1 H3 L1 H4 L12 H1 L12 H2 L12 H3 L12 H4 MobileBERT (find that although augmented with the powerful progressive knowledge transfer, our MobileBERT still degrades greatly when compared to the IB-BERT LARGE teacher. We visualize the attention distributions of the 1 st and the 12 th layers of a few models in Figure 3 for further investigation. The proposed attention transfer can help the student mimic the attention distributions of the teacher very well. Surprisingly, we find that the attention distributions in the attention heads of "MobileBERT(bare)+PD+FMT" are exactly a re-order of those of "Mobile-BERT(bare)+PD+FMT+AT" (also the teacher model), even if it has not been trained by the attention transfer objective. This phenomenon indicates that multi-head attention is a crucial and unique part of the non-linearity of BERT. Moreover, it can explain the minor improvements of Attention Transfer in ablation table 5, since the alignment of feature maps lead to the alignment of attention distributions. We have presented MobileBERT which is a task-agnostic compact variant of BERT. It is built upon a progressive knowledge transfer method and a conjugate architecture design. Standard model compression techniques including quantization and pruning can be applied to MobileBERT to further reduce the model size as well as the inference latency. In addition, although we have utilized low-rank decomposition for the embedding layer, it still accounts for a large part in the final model. We believe there is a big room for extremely compressing the embedding table . Layer-wise pre-training of neural networks can be dated back to Deep Belief Networks (DBN) and stacked auto-encoders . showed that the unsupervised pre-training of DBN helps to mitigate the difficult optimization problem of deep networks by better initializing the weights of all layers. Although they made essential breakthrough in the application of neural networks, they are widely considered to be obsolete. A more popular way today is to train deep neural networks in an end-to-end fashion. recently showed that end-to-end learning can sometimes be very inefficient. In this paper, we propose a progressive knowledge transfer scheme to combine the best of both worlds. Compared to previous layer-wise methods, we use a well-optimized wider teacher to guide the layer-wise pre-training of the narrower student, rather than a greedy layer-wise unsupervised way, which makes better use of labels and rewards. Our method also tackle the difficult training problem of end-to-end training from scratch. While much recent research has focused on improving efficient Convolutional Neural Networks (CNN) for mobile vision applications (; ; ; ;, they are usually tailored for CNN. Popular lightweight operations such as depth-wise convolution cannot be directly applied to Transformer or BERT. In the NLP literature, the most relevant work can be group LSTMs , which employs the idea of group convolution (; into Recurrent Neural Networks (RNN). Recently, compressing or accelerating Transformer or BERT has attracted much attention. apply Block-Term Tensor Decomposition on the self-attention modules of Transformer and achieve a compression of 2.5 on the machine translation task, but they don't consider how to compress the feed-forward networks, which constrains the compression ratio. use structured memory layers to replace feed-forward networks in BERT and get better perplexity by half the computation, but they cannot compress the model size. Compared to these work, Mobile-BERT reduces overheads in both self-attentions and feed-forward networks of BERT by bottleneck structure, while achieves efficiency with regard to both storage and computation. We evaluate the effectiveness of our two operational optimizations for MobileBERT introduced in Section 4.3: replacing layer normalization (LayerNorm) with NoNorm and replacing gelu activation with relu activation. We use the same experimental setting as in Section 5.5, where the models are exported to Tensorflow Lite format and evaluated on a single large core of a Pixel 3 phone with a fixed sequence length of 128. From Table 6, we can see that both NoNorm and relu are very effective in reducing the latency of MobileBERT, even if these two operational optimizations do not reduce FLOPS. This reveals the gap between the real-world inference latency and the theoretical computation overhead (i.e., FLOPS). (λ, µ, β, γ) are hyperparameters to balance the different loss terms. Specifically, we use λ = 1, µ = 100, β = 5000, γ = 5 in our all experiments. Pre-train MobileBERT For a fair comparison with original BERT, we follow the same preprocessing scheme as BERT, where we mask 15% of all WordPiece tokens in each sequence at random and use next sentence prediction. Please note that MobileBERT can be potentially further improved by several training techniques recently introduced, such as span prediction or removing next sentence prediction objective. We leave it for future work. In pre-training distillation, the hyperparameter α is used to balance the original masked language modeling loss and the distillation loss. Following , we set α to 0.5. We notice that recently there is an unpublished work 6 that also propose a task-agnosticly compressed BERT, called DistilBERT. Basically, DistilBERT is a 6-layer truncated BERT BASE, which is distilled from BERT BASE on unannotated data with masked language modeling target. The distillation process of DistilBERT is quite similar to the pre-training distillation described in Section 3.4. In comparison, in this paper, we propose a pair of conjugate architectures to help knowledge transfer and design a progressive knowledge transfer scheme which transfers the intrinsic knowledge of intermediate layers from the teacher to the student in a bottom-to-top progressive way. In this section, we provide a brief description of the tasks in the GLUE benchmark . CoLA The Corpus of Linguistic Acceptability is a collection of English acceptability judgments drawn from books and journal articles on linguistic theory. The task is to predict whether an example is a grammatical English sentence and is evaluated by Matthews correlation coefficient . SST-2 The Stanford Sentiment Treebank is a collection of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence and is evaluated by accuracy. MRPC The Microsoft Research Paraphrase Corpus is a collection of sentence pairs automatically extracted from online news sources. They are labeled by human annotations for whether the sentences in the pair are semantically equivalent. The performance is evaluated by both accuracy and F1 score. The Semantic Textual Similarity Benchmark is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair is human-annotated with a similarity score from 1 to 5. The task is to predict these scores and is evaluated by Pearson and Spearman correlation coefficients.

Represent the short summary for finding the corresponding scientific article (long)

We develop a task-agnosticlly compressed BERT, which is 4.3x smaller and 4.0x faster than BERT-BASE while achieving competitive performance on GLUE and SQuAD.

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.


Question: Represent the article for finding a single-sentence TLDR

Question: The importance weighted autoencoder (IWAE) is a popular variational-inference method which achieves a tighter evidence bound (and hence a lower bias) than standard variational autoencoders by optimising a multi-sample objective, i.e. an objective that is expressible as an integral over $K > 1$ Monte Carlo samples. Unfortunately, IWAE crucially relies on the availability of reparametrisations and even if these exist, the multi-sample objective leads to inference-network gradients which break down as $K$ is increased . This breakdown can only be circumvented by removing high-variance score-function terms, either by heuristically ignoring them (which yields the 'sticking-the-landing' IWAE (IWAE-STL) gradient from ) or through an identity from (which yields the 'doubly-reparametrised' IWAE (IWAE-DREG) gradient). In this work, we argue that directly optimising the proposal distribution in importance sampling as in the reweighted wake-sleep (RWS) algorithm from is preferable to optimising IWAE-type multi-sample objectives. To formalise this argument, we introduce an adaptive-importance sampling framework termed adaptive importance sampling for learning (AISLE) which slightly generalises the RWS algorithm. We then show that AISLE admits IWAE-STL and IWAE-DREG (i.e. the IWAE-gradients which avoid breakdown) as special cases. Let x be some observation and let z be some latent variable taking values in some space Z. These are modeled via the generative model p θ (z, x) = p θ (z)p θ (x|z) which gives rise to the marginal likelihood p θ (x) = Z p θ (z, x) dz of the model parameters θ. In this work, we analyse algorithms for variational inference, i.e. algorithms which aim to 1. learn the generative model, i.e. find a value θ which is approximately equal to the maximum-likelihood estimate (MLE) θ ml:= arg max θ p θ (x); 2. construct a tractable variational approximation q φ,x (z) of p θ (z|x) = p θ (z, x)/p θ (x), i.e. find the value φ such that q φ,x (z) is as close as possible to p θ (z|x) in some suitable sense. A few comments about this setting are in order. Firstly, as is common in the literature, we restrict our presentation to a single latent representation-observation pair (z, x) to avoid notational clutter -the extension to multiple independent observations is straightforward. Secondly, we assume that no parameters are shared between the generative model p θ (z, x) and the variational approximation q φ,x (z). This is common in neural-network applications but could be relaxed. Thirdly, our setting is general enough to cover amortised inference. For this reason, we often refer to φ as the parameters of an inference network. Two main classes of stochastic gradient-ascent algorithms for optimising ψ:= (θ, φ) which employ K ≥ 1 Monte Carlo samples ('particles') to reduce errors have been proposed. heuristically drops the problematic score-function terms from the IWAE φ-gradient. This induces bias for the IWAE objective. -IWAE-DREG. The'doubly-reparametrised' IWAE (IWAE-DREG) φ-gradient unbiasedly removes the problematic score-function terms from the IWAE φ-gradient using a formal identity. • RWS. The reweighted wake-sleep (RWS) algorithm optimises two separate objectives for θ and φ. Its gradients are approximated by self-normalised importance sampling with K particles: this induces a bias which vanishes as K → ∞. RWS can be viewed as an adaptive importance-sampling approach which iteratively improves its proposal distribution while simultaneously optimising θ via stochastic approximation. Crucially, the RWS φ-gradients do not degenerate as K → ∞. Of these two methods, the IWAE is the most popular and demonstrated empirically that RWS can break down, conjecturing that this is due to the fact that RWS does not optimise a joint objective (for θ and φ). Meanwhile, the IWAE-STL gradient performed consistently well despite lacking a firm theoretical footing. Yet, IWAE suffers from the above-mentioned φ-gradient breakdown and exhibited inferior empirical performance to RWS . Thus, it is not clear whether the multi-sample objective approach of IWAE or the adaptive importance-sampling approach of RWS is preferable. In this work, we show that directly optimising the proposal distribution, e.g. as done by RWS, is preferable to optimising the IWAE multi-sample objective because (a) the multi-sample objective typically relies on reparametrisations and, even if these are available, leads to the φ-gradient breakdown, (b) modifications of the IWAE φ-gradient which avoid this breakdown (i.e. IWAE-STL and IWAE-DREG) can be justified in a more principled manner by taking an RWS-type adaptive importance-sampling view. This was already reached by based on numerical experiments. They demonstrated that the need for reparametrisations can make IWAE inferrior to RWS e.g. for discrete latent variables. Our work complements theirs by formalising this argument. To this end, we slightly generalise the RWS algorithm to obtain a generic adaptive importance-sampling framework for variational inference which we term adaptive importance sampling for learning (AISLE) for ease of reference. We then show that AISLE admits not only RWS but also the IWAE-DREG and IWAE-STL gradients as special cases. Novel material is presented in Section 3, where we introduce the AISLEframework. From this, most of the previously proposed gradient estimators can be naturally derived in a principled manner. Importantly, the derived gradient estimators are guaranteed to not degenerate as K → ∞. Specifically, we establish the following connections. • We prove that the IWAE-STL gradient can be recovered as a special case of AISLE via a principled and novel application of the'double-reparametrisation' identity from. This indicates that the breakdown of RWS observed in may not be due to its lack of a joint objective as previously conjectured (since IWAE-STL avoided this breakdown despite having the same idealised objective as RWS). Our work also provides a theoretical foundation for IWAE-STL which was hitherto only heuristically justified as a biased IWAE-gradient. • We prove that AISLE also admits the IWAE-DREG gradient as a special case. Our derivation also makes it clear that the learning rate should be scaled as O(K) for the IWAE φ-gradient (and its modified version IWAE-DREG) unless the gradients are normalised as implicitly done by popular optimisers such as ADAM . In contrast, the learning rate for AISLE need not be scaled up with of K. • When applied to the family of α-divergences, AISLE leads to a new family of gradient estimators that generalises some previously derived in the literature. • In the supplementary materials, we provide insights into the impact of the selfnormalisation bias on some of the importance-sampling based gradient approxima-tions (Appendix A) and empirically compare the main algorithms discussed in this work (Appendix B). We stress that the focus of our work is not necessarily to derive new algorithms nor to establish which of the various special cases of AISLE is preferable. Indeed, while we compare all algorithms discussed in this work empirically on Gaussian models in the supplementary materials, we refer the reader to; for an extensive empirical comparisons of all the algorithms discussed in this work. Notation. We repeatedly employ the shorthand p(f): To keep the notation concise, we hereafter suppress dependence on the observation x, i.e. we write q φ (z):= q φ,x (z) as well as where γ θ (z):= p θ (z, x) and where 2 Background The expectation q φ (f) of a test function f: Z → R can be unbiasedly estimated by the φ, which are independent and identically distributed (IID) according to q φ. Similarly, expectations of the type π θ (f) can be approximated by the self-normalised importance sampling estimatê The notation φ, z stresses the dependence of this estimator on φ and z. The quantity w ψ (z k) are called the kth importance weight and s w k ψ is its self-normalised version. For readability, we have dropped the dependence of s w k ψ on z ∈ Z K from the notation. Remark 2. The self-normalised estimateπ θ φ, z (f) is typically not unbiased. Under mild assumptions (e.g. if sup w ψ < ∞), its bias vanishes at rate O(K −1), its standard deviation vanishes at Monte-Carlo rate Objective. The importance weighted autoencoder (IWAE), introduced by , seeks to find a value θ of the generative-model parameters θ which maximises a lower bound L K ψ on the log-marginal likelihood ('evidence'). This bound depends on the inference-network parameters φ and the number of samples, K ≥ 1: where the expectation is w.r.t. z ∼ q ⊗K φ. For any finite K, optimisation of the inferencenetwork parameters φ tightens the evidence bound. prove that for any φ we have that L K ψ ↑ log Z θ as K → ∞. If K = 1, the IWAE reduces to the variational autoencoder (VAE) from. However, for K > 1, as pointed out in; , the IWAE also constitutes another VAE on an extended space based on an auxiliary-variable construction developed in;; Lee (see, e.g. , for a review). The gradient of the IWAE objective from: The intractable quantity E G ψ (z) can be approximated unbiasedly via a vanilla Monte Carlo approach using a single (Unfortunately, this approximation typically has such a large variance that it becomes impracticably noisy . To remove this high-variance term, the well known reparametrisation trick is usually employed. It requires the following assumption. (R1) There exists a distribution q on some space E and a family of differentiable mappings, the gradient can be expressed as Here, the notation ψ indicates that one does not differentiate w ψ w.r.t. ψ. The IWAE then uses a vanilla Monte Carlo estimate of, Before proceeding, we state the following lemma, proved in Tucker et al. (2019, Section 8.1), which generalises of the well-known identity q φ (∇ φ log q φ) = 0. Lemma 1 . Under R1, for suitably integrable f ψ: Z → R, we have We now exclusively focus on the φ-portion of the IWAE gradient, ∇ iwae φ θ, z. Remark 3 (drawbacks of the IWAE φ-gradient). The gradient ∇ iwae φ θ, z has three drawbacks. The last two of these are attributable to the'score-function' terms ∇ φ log q φ (z) in the φ-gradient portion of. • Reliance on reparametrisations. A reparametrisation à la R1 is necessary to remove the high-variance term G ψ (z). For, e.g. discrete, models that violate R1, control-variate approaches or continuous relaxations have been proposed but these incur additional implementation, tuning and computation costs whilst not necessarily reducing the variance . • Vanishing signal-to-noise ratio. The φ-gradient breaks down in the sense that its signal-to-noise ratio vanishes as ). This is because ∇ iwae φ θ, z constitutes a self-normalised importance-sampling approximation of π θ (ψ − ∇ φ log q φ) = 0, an identity which directly follows from Lemma 1 with f ψ = w ψ. • Inability to achieve zero variance. As pointed out in , Two modifications of ∇ iwae φ θ, z have been proposed which (under R1) avoid the scorefunction terms in and hence (a) exhibit a stable signal-to-noise ratio as K → ∞ and (b) can achieve zero variance if q φ = π θ (because then ψ ≡ 0 since w ψ is constant). • IWAE-STL. The'sticking-the-landing' IWAE (IWAE-STL) gradient proposed by heuristically ignores the score function terms, As shown in ), this introduces an additional bias whenever K > 1. • IWAE-DREG. The'doubly-reparametrised' IWAE (IWAE-DREG) gradient proposed by removes the score-function terms through Lemma 1, The quantities ∇ iwae-dreg φ θ, z and ∇ iwae φ φ, z are equal in expectation. The reweighted wake-sleep (RWS) algorithm was proposed in. The θ-and φ-gradients read These quantities are usually intractable and therefore approximated by replacing π θ by the self-normalised importance sampling approximationπ θ φ, z (this does not require R1): Since relies on self-normalised importance sampling, Remark 2 shows that its bias relative to is of order O(1/K). Appendix A discusses the impact of this bias on the φ-gradient in more detail. The optimisation of both θ and φ is carried out simultaneously, allowing both gradients to share the same particles and weights. Nonetheless, the lack of a joint objective (for both θ and φ) is often viewed as the main drawback of RWS. rws φ θ, z in expectation and is derived by applying Lemma 1 to the latter. It reads where the function F(w):= w(1 − w) is used to transform the self-normalised importance weights s w k ψ. In high-dimensional settings, it is typically the case that the ordered selfnormalised importance weights s w are then mainly supported on the two particles with the largest self-normalised weights. and φ simultaneously is that (a) Monte Carlo samples used to approximate the θ-gradient can be re-used to approximate the φ-gradient and (b) optimising φ typically reduces the error (both in terms of bias and variance) of the θ-gradient approximation. However, adapting the proposal distribution q φ in importance-sampling schemes need not necessarily be based on minimising the (inclusive) KL-divergence. Numerous other techniques exist in the literature (e.g. ; ; ; ;) and may sometimes be preferable. Indeed, another popular approach with strong theoretical support is based on minimising the χ 2 -divergence (see, e.g., Deniz Akyildiz & Míguez, 2019). Based on this insight, we slightly generalise the RWS-objective as θ:= arg max θ log Z θ, φ:= arg min φ Dƒ(πθ q φ). Here, Dƒ(p q):= Z ƒ(p(z)/q(z))q(z) dz is some ƒ-divergence from p to q. We reiterate that alternative approaches for optimising φ (which do not minimise ƒ-divergences) could be used. However, we state for concreteness as it suffices for the remainder of this work; we call the ing algorithm adaptive importance sampling for learning (AISLE). As will become clear below, this unified framework permits a straightforward and principled derivation of robust φ-gradient estimators that do not degenerate as K → ∞. Optimisation is again performed via a stochastic gradient-ascent. The intractable θ-gradient The θ-gradient is thus the same for all algorithms discussed in this work although the IWAEparadigm views it as an unbiased gradient of a (biased) lower-bound to the evidence, while AISLE (and RWS) interpret it as a self-normalised importance-sampling (and consequently biased) approximation of the gradient ∇ θ log Z θ for the'exact' objective. In the derivations to follow, integrals of the form π θ ([F • w ψ]∇ φ log q φ ) naturally appear. These can also be expressed as Z Approximating the expectation as well as the normalising constant Z θ on the r.h.s. with the vanilla Monte Carlo method with Remark 2 shows that this approximation has a bias of order O(K −1) and a standarddeviation of order O(K −1/2). Now, most of the ƒ-divergences used for variational inference in intractable models are such that there exists a functionf: for an exponent κ ∈ R and constant C(θ) independent of φ. In other words, for a given value of θ, the optimization of the ƒ-divergence as a function of φ can be carried out without relying on the knowledge of Z θ. Writing g(y):=f (y) −f (y)/y, simple algebra then directly shows that Since the integral in is an expectation with respect to π θ, it can be approximated with selfimportance sampling, possibly multiplied an additional importance-sampling approximation Z θ φ, z of Z θ raised to some power. This leads to, Indeed, Equation applies to, leading to the reparametrised estimator where h(y) = g(y)y and g: R → R given immediately above. We now describe several particular cases. We have KL(In that case, with the notations of Section 3.3.1, we have g(y) = 1 and h (y) = 1. • AISLE-KL-NOREP/RWS. Without relying on any reparametrisation, Equation yields the following gradient, which clearly equals ∇ rws φ θ, z: • AISLE-KL. Using reparametrisation, Equation yields the gradient: We thus arrive at the following which demonstrates that IWAE-STL can be derived in a principled manner from AISLE, i.e. without the need for a multi-sample objective. θ, z. Proposition 1 is notable because it shows that IWAE-STL (which avoids the breakdown highlighted in and which can also achieve zero variance) can be derived in a principled manner from AISLE, i.e. without relying on a multi-sample objective. Proposition 1 thus provides a theoretical basis for IWAE-STL which was previously viewed as an alternative gradient for IWAE for which it is biased and only heuristically justified. Furthermore, the fact that IWAE-STL exhibited good empirical performance in even in an example in which RWS broke down, suggests that this breakdown may not be due to RWS' lack of optimising a joint objective as previously conjectured. by first replacing the exact (but intractable) φ-gradient by the self-normalised importance-sampling approximation ∇ rws φ θ, z and then applying the identity from Lemma 1. Note that this may in a variance reduction but does not change the bias of the gradient estimator. In contrast, AISLE-KL is derived by first applying Lemma 1 to the exact (RWS) φ-gradient and then approximating the ing expression. This can potentially reduce both bias and variance. Up to some irrelevant additive constant, the α-divergence between two distributions p and q is given by Z (p(z)/q(z)) α q(z) dz for some α > 1. This can also be expressed as Z κ θ Zf (w ψ (z))q φ (z) dz with κ = −α andf (y) = y α. In this case, with the notation from Section 3.3.1, we have g(y) = (α − 1)y α−1 and h (y) = α(α − 1) y α−1. Note that the case α = 2 is equivalent, up to an irrelevant additive constant, to a standard χ 2 -divergence. Minimising this divergence is natural in importance sampling since χ 2 (π θ q φ) = var z∼q φ [w ψ /Z θ] is the variance of the importance weights. • AISLE-α-NOREP. Without relying on any reparametrisation, Equation yields with the following special case which is also proportional to the'score gradient' from Dieng et al. (2017, Appendix G): • AISLE-α. Using reparametrisation, Equation becomes again with the special case ∇ This demonstrates that IWAE-DREG can be derived (up to the proportionality factor 2K) in a principled manner from AISLE, i.e. without the need for a multi-sample objective. θ, z. Note that if the implementation normalises the gradients, e.g. as effectively done by ADAM , the constant factor cancels out and AISLE-χ 2 becomes equivalent to IWAE-DREG. Otherwise (e.g. in plain stochastic gradient-ascent) this shows that the learning rate needs to be scaled as O(K) for the IWAE or IWAE-DREG φ-gradients. For the'exclusive' KL-divergence, we have KL(q φ π θ) = f (w ψ (z))q φ (z) dz + C(θ) with f (y) = log(y). In that case, with the notation from Section 3.3.1, we have h (y) = 1/y. This directly leads to the following approximation, This can be recognised as a simple average over K independent replicates of the'stickingthe-landing' estimator for VAEs proposed in Roeder et al. (2017, Equation 8). As we discuss in Appendix A, optimising this'exclusive' KL-divergence can sometimes lead to faster convergence of φ than optimising the'inclusive' KL-divergence KL(π θ q φ). However, care must be taken because minimising the exclusive divergence does not necessarily lead to well behaved or even well-defined importance weights and thus can negatively affect learning of θ (whose gradient is a self-normalised importance-sampling approximation which makes use of those weights). We have shown that the adaptive-importance sampling paradigm of the reweighted wake-sleep (RWS) is preferable to the multi-sample objective paradigm of importance weighted autoencoders (IWAEs) because the former achieves all the goals of the latter whilst avoiding its drawbacks. A On the rôle of the self-normalisation bias within RWS/AISLE Within the self-normalised importance-sampling approximation, the number of particles, K, interpolates between two extremes: • As K ↑ ∞,π θ φ, z (f) becomes an increasingly accurate approximation of π θ (f). • For K = 1, however,π θ φ, z (f) = f (z 1) reduces to a vanilla Monte Carlo approximation of q φ (f) (because the single self-normalised importance weight is always equal to 1). This leads to the following insight about the estimators ∇ aisle-kl φ θ, z and ∇ aisle-χ 2 φ θ, z. • As K ↑ ∞, these two estimators become increasingly accurate approxi-, respectively. • For K = 1, however, these two estimators reduce to vanilla Monte Carlo ap- This is similar to the standard IWAE φ-gradient which also represents a vanilla Monte Carlo approximation of −∇ φ KL(q φ π θ) if K = 1 as IWAE reduces to a VAE in this case. Characterising the small-K self-normalisation bias of the reparametrisation-free AISLE φ gradients, AISLE-KL-NOREP and AISLE-χ 2 -NOREP, is more difficult because if K = 1, they constitute vanilla Monte Carlo approximations of q φ (∇ φ log q φ) = 0. Nonetheless, Le et al. (2019, Figure 5) lends some support to the hypothesis that the small-K self-normalisation bias of these gradients also favours a minimisation of the exclusive KL-divergence. Recall that the main motivation for use of IWAEs (instead of VAEs) was the idea that we could use self-normalised importance-sampling approximations with K > 1 particles to reduce the bias of the θ-gradient relative to ∇ θ log Z θ. The error of such (self-normalised) importance-sampling approximations can be controlled by ensuring that q φ is close to π θ (in some suitable sense) in any part of the space Z in which π θ has positive probability mass. For instance, it is well known that the error will be small if the'inclusive' KL-divergence KL(π θ q φ) is small as this implies well-behaved importance weights. In contrast, a small'exclusive' KL-divergence KL(q φ π θ) is not sufficient for well-behaved importance weights because the latter only ensures that q φ is close to π θ in those parts of the space Z in which q φ has positive probability mass. Let Q:= {q φ} (which is indexed by φ) be the family of proposal distributions/the variational family. Then we can distinguish two scenarios. 1. Sufficiently expressive Q. For the moment, assume that the family Q is flexible ('expressive') enough in the sense that it contains a distribution q φ which is (at least approximately) equal to π θ and that our optimiser can reach the value φ of φ. In this case, minimising the exclusive KL-divergence can still yield well-behaved importance weights because in this case, φ:= arg min φ KL(π θ q φ) is (at least approximately) equal to arg min φ KL(q φ π θ). 2. Insufficiently expressive Q. In general, the family Q is not flexible enough in the sense that all of its members are'far away' from π θ, e.g. if the is fully factorised. In this case, minimising the exclusive KL-divergence could lead to poorly-behaved importance weights and we should optimise φ:= arg min φ KL(π θ q φ) as discussed above. Remark 4. In Scenario 1 above, i.e. for a sufficiently flexible Q, using a gradient-descent algorithm which seeks to minimise the exclusive divergence can sometimes be preferable to a gradient-descent algorithm which seeks to minimise the inclusive divergence. This is because both find (approximately) the same optimum but the latter may exhibit faster convergence in some applications. In such scenarios, the discussion in Subsection A.1 indicates that a smaller number of particles, K, could then be preferable for some of the φ-gradients because (a) the O(K −1) self-normalisation bias outweighs the O(K −1/2) standard deviation and (b) the direction of this bias may favour faster convergence. Unfortunately, simply setting K = 1 for the approximation of the φ-gradients 2 is not necessarily optimal because • even in the somewhat idealised scenario 1 above and even if the direction of the self-normalisation bias encourages faster convergence, increasing K is still desirable to reduce the variance of the gradient approximations and furthermore, even in this scenario, seeking to optimise the exclusive KL-divergence could lead to poorly behaved importance-sampling approximations of the θ-gradient whenever φ is still far away from optimal; • not using the information contained in all K particles and weights (which have already been sampled/calculated to approximate the θ-gradient) seems wasteful; • if K = 1, the reparametrisation-free AISLE φ-gradients, AISLE-KL-NOREP and AISLE-χ 2 -NOREP are simply vanilla Monte Carlo estimates of 0 and the RWS-DREG φ-gradient is then equal to 0. In these supplementary materials, we illustrate the different φ-gradient estimators (recall that all algorithms discussed in this work share the same θ-gradient estimator). Specifically, we compare the following approximations. • AISLE-KL-NOREP. The gradient for AISLE based on the KL-divergence without any further reparametrisation from i.e. this coincides with the standard RWSgradient from. This gradient does not require R1 but does not achieve zero variance even if q φ = π θ. • AISLE-KL. The gradient for AISLE based on the KL-divergence after reparametrising and exploiting the identity from Lemma 1; it is given by and coincides with the IWAE-STL-gradient from. • AISLE-χ 2 -NOREP. The gradient for AISLE based on the χ 2 -divergence without any reparametrisation given in. This gradient again does not require R1 but does not achieve zero variance even if q φ = π θ. • AISLE-χ 2. The gradient for AISLE based on the χ 2 -divergence after reparametrising and exploiting the identity from Lemma 1; it is given by and is alsow proportional to IWAE-DREG from which was stated in. When normalising the gradients (as, e.g. implicitly done by optimisers such as) the proportionality constant cancels out so that both these gradient approximations lead to computationally the same algorithm. • IWAE. The gradient for IWAE employing the reparametrisation trick from. Its sampling approximation is given in. Recall that this is the φ-gradient whose signal-to-noise ratio degenerates with K as pointed out in (and which also cannot achieve zero variance even if q φ = π θ). • IWAE-DREG. The'doubly-reparametrised' IWAE gradient from which was proposed in. It is proportional to AISLE-χ 2. • RWS-DREG. The'doubly-reparametrised' RWS φ-gradient from Hereafter, wherever necessary, we add an additional subscript to make the dependence on the observations explicit. The joint law (the 'generative model'), parametrised by θ, of the observations and latent variables then factorises as We model each latent variable-observation pair (z, x) as..,D} ∈ R D×D is assumed to be known and where I denotes the D × D-identity matrix. For any θ, with P:= (Σ −1 + I) −1 and ν θ,x:= P (Σ −1 µ + x). In particular, implies that Proposal/variational approximation. We take the proposal distributions as a fullyfactored Gaussian: where. The parameters to optimise are thus where denotes the column vector formed by the elements in the dth row of A. Furthermore, for the reparametrisation trick, we take q:= N(; 0, I), where 0 ∈ R D is a vector whose elements are all 0, so that Note that the mean of the proposal in coincides with the mean of the posterior in if A = P and b This model is similar to the one used as a benchmark in Rainforth et al. (2018, Section 4) and also in Tucker et al. (2019, Section 6.1) who specified both the generative model and the variational approximation to be isotropic Gaussians. Specifically, their setting can be recovered by taking Σ:= I and fixing c d = log(2/3)/2 so that C = 2 3 I throughout. Here, in order to investigate a slightly more realistic scenario, we also allow for the components of the latent vectors z to be correlated/dependent under the generative model. However, as the variational approximation remains restricted to being fully factored, it may fail to fully capture the uncertainty about the latent variables. φ,x (z), we then have Note that the only source of randomness in this expression is the multivariate normal random variable. Thus, by and, for any values of A and b and any K ≥ 1, the variance of the A-and b-gradient portion of AISLE-KL/IWAE-STL and AISLE-χ 2 /IWAE-DREG goes to zero as C → C = 1 2 I. In other words, in this model, these'score-function free' φ-gradients achieve (near) zero variance for the parameters governing the proposal mean as soon as the variance-parameters fall within a neighbourhood of their optimal values. Furthermore, combined with shows that for any K ≥ 1, the variance of the C-gradient portion also goes to zero as (A, b, C) → (A, b, C). A more thorough analysis of the benefits of reparametrisation-trick gradients in Gaussian settings is carried out in. Setup. We end this section by empirically comparing the algorithms from Subsection B.1. We run each of these algorithms for a varying number of particles, K ∈ {1, 10, 100}, and varying model dimensions, D ∈ {2, 5, 10}. Each of these configurations is repeated independently 100 times. Each time using a new synthetic data set consisting of N = 25 observations sampled from the generative model after generating a new'true' prior mean vector as µ ∼ N(0, I). Since all the algorithms share the same θ-gradient, we focus only on the optimisation of φ and thus simply fix θ:= θ ml throughout. We show for the following model settings. • Figure 1. The generative model is specified via Σ = I. In this case, there exists a value φ of φ such that q φ,x (z) = π θ,x (z). Note that this corresponds to Scenario 1 in Subsection A.2. • Figure 2. The generative model is specified via Σ = (0.95 |d−d |+1) (d,d)∈{1,...,D} 2. Note that in this case, the fully-factored variational approximation cannot fully mimic the dependence structure of the latent variables under the generative model. That is, in this case, q φ,x (z) = π θ,x (z) for any values of φ. Note that this corresponds to Scenario 2 in Subsection A.2. To initialise the gradient-ascent algorithm, we draw each component of the initial values φ 0 of φ IID according to a standard normal distribution. We use both plain stochastic gradient-ascent with the gradients normalised to have unit L 1 -norm (Figures 1a, 2a) and ADAM with default parameter values (Figures 1b, The total number of iterations is 10, 000; in each case, the learning-rate parameters at the ith step are i −1/2 . Figure 1 except that here, the covariance matrix Σ = (0.95 |d−e|+1) (d,e)∈{1,...,D} 2 is not a diagonal matrix. Again, note the logarithmic scaling on the second axis.

represent this text to find a much much longer article about it with say hundreds-thousands of words:

We show that most variants of importance-weighted autoencoders can be derived in a more principled manner as special cases of adaptive importance-sampling approaches like the reweighted-wake sleep algorithm.

Represent the TLDR for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.