Question: Represent the article for finding a single-sentence TLDR

Question: The task of visually grounded dialog involves learning goal-oriented cooperative dialog between autonomous agents who exchange information about a scene through several rounds of questions and answers. We posit that requiring agents to adhere to rules of human language while also maximizing information exchange is an ill-posed problem, and observe that humans do not stray from a common language, because they are social creatures and have to communicate with many people everyday, and it is far easier to stick to a common language even at the cost of some efficiency loss. Using this as inspiration, we propose and evaluate a multi-agent dialog framework where each agent interacts with, and learns from, multiple agents, and show that this in more relevant and coherent dialog (as judged by human evaluators) without sacrificing task performance (as judged by quantitative metrics). Intelligent assistants like Siri and Alexa are increasingly becoming an important part of our daily lives, be it in the household, the workplace or in public places. As these systems become more advanced, we will have them interacting with each other to achieve a particular goal BID9. We want these conversations to be interpretable to humans for the sake of transparency and ease of debugging. Having the agents communicate in natural language is one of the most universal ways of ensuring interpretability. This motivates our work on goal-driven agents which interact in coherent language understandable to humans. To that end, this paper builds on work by BID2 on goal-driven visual dialog agents. The task is formulated as a conversation between two agents, a Question (Q-) and an Answer (A-) bot. The A-Bot is given an image, while the QBot is given only a caption to the image. Both agents share a common objective, which is for the Q-Bot to form an accurate mental representation of the unseen image using which it can retrieve, rank or generate that image. This is facilitated by the exchange of 10 pairs of questions and answers between the two agents, using a shared vocabulary. BID2 trained the agents first in isolation via supervision from the VisDial dataset BID1, followed by making them interact and adapt to each other via reinforcement learning by optimizing for better task performance. While trying to maximize performance, the agents learn to communicate in non-grammatical and semantically meaningless sentences in order to maximize the exchange of information. This reduces transparency of the AI system to human observers and is undesirable. We address this problem by proposing a multi-agent dialog framework where each agent interacts with multiple agents. This is motivated by our observation that humans adhere to syntactically and semantically coherent language, which we hypothesize is because they have to interact with an entire community, and having a private language for each person would be extremely inefficient. We show that our multi-agent (with multiple Q-Bots and multiple A-Bots) dialog system in more coherent and human-interpretable dialog between agents, without compromising on task performance, which also validates our hypothesis. This makes them seem more helpful, transparent and trustworthy. We will make our code available as open-source. 1 The game involves two collaborative agents a question bot (Q-bot) and an answer bot (A-bot). The A-Bot is provided an image, I (represented as a feature embedding y gt extracted by, say, a pretrained CNN model BID13), while the Q-Bot is provided with only a caption of the image. The Q-Bot is tasked with estimating a vector representationŷ of I, which is used to retrieve that image from a dataset. Both agents receive a common penalty from the environment which is equal to the error inŷ with respect to y gt. Thus, an unlimited number of games may be simulated without human supervision, motivating the use of reinforcement learning in this framework. Our primary focus for this work is to ensure that the agents' dialog remains coherent and understandable while also being informative and improving task performance. For concreteness, an example of dialog that is informative yet incoherent: question: "do you recognize the guy and age is the adult?", answered with: "you couldn't be late teens, his". The example shows that the bots try to extract and convey as much information as possible in a single question/answer (sometimes by incorporating multiple questions or answers into a single statement). But in doing so they lose basic semantic and syntactic structure. Most of the major works which combine vision and language have traditionally been focusing on the problem of image captioning (( BID7, BID17, BID14, BID5, BID12, BID19) and visual question answering (BID0, BID20, BID4, BID18). The problem of visual dialog is relatively new and was first introduced by BID1 who also created the VisDial dataset to advance the research on visually grounded dialog. The dataset was collected by pairing two annotators on Amazon Mechanical Turk to chat about an image. They formulated the task as a'multi-round' VQA task and evaluated individual responses at each round in an image guessing setup. In a subsequent work by BID2 they proposed a Reinforcement Learning based setup where they allowed the Question bot and the Answer bot to have a dialog with each other with the goal of correctly predicting the image unseen to the Question bot. However, in their work they noticed that the reinforcement learning based training quickly lead the bots to diverge from natural language. In fact recently showed that language emerging from two agents interacting with each other might not even be interpretable or compositional. Our multi-agent framework aims to alleviate this problem and prevent the bots from developing a specialized language between them. Interleaving supervised training with reinforcement learning also helps prevent the bots from becoming incoherent and generating non-sensical dialog. Recent work has also proposed using such goal driven dialog agents for other related tasks including negotiation BID10 and collaborative drawing . We believe that our work can easily extend to those settings as well. proposed a generative-discriminative framework for visual dialog where they train only an answer bot to generate informative answers for ground truth questions. These answers were then fed to a discriminator, which was trained to rank the generated answer among a set of candidate answers. This is a major restriction of their model as it can only be trained when this additional information of candidate answers is available, which restricts it to a supervised setting. Furthermore, since they train only the answer bot and have no question bot, they cannot simulate an entire dialog which also prevents them from learning by self-play via reinforcement. BID16 further improved upon this generative-discriminative framework by formulating the discriminator as a more traditional GAN BID3, where the adversarial discriminator is tasked to distinguish between human generated and machine generated dialogs. Furthermore, unlike they modeled the discriminator using an attention network which also utilized the dialog history in predicting the next answer allowing it to maintain coherence and consistency across dialog turns. We briefly describe the agent architectures and leave the details for the appendix. The question bot architecture we use is inspired by the answer bot architecture in BID2 and but the individual units have been modified to provide more useful representations. Similar to the original architecture, our Q-Bot, shown in FIG2, also consists of 5 parts, (a) fact encoder, (b) state-history encoder, (c) caption encoder, (d) image regression network and (e) question decoder. The fact encoder is modeled using a Long-Short Term Memory (LSTM) network, which encodes the previous question-answer pair into an embedding F Q t. We modify the statehistory encoder to incorporate a two-level hierarchical encoding of the dialog. It uses the fact embedding F Q t at each time step to compute attention over the history of dialog, (F DISPLAYFORM0 and produce a history encoding S Q t . The key modification (compared to) in our architecture is the addition of a separate LSTM to compute a caption embedding C. This is key to ensuring that the hierarchical encoding does not exclusively attend on the caption while generating the history embedding, and prevents the occurrence of repetitive questions in a dialog since the encoding now has an adequate representation of the previous facts. The caption embedding is then concatenated with F Q t and S Q t, and passed through multiple fully connected layers to compute the state-history encoder embedding e t and the predicted image feature embeddingŷ t = f (S Q t). The encoder embedding, e Q t is fed to the question decoder, another LSTM, which generates the question, q t. For all LSTMs and fully connected layers in the model we use a hidden layer size of 512. The image feature vector is 4096 dimensional. The word embeddings and the encoder embeddings are 300 dimensional. The architecture for A-Bot, also inspired from, shown in FIG2, is similar to that of the Q-Bot. It has 3 components: (a) question encoder, (b) state-history encoder and (c) answer decoder. The question encoder computes an embedding, Q t for the question to be answered, q t. The history encoding (F A 1, F A 2, F A 3 ...F A t) → S A t uses a similar two-level hierarchical encoder, where the attention is computed using the question embedding Q t. The caption is passed on to the A-Bot as the first element of the history, which is why we do not use a separate caption encoder. Instead, we use the fc7 feature embedding of a pretrained VGG-16 BID13 model to compute the image embedding I. The three embeddings S A t, Q t, I are concatenated and passed through another fully connected layer to extract the encoder embedding e A t. The answer decoder, which is another LSTM, uses this embedding e A t to generate the answer a t. Similar to the Q-Bot, we use a hidden layer size of 512 for all LSTMs and fully connected layers. The image feature vector coming from the CNN is 4096 dimensional (FC7 features from VGG16). The word embeddings and the encoder embeddings are 300 dimensional. We follow the training process proposed in BID2. Two agents, a Q-Bot and an A-Bot are first trained in isolation, by supervision from the VisDial dataset. After this supervised pretraining for 15 epochs over the data, we smoothly transition the agents to learn by reinforcement via a curriculum. Specifically, for the first K rounds of dialog for each image, the agents are trained using supervision from the VisDial dataset. For the remaining 10-K rounds, however, they are trained via reinforcement learning. K starts at 9 and is linearly annealed to 0 over 10 epochs. The individual phases of training will be described below, with some details in the appendix. In the supervised part of training, both the Q-Bot and A-Bot are trained separately. Both the Q-Bot and A-Bot are trained with a Maximum Likelihood Estimation (MLE) loss computed using the ground truth questions and answers, respectively, for every round of dialog. The Q-Bot simultaneously optimizes another objective: minimizing the Mean Squared Error (MSE) loss between the true and predicted image embeddings. The ground truth dialogs and image embeddings come from the VisDial dataset. Given the true dialog history, image features and ground truth question, the A-Bot generates an answer to that question. Given the true dialog history and the previous question-answer pair, the Q-Bot is made to generate the next question to ask the A-Bot. However, there are multiple problems with this training scheme. First, MLE is known to in models that generate repetitive dialogs and often produce generic responses. Second, the agents are never allowed to interact during training. Thus, when they interact during testing, they end up facing out of distribution questions and answers, and produce unexpected responses. Third, the sequentiality of dialog is lost when the agents are trained in an isolated, supervised manner. To alleviate the issues pointed out with supervised training, we let the two bots interact with each other via self-play (no ground-truth except images and captions). This interaction is also in the form of questions asked by the Q-Bot, and answered in turn by the A-Bot, using a shared vocabulary. The state space is partially observed and asymmetric, with the Q-Bot observing {c, q 1, a 1 . . . q 10, a 10} and the A-Bot observing the same, plus the image I. Here, c is the caption, and q i, a i is the i th dialog pair exchanged where i = 1... 10. The action space for both bots consists of all possible output sequences of a specified maximum length (Q-Bot: 16, ABot: 9 as specified by the dataset) under a fixed vocabulary (size 8645). Note that these parameter values are chosen to comply with the VisDial dataset. Each action involves predicting words sequentially until a stop token is predicted, or the generated statement reaches the maximum length. Additionally, Q-Bot also produces a guess of the visual representation of the input image (VGG fc-7 embedding of size 4096). Both Q-Bot and A-Bot share the same objective and get the same reward to encourage cooperation. Information gain in each round of dialog is incentivized by setting the reward as the change in distance of the predicted image embedding to the ground-truth image representation. This means that a QA pair is of high quality only if it helps the Q-Bot make a better prediction of the image representation. Both policies are modeled by neural networks, as discussed in Section 4.However, as noted above, this RL optimization problem is ill-posed, since rewarding the agents for information exchange does not motivate them to stick to the rules and conventions of the English language. Thus, we follow the elaborate curriculum scheme described above, despite which the bots are still observed to diverge from natural language and produce non-grammatical and incoherent dialog. Thus, we propose a multi bot architecture to help the agents interact in diverse and coherent, yet informative, dialog. Learning Algorithm: A dialog round at time t consists of the following steps: 1) The Q-Bot, conditioned on the state encoding, generates a question q t, 2) A-Bot updates its state encoding with q t and then generates an answer a t, 3) Both Q-Bot and A-Bot encode the completed exchange as a fact embedding, 4) Q-Bot updates its state encoding to incorporate this fact and finally 5) Q-Bot predicts the image representation for the unseen image conditioned on its updated state encoding. Similar to BID1, we use the REIN-FORCE BID15 algorithm that updates network parameters in response to experienced rewards. The per-round rewards maximized are: DISPLAYFORM0 This reward is positive if the distance between image representation generated at time t is closer to the ground truth than the representation at time t − 1, hence incentivizing information gain at each round of dialog. The REINFORCE update rule ensures that a (q t, a t) exchange that is informative has its probabilities pushed up. Do note that the image feature regression network f is trained directly via supervised gradient updates on the L-2 loss. In this section we describe our proposed MultiAgent Dialog architecture in detail. The motivation end while 25: end procedure behind this is the observation that allowing a pair of agents to interact with each other and learn via reinforcement for too long leads to them developing an idiosyncratic private language which does not adhere to the rules of human language, and is hence not understandable by human observers. We claim that if instead of allowing a single pair of agents to interact, we were to make the agents more social, and make them interact and learn from multiple other agents, they would be disincentivized to develop a private language, and would have to conform to the common language. In particular, we create either multiple Q-bots to interact with a single A-bot, or multiple A-bots to interact with a single Q-bot. All these agents are initialized with the learned parameters from the supervised pretraining as described in Section 5.1. Then, for each batch of images from the VisDial dataset, we randomly choose a Q-bot to interact with the A-bot, or randomly choose an A-bot to interact with the Q-bot, as the case may be. The two chosen agents then have a complete dialog consisting of 10 question-answer pairs about each of those images, and update their respective weights based on the rewards received (as per Equation 1) during the conversation, using the REINFORCE algorithm. This process is repeated for each batch of images, over the entire VisDial dataset. It is important to note that histories are not shared across batches. MADF can be understood in detail using the pseudocode in Algorithm 1. We use the VisDial 0.9 dataset for our task introduced by BID1. The data is collected using Amazon Mechanical Turk by pairing 2 annotators and asking them to chat about the image as a multi round VQA setup. One of the annotators acts as the questioner and has access to only the caption of the image and has to ask questions from the other annotator who acts as the answerer and must answer the questions based on the visual information from the actual image. This dialog repeats for 10 rounds at the end of which the questioner has to guess what the image was. We perform our Model MRR Mean Rank R@1 R@5 R@10 Answer Prior BID1 0 Figure 3: The percentile scores of the ground truth image compared to the entire test set of 40k images. The X-axis denotes the dialog round number (from 1 to 10), while the Y-axis denotes the image retrieval percentile score.experiments on VisDial v0.9 (the latest available release) containing 83k dialogs on COCO-train and 40k on COCO-val images, for a total of 1.2M dialog question-answer pairs. We split the 83k into 82k for train, 1k for validation, and use the 40k as test, in a manner consistent with BID1. The caption is considered to be the first round in the dialog history. We evaluate the performance of our model using 6 metrics, proposed by BID2: 1) Mean Reciprocal Rank (MRR), 2) Mean Rank, 3) Recall@1, 4) Recall@5, 5) Recall@10 and 6) Image Retrieval Percentile. Mean Rank and MRR compute the average rank (and its reciprocal, respectively) assigned to the answer generated by the A-bot, over a set of 100 candidate answers for each question (also averaged over all the 10 rounds). Recall@k computes the percentage of answers with rank less than k. Image Retrieval percentile is a measure of how close the image prediction generated by the Q-bot is to the ground truth. All the images in the test set are ranked according to their distance from the predicted image embedding, and the rank of the ground truth embedding is used to calculate the image retrieval percentile. Table 1 compares the Mean Rank, MRR, Recall@1, Recall@5 and Recall@10 of our agent architecture and dialog framework (below the horizontal line) with previously proposed architectures (above the line). SL refers to the agents after only the isolated, supervised training of Section 5.1. RL-1Q,1A refers to a single, idiosyncratic pair of agents trained via reinforcement as in Section 5.2. RL-1Q,3A and RL-3Q,1A refer to social agents trained via our Multi-Agent Dialog framework in Section 5.3, with 1Q,3A referring to 1 Q-Bot and 3 A-Bots, and 3Q,1A referring to 3 Q-Bots and 1 A-Bot. It can be seen that our agent architectures clearly outperform all previously proposed generative architectures in MRR, Mean Rank and R@10, but not in R@1 and R@5. This indicates that our approach produces consistently good answers (as measured by MRR, Mean Rank and R@10), even though they might not be the best possible answers (as measured by R@1 and R@5). SL has the best MRR and Mean Rank, which drops drastically in RL-1Q,1A. The agents trained by MADF recover and are able to outperform all previously proposed models. Fig. 3 shows image retrieval percentile scores over dialog rounds. The percentile score decreases monotonically for SL, but is stable for all versions using RL. There are no quantitative metrics to comprehensively evaluate dialog quality, hence we do a hu-. A total of 20 evaluators (randomly chosen students) were shown the caption and the 10 QA-pairs generated by each system for one of 4 randomly chosen images, and asked to give an ordinal ranking (from 1 to 4) for each metric. If the evaluator was also given access to the image, she was asked only to evaluate metrics 3, 4 and 5, while if the evaluator was not shown the image, she was asked only to evaluate metrics 1, 2 and 5. Table 2 contains the average ranks obtained on each metric (lower is better).The convincingly prove our hypothesis that having multiple A-Bots to interact and learn from will improve the Q-Bot, and vice versa. This is because having multiple A-Bots to interact with gives the Q-Bot access to a variety of diverse dialog, leading to more stable updates with lower bias. The confirm this, with Q-Bot Relevance rank being best in RL-1Q,3A, and A-Bot Relevance rank being best in RL-3Q,1A. These two dialog systems, which were trained via MADF, also have the best overall dialog coherence by a significant margin over RL-1Q,1A and SL. We show two of the examples shown to the human evaluators in Figure 4. The trends observed in the scores given by human evaluators is also clearly visible in these examples. MADF agents are able to model the human responses much better than the other agents. It can also be seen that although the RL-1Q,1A system has greater diversity in its responses, the quality of those responses is greatly degraded, with the A-Bot's answers especially being both non-grammatical and irrelevant. In Section 5.1, we discussed how the MSE loss used in SL in models which generate repetitive dialog, which can be seen in Fig. 4. Consider the first image, where in the SL QA-generations, the Q-Bot repeats the same questions multiple times, and gets inconsistent answers from the A-Bot for the same question. By contrast, all 10 QA-generations for RL-3Q,1A are grammatically correct. The Q-Bot's questions are very relevant to the image being considered, and the A-Bot's answers appropriate and correct. In this paper we propose a novel Multi-Agent Dialog Framework (MADF), inspired from human communities, to improve the dialog quality of AI agents. We show that training 2 agents with supervised learning can lead to uninformative and repetitive dialog. Furthermore, we observe that the task performance (measured by the image retrieval percentile scores) for the system trained via supervision only deteriorates as dialog round number increases. We hypothesize that this is because the agents were trained in isolation and never allowed to interact during supervised learning, which leads to failure during testing when they encounter out of distribution samples (generated by the other agent, instead of ground truth) for the first time. We show how allowing a single pair of agents to interact and learn from each other via reinforcement learning dramatically improve their percentile scores, which additionally does not deteriorate over multiple rounds of dialog, since the agents have interacted with one another and been exposed to the other's generated questions or answers. However, the agents, in an attempt to improve task performance end up developing their own private language which does not adhere to the rules and conventions of human languages, and generates nongrammatical and non-sensical statements. As a , the dialog system loses interpretability and sociability. Figure 4: Two randomly selected images from the VisDial dataset followed by the ground truth (human) and generated dialog about that image for each of our 4 systems (SL, RL-1Q,1A, RL-1Q,3A, RL-3Q,1A). These images were also used in the human evaluation shown in Table 2.multi-agent dialog framework based on self-play reinforcement learning, where a single A-Bot is allowed to interact with multiple Q-Bots and vice versa. Through a human evaluation study, we show that this leads to significant improvements in dialog quality measured by relevance, grammar and coherence. This is because interacting with multiple agents prevents any particular pair from maximizing performance by developing a private language, since it would harm performance with all the other agents. We plan to explore several other multi bot architectural settings and perform a more thorough human evaluation for qualitative analysis of our dialog. We also plan on incorporating an explicit perplexity based reward term in our reinforcement learning setup to further improve the dialog quality. We will also experiment with using a discriminative answer decoder which uses information of the possible answer candidates to rank the generated answer with respect to all the candidate answers and use the ranking performance to train the answer decoder. Another avenue for future exploration is to use a richer image feature embedding to regress on. Currently, we use a regression network to compute the estimated image embedding which represents the Q-Bot's understanding of the image. We plan to implement an image generation GAN which can use this embedding as a latent code to generate an image which can be visualized. While the MADF in its current form only works if we have multiple Q-Bots or multiple A-Bots but not both, future work could possibly look at incorporating that into the framework, while ensuring that the updates do not become too unstable. then generate their respective embeddings which are fed into the feature regression network and the question decoder to produceŷ t and update the hidden state of the question decoder respectively. The Q-Bot is then trained by maximizing the likelihood p(q gt |E Q t) of the training data q gt, computed using the softmax probabilities given by the question decoder. Simultaneously, the Mean Squared Error (MSE) loss between the predicted image embedding and ground truth is also minimized. Effectively, the loss DISPLAYFORM0 is minimized. At time step t, the A-Bot's question encoder is fed with the ground-truth question for t, the state/history/image encoder is fed with all the ground-truth QA pairs up to t − 1 and the image I. These encoders then generate their respective embeddings which are fed into the answer decoder to produce a t. The A-Bot is trained by maximizing the likelihood p(a gt |E A t) of the training data a gt, computed using the softmax probabilities given by the answer decoder. Effectively, the loss DISPLAYFORM0 The Q-Bot is given only the caption c gt and the A-Bot is given only the image I and caption c gt as inputs. At time step t, the Q-Bot's fact encoder is fed with the generated QA pair for t − 1, the state/history encoder is fed with all the generated QA pairs up to t − 1 and the caption encoder is given the true image caption c gt as input. These encoders then generate their respective embeddings which are fed into the feature regression network and the question decoder to produceŷ t and q t respectively. The change in distance betweenŷ t and y gt due to the current QA-pair is given as a reward to Q-Bot (Eqn. 1), which it uses to train itself via REINFORCE. Simultaneously, the Mean Squared Error (MSE) loss between the predicted image embedding and ground truth is also minimized via supervision. Effectively, the loss DISPLAYFORM0 is minimized, where G t = 10−t k=0 γ k r t+k+1 indicates the Monte-Carlo return at step t, and γ is a discount factor equal to 0.99. At time step t, the A-Bot's question encoder is fed with the generated question q t, the state/history/image encoder is fed with all the generated QA pairs up to t − 1 and the image I. These encoders then generate their respective embeddings which are fed into the answer decoder to produce a t. The A-Bot also receives the same reward as the Q-Bot, and trains itself via REINFORCE. Effectively, the loss DISPLAYFORM0 is minimized, where G t = 10−t k=0 γ k r t+k+1 indicates the Monte-Carlo return at step t, and γ is a discount factor equal to 0.99.

Represent the short summary for finding the corresponding scientific article (long)

Social agents learn to talk to each other in natural language towards a goal

Represent the TLDR for finding the corresponding scientific article.

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.


Question: Represent.

Question: Posterior collapse in Variational Autoencoders (VAEs) arises when the variational distribution closely matches the uninformative prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We identify how local maxima can emerge from the marginal log-likelihood of pPCA, which yields similar local maxima for the evidence lower bound (ELBO). We show that training a linear VAE with variational inference recovers a uniquely identifiable global maximum corresponding to the principal component directions. We provide empirical evidence that the presence of local maxima causes posterior collapse in deep non-linear VAEs. Our findings help to explain a wide range of heuristic approaches in the literature that attempt to diminish the effect of the KL term in the ELBO to reduce posterior collapse. The generative process of a deep latent variable model entails drawing a number of latent factors from an uninformative prior and using a neural network to convert such factors to real data points. Maximum likelihood estimation of the parameters requires marginalizing out the latent factors, which is intractable for deep latent variable models. The influential work of BID21 and BID28 on Variational Autoencoders (VAEs) enables optimization of a tractable lower bound on the likelihood via a reparameterization of the Evidence Lower Bound (ELBO) BID18 BID4. This has created a surge of recent interest in automatic discovery of the latent factors of variation for a data distribution based on VAEs and principled probabilistic modeling BID15 BID5 BID8 BID13.Unfortunately, the quality and the number of the latent factors learned is directly controlled by the extent of a phenomenon known as posterior collapse, where the generative model learns to ignore a subset of the latent variables. Most existing work suggests that posterior collapse is caused by the KL-divergence term in the ELBO objective, which directly encourages the variational distribution to match the prior. Thus, a wide range of heuristic approaches in the literature have attempted to diminish the effect of the KL term in the ELBO to alleviate posterior collapse. By contrast, we hypothesize that posterior collapse arises due to spurious local maxima in the training objective. Surprisingly, we show that these local maxima may arise even when training with exact marginal log-likelihood. While linear autoencoders BID30 have been studied extensively BID2 BID23, little attention has been given to their variational counterpart. A well-known relationship exists between linear autoencoders and PCAthe optimal solution to the linear autoencoder problem has decoder weight columns which span the subspace defined by the principal components. The Probabilistic PCA (pPCA) model BID32 recovers the principal component subspace as the maximum likelihood solution of a Gaussian latent variable model. In this work, we show that pPCA is recovered exactly using linear variational autoencoders. Moreover, by specifying a diagonal covariance structure on the variational distribution we recover an identifiable model which at the global maximum has the principal components as the columns of the decoder. The study of linear VAEs gives us new insights into the cause of posterior collapse. Following the analysis of BID32, we characterize the stationary points of pPCA and show that the variance of the observation model directly impacts the stability of local stationary points -if the variance is too large then the pPCA objective has spurious local maxima, which correspond to a collapsed posterior. Our contributions include:• We prove that linear VAEs can recover the true posterior of pPCA and using ELBO to train linear VAEs does not add any additional spurious local maxima. Further, we prove that at its global optimum, the linear VAE recovers the principal components.• We shows that posterior collapse may occur in optimization of marginal log-likelihood, without powerful decoders. Our experiments verify the analysis of the linear setting and show that these insights extend even to high-capacity, deep, non-linear VAEs.• By learning the observation noise carefully, we are able to reduce posterior collapse. We present evidence that the success of existing approaches in alleviating posterior collapse depends on their ability to reduce the stability of spurious local maxima. Probabilistic PCA. We define the probabilitic PCA (pPCA) model as follows. Suppose latent variables z ∈ R k generate data x ∈ R n. A standard Gaussian prior is used for z and a linear generative model with a spherical Gaussian observation model for x: DISPLAYFORM0 The pPCA model is a special case of factor analysis BID3, which replaces the spherical covariance σ 2 I with a full covariance matrix. As pPCA is fully Gaussian, both the marginal distribution for x and the posterior p(z|x) are Gaussian and, unlike factor analysis, the maximum likelihood estimates of W and σ 2 are tractable BID32.Variational Autoencoders. Recently, amortized variational inference has gained popularity as a means to learn complicated latent variable models. In these models, the marginal log-likelihood, log p(x), is intractable but a variational distribution, q(z|x), is used to approximate the posterior, p(z|x), allowing tractable approximate inference. To do so we typically make use of the Evidence Lower Bound (ELBO): DISPLAYFORM1 The ELBO consists of two terms, the KL divergence between the variational distribution, q(z|x), and prior, p(z), and the expected conditional log-likelihood. The KL divergence forces the variational distribution towards the prior and so has reasonably been the focus of many attempts to alleviate posterior collapse. We hypothesize that in fact the marginal log-likelihood itself often encourages posterior collapse. In Variational Autoencoders (VAEs), two neural networks are used to parameterize q φ (z|x) and p θ (x|z), where φ and θ denote two sets of neural network weights. The encoder maps an input x to the parameters of the variational distribution, and then the decoder maps a sample from the variational distribution back to the inputs. Posterior collapse. The most consistent issue with VAE optimization is posterior collapse, in which the variational distribution collapses towards the prior: ∃i s.t. ∀x q φ (z i |x) ≈ p(z i). This reduces the capacity of the generative model, making it impossible for the decoder network to make use of the information content of all of the latent dimensions. While posterior collapse is typically described using the variational distribution as above, one can also define it in terms of the true posterior p(z|x) as: ∃i s.t. ∀x p(z i |x) ≈ p(z i).3 Related Work BID12 discuss the relationship between robust PCA methods BID6 and VAEs. In particular, they show that at stationary points the VAE objective locally aligns with pPCA under certain assumptions. We study the pPCA objective explicitly and show a direct correspondence with linear VAEs. BID12 show that the covariance structure of the variational distribution may help smooth out the loss landscape. This is an interesting whose interactions with ours is an exciting direction for future research. BID14 motivate posterior collapse through an investigation of the learning dynamics of deep VAEs. They suggest that posterior collapse is caused by the inference network lagging behind the true posterior during the early stages of training. A related line of research studies issues arising from approximate inference causing mismatch between the variational distribution and true posterior BID10 BID19 BID16. By contrast, we show that local maxima may exist even when the variational distribution matches the true posterior exactly. Alemi et al. FORMULA0 use an information theoretic framework to study the representational properties of VAEs. They show that with infinite model capacity there are solutions with equal ELBO and marginal log-likelihood which span a range of representations, including posterior collapse. We find that even with weak (linear) decoders, posterior collapse may occur. Moreover, we show that in the linear case this posterior collapse is due entirely to the marginal log-likelihood. The most common approach for dealing with posterior collapse is to anneal a weight on the KL term during training from 0 to 1 BID5 BID31 BID24 BID15 BID17. Unfortunately, this means that during the annealing process, one is no longer optimizing a bound on the log-likelihood. In addition, it is difficult to design these annealing schedules and we have found that once regular ELBO training resumes the posterior will typically collapse again (Section 5.2). propose a constraint on the KL term, which they called "free-bits", where the gradient of the KL term per dimension is ignored if the KL is below a given threshold. Unfortunately, this method reportedly has some negative effects on training stability BID26. Delta-VAEs BID26 instead choose prior and variational distributions carefully such that the variational distribution can never exactly recover the prior, allocating free-bits implicitly. Several other papers have studied alternative formulations of the VAE objective BID27 BID11 BID0. BID11 analyze the VAE objective with the goal of improving image fidelity under Gaussian observation models. Through this lens they discuss the importance of the observation noise. BID29 point out that due to the diagonal covariance used in the variational distribution of VAEs they are encouraged to pursue orthogonal representations. They use linearizations of deep networks to prove their under a modification of the objective function by explicitly ignoring latent dimensions with posterior collapse. Our formulation is distinct in focusing on linear VAEs without modifying the objective function and proving an exact correspondence between the global solution of linear VAEs and principal components. BID23 studies the optimization challenges in the linear autoencoder setting. They expose an equivalence between pPCA and Bayesian autoencoders and point out that when σ 2 is too large information about the latent code is lost. A similar phenomenon is discussed in the supervised learning setting by BID7. BID23 also show that suitable regularization allows the linear autoencoder to exactly recover the principal components. We show that the same can be achieved using linear variational autoencoders with a diagonal covariance structure. In this section we compare and analyze the optimal solutions to both pPCA and linear variational autoencoders. DISPLAYFORM0 Figure 1: Stationary points of pPCA. A zero-column of W is perturbed in the directions of two orthogonal principal components (µ5 and µ7) and the loss surface (marginal log-likelihood) is shown. The stability of the stationary points depends critically on σ 2. Left: σ 2 is able to capture both principal components. Middle: σ 2 is too large to capture one of the principal components. Right: σ 2 is too large to capture either principal component. We first discuss the maximum likelihood estimates of pPCA and then show that a simple linear VAE is able to recover the global optimum. Moreover, the same linear VAE recovers identifiability of the principle components (unlike pPCA which only spans the PCA subspace). Finally, we analyze the loss landscape of the linear VAE showing that ELBO does not introduce any additional spurious maxima. The pPCA model (Eq. FORMULA0) is a fully Gaussian linear model and thus we can compute both the marginal distribution for x and the posterior p(z | x) in closed form: DISPLAYFORM0 DISPLAYFORM1 where DISPLAYFORM2 This model is particularly interesting to analyze in the setting of variational inference as the ELBO can also be computed in closed form (see Appendix C).Stationary points of pPCA We now characterize the stationary points of pPCA, largely repeating the thorough analysis of (see Appendix A of their paper).The maximum likelihood estimate of µ is the mean of the data. We can compute W MLE and σ MLE as follows: DISPLAYFORM3 DISPLAYFORM4 Here U k corresponds to the first k principal components of the data with the corresponding eigenvalues λ 1,..., λ k stored in the k × k diagonal matrix Λ k. The matrix R is an arbitrary rotation matrix which accounts for weak identifiability in the model. We can interpret σ DISPLAYFORM5 as the average variance lost in the projection. The MLE solution is the global optima. Stability of W MLE One surprising observation is that σ 2 directly controls the stability of the stationary points of the marginal log-likelihood (see Appendix A). In Figure 1, we illustrate one such stationary point of pPCA under different values of σ 2. We computed this stationary point by taking W to have three principal components columns and zeros elsewhere. Each plot shows the same stationary point perturbed by two orthogonal eigenvectors corresponding to other principal components. The stability of the stationary points depends on the size of σ 2 -as σ 2 increases the stationary point tends towards a stable local maxima. While this example is much simpler than a non-linear VAE, we find in practice that the same principle applies. Moreover, we observed that the non-linear dynamics make it difficult to learn a smaller value of σ 2 automatically FIG5 ). We now show that linear VAEs are able to recover the globally optimal solution to Probabilistic PCA. We will consider the following VAE model, DISPLAYFORM0 where D is a diagonal covariance matrix which is used globally for all data points. While this is a significant restriction compared to typical VAE architectures, which define an amortized variance for each input point, this is sufficient to recover the global optimum of the probabilistic model. Lemma 1. The global maximum of the ELBO objective (Eq.) for the linear VAE (Eq. FORMULA9) is identical to the global maximum for the marginal log-likelihood of pPCA (Eq. FORMULA3).Proof. The global optimum of pPCA is obtained at the maximum likelihood estimate of W and σ 2, which are specified only up to an orthogonal transformation of the columns of W, i.e., any rotation R in Eq. FORMULA7 FORMULA9 is able to recover the global optimum of pPCA only when DISPLAYFORM1 k (which is diagonal) recovers the true posterior at the global optimum. In this case, the ELBO equals the marginal log-likelihood and is maximized when the decoder has weights W = W MLE. Since, ELBO lower bounds log-likelihood, then the global maximum of ELBO for the linear VAE is the same as the global maximum of marginal likelihood for pPCA.Full details are given in Appendix C. In fact, the diagonal covariance of the variational distribution allows us to identify the principal components at the global optimum. Corollary 1. The global optimum to the VAE solution has the scaled principal components as the columns of the decoder network. Proof. Follows directly from the proof of Lemma 1 and Equation 8.Finally, we can recover full identifiability by requiring D = I. We discuss this in Appendix B.We have shown that at its global optimum the linear VAE is able to recover the pPCA solution and additionally enforces orthogonality of the decoder weight columns. However, the VAE is trained with the ELBO rather than the marginal log-likelihood. The majority of existing work suggests that the KL term in the ELBO objective is responsible for posterior collapse and so we should ask whether this term introduces additional spurious local maxima. Surprisingly, for the linear VAE model the ELBO objective does not introduce any additional spurious local maxima. We provide a sketch of the proof here with full details in Appendix C. Theorem 1. The ELBO objective does not introduce any additional local maxima to the pPCA model. Proof. (Sketch) If the decoder network has orthogonal columns then the variational distribution can capture the true posterior and thus the variational objective exactly recovers the marginal log-likelihood at stationary points. If the decoder network does not have orthogonal columns then the variational distribution is no longer tight. However, the ELBO can always be increased by rotating the columns of the decoder towards orthogonality. This is because the variational distribution fits the true posterior more closely while the marginal log-likelihood is invariant to rotations of the weight columns. Thus, any additional stationary points in the ELBO objective must necessarily be saddle points. The theoretical presented in this section provide new intuition for posterior collapse in general VAEs. Our suggest that the ELBO objective, in particular the KL between the variational distribution and the prior, is not entirely responsible for posterior collapse -even exact marginal log-likelihood may suffer. The evidence for this is two-fold. We have shown that marginal log-likelihood may have spurious local maxima but also that in the linear case the ELBO objective does not add any additional spurious local maxima. Rephrased, in the linear setting the problem lies entirely with the probabilistic model. FIG0 The marginal log-likelihood and optimal ELBO of MNIST pPCA solutions over increasing hidden dimension. Green represents the MLE solution (global maximum), the red dashed line is the optimal ELBO solution which matches the global optimum. The blue line shows the marginal log-likelihood of the solutions using the full decoder weights when σ 2 is fixed to its MLE solution for 50 hidden dimensions. In this section we present empirical evidence found from studying two distinct claims. First, we verified our theoretical analysis of the linear VAE model. Second, we explored to what extent these insights apply to deep non-linear VAEs. In FIG0 we display the likelihood values for various optimal solutions to the pPCA model trained on the MNIST dataset. We plot the maximum log-likelihood and numerically verify that the optimal ELBO solution is able to exactly match this (Lemma 1). We also evaluated the model with all principal components used but with a fixed value of σ 2 corresponding to the MLE solution for 50 hidden dimensions. This is equivalent to σ 2 ≈ λ 222. Here the log-likelihood is optimal at σ 2 = 50 as expected, but interestingly the likelihood decreases for 300 hidden dimensions -including the additional principal components has made the solution worse under marginal log-likelihood. We explored how well the analysis of the linear VAEs extends to deep non-linear models. To do so, we trained VAEs with Gaussian observation models on the MNIST dataset. This is a fairly uncommon choice of model for this dataset, which is nearly binary, but it provides a good setting for us to investigate our theoretical findings. FIG3 shows the cumulative distribution of the per-dimension KL divergence between the variational distribution and the prior at the end of training. We observe that using a smaller value of σ 2 prevents the posterior from collapsing and allows the model to achieve a substantially higher ELBO. It is possible that the difference in ELBO is due entirely to the change of scale introduced by σ 2 and not because of differences in the learned representations. To test this hypothesis we took each of the trained models and optimized for σ 2 while keeping all other parameters fixed (TAB1 . As expected, the ELBO increased but the relative ordering remained the same with a significant gap still present. The final model is evaluated on the training set. We also tuned σ 2 to the trained model and reevaluated to confirm that the difference in loss is due to differences in latent representations. The role of KL-annealing An alternative approach to tuning σ 2 is to scale the KL term directly by a coefficient, β. For β < 1 this provides a loose lowerbound on the ELBO but for appropriate choices of β and learning rate, this scheme can be made equivalent to tuning σ 2 . In this section we explore this technique. We found that KL-annealing may provide temporary relief from posterior collapse but that if σ 2 is not appropriately tuned then ultimately ELBO training will recover the default solution. In FIG4 we show the proportion of units collapsed by threshold for several fixed choices of σ 2 when β is annealed from 0 to 1 over the first 100 epochs. The solid lines correspond to the final model while the dashed line corresponds to the model at 80 epochs of training. Early on, KL-annealing is able to reduce posterior collapse but ultimately we recover the ELBO solution from FIG3 .After finding that KL-annealing alone was insufficient to prevent posterior collapse we explored KL annealing while learning σ 2 . Based on our analysis in the linear case we expect that this should work well: while β is small the model should be able to learn to reduce σ 2 . To test this, we trained the same VAE as above on MNIST data but this time we allowed σ 2 to be learned. The are presented in FIG5 . We trained first using the standard ELBO objective and then again using KL-annealing. The ELBO objective learns to reduce σ 2 but ultimately learns a solution with a large degree of posterior collapse. Using KL-annealing, the VAE is able to learn a much smaller σ 2 value and ultimately reduces posterior collapse. Interestingly, despite significantly differing representations, these two models have approximately the same final training ELBO. This is consistent with the analysis of BID0, who showed that there can exist solutions equal under ELBO with differing posterior distributions. We trained deep convolutional VAEs with 500 hidden dimensions on images from the CelebA dataset (resized to 64x64). In FIG6 we show the training ELBO for the standard ELBO objective and training with KL-annealing. In each case, σ 2 is learned online. As in FIG5, KL-Annealing enabled the VAE to learn a smaller value of σ 2 which corresponded to a better final ELBO value and reduced posterior collapse FIG7 ). By analyzing the correspondence between linear VAEs and pPCA we have made significant progress towards understanding the causes of posterior collapse. We have shown that for simple linear VAEs posterior collapse is caused by spurious local maxima in the marginal log-likelihood and we demonstrated empirically that the same local maxima seem to play a role when optimizing deep non-linear VAEs. In future work, we hope to extend this analysis to other observation models and provide theoretical support for the non-linear case. Here we briefly summarize the analysis of BID32 with some simple additional observations. We recommend that interested readers study Appendix A of BID32 for the full details. We begin by formulating the conditions for stationary points of xi log p(x i): DISPLAYFORM0 Where S denotes the sample covariance matrix (assuming we set µ = µ M LE, which we do throughout), and C = WW T + σ 2 I (note that the dimensionality is different to M). There are three possible solutions to this equation, W = 0, C = S, or the more general solutions. FORMULA0 and are not particularly interesting to us, so we focus herein on.We can write W = ULV T using its singular value decomposition. Substituting back into the stationary points equation, we recover the following: DISPLAYFORM1 Noting that L is diagonal, if the j th singular value (l j) is non-zero, this gives Su j = (σ 2 +l 2 j)u j, where u j is the j th column of U. Thus, u j is an eigenvector of S with eigenvalue DISPLAYFORM2 Thus, all potential solutions can be written as, W = U q (K q − σ 2 I) 1/2 R, with singular values written as k j = σ 2 or σ 2 + l 2 j and with R representing an arbitrary orthogonal matrix. From this formulation, one can show that the global optimum is attained with σ 2 = σ 2 M LE and U q and K q chosen to match the leading singular vectors and values of S. Consider stationary points of the form, W = U q (K q − σ 2 I) 1/2 where U q contains arbitrary eigenvectors of S. In the original pPCA paper they show that all solutions except the leading principal components correspond to saddle points in the optimization landscape. However, this analysis depends critically on σ 2 being set to the true maximum likelihood estimate. Here we repeat their analysis, considering other (fixed) values of σ 2.We consider a small perturbation to a column of W, of the form u j. To analyze the stability of the perturbed solution, we check the sign of the dot-product of the perturbation with the likelihood gradient at w i + u j. Ignoring terms in 2 we can write the dot-product as, DISPLAYFORM0 Now, C −1 is positive definite and so the sign depends only on λ j /k i − 1. The stationary point is stable (local maxima) only if the sign is negative. If k i = λ i then the maxima is stable only when λ i > λ j, in words, the top q principal components are stable. However, we must also consider the case k = σ 2. BID32 show that if σ 2 = σ 2 M LE, then this also corresponds to a saddle point as σ 2 is the average of the smallest eigenvalues meaning some perturbation will be unstable (except in a special case which is handled separately).However, what happens if σ 2 is not set to be the maximum likelihood estimate? In this case, it is possible that there are no unstable perturbation directions (that is, λ j < σ 2 for too many j). In this case when σ 2 is fixed, there are local optima where W has zero-columnsthe same solutions that we observe in non-linear VAEs corresponding to posterior collapse. Note that when σ 2 is learned in non-degenerate cases the local maxima presented above become saddle points where σ 2 is made smaller by its gradient. In practice, we find that even when σ 2 is learned in the non-linear case local maxima exist. Linear autoencoders suffer from a lack of identifiability which causes the decoder columns to span the principal component subspace instead of recovering it. Here we show that linear VAEs are able to recover the principal components up to scaling. We once again consider the linear VAE from Eq. FORMULA9: DISPLAYFORM0 The output of the VAE,x is distributed as, DISPLAYFORM1 Therefore, the linear VAE is invariant to the following transformation: DISPLAYFORM2 where A is a diagonal matrix with non-zero entries so that D is well-defined. We see that the direction of the columns of W are always identifiable, and thus the principal components can be exactly recovered. Moreover, we can recover complete identifiability by fixing D = I, so that there is a unique global maximum. Here we present details on the analysis of the stationary points of the ELBO objective. To begin, we first derive closed form solutions to the components of the marginal log-likelihood (including the ELBO). The VAE we focus on is the one presented in Eq. FORMULA9, with a linear encoder, linear decoder, Gaussian prior, and Gaussian observation model. Remember that one can express the marginal log-likelihood as: DISPLAYFORM0 Each of the terms (A-C) can be expressed in closed form for the linear VAE. Note that the KL term (A) is minimized when the variational distribution is exactly the true posterior distribution. This is possible when the columns of the decoder are orthogonal. The term (B) can be expressed as, DISPLAYFORM1 The term (C) can be expressed as, DISPLAYFORM2 Noting that Wz ∼ N WV(x − µ), WDW T, we can compute the expectation analytically and obtain, DISPLAYFORM3 To compute the stationary points we must take derivatives with respect to µ, D, W, V, σ 2. As before, we have µ = µ M LE at the global maximum and for simplicity we fix µ here for the remainder of the analysis. Taking the marginal likelihood over the whole dataset, at the stationary points we have, DISPLAYFORM4 The above are computed using standard matrix derivative identities (Petersen et al.). These equations yield the expected solution for the variational distribution directly. From Eq. we compute DISPLAYFORM5 recovering the true posterior mean in all cases and getting the correct posterior covariance when the columns of W are orthogonal. We will now proceed with the proof of Theorem 1. Theorem 1. The ELBO objective does not introduce any additional local maxima to the pPCA model. Proof. If the columns of W are orthogonal then the marginal log-likelihood is recovered exactly at all stationary points. This is a direct consequence of the posterior mean and covariance being recovered exactly at all stationary points so that is zero. We must give separate treatment to the case where there is a stationary point without orthogonal columns of W. Suppose we have such a stationary point, using the singular value decomposition we can write W = ULR T, where U and R are orthogonal matrices. Note that log p(x) is invariant to the choice of R BID32. However, the choice of R does have an effect on the first term of Eq.: this term is minimized when R = I, and thus the ELBO must increase. To formalize this argument, we compute at a stationary point. From above, at every stationary point the mean of the variational distribution exactly matches the true posterior. Thus the KL simplifies to: DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 DISPLAYFORM9 where M = diag(W T W) + σ 2 I. Now consider applying a small rotation to W: W → WR. As the optimal D and V are continuous functions of W, this corresponds to a small perturbation of these parameters too for a sufficiently small rotation. Importantly, log det M remains fixed for any orthogonal choice of R but log det M does not. Thus, we choose R to minimize this term. In this manner, shrinks meaning that the ELBO (-2)+ must increase. Thus if the stationary point existed, it must have been a saddle point. We now describe how to construct such a small rotation matrix. First note that without loss of generality we can assume that det(R) = 1. (Otherwise, we can flip the sign of a column of R and the corresponding column of U.) And additionally, we have WR = UL, which is orthogonal. The Special Orthogonal group of determinant 1 orthogonal matrices is a compact, connected Lie group and therefore the exponential map from its Lie algebra is surjective. This means that we can find an upper-triangular matrix B, such that DISPLAYFORM10 T )}, where n is an integer chosen to ensure that the elements of B are within > 0 of zero. This matrix is a rotation in the direction of R which we can make arbitrarily close to the identity by a suitable choice of. This is verified through the Taylor series expansion of DISPLAYFORM11. Thus, we have identified a small perturbation to W (and D and V) which decreases the posterior KL (A) but keeps the marginal log-likelihood constant. Thus, the ELBO increases and the stationary point must be a saddle point. We would like to extend our linear analysis to the case where we have a Bernoulli observation model, as this setting also suffers severely from posterior collapse. The analysis may also shed light on more general categorical observation models which have also been used. Typically, in these settings a continuous latent space is still used (for example, Bowman et al. FORMULA0).We will consider the following model, p(z) = N (0, I), p(x|z) = Bernoulli(y), y = σ(Wz + µ)where σ denotes the sigmoid function, σ(y) = 1/(1 + exp(−y)) and we assume an independent Bernoulli observation model over x. Unfortunately, under this model it is difficult to reason about the stationary points. There is no closed form solution for the marginal likelihood p(x) or the posterior distribution p(z|x).Numerical integration methods exist which may make it easy to evaluate this quantity in practice but they will not immediately provide us a good gradient signal. We can compute the density function for y using the change of variables formula. Noting that Wz + µ ∼ N (µ, WW T), we recover the following logit-Normal distribution: DISPLAYFORM0 We can write the marginal likelihood as, DISPLAYFORM1 = E z y(z) DISPLAYFORM2 where (·) x is taken to be elementwise. Unfortunately, the expectation of a logit-normal distribution has no closed form BID1 ) and so we cannot tractably compute the marginal likelihood. Similarly, under ELBO we need to compute the expected reconstruction error. This can be written as, E q(z|x) [log p(x|z)] = y(z)x (1 − y(z)) 1−x N (z; V(x − µ), D)dz,another intractable integral. Visualizing stationary points of pPCA For this experiment we computed the pPCA MLE using a subset of 10000 random training images from the MNIST dataset. We evaluate and plot the marginal log-likelihood in closed form on this same subset. MNIST VAE The VAEs we trained on MNIST all had the same architecture: 784-1024-512-k-512-1024-784. The VAE parameters were optimized jointly using the Adam optimizer BID20. We trained the VAE for 1000 epochs total, keeping the learning rate fixed throughout. We performed a grid search over values for the learning rate and reported for the model which achieved the best training ELBO.CelebA VAE We used the convolutional architecture proposed by BID15. Otherwise, the experimental procedure followed that of the MNIST VAEs. We also trained convolutional VAEs on the CelebA dataset using fixed choices of σ 2. As expected, the same general pattern emerged as in FIG1. Reconstructions from the KL-Annealed model are shown in FIG9. We also show the output of interpolating in the latent space in FIG10. To produce the latter plot, we compute the variational mean of 3 input points (top left, top right, bottom left) and interpolate linearly between. We also extrapolate out to a fourth point (bottom right), which lies on the plane defined by the other points.

represent this text to find a much much longer article about it with say hundreds-thousands of words!

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR!

Question: Transformers have achieved state-of-the-art on a variety of natural language processing tasks. Despite good performance, Transformers are still weak in long sentence modeling where the global attention map is too dispersed to capture valuable information. In such case, the local/token features that are also significant to sequence modeling are omitted to some extent. To address this problem, we propose a Multi-scale attention model (MUSE) by concatenating attention networks with convolutional networks and position-wise feed-forward networks to explicitly capture local and token features. Considering the parameter size and computation efficiency, we re-use the feed-forward layer in the original Transformer and adopt a lightweight dynamic convolution as implementation. Experimental show that the proposed model achieves substantial performance improvements over Transformer, especially on long sentences, and pushes the state-of-the-art from 35.6 to 36.2 on IWSLT 2014 German to English translation task, from 30.6 to 31.3 on IWSLT 2015 English to Vietnamese translation task. We also reach the state-of-art performance on WMT 2014 English to French translation dataset, with a BLEU score of 43.2. In recent years, Transformer has been remarkably adept at sequence learning tasks like machine translation , text classification (;, language modeling (b;, etc. It is solely based on an attention mechanism that captures global dependencies between input tokens, dispensing with recurrence and convolutions entirely. The key idea of the self-attention mechanism is updating token representations based on a weighted sum of all input representations. However, recent research has shown that the Transformer has surprising shortcomings in long sequence learning, exactly because of its use of self-attention. As shown in Figure 1 (a), in the task of machine translation, the performance of Transformer drops with the increase of the source sentence length, especially for long sequences. The reason is that the attention can be over-concentrated and disperse, as shown in Figure 1 (b), and only a small number of tokens are represented by attention. It may work fine for shorter sequences, but for longer sequences, it causes insufficient representation of information and brings difficulty for the model to comprehend the source information intactly. In recent work, local attention that constrains the attention to focus on only part of the sequences (; a) is used to address this problem. However, it costs self-attention the ability to capture long-range dependencies and also does not demonstrate effectiveness in sequence to sequence learning tasks. To build a module with both inductive bias of local and global context modelling in sequence to sequence learning, we hybrid self-attention with convolution and present Parallel multi-scale attention called MUSE. It encodes inputs into hidden representations and then applies self-attention and depth-separable convolution transformations in parallel. The convolution compensates for the in- The left figure shows that the performance drops largely with the increase of sentence length on the De-En dataset. The right figure shows the attention map from the 3-th encoder layer. As we can see, the attention map is too dispersed to capture sufficient information. For example, "[EOS]", contributing little to word alignment, is surprisingly over attended. sufficient use of local information while the self-attention focuses on capturing the dependencies. Moreover, this parallel structure is highly extensible, i.e., new transformations can be easily introduced as new parallel branches, and is also favourable to parallel computation. The main contributions are summarized as follows: • We find that the attention mechanism alone suffers from dispersed weights and is not suitable for long sequence representation learning. The proposed method tries to address this problem and achieves much better performance on generating long sequence. • We propose a parallel multi-scale attention and explore a simple but efficient method to successfully combine convolution with self-attention all in one module. • MUSE outperforms all previous models with same training data and the comparable model size, with state-of-the-art BLEU scores on three main machine translation tasks. • The proposed method enables parallel representation learning. Experiments show that the inference speed can be increased by 31% on GPUs. Like other sequence-to-sequence models, MUSE also adopts an encoder-decoder framework. The encoder takes a sequence of word embeddings (x 1, · · ·, x n) as input where n is the length of input. It transfers word embeddings to a sequence of hidden representation z = (z 1, · · ·, z n). Given z, the decoder is responsible for generating a sequence of text (y 1, · · ·, y m) token by token. The encoder is a stack of N MUSE modules. Residual mechanism and layer normalization are used to connect two adjacent layers. The decoder is similar to encoder, except that each MUSE module in the decoder not only captures features from the generated text representations but also performs attention over the output of the encoder stack through additional context attention. Residual mechanism and layer normalization are also used to connect two modules and two adjacent layers. The key part in the proposed model is the MUSE module, which contains three main parts: selfattention for capturing global features, depth-wise separable convolution for capturing local features, and a position-wise feed-forward network for capturing token features. The module takes the output of (i − 1) layer as input and generates the output representation in a fusion way: where "Attention" refers to self-attention, "Conv" refers to dynamic convolution, "Pointwise" refers to a position-wise feed-forward network. The followings list the details of each part. Figure 2: Multi-scale attention hybrids point-wise transformation, convolution, and self-attention to learn multi-scale sequence representations in parallel. We project convolution and self-attention into the same space to learn contextual representations. We also propose MUSE-simple, a simple version of MUSE, which generates the output representation similar to the MUST model except for that it dose not the include convolution operation: 2.1 ATTENTION MECHANISM FOR GLOBAL CONTEXT REPRESENTATION Self-attention is responsible for learning representations of global context. For a given input sequence X, it first projects X into three representations, key K, query Q, and value V. Then, it uses a self-attention mechanism to get the output representation: Where W O, W Q, W K, and W V are projection parameters. The self-attention operation σ is the dot-production between key, query, and value pairs: Note that we conduct a projecting operation over the value in our self-attention mechanism V 1 = V W V here. We introduce convolution operations into MUSE to capture local context. To learn contextual sequence representations in the same hidden space, we choose depth-wise convolution (we denote it as DepthConv in the experiments) as the convolution operation because it includes two separate transformations, namely, point-wise projecting transformation and contextual transformation. It is because that original convolution operator is not separable, but DepthConv can share the same point-wise projecting transformation with self-attention mechanism. We choose dynamic convolution (a), the best variant of DepthConv, as our implementation. Each convolution sub-module contains multiple cells with different kernel sizes. They are used for capturing different-range features. The output of the convolution cell with kernel size k is: where W V and W out are parameters, W V is a point-wise projecting transformation matrix. The Depth conv refers to depth convolution in the work of Wu et al. (2019a). For an input sequence X, the output O is computed as: where d is the hidden size. Note that we conduct the same projecting operation over the input in our convolution mechanism V 2 = XW V here with that in self-attention mechanism. Shared projection To learn contextual sequence representations in the same hidden space, the projection in the self-attention mechanism V 1 = V W V and that in the convolution mechanism V 2 = XW V is shared. because the shared projection can project the input feature into the same hidden space. If we conduct two independent projection here: are two parameter matrices, we call it as separate projection. We will analyze the necessity of applying shared projection here instead of separate projection. We introduce a gating mechanism to automatically select the weight of different convolution cells. To learn token level representations, MUSE concatenates an self-attention network with a positionwise feed-forward network at each layer. Since the linear transformations are the same across different positions, the position-wise feed-forward network can be seen as a token feature extractor. where W 1, b 1, W 2, and b 2 are projection parameters. We evaluate MUSE on four machine translation tasks. This section describes the datasets, experimental settings, detailed , and analysis. The WMT 2014 English-French translation dataset, consisting of 36M sentence pairs, is adopted as a big dataset to test our model. We use the standard split of development set and test set. We use newstest2014 as the test set and use newstest2012 +newstest2013 as the development set. , we also adopt a joint source and target BPE factorization with the vocabulary size of 40K. For medium dataset, we borrow the setup of and adopt the WMT 2014 English-German translation dataset which consists of 4.5M sentence pairs, the BPE vocabulary size is set to 32K. The test and validation datasets we used are the same as. IWSLT De-En and En-Vi datasets Besides, we perform experiments on two small IWSLT datasets to test the small version of MUSE with other comparable models. The IWSLT 2014 German-English translation dataset consists of 160k sentence pairs. We also adopt a joint source and target BPE factorization with the vocabulary size of 32K. The IWSLT 2015 English-Vietnamese translation dataset consists of 133K training sentence pairs. For the En-Vi task, we build a dictionary including all source and target tokens. The vocabulary size for English is 17.2K, and the vocabulary size for the Vietnamese is 6.8K. For fair comparisons, we only compare models reported with the comparable model size and the same training data. We do not compare Wu et al. (2019b) because it is an ensemble method. We build MUSE-base and MUSE-large with the parameter size comparable to Transformer-base and Transformer-large. We adopt multi-head attention as implementation of selfattention in MUSE module. The number of attention head is set to 4 for MUSE-base and 16 for MUSE-large. We also add the network architecture built by MUSE-simple in the similar way into the comparison. MUSE consist of 12 residual blocks for encoder and 12 residual blocks for decoder, the dimension is set to 384 for MUSE-base and 768 for MUSE-large. The hidden dimension of non linear transformation is set to 768 for MUSE-base and 3072 for MUSE-large. The MUSE-large is trained on 4 Titan RTX GPUs while the MUSE-base is trained on a single NVIDIA RTX 2080Ti GPU. The batch size is calculated at the token level, which is called dynamic batching . We adopt dynamic convolution as the variant of depth-wise separable convolution. We tune the kernel size on the validation set. For convolution with a single kernel, we use the kernel size of 7 for all layers. In case of dynamic selected kernels, the kernel size is 3 for small kernels and 15 for large kernels for all layers. The training hyper-parameters are tuned on the validation set. , we use Adam optimizer with a learning rate of 0.001. We use the warmup mechanism and invert the learning rate decay with warmup updates of 4K. For the De-En dataset, we train the model for 20K steps with a batch size of 4K. The parameters are updated every 4 steps. The dropout rate is set to 0.4. For the En-Vi dataset, we train the model for 10K steps with a batch size of 4K. The parameters are also updated every 4 steps. The dropout rate is set to 0.3. We save checkpoints every epoch and average the last 10 checkpoints for inference. During inference, we adopt beam search with a beam size of 5 for De-En, En-Fr and En-Vi translation tasks. The length penalty is set to 0.8 for En-Fr according to the validation , 1 for the two small datasets following the default setting of. We do not tune beam width and length penalty but use the setting reported in. The BLEU 1 metric is adopted to evaluate the model performance during evaluation. As shown in Table 1, MUSE outperforms all previously models on En-De and En-Fr translation, including both state-of-the-art models of stand alone self-attention , and convolutional models (; a). This shows that either self-attention or convolution alone is not enough for sequence to sequence learning. The proposed parallel multi-scale attention improves over them both on En-De and En-Fr. En-De En-Fr ConvSeq2seq 25.2 40.5 SliceNet 26.1 -Transformer 28.4 41.0 Weighted Transformer 28.9 41.4 Layer-wise Coordination 29.1 -Transformer (relative position) 29.2 41.5 Transformer 29 Relative position or local attention constraints bring improvements over origin self-attention model, but parallel multi-scale outperforms them. MUSE can also scale to small model and small datasets, as depicted in Table 2, MUSE-base pushes the state-of-the-art from 35.7 to 36.3 on IWSLT De-En translation dataset. It is shown in Table 1 and Table 2 that MUSE-simple which contains the basic idea of parallel multi-scale attention achieves state-of-the-art performance on three tasks. In this subsection we compare MUSE and its variants on IWSLT 2015 De-En translation to answer the question. Does concatenating self-attention with convolution certainly improve the model? To bridge the gap between point-wise transformation which learns token level representations and self-attention which learns representations of global context, we introduce convolution to enhance our multi-scale attention. As we can see from the first experiment group of Table 3, convolution is important in the parallel multi-scale attention. However, it is not easy to combine convolution and self-attention in one module to build better representations on sequence to sequence tasks. As shown in the first line of both second and third group of Table 3, simply learning local representations by using convolution or depth-wise separable convolution in parallel with self-attention harms the performance. Furthermore, combining depth-wise separable convolution (in this work we choose its best variant dynamic convolution as implementation) is even worse than combining convolution. Conv and self-attention? We conjecture that convolution and self-attention both learn contextual sequence representations and they should share the point transformation and perform the contextual transformation in the same hidden space. We first project the input to a hidden representation and perform a variant of depth-wise convolution and self-attention transformations in parallel. The fist two experiments in third group of Table 3 show that validating the utility of sharing Projection in parallel multi-scale attention, shared projection gain 1.4 BLEU scores over separate projection, and bring improvement of 0.5 BLEU scores over MUSE-simple (without DepthConv). How much is the kernel size? Comparative experiments show that the too large kernel harms performance both for DepthConv and convolution. Since there exists self-attention and point-wise transformations, simply applying the growing kernel size schedule proposed in SliceNet doesn't work. Thus, we propose to use dynamically selected kernel size to let the learned network decide the kernel size for each layer. Parallel multi-scale attention brings time efficiency on GPUs The underlying parallel structure (compared to the sequential structure in each block of Transformer) allows MUSE to be efficiently computed on GPUs. For example, we can combine small matrices into large matrices, and while it does not reduce the number of actual operations, it can be better paralleled by GPUs to speed up computation. Concretely, for each MUSE module, we first concentrate W Q, W K, W V of selfattention and W 1 of point feed-forward transformation into a single encoder matrix W Enc, and then perform transformation such as self-attention, depth-separable convolution, and nonlinear transformation, in parallel, to learn multi-scale representations in the hidden layer. W O, W 2, W out can also be combined a single decoder matrix W Dec. The decoder of sequence to sequence architecture can be implemented similarly. In Table 4, we conduct comparisons to show the speed gains with the aforementioned implementation, the batch size is set to one sample per batch to simulate online inference environment. Under the settings, where the numbers of parameters are similar for MUSE and Transformer, about 31% increase in inference speed can be obtained. The experiments use MUSE with 6 MUSE-simple modules and Transformer with 6 base blocks. The hidden size is set to 512. It is worth noticing that for the MUSE structure used in the main experiments, ideally a similar speedup can be witnessed if the computing device is powerful enough. However, such is not the case in our preliminary experiments. We also need to point out the implementation is far from fully optimized and the are only meant to demonstrate the feasibility of the procedure. Inference Speed (tokens/s) Transformer 132 MUSE 173 Acceleration 31% Parallel multi-scale attention generates much better long sequence As demonstrated in Figure 3, MUSE generates better sequences of various length than self-attention, but it is remarkably adept at generate long sequence, e.g. for sequence longer than 100, MUSE is two times better. Lower layers prefer local context and higher layers prefer more contextual representations MUSE contains multiple dynamic convolution cells, whose streams are fused by a gated mechanism. The weight for each dynamic cell is a scalar. Here we analyze the weight of different dynamic convolution cells in different layers. Figure 4 shows that as the layer depth increases, the weight of dynamic convolution cells with small kernel sizes gradually decreases. It demonstrates that lower layers prefer local features while higher layers prefer global features. It is corresponding to the finding in. MUSE not only gains BLEU scores, but also generates more reasonable sentences and increases the translation quality. We conduct the case study on the De-En dataset and the cases are shown in Table 5. In case 1, although the baseline transformer translates many correct words according to the source sentence, the translated sentence is not fluent at all. It indicates that Transformer does not capture the relationship between some words and their neighbors, such as "right" and "clap". By contrast, MUSE captures them well by combining local convolution with global selfattention. In case 2, the cause adverbial clause is correctly translated by MUSE while transformer misses the word "why" and fails to translate it. Sequence to sequence learning is an important task in machine learning. It evolves understanding and generating sequence. Machine translation is the touchstone of sequence to sequence learning. Traditional approaches usually adopt long-short term memory networks to learn the representation of sequences. However, these models either are built upon auto-regressive structures requiring longer encoding time or perform worse on real-world natural language processing tasks. Recent studies explore convolutional neural networks (CNN) or self-attention to support high-parallel sequence modeling and does not require auto-regressive structure during encoding, thus bringing large efficiency improvements. They are strong at capturing local or global dependencies. There are several studies on combining self-attention and convolution. However, they do not surpass both convectional and self-attention mechanisms. Sukhbaatar et al. (2019b) propose to augment convolution with self attention by directly concentrating them in computer vision tasks. However, as demonstrated in Table 3 there method does not work for sequence to sequence learning task. Since state-of-the-art models on question answering tasks still consist on self-attention and do no adopt ideas in QAnet . Both self-attention and convolution (a) outperforms Evolved transformer by near 2 BLEU scores on En-Fr translation. It seems that learning global and local context through stacking self-attention and convolution layers does not beat either self-attention or convolution models. In contrast, the proposed parallel multiscale attention outperforms previous convolution or self-attention based models on main translation tasks, showing its effectiveness for sequence to sequence learning. Although the self-attention mechanism has been prevalent in sequence modeling, we find that attention suffers from dispersed weights especially for long sequences, ing from the insufficient local information. To address this problem, we present Parallel Multi-scale Attention (MUSE) and MUSE-simple. MUSE-simple introduces the idea of parallel multi-scale attention into sequence to sequence learning. And MUSE fuses self-attention, convolution, and point-wise transformation together to explicitly learn global, local and token level sequence representations. Especially, we find from empirical that the shared projection plays important part in its success, and is essential for our multiscale learning. Beyond the inspiring new state-of-the-art on three machine translation datasets, detailed analysis and model variants also verify the effectiveness of MUSE. In future work, we would like to explore the detailed effects of shared projection on contextual representation learning. We are exited about future of parallel multi-scale attention and plan to apply this simple but effective idea to other tasks including image and speech. A.1 CASE STUDY Source wenn sie denken, dass die auf der linken seite jazz ist und die, auf der rechten seite swing ist, dann klatschen sie bitte. Target if you think the one on the left is jazz and the one on the right is swing, clap your hands. Transformer if you think it's jazz on the left, and those on the right side of the swing are clapping, please. MUSE if you think the one on the left is jazz, and the one on the right is swing, please clap. Case 2 Source und deswegen haben wir uns entschlossen in berlin eine halle zu bauen, in der wir sozusagen die elektrischen verhältnisse der insel im maßstab eins zu drei ganz genau abbilden können. Target and that's why we decided to build a hall in berlin, where we could precisely reconstruct, so to speak, the electrical ratio of the island on a one to three scale. Transformer and so in berlin, we decided to build a hall where we could sort of map the electrical proportions of the island at scale one to three very precisely. and that's why we decided to build a hall in berlin, where we can sort of map the electric relationship of the island at the scale one to three very precisely. Table 5: Case study on the De-En dataset. The blue bolded words denote the wrong translation and red bolded words denote the correct translation. In case 1, transformer fails to capture the relationship between some words and their neighbors, such as "right" and "clap". In case 2, the cause adverbial clause is correctly translated by MUSE while transformer misses the word "why" and fails to translate it.

Represent the TLDR for finding the corresponding scientific article

This paper propose a new model  which combines multi scale information for sequence to sequence learning.

We proposed SesameBERT, a generalized fine-tuning method that enables the extraction of global information among all layers through Squeeze and Excitation and enriches local information by capturing neighboring contexts via Gaussian blurring.