Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in most practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters. In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the co-natural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low resource scenario. It is good to have an end to journey toward; but it is the journey that matters, in the end. Endowing machine learning models with the capability to learn a variety of tasks in a sequential manner is critical to obtain agents that are both versatile and persistent. However, continual learning of multiple tasks is hampered by catastrophic forgetting , the tendency of previously acquired knowledge to be overwritten when learning a new task. Techniques to mitigate catastrophic forgetting can be roughly categorized into 3 lines of work (see for a comprehensive overview): 1. regularization-based approaches, where forgetting is mitigated by the addition of a penalty term in the learning objective (; Chaudhry et al. (2018a), inter alia), 2. dynamic architectures approaches, which incrementally increase the model's capacity to accomodate the new tasks , and 3. memorybased approaches, which retain data from learned tasks for later reuse (; b; . Among these, regularization-based approaches are particularly appealing because they do not increase the model size and do not require access to past data. This is particularly relevant to real-world scenarios where keeping data from previous training tasks may be impractical because of infrastructural or privacy-related reasons. Moreover, they are of independent intellectual interest because of their biological inspiration rooted in the idea of synaptic consolidation . A good regularizer ensures that, when learning a new task, gradient descent will ultimately converge to parameters that yield good on the new task while preserving performance on previously learned tasks. Critically, this is predicated upon successful optimization of the regularized objective, a fact that has been largely taken for granted in previous work. Non-convexity of the loss function, along with noise in the data (due to small or biased datasets) or in the gradients (due to stochastic gradient descent), can yield optimization trajectories -and ultimately convergence points -that are highly non-deterministic, even for the same starting parameters. As we demonstrate in this paper, this can cause unintended catastrophic forgetting along the optimization path. This is illustrated in a toy setting in Figure 1: a two parameter model is trained to perform task T 2 (an arbitrary bi-modal loss function) after having learned task T 1 (a logistic regression task). Standard finetuning, even in T 2 Finetuning EWC Co-natural finetuning Figure 1: On the importance of trajectories: an example with 2-dimensional logistic regression. Having learned task T 1, the model is trained on T 2 with two different objectives: minimizing the loss on T 2 (Finetuning) and a regularized objective (EWC;). We add a small amount of Gaussian noise to gradients in order to simulate the stochasticity of the trajectory. Plain finetuning and EWC often converge to a solution with high loss for T 1, but the co-natural optimization trajectory consistently converges towards the optimum with lowest loss for T 1. the presence of a regularized objective (EWC;), quickly changes the loss of T 1 and tends converge to a solution with high T 1 loss. We propose to remedy this issue by regularizing the optimization trajectory itself, specifically by preconditioning gradient descent with the empirical Fisher information of previously learned tasks (§3). This yields what we refer to as a co-natural gradient, an update rule inspired by the natural gradient , but taking the Fisher information of previous tasks as a natural Riemannian metric 2 of the parameter space, instead of the Fisher information of the task being optimized for. When we introduce our proposed co-natural gradient for the toy example of Figure 1, the learning trajectory follows a path that changes the loss on T 1 much more slowly, and tends to converges to the optimum that incurs the lowest performance degradation on T 1. We test the validity of our approach in a continual learning scenario (§4). We show that the co-natural gradient consistently reduces forgetting in a variety of existing continual learning approaches by a factor of ≈ 1.5 to 9, and greatly improves performance over simple finetuning, without modification to the training objective. We further investigate the special case of transfer learning in a two-task, low-resource scenario. In this specific case, control over the optimization trajectory is particularly useful because the optimizer has to rely on early stopping to prevent overfitting to the meager amount of training data in the target task. We show that the co-natural gradient yields the best trade-offs between source and target domain performance over a variety of hyper-parameters (§5). We first give a brief overview of the continual learning paradigm and existing approaches for overcoming catastrophic forgetting. Let us define a task as a triplet containing an input space X and an output space Y, both measurable spaces, as well as a distribution D over X × Y. In general, learning a task will consist of training a model to approximate the conditional distribution p(y | x) induced by D. Consider a probabilistic model p θ parametrized by θ ∈ R d where d is the size of the model, trained to perform a source task S = X S, Y S, D S to some level of performance, yielding parameters θ S. In the most simple instance of continual learning, we are tasked with learning a second target task T = X T, Y T, D T. In general in a multitask setting, it is not the case that the input or output spaces are the same. The discrepancy between input/output space can be addressed in various ways, e.g. by adding a minimal number of task-specific parameters (for example, different softmax layers for different label sets). To simplify exposition, we set these more specific considerations aside for now, and assume that X S = X T and Y S = Y T. At any given point during training for task T, our objective will be to minimize the loss function L T (θ) -generally the expected log-likelihood E x,y∼D T [− log p θ (y | x)]. Typically, this will be performed by iteratively adding incremental update vectors δ ∈ R d to the parameters θ ←− θ + δ. In this paper, we focus on those models that have a fixed architecture over the course of continual learning. The study of continual learning for models of fixed capacity can be split into two distinct (but often overlapping) streams of work: Regularization-based approaches introduce a penalty in the loss function L T, typically quadratic, pushing the weights θ back towards θ S: where Ω S is a matrix, typically diagonal, that encodes the respective importance of each parameter with respect to task S, and λ is a regularization strength hyper-parameter. Various choices have been proposed for Ω S; the diagonal empirical Fisher information matrix , or pathintegral based importance measures (; a). More elaborate regularizers have been proposed based on e.g. a Bayesian formulation of continual learning or a distillation term . The main advantage of these approaches is that they do not rely on having access to training data of previous tasks. Memory-based approaches store data from previously seen tasks for re-use in continued learning, either as a form of constraint, by e.g. ensuring that training on the new task doesn't increase the loss on previous tasks (; b), or for replay i.e. by retraining on instances from previous tasks (; ; b; a). Various techniques have been proposed for the selection of samples to store in the memory (; b) or for retrieval of the samples to be used for replay Aljundi et al. (2019a). All of these methods rely on stochastic gradient descent to optimize their regularized objective or to perform experience replay, with the notable exception of GEM (; b), where the gradients are projected onto the orthogonal complement of previous task's gradients. However, this method has been shown to perform poorly in comparison with simple replay , and it still necessitates access to data from previous tasks. After briefly recalling how the usual update is obtained in gradient descent, we derive a new, conatural update designed to better preserve the distribution induced by the model over previous tasks. At point θ in the parameter space, gradient descent finds the optimal update δ that is small and locally minimizes the decrease in loss Traditionally this can be formulated as minimizing the Lagrangian: with Lagrangian multiplier µ > 0. Minimizing L for δ yields the well-known optimal update δ *: 1 2µ corresponds to the learning rate (see Appendix A.1 for the full derivation). The δ 2 term in L implicitly expresses the underlying assumption that the best measure of distance between parameters θ and θ + δ is the Euclidean distance. In a continual learning setting however, the quantity we are most interested in preserving is the probability distribution that θ models on the source task S: Therefore, a more natural distance between θ and θ + δ is the Kullback-Leibler divergence KL(p S θ p S θ+δ) . For preventing catastrophic forgetting along the optimization path, we incorporate incorporate this KL term into the Lagrangian L itself: Doing so means that the optimization trajectory will tend to follow the direction that changes the distribution of the model the least. Notably, this is not a function of the previous objective L S, so knowledge of the original training objective is not necessary during continual learning (which is typically the case in path-integral based regularization methods or experience replay ). Presuming that δ is small, we can perform a second order Taylor approximation of the function δ → KL(p where F S θ is the Hessian of the KL divergence around θ. A crucial, well-known property of this matrix is that it coincides with the Fisher information matrix (the expectation being taken over the model's distribution p θ ; see Appendix A.1 for details). This is appealing from a computational perspective because the Fisher can be computed by means of first order derivatives only. Minimizing for δ yields the following optimal update: where coefficients µ and ν are folded into two hyper-parameters: the learning rate λ and a damping coefficient α (the step-by-step derivation can be found in Appendix A.1). In practice, especially with low damping coefficients, it is common to obtain updates that are too large (typically when some parameters have no effect on the KL divergence). To address this, we re-normalize δ * to have the same norm as the original gradient, ∇L T. For computational reasons, we will make 3 key practical approximations to the Fisher:: we maintain the Fisher computed at θ S, instead of recomputing F S at every step of training. This relieves us of the computational burden of updating the Fisher for every new value of θ. This approximation (shared by previous work, e.g. ; Chaudhry et al. (2018a) ) is only valid insofar as θ S and θ are close. Empirically we observe that this still leads to good . S is diagonal: this is a common approximation in practice with two appealing properties. First, this makes it possible to store the d diagonal Fisher coefficients in memory. Second, this trivializes the inverse operation (simply invert the diagonal elements). this common approximation replaces the expectation under the model's distribution by the expected log-likelihood of the true distribution: T ] (mind the subscript). This is particularly useful in tasks with a large or unbounded number of classes (e.g. structured prediction), where summing over all possible outputs is intractable. We can then compute the diagonal of the empirical Fisher using Monte Carlo sampling: 2 with (x i, y i) sampled from D S (we use N = 1000 for all experiments). This formulation bears many similarities with the natural gradient from , which also uses the KL divergence as a metric for choosing the optimal update δ *. There is a however a crucial difference, both in execution and purpose: where the natural gradient uses knowledge of the curvature of the KL divergence of D T to speed-up convergence, our proposed method leverages the curvature of the KL divergence on D S to slow-down divergence from p S θ S. To highlight the resemblance and complementarity between these two concepts, we refer to the new update as the co-natural gradient. In a continual learning scenario, we are confronted with a large number of tasks T 1... T n presented in sequential order. When learning T n, we can change the Lagrangian L from 5 to incorporate the constraints for all previous tasks T 1... T n−1: This in turn changes the Fisher in Eq. 8 toF n−1:= The choice of the coefficients ν i is crucial. Setting all ν i to the same value, i.e. assigning the same importance to all tasks is suboptimal for a few reasons. First and foremost, it is unreasonable to expect of a model with finite capacity to remember an unbounded number of tasks (as tasks "fill-up" the model capacity, F n−1 is likely to become more "homogeneous"). Second, as training progresses and θ changes, our approximation that is less and less likely to hold. We address this issue in the same fashion as , by keeping a rolling exponential average of the Fisher matrices: In this case, previous tasks are gracefully forgotten at an exponential rate controlled by γ. We account for the damping α term in Eq. 7 by settingF 0:= α γ I. In preliminary experiments, we have found γ = 0.9 to yield consistently good , and use this value in all presented experiments. To corroborate our hypothesis that controlling the optimization trajectory with the co-natural gradient reduces catastrophic forgetting, we perform experiments on two continual learning testbeds: • Split CIFAR: The CIFAR100 dataset, split into 20 independent 5-way classification tasks. Similarly to Chaudhry et al. (2018b), we use a smaller version of the ResNet architecture . • Omniglot: the Omniglot dataset consists of 50 independent character recognition datasets on different alphabet. We adopt the setting of and consider each alphabet as a separate task. 4 On this dataset we use the same small CNN architecture as. • Split MiniImageNet: The MiniImageNet dataset (a subset of the popular ImageNet dataset 5; ). Split the dataset into 20 independent 5-way classification tasks, similarly to Split CIFAR, and use the same smaller ResNet. We adopt the experimental setup from: in each dataset we create a "validation set" of 3 tasks, used to select the best hyper-parameters, and keep the remaining tasks for evaluation. This split is chosen at random and kept the same across all experiments. In these datasets, the nature and possibly the number of classes for each task changes. We account for this by training a separate softmax layer for each task, and apply continual learning only to the remaining, "feature-extraction" part of the model. We report along two common metrics for continual learning: average accuracy, the accuracy at the end of training averaged over all tasks, and forgetting. Forgetting is defined in Chaudhry et al. (2018a) as the difference in performance on a task between the current model and the best performing model on this task. Formally if A T t represents the accuracy on task T at step t of training, the forgetting F T t at step t is defined as F We implement the co-natural update rule on top of 3 baselines: • Finetuning: Simply train the model on the task at hand, without any form of regularization. • EWC: Proposed by , it is a simple but effective quadratic regularization approach. While neither the most recent nor sophisticate regularization technique, it is a natural baseline for us to compare to in that it also consists in a Fisher-based penalty -although in the For visibility we only show accuracies for every fifth task. The rectangular shaded regions delineate the period during which each task is being trained upon; with the exception of ER, this is the only period the model has access to the data for this task. loss function instead of the optimization dynamics. We also use the rolling Fisher described in Section 3.4, making our EWC baseline equivalent to the superior online EWC introduced by. • ER: Experience replay with a fixed sized episodic memory proposed by. While not directly comparable to EWC in that it presupposes access to data from previous tasks, ER is a simple approach that boasts the best performances on a variety of benchmarks . In all experiments, we use memory size 1,000 with reservoir sampling. Training proceeds as follows: we perform exhaustive search on all the hyper-parameter combinations using the validation tasks. Every combination is reran 3 times (the order of tasks, model initialization and order of training examples changes with each restart), and rated by accuracy averaged over tasks and restarts. We then evaluate the best hyper-parameters by continual training on the evaluation tasks. Results are reported over 5 random restarts (3 for MiniImageNet), and we control for statistical significance using a paired t-test (we pair together runs with the same task ordering). We refer to Appendix A.2 for more details regarding fine-grained design choices. The upper half of Table 1 reports the average accuracy of all the tasks at the end of training (higher is better). We observe that the co-natural gradient always improves greatly over simple finetuning, and occasionally over EWC and ER. We note that on both datasets, bare-bone co-natural finetuning matches or exceeds the performance of EWC and ER even though it requires strictly fewer resources (no need to store the previous parameters as in EWC, or data in ER). Even more appreciable is the effect of the co-natural trajectories on forgetting, as shown in the lower half of Table 1. As evidenced by the in the lowest row, using the co-natural gradient systematically in large drops in forgetting across all approaches and both datasets, even when the average accuracy is not increased. To get a qualitative assessment of the learning trajectories that yield such , we visualize the accuracy curves of 10 out of the 47 evaluation tasks of Omniglot in Figure 2. We observe that previous approaches do poorly at keeping stable levels of performance over a long period of time (especially for tasks learned early in training), a problem that is largely resolved by the co-natural preconditioning. This seems to come at the cost of more intransigence (a), i.e. some of the later tasks are not being learnt properly. In models of fixed capacity, there is a natural trade-off between intransigence and forgetting (see also the "stability-plasticity" dilemma in neuroscience). Our position the co-natural gradient as a strong lowforgetting/moderate intransigence basis for future work. In this section we take a closer look at the specific case of adapting a model from a single task to another, when we only have access to a minimal amount of data in the target task. In this case, controlling the learning trajectory is particularly important because the model is being trained on an unreliable sample of the true distribution of the target task, and we have to rely on early-stopping to prevent overfitting. We show that using the co-natural gradient during adaptation helps both at preserving source task performance and reach higher overall target task performance. We perform experiments on two different scenarios: Image classification We take MiniImagenet as a source task and CUB (a 200-way birds species classification dataset;) as a target task. To guarantee a strong base model despite the small size of MiniImageNet, we start off from a ResNet18 model pretrained on the full ImageNet, which we retrofit to MiniImageNet by replacing the last fully connected layer with a separate linear layer regressed over the MiniImageNet training data. To simulate a low-resource setting, we sub-sample the CUB training set to 200 images (≈ 1 per class). Scores for these tasks are reported in terms of accuracy. Machine translation We consider adaptation of an English to French model trained on WMT15 (a dataset of parallel sentences crawled from parliamentary proceedings, news commentary and web page crawls;) to MTNT (a dataset of Reddit comments;). Our model is a Transformer pretrained on WMT15. Similarly to CUB, we simulate a low-resource setting by taking a sub-sample of 1000 sentence pairs as a training set. Scores for these two datasets are reported in terms of BLEU score. 6 Here we do not allow any access to data in the source task when training on the target task. We compare four methods Finetuning (our baseline), Co-natural finetuning, EWC (which has been proven effective for domain adaptation, see) and Co-natural EWC. Given that different methods might lead to different trade-offs between source and target task performance, with some variation depending on the hyper-parameters (e.g. learning rate, regularization strength. . .), we take inspiration from and graphically report for all hyper-parameter configuration of each method on the 2 dimensional space defined by the score on source and target tasks 7. Additionally, we highlight the Pareto frontier of each method i.e. the set of configurations that are not strictly worse than any other configuration for the same model. The adaptation for both scenarios are reported in Figure 3. We find that in both cases, the co-natural gradient not only helps preserving the source task performance, but to some extent it also allows the model to reach better performance on the target task as well. We take this to corroborate our starting hypothesis: while introducing a regularizer does help, controlling the optimization dynamics actively helps counteract overfitting to the very small amount of training data, because the co-natural pre-conditioning makes it harder for stochastic gradient descent to push the model towards directions that would also hurt the source task. We have presented the co-natural gradient, a technique that regularizes the optimization trajectory of models trained in a continual setting. We have shown that the co-natural gradient stands on its own as an efficient approach for overcoming catastrophic forgetting, and that it effectively complements and stabilizes other existing techniques at a minimal cost. We believe that the co-natural gradientand more generally, trajectory regularization -can serve as a solid bedrock for building agents that learn without forgetting. We solve the Lagrangian from Eq. 6 in a similar fashion as in A.1.1. First we compute its gradient and Hessian: ∇L = ∇L T + 2µδ + 2νF This section is intended to facilitate the reproduction of our . The full details can be found with our code at anonymized url. We split the dataset into 20 disjoint sub-tasks with each 5 classes, 2500 training examples and 500 test examples. This split, performed at random, is kept the same across all experiments, only the order of these tasks is changed. During continual training, we train the model for one epoch on each task with batch size 10, following the setup in Chaudhry et al. (2018b). We consider each alphabet as a separate task, and split each task such that every character is present 12, 4 and 4 times in the training, validation and test set respectively (out of the 20 images for each character). During continual training, we train for 2500 steps with batch size 32 (in keeping with). We ignore the validation data and simply evaluate on the test set at the end of training. For each method, we perform grid-search over the following parameter values: • Learning rate (all methods): 0.1, 0.03, 0.01 • Regularization strength (EWC, Co-natural EWC): 0.5, 1, 5 • Fisher damping coefficient (Co-natural finetuning, Co-natural EWC): 0,1.0,0.1 for Split CIFAR and 0,0.1,0.01 for Omniglot For ER, we simply set the batch size to the same value as standard training (10 and 32 for Split CIFAR and Omniglot respectively). Note that whenever applicable, we re-normalize the diagonal Fisher so that the sum of its weights is equal to the number of parameters in the model. This is so that the hyper-parameter choice is less dependent on the size of the model. In particular this means that the magnitude of each diagonal element is much bigger, which is why we do grid-search over smaller regularization parameters for EWC.

represent the tldr for finding the corresponding scientific article

Regularizing the optimization trajectory with the Fisher information of old tasks reduces catastrophic forgetting greatly

Represent the short summary for finding the corresponding scientific article (long)

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We study the problem of generating source code in a strongly typed, Java-like programming language, given a label (for example a set of API calls or types) carrying a small amount of information about the code that is desired. The generated programs are expected to respect a `"realistic" relationship between programs and labels, as exemplified by a corpus of labeled programs available during training. Two challenges in such *conditional program generation* are that the generated programs must satisfy a rich set of syntactic and semantic constraints, and that source code contains many low-level features that impede learning. We address these problems by training a neural generator not on code but on *program sketches*, or models of program syntax that abstract out names and operations that do not generalize across programs. During generation, we infer a posterior distribution over sketches, then concretize samples from this distribution into type-safe programs using combinatorial techniques. We implement our ideas in a system for generating API-heavy Java code, and show that it can often predict the entire body of a method given just a few API calls or data types that appear in the method. Neural networks have been successfully applied to many generative modeling tasks in the recent past BID22 BID11 BID33. However, the use of these models in generating highly structured text remains relatively understudied. In this paper, we present a method, combining neural and combinatorial techniques, for the condition generation of an important category of such text: the source code of programs in Java-like programming languages. The specific problem we consider is one of supervised learning. During training, we are given a set of programs, each program annotated with a label, which may contain information such as the set of API calls or the types used in the code. Our goal is to learn a function g such that for a test case of the form (X, Prog) (where Prog is a program and X is a label), g(X) is a compilable, type-safe program that is equivalent to Prog. This problem has immediate applications in helping humans solve programming tasks BID12 BID26. In the usage scenario that we envision, a human programmer uses a label to specify a small amount of information about a program that they have in mind. Based on this information, our generator seeks to produce a program equivalent to the "target" program, thus performing a particularly powerful form of code completion. Conditional program generation is a special case of program synthesis BID19 BID32, the classic problem of generating a program given a constraint on its behavior. This problem has received significant interest in recent years BID2 BID10. In particular, several neural approaches to program synthesis driven by input-output examples have emerged BID3 BID23 BID5. Fundamentally, these approaches are tasked with associating a program's syntax with its semantics. As doing so in general is extremely hard, these methods choose to only generate programs in highly controlled domainspecific languages. For example, BID3 consider a functional language in which the only data types permitted are integers and integer arrays, control flow is linear, and there is a sum total of 15 library functions. Given a set of input-output examples, their method predicts a vector of binary attributes indicating the presence or absence of various tokens (library functions) in the target program, and uses this prediction to guide a combinatorial search for programs. In contrast, in conditional program generation, we are already given a set of tokens (for example library functions or types) that appear in a program or its metadata. Thus, we sidestep the problem of learning the semantics of the programming language from data. We ask: does this simpler setting permit the generation of programs from a much richer, Java-like language, with one has thousands of data types and API methods, rich control flow and exception handling, and a strong type system? While simpler than general program synthesis, this problem is still highly nontrivial. Perhaps the central issue is that to be acceptable to a compiler, a generated program must satisfy a rich set of structural and semantic constraints such as "do not use undeclared variables as arguments to a procedure call" or "only use API calls and variables in a type-safe way". Learning such constraints automatically from data is hard. Moreover, as this is also a supervised learning problem, the generated programs also have to follow the patterns in the data while satisfying these constraints. We approach this problem with a combination of neural learning and type-guided combinatorial search BID6. Our central idea is to learn not over source code, but over tree-structured syntactic models, or sketches, of programs. A sketch abstracts out low-level names and operations from a program, but retains information about the program's control structure, the orders in which it invokes API methods, and the types of arguments and return values of these methods. We propose a particular kind of probabilistic encoder-decoder, called a Gaussian Encoder-Decoder or GED, to learn a distribution over sketches conditioned on labels. During synthesis, we sample sketches from this distribution, then flesh out these samples into type-safe programs using a combinatorial method for program synthesis. Doing so effectively is possible because our sketches are designed to contain rich information about control flow and types. We have implemented our approach in a system called BAYOU. 1 We evaluate BAYOU in the generation of API-manipulating Android methods, using a corpus of about 150,000 methods drawn from an online repository. Our experiments show that BAYOU can often generate complex method bodies, including methods implementing tasks not encountered during training, given a few tokens as input. Now we define conditional program generation. Assume a universe P of programs and a universe X of labels. Also assume a set of training examples of the form {(X 1, Prog 1), (X 2, Prog 2),...}, where each X i is a label and each Prog i is a program. These examples are sampled from an unknown distribution Q(X, Prog), where X and Prog range over labels and programs, respectively. 2 We assume an equivalence relation Eqv ⊆ P × P over programs. If (Prog 1, Prog 2) ∈ Eqv, then Prog 1 and Prog 2 are functionally equivalent. The definition of functional equivalence differs across applications, but in general it asserts that two programs are "just as good as" one another. The goal of conditional program generation is to use the training set to learn a function g: X → P such that the expected value E[I((g(X), Prog) ∈ Eqv )] is maximized. Here, I is the indicator function, returning 1 if its boolean argument is true, and 0 otherwise. Informally, we are attempting to learn a function g such that if we sample (X, Prog) ∼ Q(X, P rog), g should be able to reconstitute a program that is functionally equivalent to Prog, using only the label X. In this paper, we consider a particular form of conditional program generation. We take the domain P to be the set of possible programs in a programming language called AML that captures the essence of API-heavy Java programs (see Appendix A for more details). AML includes complex control flow such as loops, if-then statements, and exceptions; access to Java API data types; and calls to Java API methods. AML is a strongly typed language, and by definition, P only includes programs DISPLAYFORM0 Figure 1: Programs generated by BAYOU with the API method name readLine as a label. Names of variables of type T whose values are obtained from the environment are of the form $T.that are type-safe. 3 To define labels, we assume three finite sets: a set Calls of possible API calls in AML, a set Types of possible object types, and a set Keys of keywords, defined as words, such as "read" and "file", that often appear in textual descriptions of what programs do. The space of possible labels is X = 2 Calls × 2 T ypes × 2 Keys (here 2 S is the power set of S).Defining Eqv in practice is tricky. For example, a reasonable definition of Eqv is that (Prog 1, Prog 2) ∈ Eqv iff Prog 1 and Prog 2 produce the same outputs on all inputs. But given the richness of AML, the problem of determining whether two AML programs always produce the same output is undecidable. As such, in practice we can only measure success indirectly, by checking whether the programs use the same control structures, and whether they can produce the same API call sequences. We will discuss this issue more in Section 6. Consider the label X = (X Calls, X Types, X Keys) where X Calls = {readLine} and X Types and X Keys are empty. Figure 1 (a) shows a program that our best learner stochastically returns given this input. As we see, this program indeed reads lines from a file, whose name is given by a special variable $String that the code takes as input. It also handles exceptions and closes the reader, even though these actions were not directly specified. Although the program in Figure 1 -(a) matches the label well, failures do occur. Sometimes, the system generates a program as in Figure 1 -(b), which uses an InputStreamReader rather than a FileReader. It is possible to rule out this program by adding to the label. Suppose we amend X T ypes so that X T ypes = {FileReader}. BAYOU now tends to only generate programs that use FileReader. The variations then arise from different ways of handling exceptions and constructing FileReader objects (some programs use a String argument, while others use a File object). Our approach is to learn g via maximum conditional likelihood estimation (CLE). That is, given a distribution family P (P rog|X, θ) for a parameter set θ, we choose θ * = arg max θ i log P (Prog i | X i, θ). Then, g(X) = arg max Prog P (Prog|X, θ *).The key innovation of our approach is that here, learning happens at a higher level of abstraction than (X i, Prog i) pairs. In practice, Java-like programs contain many low-level details (for example, variable names and intermediate ) that can obscure patterns in code. Further, they contain complicated semantic rules (for example, for type safety) that are difficult to learn from data. In contrast, these are relatively easy for a combinatorial, syntax-guided program synthesizer BID2 to deal with. However, synthesizers have a notoriously difficult time figuring out the correct "shape" of a program (such as the placement of loops and conditionals), which we hypothesize should be relatively easy for a statistical learner. Specifically, our approach learns over sketches: tree-structured data that capture key facets of program syntax. A sketch Y does not contain low-level variable names and operations, but carries information about broadly shared facets of programs such as the types and API calls. During generation, a program synthesizer is used to generate programs from sketches produced by the learner. Let the universe of all sketches be denoted by Y. The sketch for a given program is computed by applying an abstraction function α: P → Y. We call a sketch Y satisfiable, and write sat(Y), if α −1 (Y) = ∅. The process of generating (type-safe) programs given a satisfiable sketch Y is probabilistic, and captured by a concretization distribution P (Prog | Y, sat(Y)). We require that for all programs Prog and sketches Y such that sat(Y), we have DISPLAYFORM0 Importantly, the concretization distribution is fixed and chosen heuristically. The alternative of learning this distribution from source code poses difficulties: a single sketch can correspond to many programs that only differ in superficial details, and deciding which differences between programs are superficial and which are not requires knowledge about program semantics. In contrast, our heuristic approach utilizes known semantic properties of programming languages like oursfor example, that local variable names do not matter, and that some algebraic expressions are semantically equivalent. This knowledge allows us to limit the set of programs that we generate. Let us define a random variable Y = α(Prog). We assume that the variables X, Y and Prog are related as in the Bayes net in Figure 2. Specifically, given Y, Prog is conditionally independent of X. Further, let us assume a distribution family P (Y |X, θ) parameterized on θ. DISPLAYFORM1 DISPLAYFORM2 Our problem now simplifies to learning over sketches, i.e., finding FIG1 shows the full grammar for sketches in our implementation. Here, τ 0, τ 1,... range over a finite set of API data types that AML programs can use. A data type, akin to a Java class, is identified with a finite set of API method names (including constructors), and a ranges over these names. Note that sketches do not contain constants or variable names. DISPLAYFORM3 A full definition of the abstraction function for AML appears in Appendix B. As an example, API calls in AML have the syntax "call e.a(e 1, . . ., e k)", where a is an API method, the expression e evaluates to the object on which the method is called, and the expressions e 1,..., e k evaluate to the arguments of the method call. We abstract this call into an abstract method call "call τ.a(τ 1, . . ., τ k)", where τ is the type of e and τ i is the type of e i. The keywords skip, while, if-then-else, and trycatch preserve information about control flow and exception handling. Boolean conditions Cseq are replaced by abstract expressions: lists whose elements abstract the API calls in Cseq. Now we describe our learning approach. Equation 1 leaves us with the problem of computing arg max θ i log P (Y i |X i, θ), when each X i is a label and Y i is a sketch. Our answer is to utilize an encoder-decoder and introduce a real vector-valued latent variable Z to stochastically link labels and sketches: DISPLAYFORM0 is realized as a probabilistic decoder mapping a vector-valued variable to a distribution over trees. We describe this decoder in Appendix C. As for P (Z|X, θ), this distribution can, in principle, be picked in any way we like. In practice, because both P (Y |Z, θ) and P (Z|X, θ) have neural components with numerous parameters, we wish this distribution to regularize the learner. To provide this regularization, we assume a Normal (0, I) prior on Z.Recall that our labels are of the form X = (X Calls, X T ypes, X Keys), where X Calls, X Types, and X Keys are sets. Assuming that the j-th elements X Calls,j, X Types,j, and X Keys,j of these sets are generated independently, and assuming a function f for encoding these elements, let: DISPLAYFORM1 That is, the encoded value of each X Types,j, X Calls,j or X Keys,j is sampled from a high-dimensional Normal distribution centered at Z. If f is 1-1 and onto with the set R m then from Normal-Normal conjugacy, we have: DISPLAYFORM2 1+n I, where DISPLAYFORM3 Keys. Here, n Types is the number of types supplied, and n Calls and n Keys are defined similarly. Note that this particular P (Z|X, θ) only follows directly from the Normal (0, I) prior on Z and Normal likelihood P (X|Z, θ) if the encoding function f is 1-1 and onto. However, even if f is not 1-1 and onto (as will be the case if f is implemented with a standard feed-forward neural network) we can still use this probabilistic encoder, and in practice we still tend to see the benefits of the regularizing prior on Z, with P (Z) distributed approximately according to a unit Normal. We call this type of encoder-decoder, with a single, Normally-distributed latent variable Z linking the input and output, a Gaussian encoder-decoder, or GED for short. Now that we have chosen P (X|Z, θ) and P (Y |Z, θ), we must choose θ to perform CLE. Note that: DISPLAYFORM4 where the ≥ holds due to Jensen's inequality. Hence, L(θ) serves as a lower bound on the loglikelihood, and so we can compute θ * = arg max θ L(θ) as a proxy for the CLE. We maximize this lower bound using stochastic gradient ascent; as P (Z|X i, θ) is Normal, we can use the reparameterization trick common in variational auto-encoders BID14 while doing so. The parameter set θ contains all of the parameters of the encoding function f as well as σ Types, σ Calls, and σ Keys, and the parameters used in the decoding distribution funciton P (Y |Z, θ). The final step in our algorithm is to "concretize" sketches into programs, following the distribution P (Prog|Y). Our method of doing so is a type-directed, stochastic search procedure that builds on combinatorial methods for program synthesis BID28 BID6.Given a sketch Y, our procedure performs a random walk in a space of partially concretized sketches (PCSs). A PCS is a term obtained by replacing some of the abstract method calls and expressions in a sketch by AML method calls and AML expressions. For example, the term "x 1.a(x 2); τ 1.b(τ 2)", which sequential composes an abstract method call to b and a "concrete" method call to a, is a PCS. The state of the procedure at the i-th point of the walk is a PCS H i. The initial state is Y.Each state H has a set of neighbors Next(H). This set consists of all PCS-s H that are obtained by concretizing a single abstract method call or expression in H, using variable names in a way that is consistent with the types of all API methods and declared variables in H.The (i + 1)-th state in a walk is a sample from a predefined, heuristically chosen distribution P (H i+1 | H i,). The only requirement on this distribution is that it assigns nonzero probability to a state iff it belongs to Next(H i). In practice, our implementation of this distribution prioritizes programs that are simpler. The random walk ends when it reaches a state H * that has no neighbors. If H * is fully concrete (that is, an AML program), then the walk is successful and H * is returned as a sample. If not, the current walk is rejected, and a fresh walk is started from the initial state. Recall that the concretization distribution P (Prog|Y) is only defined for sketches Y that are satisfiable. Our concretization procedure does not assume that its input Y is satisfiable. However, if Y is not satisfiable, all random walks that it performs end with rejection, causing it to never terminate. While the worst-case complexity of this procedure is exponential in the generated programs, it performs well in practice because of our chosen language of sketches. For instance, our search does not need to discover the high-level structure of programs. Also, sketches specify the types of method arguments and return values, and this significantly limits the search space. Now we present an empirical evaluation of the effectiveness of our method. The experiments we describe utilize data from an online repository of about 1500 Android apps (and, 2017). We decompiled the APKs using JADX BID29 to generate their source code. Analyzing about 100 million lines of code that were generated, we extracted 150,000 methods that used Android APIs or the Java library. We then pre-processed all method bodies to translate the code from Java to AML, preserving names of relevant API calls and data types as well as the high-level control flow. Hereafter, when we say "program" we refer to an AML program. Figure 4: Statistics on labelsFrom each program, we extracted the sets X Calls, X Types, and X Keys as well as a sketch Y. Lacking separate natural language dscriptions for programs, we defined keywords to be words obtained by splitting the names of the API types and calls that the program uses, based on camel case. For instance, the keywords obtained from the API call readLine are "read" and "line". As API method and types in Java tend to be carefully named, these words often contain rich information about what programs do. Figure 4 gives some statistics on the sizes of the labels in the data. From the extracted data, we randomly selected 10,000 programs to be in the testing and validation data each. We implemented our approach in our tool called BAYOU, using TensorFlow BID1 to implement the GED neural model, and the Eclipse IDE for the abstraction from Java to the language of sketches and the combinatorial concretization. In all our experiments we performed cross-validation through grid search and picked the best performing model. Our hyper-parameters for training the model are as follows. We used 64, 32 and 64 units in the encoder for API calls, types and keywords, respectively, and 128 units in the decoder. The latent space was 32-dimensional. We used a mini-batch size of 50, a learning rate of 0.0006 for the Adam gradient-descent optimizer BID13, and ran the training for 50 epochs. The training was performed on an AWS "p2.xlarge" machine with an NVIDIA K80 GPU with 12GB GPU memory. As each sketch was broken down into a set of production paths, the total number of data points fed to the model was around 700,000 per epoch. Training took 10 hours to complete. To visualize clustering in the 32-dimensional latent space, we provided labels X from the testing data and sampled Z from P (Z|X), and then used it to sample a sketch from P (Y |Z). We then used t-SNE BID17 to reduce the dimensionality of Z to 2-dimensions, and labeled each point with the API used in the sketch Y. Figure 5 shows this 2-dimensional space, where each label has been coded with a different color. It is immediately apparent from the plot that the model has learned to cluster the latent space neatly according to different APIs. Some APIs such as java.io have several modes, and we noticed separately that each mode corresponds to different usage scenarios of the API, such as reading versus writing in this case. To evaluate prediction accuracy, we provided labels from the testing data to our model, sampled sketches from the distribution P (Y |X) and concretized each sketch into an AML program using our combinatorial search. We then measured the number of test programs for which a program that is equivalent to the expected one appeared in the top-10 from the model. As there is no universal metric to measure program equivalence (in fact, it is an undecidable problem in general), we used several metrics to approximate the notion of equivalence. We defined the following metrics on the top-10 programs predicted by the model:M1. This binary metric measures whether the expected program appeared in a syntactically equivalent form in the . Of course, an impediment to measuring this is that the names of variables used in the expected and predicted programs may not match. It is neither reasonable nor useful for any model of code to learn the exact variable names in the training data. Therefore, in performing this equivalence check, we abstract away the variable names and compare the rest of the program's Abstract Syntax Tree (AST) instead. M2. This metric measures the minimum Jaccard distance between the sets of sequences of API calls made by the expected and predicted programs. It is a measure of how close to the original program were we able to get in terms of sequences of API calls. M3. Similar to metric M2, this metric measures the minimum Jaccard distance between the sets of API calls in the expected and predicted programs. M4. This metric computes the minimum absolute difference between the number of statements in the expected and sampled programs, as a ratio of that in the former. M5. Similar to metric M4, this metric computes the minumum absolute difference between the number of control structures in the expected and sampled programs, as a ratio of that in the former. Examples of control structures are branches, loops, and try-catch statements. To evaluate our model's ability to predict programs given a small amount of information about its code, we varied the fraction of the set of API calls, types, and keywords provided as input from the testing data. We experimented with 75%, 50% and 25% observability in the testing data; the median number of items in a label in these cases were 9, 6, and 2, respectively. Figure 6: Accuracy of different models on testing data. GED-AML and GSNN-AML are baseline models trained over AML ASTs, GED-Sk and GSNN-Sk are models trained over sketches. In order to compare our model with state-of-the-art conditional generative models, we implemented the Gaussian Stochastic Neural Network (GSNN) presented by BID30, using the same tree-structured decoder as the GED. There are two main differences: (i) the GSNN's decoder is also conditioned directly on the input label X in addition to Z, which we accomplish by concatenating its initial state with the encoding of X, (ii) the GSNN loss function has an additional KL-divergence term weighted by a hyper-parameter β. We subjected the GSNN to the same training and crossvalidation process as our model. In the end, we selected a model that happened to have very similar hyper-parameters as ours, with β = 0.001. In order to evaluate the effect of sketch learning for program generation, we implemented and compared with a model that learns directly over programs. Specifically, the neural network structure is exactly the same as ours, except that instead of being trained on production paths in the sketches, the model is trained on production paths in the ASTs of the AML programs. We selected a model that had more units in the decoder compared to our model, as the AML grammar is more complex than the grammar of sketches. We also implemented a similar GSNN model to train over AML ASTs directly. Figure 6 shows the collated of this evaluation, where each entry computes the average of the corresponding metric over the 10000 test programs. It takes our model about 8 seconds, on average, to generate and rank 10 programs. When testing models that were trained on AML ASTs, namely the GED-AML and GSNN-AML models, we observed that out of a total of 87,486 AML ASTs sampled from the two models, 2525 (or 3%) ASTs were not even well-formed, i.e., they would not pass a parser, and hence had to be discarded from the metrics. This number is 0 for the GED-Sk and GSNN-Sk models, meaning that all AML ASTs that were obtained by concretizing sketches were well-formed. In general, one can observe that the GED-Sk model performs best overall, with GSNN-Sk a reasonable alternative. We hypothesize that the reason GED-Sk performs slightly better is the regularizing prior on Z; since the GSNN has a direct link from X to Y, it can choose to ignore this regularization. We would classify both these models as suitable for conditional program generation. However, the other two models GED-AML and GSNN-AML perform quite worse, showing that sketch learning is key in addressing the problem of conditional program generation. To evaluate how well our model generalizes to unseen data, we gather a subset of the testing data whose data points, consisting of label-sketch pairs (X, Y), never occurred in the training data. We then evaluate the same metrics in Figure 6 (a)-(e), but due to space reasons we focus on the 50% observability column. Figure 6 (f) shows the of this evaluation on the subset of 5126 (out of 10000) unseen test data points. The metrics exhibit a similar trend, showing that the models based on sketch learning are able to generalize much better than the baseline models, and that the GED-Sk model performs the best. Unconditional, corpus-driven generation of programs has been studied before BID18; BID4, as has the generation of code snippets conditioned on a context into which the snippet is merged BID21 BID26 BID20. These prior efforts often use models like n-grams BID21 and recurrent neural networks BID26 that are primarily suited to the generation of straight-line programs; almost universally, they cannot guarantee semantic properties of generated programs. Among prominent exceptions, BID18 use log-bilinear tree-traversal models, a class of probabilistic pushdown automata, for program generation. BID4 study a generalization of probabilistic grammars known as probabilistic higher-order grammars. Like our work, these papers address the generation of programs that satisfy rich constraints such as the type-safe use of names. In principle, one could replace our decoder and the combinatorial concretizer, which together form an unconditional program generator, with one of these models. However, given our experiments, doing so is unlikely to lead to good performance in the end-to-end problem of conditional program generation. There is a line of existing work considering the generation of programs from text BID34 BID16 BID25. These papers use decoders similar to the one used in BAYOU, and since they are solving the text-to-code problem, they utilize attention mechanisms not found in BAYOU. Those attention mechanisms could be particularly useful were BAYOU extended to handle natural language evidence. The fundamental difference between these works and BAYOU, however, is the level of abstraction at which learning takes place. These papers attempt to translate text directly into code, whereas BAYOU uses neural methods to produce higher-level sketches that are translated into program code using symbolic methods. This two-step code generation process is central to BAYOU. It ensures key semantic properties of the generated code (such as type safety) and by abstracting away from the learner many lower-level details, it may make learning easier. We have given experimental evidence that this approach can give better than translating directly into code. BID15 propose a variational autoencoder for context-free grammars. As an autoencoder, this model is generative, but it is not a conditional model such as ours. In their application of synthesizing molecular structures, given a particular molecular structure, their model can be used to search the latent space for similar valid structures. In our setting, however, we are not given a sketch but only a label for the sketch, and our task is learn a conditional model that can predict a whole sketch given a label. Conditional program generation is closely related to program synthesis BID10, the problem of producing programs that satisfy a given semantic specification. The programming language community has studied this problem thoroughly using the tools of combinatorial search and symbolic reasoning BID2 BID31 BID8 BID6. A common tactic in this literature is to put syntactic limitations on the space of feasible programs BID2. This is done either by adding a human-provided sketch to a problem instance BID31, or by restricting synthesis to a narrow DSL BID8 BID24.A recent body of work has developed neural approaches to program synthesis. Terpret and Neural Forth BID27 use neural learning over a set of user-provided examples to complete a user-provided sketch. In neuro-symbolic synthesis BID23 and RobustFill BID5, a neural architecture is used to encode a set of input-output examples and decode the ing representation into a Flashfill program. DeepCoder BID3 uses neural techniques to speed up the synthesis of Flashfill programs. These efforts differ from ours in goals as well as methods. Our problem is simpler, as it is conditioned on syntactic, rather than semantic, facets of programs. This allows us to generate programs in a complex programming language over a large number of data types and API methods, without needing a human-provided sketch. The key methodological difference between our work and symbolic program synthesis lies in our use of data, which allows us to generalize from a very small amount of specification. Unlike our approach, most neural approaches to program synthesis do not combine learning and combinatorial techniques. The prominent exception is Deepcoder BID3, whose relationship with our work was discussed in Section 1. We have given a method for generating type-safe programs in a Java-like language, given a label containing a small amount of information about a program's code or metadata. Our main idea is to learn a model that can predict sketches of programs relevant to a label. The predicted sketches are concretized into code using combinatorial techniques. We have implemented our ideas in BAYOU, a system for the generation of API-heavy code. Our experiments indicate that the system can often generate complex method bodies from just a few tokens, and that learning at the level of sketches is key to performing such generation effectively. An important distinction between our work and classical program synthesis is that our generator is conditioned on uncertain, syntactic information about the target program, as opposed to hard constraints on the program's semantics. Of course, the programs that we generate are type-safe, and therefore guaranteed to satisfy certain semantic constraints. However, these constraints are invariant across generation tasks; in contrast, traditional program synthesis permits instance-specific semantic constraints. Future work will seek to condition program generation on syntactic labels as well as semantic constraints. As mentioned earlier, learning correlations between the syntax and semantics of programs written in complex languages is difficult. However, the approach of first generating and then concretizing a sketch could reduce this difficulty: sketches could be generated using a limited amount of semantic information, and the concretizer could use logic-based techniques BID2 BID10 to ensure that the programs synthesized from these sketches match the semantic constraints exactly. A key challenge here would be to calibrate the amount of semantic information on which sketch generation is conditioned. A THE AML LANGUAGE AML is a core language that is designed to capture the essence of API usage in Java-like languages. Now we present this language. DISPLAYFORM0 AML uses a finite set of API data types. A type is identified with a finite set of API method names (including constructors); the type for which this set is empty is said to be void. Each method name a is associated with a type signature (τ 1, . . ., τ k) → τ 0, where τ 1,..., τ k are the method's input types and τ 0 is its return type. A method for which τ 0 is void is interpreted to not return a value. Finally, we assume predefined universes of constants and variable names. The grammar for AML is as in FIG4. Here, x, x 1,... are variable names, c is a constant, and a is a method name. The syntax for programs Prog includes method calls, loops, branches, statement sequencing, and exception handling. We use variables to feed the output of one method into another, and the keyword let to store the return value of a call in a fresh variable. Exp stands for (objectvalued) expressions, which include constants, variables, method calls, and let-expressions such as "let x = Call: Exp", which stores the return value of a call in a fresh variable x, then uses this binding to evaluate the expression Exp. (Arithmetic and relational operators are assumed to be encompassed by API methods.)The operational semantics and type system for AML are standard, and consequently, we do not describe these in detail. We define the abstraction function α for the AML language in Figure 9. In this section we present the details of the neural networks used by BAYOU. The task of the neural encoder is to implement the encoding function f for labels, which accepts an element from a label, say X Calls,i as input and maps it into a vector in d-dimensional space, where d is the dimensionality of the latent space of Z. To achieve this, we first convert each element X Calls,i into its one-hot vector representation, denoted X Calls,i. Then, let h be the number of neural hidden DISPLAYFORM0 DISPLAYFORM1 be real-valued weight and bias matrices of the neural network. The encoding function f (X Calls,i) can be defined as follows: DISPLAYFORM2 where tanh is a non-linearity defined as tanh(x) = 1−e −2x 1+e −2x. This would map any given API call into a d-dimensional real-valued vector. The values of entries in the matrices W h, b h, W d and b d will be learned during training. The encoder for types can be defined analogously, with its own set of matrices and hidden state. The task of the neural decoder is to implement the sampler for Y ∼ P (Y |Z). This is implemented recursively via repeated samples of production rules Y i in the grammar of sketches, drawn as DISPLAYFORM0 The generation of each Y i requires the generation of a new "path" from a series of previous "paths", where each path corresponds to a series of production rules fired in the grammar. As a sketch is tree-structured, we use a top-down tree-structured recurrent neural network similar to BID35, which we elaborate in this section. First, similar to the notion of a "dependency path" in BID35, we define a production path as a sequence of pairs (v 1, e 1), (v 2, e 2),..., (v k, e k) where v i is a node in the sketch (i.e., a term in the grammar) and e i is the type of edge that connects v i with v i+1. Our representation has two types of edges: sibling and child. A sibling edge connects two nodes at the same level of the tree and under the same parent node (i.e., two terms in the RHS of the same rule). A child edge connects a node with another that is DISPLAYFORM1 1 + e −2x and DISPLAYFORM2 for j ∈ 1... K Figure 11: Computing the hidden state and output of the decoder one level deeper in the tree (i.e., the LHS with a term in the RHS of a rule). We consider a sequence of API calls connected by sequential composition as siblings. The root of the entire tree is a special node named root, and so the first pair in all production paths is (root, child). The last edge in a production path is irrelevant (·) as it does not connect the node to any subsequent nodes. As an example, consider the sketch in FIG3 (a), whose representation as a tree for the decoder is shown in Figure 10. For brevity, we use s and c for sibling and child edges respectively, abbreviate some classnames with uppercase letters in their name, and omit the first pair (root, c) that occurs in all paths. There are four production paths in the tree of this sketch:Let h be the number of hidden units in the decoder, and |G| be the size of the decoder's output vocabulary, i.e., the total number of terminals and non-terminals in the grammar of sketches. and b e v ∈ R h be the input weight and bias matrices, and W e y ∈ R h×|G| and b e y ∈ R |G| be the output weight and bias matrices, where e is the type of edge: either c (child) or s (sibling). We also use "lifting" matrices W l ∈ R d×h and b l ∈ R h, to lift the d-dimensional vector Z onto the (typically) higher-dimensional hidden state space h of the decoder. Let h i and y i be the hidden state and output of the network at time point i. We compute these quantities as given in Figure 11, where tanh is a non-linear activation function that converts any given value to a value between -1 and 1, and softmax converts a given K-sized vector of arbitrary values to another K-sized vector of values in the range that sum to 1-essentially a probability distribution. The type of edge at time i decides which RNN to choose to update the (shared) hidden state h i and the output y i. Training consists of learning values for the entries in all the W and b matrices. During training, v i, e i and the target output are known from the data point, and so we optimize a standard cross-entropy loss function (over all i) between the output y i and the target output. During inference, P (v i+1 |Y i, Z) is simply the probability distribution y i, the of the softmax. A sketch is obtained by starting with the root node pair (v 1, e 1) = (root, child), recursively applying Equation 2 to get the output distribution y i, sampling a value for v i+1 from y i, and growing the tree by adding the sampled node to it. The edge e i+1 is provided as c or s depending on the v i+1 that was Figure 12: 2-dimensional projection of latent space of the GSNN-Sk model sampled. If only one type of edge is feasible (for instance, if the node is a terminal in the grammar, only a sibling edge is possible with the next node), then only that edge is provided. If both edges are feasible, then both possibilities are recursively explored, growing the tree in both directions. Remarks. In our implementation, we generate trees in a depth-first fashion, by exploring a child edge before a sibling edge if both are possible. If a node has two children, a neural encoding of the nodes that were generated on the left is carried onto the right sub-tree so that the generation of this tree can leverage additional information about its previously generated sibling. We refer the reader to Section 2.4 of BID35 for more details. In this section, we provide of additional experimental evaluation. Similar to the visualization of the 2-dimensional latent space in Figure 5, we also plotted the latent space of the GSNN-Sk model trained on sketches. Figure 12 shows this plot. We observed that the latent space is clustered, relatively, more densely than that of our model (keep in mind that the plot colors are different when comparing them). To give a sense of the quality of the end-to-end generation, we present and discuss a few usage scenarios for our system, BAYOU. In each scenario, we started with a set of API calls, types or keywords as labels that indicate what we (as the user) would like the generated code to perform. We then pick a single program in the top-5 returned by BAYOU and discuss it. FIG1 shows three such example usage scenarios. In the first scenario, we would like the system to generate a program to write something to a file by calling write using the type FileWriter. With this label, we invoked BAYOU and it returned with a program that actually accomplishes the task. Note that even though we only specified FileWriter, the program uses it to feed a BufferedWriter to write to a file. This is an interesting pattern learned from data, that file reads and writes in Java often take place in a buffered manner. Also note that the program correctly flushes the buffer before closing it, even though none of this was explicitly specified in the input. In the second scenario, we would like the generated program to set the title and message of an Android dialog. This time we provide no API calls or types but only keywords. With this, BAYOU generated a program that first builds an Android dialog box using the helper class AlertDialog. Builder, and does set its title and message. In addition, the program also adds a

Represent the short summary for finding the corresponding scientific article (long)!

We give a method for generating type-safe programs in a Java-like language, given a small amount of syntactic information about the desired code.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We propose an approach for sequence modeling based on autoregressive normalizing flows. Each autoregressive transform, acting across time, serves as a moving reference frame for modeling higher-level dynamics. This technique provides a simple, general-purpose method for improving sequence modeling, with connections to existing and classical techniques. We demonstrate the proposed approach both with standalone models, as well as a part of larger sequential latent variable models. Results are presented on three benchmark video datasets, where flow-based dynamics improve log-likelihood performance over baseline models. Data often contain sequential structure, providing a rich signal for learning models of the world. Such models are useful for learning self-supervised representations of sequences and planning sequences of actions . While sequential models have a longstanding tradition in probabilistic modeling , it is only recently that improved computational techniques, primarily deep networks, have facilitated learning such models from high-dimensional data , particularly video and audio. Dynamics in these models typically contain a combination of stochastic and deterministic variables (; ; ;), using simple distributions (e.g. Gaussian) to directly model the likelihood of data observations. However, attempting to capture all sequential dependencies with relatively unstructured dynamics may make it more difficult to learn such models. Intuitively, the model should use its dynamical components to track changes in the input instead of simultaneously modeling the entire signal. Rather than expanding the computational capacity of the model, we seek a method for altering the representation of the data to provide a more structured form of dynamics. To incorporate more structured dynamics, we propose an approach for sequence modeling based on autoregressive normalizing flows , consisting of one or more autoregressive transforms in time. A single transform is equivalent to a Gaussian autoregressive model. However, by stacking additional transforms or latent variables on top, we can arrive at more expressive models. Each autoregressive transform serves as a moving reference frame in which higher-level structure is modeled. This provides a general mechanism for separating different forms of dynamics, with higher-level stochastic dynamics modeled in the simplified space provided by lower-level deterministic transforms. In fact, as we discuss, this approach generalizes the technique of modeling temporal derivatives to simplify dynamics estimation . We empirically demonstrate this approach, both with standalone autoregressive normalizing flows, as well as by incorporating these flows within more flexible sequential latent variable models. While normalizing flows have been applied in a few sequential contexts previously, we emphasize the use of these models in conjunction with sequential latent variable models. We present experimental on three benchmark video datasets, showing improved quantitative performance in terms of log-likelihood. In formulating this general technique for improving dynamics estimation in the framework of normalizing flows, we also help to contextualize previous work. Figure 1: Affine Autoregressive Transforms. Computational diagrams for forward and inverse affine autoregressive transforms . Each y t is an affine transform of x t, with the affine parameters potentially non-linear functions of x <t. The inverse transform is capable of converting a correlated input, x 1:T, into a less correlated variable, y 1:T. Consider modeling discrete sequences of observations, x 1:T ∼ p data (x 1:T), using a probabilistic model, p θ (x 1:T), with parameters θ. Autoregressive models use the chain rule of probability to express the joint distribution over all time steps as the product of T conditional distributions. Because of the forward nature of the world, as well as for handling variable-length sequences, these models are often formulated in forward temporal order: Each conditional distribution, p θ (x t |x <t), models the temporal dependence between time steps, i.e. a prediction of the future. For continuous variables, we often assume that each distribution takes a relatively simple form, such as a diagonal Gaussian density: where µ θ (·) and σ θ (·) are functions denoting the mean and standard deviation, often sharing parameters over time steps. While these functions may take the entire past sequence of observations as input, e.g. through a recurrent neural network, they may also be restricted to a convolutional window (van den a). Autoregressive models can also be applied to non-sequential data (van den b), where they excel at capturing local dependencies. However, due to their restrictive distributional forms, such models often struggle to capture higher-level structure. Autoregressive models can be improved by incorporating latent variables, often represented as a corresponding sequence, z 1:T. Classical examples include Gaussian state space models and hidden Markov models . The joint distribution, p θ (x 1:T, z 1:T), has the following form: Unlike the simple, parametric form in Eq. 2, evaluating p θ (x t |x <t) now requires integrating over the latent variables, yielding a more flexible distribution. However, performing this integration in practice is typically intractable, requiring approximate inference techniques, like variational inference . Recent works have parameterized these models with deep neural networks, e.g. (; ; ;), using amortized variational inference for inference and learning. Typically, the conditional likelihood, p θ (x t |x <t, z ≤t), and the prior, p θ (z t |x <t, z <t), are Gaussian densities, with temporal conditioning handled through deterministic recurrent networks and the stochastic latent variables. Such models have demonstrated success in audio and video modeling (; ; ; ;). However, design choices for these models remain an active area of research, with each model proposing new combinations of deterministic and stochastic dynamics. Our approach is based on affine autoregressive normalizing flows . Here, we review this basic concept, continuing with the perspective of temporal sequences, however, it is worth noting that these flows were initially developed and demonstrated in static settings. noted that sampling from an autoregressive Gaussian model is an invertible transform, ing in a normalizing flow . Flow-based models transform between simple and complex probability distributions while maintaining exact likelihood evaluation. To see their connection to autoregressive models, we can express sampling a Gaussian random variable, x t ∼ p θ (x t |x <t) (Eq. 2), using the reparameterization trick : where y t ∼ N (y t ; 0, I) is an auxiliary random variable and denotes element-wise multiplication. Thus, x t is an invertible transform of y t, with the inverse given as where division is performed element-wise. The inverse transform in Eq. 6 acts to normalize (hence, normalizing flow) and therefore decorrelate x 1:T. Given the functional mapping between y t and x t in Eq. 5, the change of variables formula converts between probabilities in each space: log p θ (x 1:T) = log p θ (y 1:T) − log det ∂x 1:T ∂y 1:T. By the construction of Eqs. 5 and 6, the Jacobian in Eq. 7 is triangular, enabling efficient evaluation as the product of diagonal terms: where i denotes the observation dimension, e.g. pixel. For a Gaussian autoregressive model, p θ (y 1:T) = N (y 1:T ; 0, I). With these components, the change of variables formula (Eq. 7) provides an equivalent method for sampling and evaluating the model, p θ (x 1:T), from Eqs. 1 and 2. We can improve upon this simple set-up by chaining together multiple transforms, effectively ing in a hierarchical autoregressive model. Letting y m 1:T denote the variables after the m th transform, the change of variables formula for M transforms is log p θ (x 1:T) = log p θ (y Autoregressive flows were initially considered in the contexts of variational inference and generative modeling . These approaches are, in fact, generalizations of previous approaches with affine transforms. While autoregressive flows are well-suited for sequential data, as mentioned previously, these approaches, as well as many recent approaches (; ;), were initially applied in static settings, such as images. More recent works have started applying flow-based models to sequential data. For instance, van den and distill autoregressive speech models into flow-based models. by using flows to model dynamics of continuous latent variables. Like these recent works, we apply flow-based models to sequential data. However, we demonstrate that autoregressive flows can serve as a useful, general-purpose technique for improving sequence modeling as components of sequential latent variable models. To the best of our knowledge, our work is the first to focus on the aspect of using flows to pre-process sequential data to improve downstream dynamics modeling. Finally, we utilize affine flows (Eq. 5) in this work. This family of flows includes methods like NICE, RealNVP , IAF , MAF , and GLOW . However, there has been recent work in non-affine flows (; ;), which may offer further flexibility. We chose to investigate affine flows for their relative simplicity and connections to previous techniques, however, the use of non-affine flows could in additional improvements. We now describe our approach for sequence modeling with autoregressive flows. Although the core idea is a relatively straightforward extension of autoregressive flows, we show how this simple technique can be incorporated within autoregressive latent variable models (Section 2.2), providing a general-purpose approach for improving dynamics modeling. We first motivate the benefits of affine autoregressive transforms in the context of sequence modeling with a simple example. Consider the discrete dynamical system defined by the following set of equations: where w t ∼ N (w t ; 0, Σ). We can express x t and u t in probabilistic terms as Physically, this describes the noisy dynamics of a particle with momentum and mass 1, subject to Gaussian noise. That is, x represents position, u represents velocity, and w represents stochastic forces. If we consider the dynamics at the level of x, we can use the fact that Thus, we see that in the space of x, the dynamics are second-order Markov, requiring knowledge of the past two time steps. However, at the level of u (Eq. 13), the dynamics are first-order Markov, requiring only the previous time step. Yet, note that u t is, in fact, an affine autoregressive transform of x t because u t = x t −x t−1 is a special case of the general form. In Eq. 10, we see that the Jacobian of this transform is ∂x t /∂u t = I, so, from the change of variables formula, we have p(x t |x t−1, x t−2) = p(u t |u t−1). In other words, an affine autoregressive transform has allowed us to convert a second-order Markov system into a first-order Markov system, thereby simplifying the dynamics. Continuing this process to move to w t = u t − u t−1, we arrive at a representation that is entirely temporally decorrelated, i.e. no dynamics, because p(w t) = N (w t ; 0, Σ). A sample from this system is shown in Figure 2, illustrating this process of temporal decorrelation. The special case of modeling temporal changes, u t = x t −x t−1 = ∆x t, is a common pre-processing technique; for recent examples, see;;. In fact, ∆x t is a finite differences approximation of the generalized velocity of x, a classic modeling technique in dynamical models and control , redefining the state-space to be first-order Markov. Affine autoregressive flows offer a generalization of this technique, allowing for non-linear transform parameters and flows consisting of multiple transforms, with each transform serving to successively decorrelate the input sequence in time. In analogy with generalized velocity, each transform serves as a moving reference frame, allowing us to focus model capacity on less correlated fluctuations rather than the highly temporally correlated raw signal. We apply autoregressive flows across time steps within a sequence, x 1:T ∈ R T ×D. That is, the observation at each time step, x t ∈ R D, is modeled as an autoregressive function of past observations, x <t ∈ R t−1×D, and a random variable, y t ∈ R D (Figure 3a). We consider flows of the form given in Eq. 5, where µ θ (x <t) and σ θ (x <t) are parameterized by neural networks. In constructing chains of flows, we denote the shift and scale functions at the m th transform as µ m θ (·) and σ m θ (·) respectively. We then calculate y m using the corresponding inverse transform: After the final (M th) transform, the base distribution, p θ (y M 1:T), can range from a simple distribution, e.g. N (y M 1:T ; 0, I), in the case of a flow-based model, up to more complicated distributions in the case of other latent variable models (Section 3.3). While flows of greater depth can improve model capacity, such transforms have limiting drawbacks. In particular, 1) they require that the outputs maintain the same dimensionality as the inputs, R T ×D, 2) they are restricted to affine transforms, and 3) these transforms operate element-wise within a time step. As we discuss in the next section, we can combine autoregressive flows with non-invertible sequential latent variable models (Section 2.2), which do not have these restrictions. We can use autoregressive flows as a component in parameterizing the dynamics within autoregressive latent variable models. To simplify notation, we consider this set-up with a single transform, but a chain of multiple transforms (Section 3.2) can be applied within each flow. Let us consider parameterizing the conditional likelihood, p θ (x t |x <t, z ≤t), within a latent variable model using an autoregressive flow (Figure 3b). To do so, we express a base conditional distribution for y t, denoted as p θ (y t |y <t, z ≤t), which is then transformed into x t via the affine transform in Eq. 5. We have written p θ (y t |y <t, z ≤t) with conditioning on y <t, however, by removing temporal correlations to arrive at y 1:T, our hope is that these dynamics can be primarily modeled through z 1:T. Using the change of variables formula, we can express the latent variable model's log-joint distribution as log p θ (x 1:T, z 1:T) = log p θ (y 1:T, z 1:T) − log det where the joint distribution over y 1:T and z 1:T, in general, is given as Note that the latent prior, p θ (z t |y <t, z <t), can be equivalently conditioned on x <t or y <t, as there is a one-to-one mapping between these variables. We could also consider parameterizing the prior with autoregressive flows, or even constructing a hierarchy of latent variables. However, we leave these extensions for future work, opting to first introduce the basic concept here. Training a latent variable model via maximum likelihood requires marginalizing over the latent variables to evaluate the marginal log-likelihood of observations: log p θ (x 1:T) = log p θ (x 1:T, z 1:T)dz 1:T. This marginalization is typically intractable, requiring the use of approximate inference methods. Variational inference introduces an approximate posterior distribution, q(z 1:T |x 1:T), which provides a lower bound on the marginal log-likelihood: referred to as the evidence lower bound (ELBO). Often, we assume q(z 1:T |x 1:T) is a structured distribution, attempting to explicitly capture the model's temporal dependencies across z 1:T. We can consider both filtering or smoothing inference, however, we focus on the case of filtering, with The conditional dependencies in q can be modeled through a direct, amortized function, e.g. using a recurrent network , or through optimization. Again, note that we can condition q on x ≤t or y ≤t, as there exists a one-to-one mapping between these variables. With the model's joint distribution (Eq. 16) and approximate posterior (Eq. 19), we can then evaluate the ELBO. We derive the ELBO for this set-up in Appendix A, yielding This expression makes it clear that a flow-based conditional likelihood amounts to learning a latent variable model on top of the intermediate learned space provided by y, with an additional factor in the objective penalizing the scaling between x and y. From top to bottom, each figure shows 1) the original frames, x t, 2) the predicted shift, µ θ (x <t), for the frame, 3) the predicted scale, σ θ (x <t), for the frame, and 4) the noise, y t, obtained from the inverse transform. We demonstrate and evaluate the proposed framework on three benchmark video datasets: Moving MNIST , KTH Actions , and BAIR Robot Pushing . Experimental setups are described in Section 4.1, followed by a set of qualitative experiments in Section 4.2. In Section 4.3, we provide quantitative comparisons across different model classes. Further implementation details and visualizations can be found in Appendix B. Anonymized code is available at the following link. We implement three classes of models: 1) standalone autoregressive flow-based models, 2) sequential latent variable models, and 3) sequential latent variable models with flow-based conditional likelihoods. Flows are implemented with convolutional networks, taking in a fixed window of previous frames and outputting shift, µ θ, and scale, σ θ, parameters. The sequential latent variable models consist of convolutional and recurrent networks for both the encoder and decoder networks, following the basic form of architecture that has been previously employed in video modeling (; ;). In the case of a regular sequential latent variable model, the conditional likelihood is a Gaussian that models the frame, x t. In the case of a flow-based conditional likelihood, we model the noise variable, y t, with a Gaussian. In our experiments, the flow components have vastly fewer parameters than the sequential latent variable models. In addition, for models with flow-based conditional likelihoods, we restrict the number of parameters to enable a fairer comparison. These models have fewer parameters than the baseline sequential latent variable models (with non-flow-based conditional likelihoods). See Appendix B for parameter comparisons and architecture details. Finally, flow-based conditional likelihoods only add a constant computational cost per time-step, requiring a single forward pass per time step for both evaluation and generation. To better understand the behavior of autoregressive flows on sequences, we visualize each component as an image. In Figure 4, we show the data, x t, shift, µ θ, scale, σ θ, and noise variable, y t, for standalone flow-based models (left) and flow-based conditional likelihoods (right) on random sequences from the Moving MNIST and BAIR Robot Pushing datasets. Similar visualizations for KTH Actions are shown in Figure 8 in the Appendix. In Figure 9 in the Appendix, we also visualize these quantities for a flow-based conditional likelihood with two transforms. From these visualizations, we can make a few observations. The shift parameters (second row) tend to capture the static , blurring around regions of uncertainty. The scale parameters (third row), on the other hand, tend to focus on regions of higher uncertainty, as expected. The ing noise variables (bottom row) display any remaining structure not modeled by the flow. In comparing standalone flow-based models with flow-based conditional likelihoods in sequential latent variable models, we see that the latter qualitatively contains more structure in y, e.g. dots (Figure 4b, fourth row) or sharper edges (Figure 4d, fourth row). This is expected, as the noise distribution is more expressive in this case. With a relatively simple dataset, like Moving MNIST, a single flow can reasonably decorrelate the input, yielding white noise images (Figure 4a, fourth row). However, with natural image datasets like KTH Actions and BAIR Robot Pushing, a large degree of structure is still present in these images, motivating the use of additional model capacity to model this signal. In Appendix C.1, we quantify the degree of temporal decorrelation performed by flow-based models by evaluating the empirical correlation between frames at successive time steps for both the data, x, and the noise variables, y. In Appendix C.2, we provide additional qualitative . Log-likelihood for each model class are shown in Table 1. We report the average test loglikelihood in nats per pixel per channel for flow-based models and the lower bound on this quantity for sequential latent variable models. Standalone flow-based models perform surprisingly well, even outperforming sequential latent variable models in some cases. Increasing flow depth from 1 to 2 generally in improved performance. Sequential latent variable models with flow-based conditional likelihoods outperform their baseline counterparts, despite having fewer parameters. One reason for this disparity is overfitting. Comparing with the training performance reported in Table 3, we see that sequential latent variable models with flow-based conditional likelihoods overfit less. This is particularly apparent on KTH Actions, which contains training and test sets with a high degree of separation (different identities and activities). This suggests that removing static components, like s, yields a reconstruction space that is better for generalization. The quantitative in Table 1 are for a representative sequential latent variable model with a standard convolutional encoder-decoder architecture and fully-connected latent variables. However, many previous works do not evaluate proper lower bounds on log-likelihood, using techniques like down-weighting KL divergences (; ;). Indeed, train SVG with a proper lower bound and report a lower bound of −2.86 nats per pixel on KTH Actions, on-par with our . report log-likelihood on BAIR Robot Pushing, obtaining −1.3 nats per pixel, substantially higher than our . However, their model is significantly larger than the models presented here, consisting of 3 levels of latent variables, each containing 24 steps of flows. We have presented a technique for improving sequence modeling based on autoregressive normalizing flows. This technique uses affine transforms to temporally decorrelate sequential data, thereby simplifying the estimation of dynamics. We have drawn connections to classical approaches, which involve modeling temporal derivatives. Finally, we have empirically shown how this technique can improve sequential latent variable models. Consider the model defined in Section 3.3.1, with the conditional likelihood parameterized with autoregressive flows. That is, we parameterize The joint distribution over all time steps is then given as To perform variational inference, we consider a filtering approximate posterior of the form We can then plug these expressions into the evidence lower bound: Finally, in the filtering setting, we can rewrite the expectation, bringing it inside of the sum (see ;): Because there exists a one-to-one mapping between x 1:T and y 1:T, we can equivalently condition the approximate posterior and the prior on y, i.e. We store a fixed number of past frames in the buffer of each transform, to generate the shift and scale for the transform. For each stack of flow, 4 convolutional layers with kernel size, stride 1 and padding 1 are applied first on each data observation in the buffer, preserving the data shape. The outputs are concatenated along the channel dimension and go through another four convolutional layers also with kernel size, stride 1 and padding 1. Finally, separate convolutional layers with the same kernel size, stride and padding are used to generate shift and scale respectively. For latent variable models, we use a DC-GAN structure , with 4 layers of convolutional layers of kernel size, stride 2 and padding 1 before another convolutional layer of kernel size, stride 1 and no padding to encode the data. The encoded data is sent to an LSTM followed by fully connected layers to generate the mean and log-variance for estimating the approximate posterior distribution of the latent variable, z t. The conditional prior distribution is modeled with another LSTM followed by fully connected layers, taking the previous latent variable as input. The decoder take the inverse structure of the encoder. In the SLVM, we use 2 LSTM layers for modelling the conditional prior and approximate posterior distributions, while in the combined model we use 1 LSTM layer for each. We use the Adam optimizer with a learning rate of 1 × 10 −4 to train all the models. For Moving MNIST, we use a batch size of 16 and train for 200, 000 iterations for latent variable models and 100, 000 iterations for flow-based and latent variable models with flow-based likelihoods. For BAIR Robot Pushing, we use a batch size of 8 and train for 200, 000 iterations for all models. For KTH dataset we use a batch size of 8 and train for 90, 000 iterations for all models. Batch norm is applied to all convolutional layers that do not directly generate distribution or transform parameters. We randomly crop sequence of length 13 from all sequences and evaluate on the last 10 frames. (For 2-flow models we crop sequence of length 16 to fill up all buffers.) Anonymized code is available at the following link. fc fc (Figure 6 : Model Architecture Diagrams. Diagrams are shown for the (a) approximate posterior, (b) conditional prior, and (c) conditional likelihood of the sequential latent variable model. conv denotes a convolutional layer, LSTM denotes a long short-term memory layer, fc denotes a fullyconnected layer, and t conv denotes a transposed convolutional layer. For conv and t conv layers, the numbers in parentheses respectively denote the number of filters, filter size, stride, and padding of the layer. For fc and LSTM layers, the number in parentheses denotes the number of units. C ADDITIONAL EXPERIMENTAL The qualitative in Figures 4 and 8 demonstrate that flows are capable of removing much of the structure of the observations, ing in whitened noise images. To quantitatively confirm the temporal decorrelation ing from this process, we evaluate the empirical correlation between successive frames, averaged over spatial locations and channels, for the data observations and noise variables. This is an average normalized version of the auto-covariance of each signal with a time delay of 1 time step. Specifically, we estimate the temporal correlation as where x (i,j,k) denotes the value of the image at location (i, j) and channel k, µ (i,j,k) denotes the mean of this dimension, and σ (i,j,k) denotes the standard deviation of this dimension. H, W, and C respectively denote the height, width, and number of channels of the observations. We evaluated this quantity for data examples, x, and noise variables, y, for SLVM w/ 1-AF. The for training sequences are shown in Table 4. In Figure 7, we plot this quantity during training for KTH Actions. We see that flows do indeed in a decrease in temporal correlation. Note that because correlation is a measure of linear dependence, one cannot conclude from these alone From top to bottom, each figure shows 1) the original frames, x t, 2) the predicted shift, µ θ (x <t), for the frame, 3) the predicted scale, σ θ (x <t), for the frame, and 4) the noise, y t, obtained from the inverse transform. Figure 10: Generated Moving MNIST Samples. Samples frame sequences generated from a 2-AF model. Figure 11: Generated BAIR Robot Pushing Samples. Samples frame sequences generated from SLVM w/ 1-AF.

We show how autoregressive flows can be used to improve sequential latent variable models.

Represent the TLDR for finding the corresponding scientific article.

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.