Question: represent the article for finding a single-sentence tldr

Question: The interpretability of an AI agent's behavior is of utmost importance for effective human-AI interaction. To this end, there has been increasing interest in characterizing and generating interpretable behavior of the agent. An alternative approach to guarantee that the agent generates interpretable behavior would be to design the agent's environment such that uninterpretable behaviors are either prohibitively expensive or unavailable to the agent. To date, there has been work under the umbrella of goal or plan recognition design exploring this notion of environment redesign in some specific instances of interpretable of behavior. In this position paper, we scope the landscape of interpretable behavior and environment redesign in all its different flavors. Specifically, we focus on three specific types of interpretable behaviors -- explicability, legibility, and predictability -- and present a general framework for the problem of environment design that can be instantiated to achieve each of the three interpretable behaviors. We also discuss how specific instantiations of this framework correspond to prior works on environment design and identify exciting opportunities for future work. The design of human-aware AI agents must ensure that its decisions are interpretable to the human in the loop. Uninterpretable behavior can lead to increased cognitive load on the human -from reduced trust, productivity to increased risk of danger around the agent BID7. BID5 emphasises in the Roadmap for U.S. Robotics -"humans must be able to read and recognize agent activities in order to interpret the agent's understanding". The agent's behavior may be uninterpretable if the human: has incorrect notion of the agent's beliefs and capabilities BID15 BID1 ) FORMULA3 is unaware of the agent's goals and rewards BID6 BID12 cannot predict the agent's plan or policy BID8 BID12. Thus, in order to be interpretable, the agent must take into account the human's expectations of its behavior -i.e. the human mental model BID0 ). There are many ways in which considerations of the human mental model can affect agent behavior. * equal contribution There has been significant interest recently in characterizing different notions of interpretable behavior of a human-aware AI agent. Three important properties of interpretable behaviors emerge, namely - explicability: when the agent behavior conforms to the expectations of the human; legibility: when the agent behavior reveals its objectives or intentions to the observer; and predictability: when the (remaining) agent behavior can be precisely predicted by the human. In existing works, the generation of these behaviors was explored from the point of view of the agent -i.e. the agent altered its planning process by using its estimation of the mental model of the human in the loop in order to exhibit the desired behavior. We refer the reader to ) for a detailed treatise of these concepts. A parallel thread of work, under the umbrella of goal and plan recognition design, has looked at the notion of changing the environment of an agent to increase interpretability of the behaviors available to an agent. The design of environment can be optimized in order to maximize (or minimize) some objective for the actor (for example, optimalcost to a goal, desired behavioral property) BID16 BID9 ). An environment design problem takes the initial environment configuration as input, along with set of modifications allowed in the environment and outputs a sequence of modifications, which can be applied to the initial environment to derive a new environment in which the desired objective is optimized. The problem of environment design is more suited for structured settings where an actor performs repetitive objectives (for example, on factory floors, etc). It is also suited for settings involving multiple actors, where the expectations of the observers are the same but there are multiple actors in the environment, making environment design an effective choice (for example, in restaurants with waiter robots where the human customers have same expectations).Goal (and Plan) Recognition Design In existing work, the concept of environment design for planning agents has been studied in the concept of goal (or plan) recognition The domain can be updated for more explicable behavior by disabling the robot coffee holder; (c) To induce legible behavior, we can add dividing walls to constrain the agent and help the observer reduce uncertainty in their mental model; and (d) To induce predictable behavior we can reduce uncertainty about the order of pickup by including a tray that allows the agent to pick up the objects in any order.design BID10 BID13 in order to make the goals (or plans) of an actor easier to recognize to the observer. Immediately, this should remind the reader of legibility (and predictability) introduced above. The goal of this paper is to bridge the gap between these two parallel threads of work and explore the full spectrum of interpretable behavior in the context of environment design. We will discuss how existing work in environment design fits into this narrative and highlight much needed gaps in existing work as exciting avenues for future work. Adopting the reasoning capabilities of an agent to deal with the human mental model, as done in classical human-aware decision-making versus the design of environments are indeed complimentary approaches towards achieving the same purpose: behavior of the agent that is more interpretable to the observer. While one could conceive of the most general framework that accounts for both, it is useful to recognize that these have their own unique set of features. Perhaps, the biggest advantage of environment design is that the process of generation of interpretable behavior is offloaded from the actor onto the design process. In other words, the computational overhead of generating interpretable behavior -in having to deal with the human mental model and reason in the space of models -is now part of the design process only, which can be done offline. We will see later, in our formulation, how the actor in the "Design for Interpretability" is modeled as a cost-optimal agent that is able to produce interpretable behavior while still planning in the traditional sense. In addition, design also means that the observer does not have to count on the agent to be cooperative and interpretable, and instead can deal with adversarial agents as well. At the end of this paper, we will see that this advantage does come with a caveat. In general, the notion of interpretable behavior is complemented by communication: e.g. authors in balance considerations of explanations and explicability, while authors in ) balance out intention projection actions for the sake of legibility and predictability. Communication operates in model space by being able to change the observer's beliefs. Design, also operating in model space, can be seen as an alternative to communication that complements the notion of interpretable behavior. Consider an office setting where an office assistant robot, responsible for delivering items such as coffee or mail to the employees, is about to be deployed (FIG0). The robot (actor) will be supervised by office security guards (observer) who have worked with previous generation office assistant robots and have some expectations regarding their functions. In particular, they expect the robot to carry one item at a time (i.e. either mail or coffee) and each robot generally has a strong preference on the order in which it picks up these items (though the order changes from robot to robot). Unknown to the guards, the new model adds more flexibility to the robot by removing the need for the robots to have fixed preference on the order to pick up items and installs a coffee cup holder that allows the robot to carry both mail and coffee at the same time. Now if we allow the new robot to simply act optimally in the original setting, it would unnecessary confuse the observers. If the robot was built to generate interpretable behavior, it will change its behavior (and possibly settle for suboptimal decisions in its own model) in order to conform to expectations or it will provide explanations that address these model differences. However, the same effect can be achieved if the designers who are deploying the robot also designed the environment to ensure that decisions of the new robot remain interpretable to the occupants of the office. If the designers wish to prioritize explicability, then the change that they would need to make would be to disable the coffee holder, this will cause the robot to choose one of the items first, deliver it and then move on to the second one. For explicability, it does not matter which one the robot chooses as the user would simply assume that the order chosen by the robot is the one enforced by the robot's model. As for legibility, the aim is to help the user differentiate between the models as early as possible, one way to do it would be to disable the coffee holder and then build introduce obstacles as shown in FIG0. Finally, for predictability, the focus is to allow the user to be able to predict the entire plan as early as possible. One possible design for this scenario is to disable the coffee holder and provide the robot with a tray that allows the robot to carry both items at the same time. The observer can see the tray and realizes the robot can place both items in the tray and the order of picking up no longer matters. In predictability, we may need to add additional obstacles to further restrict the space of possible plans that can be done by the robot FIG0 ). An interpretable decision making problem involves two entities: an actor (A) and an observer (O). The actor operates in an environment while being observed by the observer. Definition 1. An interpretable decision making problem is a tuple, P Int = P A, P O, Int, where: -P A is the decision making problem of the actor A DISPLAYFORM0 O } is observer's mental model of the actor, represented by a set of possible decision making problems that the observer believes that the actor may be solving.-Int: Π → R is the interpretability score that is used to evaluate agent plans (where Π is the space of plans)Interestingly, we do not require that P A ∈ P O -i.e. the problems in P O can be different from P A in all possible aspects (e.g. state space, action space, initial state and goals). The solution to P Int is a plan or policy that not only solves P A but also satisfies some desired properties of interpretable behaviors (measured through the interpretability score). The score could reflect properties like explicability, legibility or predictability of the plan. Explicability The actor's behavior is considered explicable if it aligns with at least one of the observer's expected plans (as per their mental model). The set of plans expected by the observer consists of all the cost-optimal solutions for problems in P O. The target of explicability is thus to generate behavior that belongs to this set of expected plans. Legibility With legibility, the objective of the actor is to inform the observer about its model -i.e. reduce the size of P O. An actor's behavior is said to be perfectly legible if it can be derived from only one model in P O. The longer it takes for a plan prefix to achieve this, the worse is the plan's legibility. This notion of interpretability thus helps the observer narrow down their belief over the possible actor models as quickly as possible. Predictability The objective of the actor with predictability is to generate the most disambiguating behavior -i.e. given the actor's plan prefix, the observer should be able to predict its completion. These predictions would be in terms of cost-optimal completions of a given prefix in the possible problems in the mental model. This means that if there exists the same unique completion in all of the models then the plan is predictable even though not legible. The shorter the length of the disambiguating plan prefix, the better the predictability of the plan. An empty prefix would thus correspond to the most predictable plan. In this section, we present a general formulation for the design problem for interpretable behaviors. Given an environment design, we assume that the actor is a rational agent and therefore is incentivized to generate cost-optimal plans. Let the set of cost optimal plans of the actor be Π * P A. A costoptimal plan solution to P A can exist anywhere on the spectrum of interpretability from high to low. Therefore, we need a measure to quantify the interpretability score for the actor's set of cost-optimal plans. To that end, we introduce the worst-case interpretability score wci as follows: Definition 2. The worst-case interpretability score wci(·), for P Int is defined as DISPLAYFORM0 Int(·) is instantiated for each type of interpretable behavior separately and is discussed in detail at the end of this section. The higher the interpretablity score, the better the interpretability of the behavior (in terms of either of three properties). Therefore, the worst-case interpretability score is the minimum interpretability score of a cost-optimal plan of the actor. We can now define the design problem for interpretability. When a modification is applied to the environment, both the actor's decision making problem and the observer's mental model are modified, thereby changing the worst-case interpretability score of the actor's cost-optimal plans for the given decision making problem. Let P denote the set of valid configurations in the real environment. Although P A ∈ P, problems in P O might not necessarily be in P if the observer has incorrect or infeasible notions about the actor's model. Therefore, we represent the set of configurations that the observer thinks are possible as P, and P O ⊆ P. Definition 3. The design problem for interpretability, DPInt, is a tuple DISPLAYFORM1 A ∈ P and P O 0 ⊆ P are the initial models.• ∆ is the set of modifications that can be applied to the environment. ξ is a sequence of modifications.• Λ A: ∆ × P → P and Λ O: ∆ × P → P are the model transition function that specify the ing model after applying a modification to the existing models. The set of possible modifications includes modifying the set of states, action preconditions, action effects, action costs, initial state and goal. Each modification ξ ∈ ∆ is associated with a cost, such that, C(ξ) = ξi∈ξ C(ξ). After applying ξ to both P 0 A and P O 0, the ing actor decision making problem model and observer mental model are represented as P |ξ| A and P O |ξ| respectively. Let P |ξ| Int be the modified interpretable decision making problem after applying the modification ξ to P Int. Our objective here is to solve DP-Int such that the worst-case interpretability score of P Int is maximized. Apart from that, the design cost of ξ has to be minimized, as well as the cost of a plan π A that solves P |ξ| A. Definition 4. A solution to DP-Int, is a sequence of modifications ξ with DISPLAYFORM2 This completes the general framework of design for interpretability. In the following, we will look at specific instances of design for the different notions of interpretability. In order to be explicable, the actor's plan has to be consistent with the observer's expectations of it. The observer has an implicit assumption that the actor is a rational agent. Therefore the set of plans expected by the observer includes the cost-optimal plans for all the planning models in the observer's mental model. Let Π * P O be the set of expected plans for the observer. Given the set of expected plans, the explicability of the actor's plan depends on how different it is from the expected plans. In order to quantify the explicability of a plan, we introduce the following scoring function: Definition 5. The explicability score Exp(·) of an actor's plan π A that solves P A is defined as follows: DISPLAYFORM0 Here δ P O (·) computes the distance between two plans with respect to the observer's mental model. For example, the distance function could compute a cost-based difference between the two plans in the observer's mental model. Plugging this scoring function in Equation 2 allows us to instantiate the design problem for explicability. In order to be legible, the actor's plan has to reveal its problem to the observer as early on as possible. Therefore, the legibility of a plan is inversely proportional to the length of its shortest prefix that has unique cost optimal completion for more than one problem in the observer's mental model. Definition 6. The legibility score Leg(·) of an actor's plan, π A, that solves P A is defined as follows: DISPLAYFORM0 with unique cost optimal completion ofπ A in each model, andΠ P A is the set of all prefixes of π A. Plugging this scoring function in Equation 2 allows us to instantiate the design problem for legibility. Goal Recognition Design The work on goal recognition design (GRD) BID10 ) is a special case of the design problem for legibility. The GRD problem involves an actor and an observer where the observer's mental model consists of planning models that have the exact same state space, actions and initial state as the actor's planning model. However, each planning model in the observer's mental model has a different goal. The actor's true goal is one of them, and the objective of GRD problem is to redesign the environment, such that, the true goal of the actor is revealed to the observer as early as possible. The interpretability problem defined here is a general one, where the observer's mental model can be different in all possible ways from the actor's actual planning model. In order to be predictable, the plan has to be the mostdisambiguating plan among the set of plans the observer is considering -i.e. the observer should be able to predict the rest of the plan after seeing the prefix. Therefore, predictability of a plan is inversely proportional to the length of its shortest prefix which ensures only one optimal completion solving only a single problem in the observer's mental model. We can quantify the predictability score as follows: Definition 7. The predictability score P red(·) of an actor's plan π A that solves P A is defined as follows: DISPLAYFORM0 where π is an optimal completion ofπ A, andΠ P A is the set of all prefixes of a plan π A. Plugging this scoring function in Equation 2 allows us to instantiate the design problem for predictability. The predictability problem corresponds to the plan recognition design (PRD) problem BID13. However, our proposed framework in terms of possible observer models subsumes the plan library based approaches in being able to support a generative model of observer expectations. We will now highlight limitations of the proposed framework and discuss how they may be extended in the future. Multiple decision making problems. The problem of environment design, as studied in this paper, is suitable for settings where the actor performs a single repetitive task. However, our formulation can be easily extended to handle an array of tasks that the agent performs in its environment by considering a set of decision making problems for the actor, where the worst-case score is decided by taking either minimum (or average) over the wci(·) for the set of problems. Interpretability Score. The three properties of interpretable agent behavior are not mutually exclusive. A plan can be explicable, legible and predictable at the same time. In general, a plan can have any combination of the three properties. In Equation 2, Int(·) uses one of these properties at a time. In order to handle more than one property at a time, one could formulate Int(·) as a linear combination of the three properties. In general, the design objective would be to minimize the worst-case interpretability score such that the scores for each property are maximized in the modified environment, or at least allow the designer pathways to trade off among potentially competing metrics. Cost of the agent. In Section 1.3 we mentioned an advantage of the design process in the context of interpretabilitythe ability to offload the computational load on the actor, in having to reason about the observer model, to the offline design stage. However, there is never any free lunch. The effect of environment design is more permanent than operating on the human mental model. That is to say, interpretable behavior while targeted for a particular human in the loop or for a particular interaction, does not (usually) affect the actor going forward. However, in case of design of environment, the actor has to live with the design decisions for the rest of its life. That means, for example, if the environment has been designed to promote explicable behavior, the actor would be incurring additional cost for its behaviors (than it would have had in the original environment). This also affects not only a particular decision making problem at hand, but also everything that the actor does in the environment, and for all the agents it interacts with. As such there is a "loss of autonomy" is some sense due to environment design, the cost of which can and should be incorporated in the design process.

We present an approach to redesign the environment such that uninterpretable agent behaviors are minimized or eliminated.

Represent the TLDR for finding the corresponding scientific article

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Inference models, which replace an optimization-based inference procedure with a learned model, have been fundamental in advancing Bayesian deep learning, the most notable example being variational auto-encoders (VAEs). In this paper, we propose iterative inference models, which learn how to optimize a variational lower bound through repeatedly encoding gradients. Our approach generalizes VAEs under certain conditions, and by viewing VAEs in the context of iterative inference, we provide further insight into several recent empirical findings. We demonstrate the inference optimization capabilities of iterative inference models, explore unique aspects of these models, and show that they outperform standard inference models on typical benchmark data sets. Generative models present the possibility of learning structure from data in unsupervised or semisupervised settings, thereby facilitating more flexible systems to learn and perform tasks in computer vision, robotics, and other application domains with limited human involvement. Latent variable models, a class of generative models, are particularly well-suited to learning hidden structure. They frame the process of data generation as a mapping from a set of latent variables underlying the data. When this mapping is parameterized by a deep neural network, the model can learn complex, non-linear relationships, such as object identities and dynamics . However, performing exact posterior inference in these models is computationally intractable, necessitating the use of approximate inference methods. Variational inference is a scalable approximate inference method, transforming inference into a non-convex optimization problem. Using a set of approximate posterior distributions, e.g. Gaussians, variational inference attempts to find the distribution that most closely matches the true posterior. This matching is accomplished by maximizing a lower bound on the marginal log-likelihood, or model evidence, which can also be used to learn the model parameters. The ensuing expectation-maximization procedure alternates between optimizing the approximate posteriors and model parameters (; ;). Amortized inference avoids exactly computing optimized approximate posterior distributions for each data example, instead learning a separate inference model to perform this task. Taking the data example as input, this model outputs an estimate of the corresponding approximate posterior. When the generative and inference models are parameterized with neural networks, the ing set-up is referred to as a variational auto-encoder (VAE) .We introduce a new class of inference models, referred to as iterative inference models, inspired by recent work in learning to learn . Rather than directly mapping the data to the approximate posterior, these models learn how to iteratively estimate the approximate posterior by repeatedly encoding the corresponding gradients, i.e. learning to infer. With inference computation distributed over multiple iterations, we conjecture that this model set-up should provide improved inference estimates over standard inference models given sufficient model capacity. Our work is presented as follows: Section 2 contains on latent variable models, variational inference, and inference models; Section 3 motivates and introduces iterative inference models; Section 4 presents this approach for latent Gaussian models, showing that a particular form of iterative inference models reduces to standard inference models under mild assumptions; Section 5 contains empirical ; and Section 6 concludes our work. Latent variable models are generative probabilistic models that use local (per data example) latent variables, z, to model observations, x, using global (across data examples) parameters, θ. A model is defined by the joint distribution p θ (x, z) = p θ (x|z)p θ (z), which is composed of the conditional likelihood and the prior. Learning the model parameters and inferring the posterior p(z|x) are intractable for all but the simplest models, as they require evaluating the marginal likelihood, p θ (x) = p θ (x, z)dz, which involves integrating the model over z. For this reason, we often turn to approximate inference methods. Variational inference reformulates this intractable integration as an optimization problem by introducing an approximate posterior 1 q(z|x), typically chosen from some tractable family of distributions, and minimizing the KL-divergence from the true posterior, D KL (q(z|x)||p(z|x)). This quantity cannot be minimized directly, as it contains the true posterior. Instead, the KL-divergence can be decomposed into DISPLAYFORM0 where L is the evidence lower bound (ELBO), which is defined as:L ≡ E z∼q(z|x) [log p θ (x, z) − log q(z|x)] = E z∼q(z|x) [log p θ (x|z)] − D KL (q(z|x)||p θ (z)).Briefly, the first term in eq. 3 can be considered as a reconstruction term, as it expresses how well the output fits the data example. The second term can be considered as a regularization term, as it quantifies the dissimilarity between the latent representation and the prior. Because log p θ (x) is not a function of q(z|x), in eq. 1 we can minimize D KL (q(z|x)||p(z|x)), thereby performing approximate inference, by maximizing L w.r.t. q(z|x). Likewise, because D KL (q(z|x)||p(z|x)) is non-negative, L is a lower bound on log p θ (x), meaning that if we have inferred an optimal q(z|x), learning corresponds to maximizing L w.r.t. θ. The optimization procedures involved in inference and learning, when implemented using conventional gradient ascent techniques, are respectively the expectation and maximization steps of the variational EM algorithm (; ;), which alternate until convergence. When q(z|x) takes a parametric form, the expectation step for data example x (i) involves finding a set of distribution parameters, λ (i), that are optimal. With a factorized Gaussian density over continuous variables, i.e. DISPLAYFORM0 q ), this entails repeatedly estimating the stochastic gradients DISPLAYFORM1. This direct optimization procedure, which is repeated for each example, is not only computationally costly for expressive generative models and large data sets, but also sensitive to step sizes and initial conditions. Amortized inference replaces the optimization of each set of local approximate posterior parameters, λ (i), with the optimization of a set of global parameters, φ, contained within an inference model. Taking x (i) as input, this model directly outputs estimates of λ (i). Sharing the inference model across data examples allows for an efficient algorithm, in which φ and θ can be updated jointly. The canonical example, the variational auto-encoder (VAE) , employs the reparameterization trick to propagate stochastic gradients from the generative model to the inference model, both of which are parameterized by neural networks. The formulation has an intuitive interpretation: the inference model encodes x into q(z|x), and the generative model decodes samples from q(z|x) into p(x|z). Throughout the rest of this paper, we refer to inference models of this form as standard inference models. Optimization surface of L (in nats) for a 2-D latent Gaussian model and a particular MNIST data example. Shown on the plot are the MAP (optimal estimate), the output of a standard inference model (VAE), and an expectation step trajectory of variational EM using stochastic gradient ascent. The plot on the right shows the estimates of each inference scheme near the optimum. The expectation step arrives at a better final inference estimate than the standard inference model. In Section 3.2, we introduce our contribution, iterative inference models. We first motivate our approach in Section 3.1 by interpreting standard inference models in VAEs as optimization models, i.e. models that learn to perform optimization. Using insights from other optimization models, this interpretation extends and improves upon standard inference models. As described in Section 2.1, variational inference transforms inference into the maximization of L w.r.t. the parameters of q(z|x), constituting the expectation step of the variational EM algorithm. In general, this is a non-convex optimization problem, making it somewhat surprising that an inference model can learn to output reasonable estimates of q(z|x) across data examples. Of course, directly comparing inference schemes is complicated by the fact that generative models adapt to accommodate their approximate posteriors. Nevertheless, inference models attempt to replace traditional optimization techniques with a learned mapping from x to q(z|x).We demonstrate this point in FIG0 by visualizing the optimization surface of L defined by a trained 2-D latent Gaussian model and a particular data example, in this case, a binarized MNIST digit. To visualize the surface, we use a 2-D point estimate as the approximate posterior, q(z|x) = δ(z = µ q), where µ q = (µ 1, µ 2) ∈ R 2 and δ is the Dirac delta function. See Appendix C.1 for further details. Shown on the plot are the MAP (i.e. optimal) estimate, the estimate from a trained inference model, and an expectation step trajectory using stochastic gradient ascent on µ q. The expectation step arrives at a better final estimate, but it requires many iterations and is dependent on the step size and initial estimate. The inference model outputs a near-optimal estimate in one forward pass without hand tuning (other than the architecture), but it is restricted to this single estimate. Note that the inference model does not attain the optimal estimate, ing in an "amortization gap" .This example illustrates how inference models differ from conventional optimization techniques. Despite having no convergence guarantees on inference optimization, inference models have been shown to work well empirically. However, by learning a direct mapping from x to q(z|x), standard inference models are restricted to only single-step estimation procedures, which may yield worse inference estimates. The ing large amortization gap then limits the quality of the accompanying generative model. To improve upon this paradigm, we take inspiration from the area of learning to learn, where showed that an optimizer model, instantiated as a recurrent neural network, can learn to optimize the parameters of an optimizee model, another neural network,. θ refers to the generative model (decoder) parameters. ∇ λ L denotes the gradients of the ELBO w.r.t. the distribution parameters, λ, of the approximate posterior, q(z|x). Iterative inference models learn to perform approximate inference optimization by using these gradients and a set of inference model (encoder) parameters, φ. See FIG6 for a similar set of diagrams with unrolled computational graphs.for various tasks. The optimizer model receives the optimizee's parameter gradients and outputs updates to these parameters to improve the optimizee's loss. Because the computational graph is differentiable, the optimizer itself can also be learned. Optimization models can learn to adaptively adjust update step sizes, potentially speeding up and improving optimization. focus primarily on parameter optimization (i.e. learning), we apply an analogous approach to inference optimization in latent variable models. We refer to this class of optimization models as iterative inference models, as they are inference models that iteratively update their approximate posterior estimates. Our work differs from that of in three distinct ways: variational inference is a qualitatively different optimization problem, involving amortization across data examples rather than learning tasks; we utilize nonrecurrent optimization models, providing a more computationally efficient model that breaks the assumption that previous gradient information is essential for learned optimization; and we provide a novel model formulation that approximates gradient steps using locally computed errors on latent and observed variables (see Section 4.1). We formalize our approach in the following section. We present iterative inference models starting from the context of standard inference models. For a standard inference model f with parameters φ, the estimate of the approximate posterior distribution parameters λ (i) for data example x (i) is of the form: DISPLAYFORM0 We propose to instead use an iterative inference model, also denoted as f with parameters φ. With DISPLAYFORM1 t; θ) as the ELBO for data example x (i) at inference iteration t, the model uses DISPLAYFORM2 t, to output updated estimates of λ (i): DISPLAYFORM3 where DISPLAYFORM4 t is the estimate of λ (i) at inference iteration t. We use f t to highlight that the form of f at iteration t may depend on hidden states within the iterative inference model, such as those found within recurrent neural networks. See Figures 2 and 8 for schematic comparisons of iterative inference models with variational EM and standard inference models. As with standard inference models, the parameters of an iterative inference model can be updated using stochastic estimates of ∇ φ L, obtained through the reparameterization trick or other methods. Model parameter updating is typically performed using standard optimization techniques. Note that eq. 5 is in a general form and contains, as a special case, the residual updating scheme used in. We now describe an example of iterative inference models for latent Gaussian generative models, deriving the gradients to understand the source of the approximate posterior updates. Latent Gaussian models are latent variable models with Gaussian prior distributions over latent variables: DISPLAYFORM0 This class of models is often used in VAEs and is a common choice for representing continuous-valued latent variables. While the approximate posterior can be any probability density, it is typically also chosen as Gaussian: q(z|x) = N (z; µ q, diag σ 2 q). With this choice, λ (i) corresponds to {µ DISPLAYFORM1 q} for example x (i). Dropping the superscript (i) to simplify notation, we can express eq. 5 for this model as: DISPLAYFORM2 DISPLAYFORM3 where f µq t and f σ 2 q t are the iterative inference models for updating µ q and σ 2 q respectively. For continuous observations, we can use a Gaussian output density: DISPLAYFORM4 is a non-linear function of z, and σ 2 x is a global parameter, a common assumption in these models. The approximate posterior parameter gradients for this model are (see Appendix A): DISPLAYFORM5 where ∼ N (0, I) is the auxiliary noise variable from the reparameterization trick, denotes element-wise multiplication, and all division is performed element-wise. In Appendix A, we also derive the corresponding gradients for a Bernoulli output distribution, which take a similar form. Although we only derive gradients for these two output distributions, note that iterative inference models can be used with any distribution form. We now briefly discuss the terms in eqs. 8 and 9. Re-expressing the reparameterized latent variable as z = µ q + σ q, the gradients have two shared terms, (x − µ x)/σ 2 x and (z − µ p)/σ 2 p, the precision-weighted errors at the observed ("bottom-up") and latent ("top-down") levels respectively. The terms ∂µx ∂µq and ∂µx ∂σ 2 q are the Jacobian matrices of µ x w.r.t. the approximate posterior parameters, which effectively invert the output model. Understanding the significance of each term, in the following section we provide an alternative formulation of iterative inference models for latent Gaussian generative models. The approximate posterior gradients are inherently stochastic, arising from the fact that evaluating L involves approximating expectations (eq. 2) using Monte Carlo samples of z ∼ q(z|x). As these estimates always contain some degree of noise, a close approximation to these gradients should also suffice for updating the approximate posterior parameters. The motivations for this are two-fold: approximate gradients may be easier to compute, especially in an online setting, and by encoding more general terms, the inference model may be able to approximate higher-order approximate posterior derivatives, allowing for faster convergence. We now provide an alternative formulation of iterative inference models for latent Gaussian models that approximates gradient information. With the exception of ∂µx ∂µq and ∂µx ∂σ 2 q, all terms in eqs. 8 and 9 can be easily computed using x and the distribution parameters of p(x|z), p(z), and q(z|x). Likewise, higher-order approximate posterior derivatives consist of these common terms as well as higher-order derivatives of the output model. As the output model derivatives are themselves functions, by encoding only the common terms, we can offload these (approximate) derivative calculations onto the iterative inference model. Again dropping the superscript (i), one possible set-up is formulated as follows: DISPLAYFORM0 DISPLAYFORM1 where, in the case of a Gaussian output density, the stochastic error terms are defined as DISPLAYFORM2. This encoding scheme resembles the approach taken in DRAW (Gregor et al. FORMULA0), where reconstruction errors, x − µ t,x, are iteratively encoded. However, DRAW and later variants (Gregor et al. FORMULA0) do not explicitly account for latent errors, ε z,t, or approximate posterior estimates. If possible, these terms must instead be implicitly handled by the inference model's hidden states. In Section 5.2, we demonstrate that iterative inference models of this form do indeed learn to infer. Unlike gradient encoding iterative inference models, these error encoding models do not require gradients at test time and they empirically perform well even with few inference iterations. Under a certain set of assumptions, single-iteration iterative inference models of the derivative approximating form proposed in Section 4.1 are equivalent to standard inference models, as used in conventional VAEs. Specifically, assuming:1. the initial approximate posterior estimate is a global constant: DISPLAYFORM0 we are in the limit of infinite samples of the initial auxiliary variable 0, then the initial approximate posterior estimate (µ q,0, σ 2 q,0) and initial latent error (ε z,0) are constants and the initial observation error (ε x,0) is a constant affine transformation of the observation (x). When the inference model is a neural network, then encoding x or an affine transformation of x is equivalent (assuming the inputs are properly normalized). Therefore, eqs. 10 and 11 simplify to that of a standard inference model, eq. 4. From this perspective, standard inference models can be interpreted as single-step optimization models that learn to approximate derivatives at a single latent point. In the following section, we consider the case in which the second assumption is violated; iterative inference models naturally handle this case, whereas standard inference models do not. Hierarchical latent variable models contain higher level latent variables that provide empirical priors on lower level variables; p θ (z) is thus observation-dependent (see Figure 7 in Appendix A.6). The approximate posterior gradients for an intermediate level in a hierarchical latent Gaussian model (see Appendix A.6) take a similar form as eqs. 8 and 9, comprising bottom-up errors from lower variables and top-down errors from higher variables. Iterative inference models encode both of these errors, either directly or through the gradient. However, standard inference models, which map x and lower latent variables to each level of latent variables, can only approximate bottom-up information. Lacking top-down prior information, these models must either use a less expressive prior or output poor approximate posterior estimates. Sønderby et al. identified this phenomenon, proposing a "top-down inference" technique. Iterative inference models formalize and extend this technique. We performed experiments using latent Gaussian models trained on MNIST, Omniglot (Lake et al. FORMULA14). MNIST and Omniglot were dynamically binarized and modeled with Bernoulli output distributions, and Street View House Numbers and CIFAR-10 were modeled with Gaussian output distributions, using the procedure from. All experiments presented here use fully-connected neural networks. Reported values of L were estimated using 1 sample (Figures 3, 5, 6), and reported values of − log p(x) were estimated using 5,000 importance weighted samples TAB0. Additional experiment details, including model architectures and optimizers, can be found in Appendix C. We present additional experiments on text data in Appendix D. Source code will be released online. To confirm the ability of iterative inference models to optimize the approximate posterior, we tested these models in the simplified setting of a 2D latent Gaussian model, trained on MNIST, with a point estimate approximate posterior. The generative model architecture and approximate posterior form are identical to those used in Section 3.1 (see Appendix C.1). Here we show a from encoding x and ∇ µq L through a feedforward neural network. In Figure 3, we visualize an optimization trajectory taken by this model for a particular test example. Despite lacking convergence guarantees, the model learns to adaptively adjust inference update step sizes to navigate the optimization surface, arriving and remaining at a near-optimal approximate posterior estimate for this example. Approximate inference optimization can also be visualized through data reconstructions. In eq. 3, the reconstruction term encourages q(z|x) to represent outputs that closely match the data examples. As this is typically the dominant term in L, during inference optimization, the output reconstructions should improve in terms of visual quality, more closely matching x. We demonstrate this phenomenon with iterative inference models for several data sets in Figure 4 (see Appendix C.2 for additional reconstructions.). Reconstruction quality noticeably improves during inference. We highlight two unique aspects of iterative inference models: direct improvement with additional samples and inference iterations. These aspects provide two advantageous qualitative differences over standard inference models. Additional approximate posterior samples provide more precise gradient estimates, potentially allowing an iterative inference model to output more precise updates. To verify this, we trained standard and iterative inference models on MNIST using 1, 5, 10, and 20 approximate posterior samples. Iterative inference models were trained by encoding the data (x) and approximate posterior gradients (∇ λ L) for 5 iterations. The are shown in FIG4, where we observe that the iterative inference model improves by more than 1 nat with additional samples, while the standard inference model improves by roughly 0.5 nats. We investigated the effect of training with additional inference iterations while encoding approximate posterior gradients (∇ λ L) or errors (ε x, ε z), with or without the data (x). Section 4 and Appendix A define these terms. Note that the encoded terms affect the number of input parameters to the inference model. Here, the iterative inference model that only encodes ∇ λ L has fewer input parameters than a standard inference model, whereas the models that encode errors or data have strictly more input parameters. Experiments were performed on MNIST, with for 2, 5, 10, and 16 inference iterations in FIG4. All encoding schemes outperformed standard inference models with the same architecture, which we found to be consistent over a range of architectures. Encoding the data was beneficial, allowing the inference model to trade off between learning a direct and iterative mapping. Encoding errors allows the iterative inference model to approximate higher order derivatives (Section 4.1), which we observe helps when training with fewer inference iterations. However, it appears that these approximations are less helpful with additional iterations, where derivative approximation errors likely limit performance. TAB0 contains the estimated marginal log-likelihood on MNIST and CIFAR-10 for standard and iterative inference models, including hierarchical inference models. Iterative inference models were trained by encoding the data and errors for 5 inference iterations. With the same architecture, iterative inference models outperform their standard counterparts. See Appendix C.5 for details and discussion. We also compared the inference optimization performance of iterative inference models with variational EM expectation steps using various optimizers. In Figure 6, we observe that the iterative inference model empirically converges substantially faster to better estimates, even with only local gradient information. See Appendix C.6 for details and discussion. To summarize, iterative inference models outperform standard inference models in terms of inference capabilities, yet are far more computationally efficient than variational EM. Consider a latent variable model, p θ (x, z) = p θ (x|z)p θ (z), where the prior on z is a factorized Gaussian density, p θ (z) = N (z; µ p, diag σ 2 x), and the conditional likelihood, p θ (x|z), is Bernoulli for binary observations or Gaussian for continuous observations. We introduce an approximate posterior distribution, q(z|x), which can be any parametric probability density defined over real values. Here, we assume that q also takes the form of a factorized Gaussian density, q(z|x) = N (z; µ q, diag σ 2 q). The objective during variational inference is to maximize L w.r.t. the parameters of q(z|x), i.e. µ q and σ DISPLAYFORM0 To solve this optimization problem, we will inspect the gradients ∇ µq L and ∇ σ 2 q L, which we now derive. The objective can be written as: DISPLAYFORM1 Plugging in p θ (z) and q(z|x): DISPLAYFORM2 Since expectation and differentiation are linear operators, we can take the expectation and derivative of each term individually. We can write the log-prior as: DISPLAYFORM0 where n z is the dimensionality of z. We want to evaluate the following terms: DISPLAYFORM1 and DISPLAYFORM2 To take these derivatives, we will use the reparameterization trick to re-express z = µ q + σ q, where ∼ N (0, I) is an auxiliary standard Gaussian variable, and denotes the element-wise product. We can now perform the expectations over, allowing us to bring the gradient operators inside the expectation brackets. The first term in eqs. 17 and 18 does not depend on µ q or σ 2 q, so we can write: DISPLAYFORM3 and DISPLAYFORM4 To simplify notation, we define the following term: DISPLAYFORM5 allowing us to rewrite eqs. 19 and 20 as: DISPLAYFORM6 and DISPLAYFORM7 We must now find ∂ξ ∂µq and DISPLAYFORM8 and DISPLAYFORM9 where division is performed element-wise. Plugging eqs. 24 and 25 back into eqs. 22 and 23, we get: DISPLAYFORM10 and DISPLAYFORM11 Putting everything together, we can express the gradients as: DISPLAYFORM12 and DISPLAYFORM13 A.3 GRADIENT OF THE LOG-APPROXIMATE POSTERIOR We can write the log-approximate posterior as: DISPLAYFORM14 where n z is the dimensionality of z. Again, we will use the reparameterization trick to re-express the gradients. However, notice what happens when plugging the reparameterized z = µ q + σ q into the second term of eq. 30: DISPLAYFORM15 This term does not depend on µ q or σ 2 q. Also notice that the first term in eq. 30 depends only on σ 2 q. Therefore, the gradient of the entire term w.r.t. µ q is zero: DISPLAYFORM16 The gradient w.r.t. σ 2 q is DISPLAYFORM17 Note that the expectation has been dropped, as the term does not depend on the value of the sampled z. Thus, the gradient of the entire term w.r.t. σ 2 q is: DISPLAYFORM18 The form of the conditional likelihood will depend on the data, e.g. binary, discrete, continuous, etc. Here, we derive the gradient for Bernoulli (binary) and Gaussian (continuous) conditional likelihoods. Bernoulli Output Distribution The log of a Bernoulli output distribution takes the form: DISPLAYFORM0 where µ x = µ x (z, θ) is the mean of the output distribution. We drop the explicit dependence on z and θ to simplify notation. We want to compute the gradients DISPLAYFORM1 and DISPLAYFORM2 Again, we use the reparameterization trick to re-express the expectations, allowing us to bring the gradient operators inside the brackets. Using z = µ q + σ q, eqs. 36 and 37 become: DISPLAYFORM3 and DISPLAYFORM4 where µ x is re-expressed as function of µ q, σ 2 q,, and θ. Distributing the gradient operators yields: DISPLAYFORM5 and DISPLAYFORM6 Taking the partial derivatives and combining terms gives: DISPLAYFORM7 and DISPLAYFORM8 Gaussian Output Density The log of a Gaussian output density takes the form: DISPLAYFORM9 where µ x = µ x (z, θ) is the mean of the output distribution and σ 2 x = σ 2 x (θ) is the variance. We assume σ 2 x is not a function of z to simplify the derivation, however, using σ 2 x = σ 2 x (z, θ) is possible and would simply in additional gradient terms in ∇ µq L and ∇ σ 2 q L. We want to compute the gradients DISPLAYFORM10 and DISPLAYFORM11 The first term in eqs. 45 and 46 is zero, since σ 2 x does not depend on µ q or σ 2 q. To take the gradients, we will again use the reparameterization trick to re-express z = µ q + σ q. We now implicitly express µ x as µ x (µ q, σ 2 q, θ). We can then write: DISPLAYFORM12 and DISPLAYFORM13 To simplify notation, we define the following term: DISPLAYFORM14 allowing us to rewrite eqs. 47 and 48 as DISPLAYFORM15 and DISPLAYFORM16 We must now find ∂ξ ∂µq and DISPLAYFORM17 and DISPLAYFORM18 Plugging these expressions back into eqs. 50 and 51 gives DISPLAYFORM19 and DISPLAYFORM20 Despite having different distribution forms, Bernoulli and Gaussian output distributions in approximate posterior gradients of a similar form: the Jacobian of the output model multiplied by a weighted error term. A.5 SUMMARY Putting the gradient terms from log p θ (x|z), log p θ (z), and log q(z|x) together, we arrive at Bernoulli Output Distribution: DISPLAYFORM21 Gaussian Output Distribution: Figure 7: Plate notation for a hierarchical latent variable model consisting of L levels of latent variables. Variables at higher levels provide empirical priors on variables at lower levels. With data-dependent priors, the model has more flexibility in representing the intricacies of each data example. DISPLAYFORM22 Hierarchical latent variable models factorize the latent variables over multiple levels, z = {z 1, z 2, . . ., z L}. Latent variables at higher levels provide empirical priors on latent variables at lower levels. For an intermediate latent level, we use the notation DISPLAYFORM0 DISPLAYFORM1. FORMULA0 Notice that these gradients take a similar form to those of a one-level latent variable model. The first terms inside each expectation can be interpreted as a "bottom-up" gradient coming from reconstruction errors at the level below. The second terms inside the expectations can be interpreted as "top-down" errors coming from priors generated by the level above. The last term in the variance gradient expresses a form of regularization. Standard hierarchical inference models only contain bottom-up information, and therefore have no way of estimating the second term in each of these gradients. Equation 5 provides a general form for an iterative inference model. Here, we provide specific implementation details for these models. Code for reproducing the experiments will be released online. As mentioned in , gradients can be on vastly different scales, which is undesirable for training neural networks. To handle this issue, we adopt the technique they proposed: replacing ∇ λ L with the concatenation of [α log(|∇ λ L| +), sign(∇ λ L)], where α is a scaling constant and is a small constant for numerical stability. This is performed for both parameters in λ = {µ q, log σ 2 q}. When encoding the errors, we instead input the concatenation of [ε x, ε z] (see section 4.1 for definitions of these terms). As we use global variances on the output and prior densities, we drop σ 2 x and σ 2 p from these expressions because they are constant across all examples. We also found it beneficial to encode the current estimates of µ q and log σ 2 q. We end by again noting that encoding gradients or errors over successive iterations can be difficult, as the distributions of these inputs change quickly during both learning and inference. Work remains to be done in developing iterative encoding architectures that handle this aspect more thoroughly, perhaps through some form of input normalization or saturation. For the output form of these models, we use a gated updating scheme, sometimes referred to as a "highway" connection . Specifically, approximate posterior parameters are updated according to DISPLAYFORM0 where represents element-wise multiplication and DISPLAYFORM1 is the gating function for λ at time t, which we combine with the iterative inference model f t. We found that this yielded improved performance and stability over the residual updating scheme used in. In our experiments with latent Gaussian models, we found that means tend to receive updates over many iterations, whereas variances (or log variances) tend to receive far fewer updates, often just a single large update. Further work could perhaps be done in developing schemes that update these two sets of parameters differently. We parameterize iterative inference models as neural networks. Although exclusively use recurrent neural networks, we note that optimization models can also be instantiated with feed-forward networks. Note that even with a feed-forward network, because the entire model is run over multiple iterations, the model is technically a recurrent network, though quite different from the standard RNN formulation. RNN iterative inference models, through hidden or memory states, are able to account for non-local curvature information, analogous to momentum or other moment terms in conventional optimization techniques. Feed-forward networks are unable to capture and utilize this information, but purely local curvature information is still sufficient to update the output estimate, e.g. vanilla stochastic gradient descent. propagate optimizer parameter gradients (∇ φ L) from the optimizee's loss at each optimization step, giving each step equal weight. We take the same approach; we found it aids in training recurrent iterative inference models and is essential for training feed-forward iterative inference models. With a recurrent model, ∇ φ L is calculated using stochastic backpropagation through time. With a feedforward model, we accumulate ∇ φ L at each step using stochastic backpropagation, then average over the total number of steps. The advantage of using a feed-forward iterative inference model is that it maintains a constant memory footprint, as we do not need to keep track of gradients across iterations. However, as mentioned above, this limits the iterative inference model to only local optimization information. Overall, we found iterative inference models were not difficult to train. Almost immediately, these models started learning to improve their estimates. As noted by , some care must be taken to ensure that the input gradients stay within a reasonable range. We found their log transformation trick to work well in accomplishing this. We also observed that the level of stochasticity in the gradients played a larger role in inference performance for iterative inference models. For instance, in the Gaussian case, we noticed a sizable difference in performance between approximating the KL-divergence and evaluating it analytically. This difference was much less noticeable for standard inference models. In all experiments, inference model and generative model parameters were learned jointly using the AdaM optimizer (Kingma & Ba FORMULA0). The learning rate was set to 0.0002 for both sets of parameters and all other optimizer parameters were set to their default values. Learning rates were decayed exponentially by a factor of 0.999 at every epoch. All models utilized exponential linear unit (ELU) activation functions , although we found other non-linearities to work as well. Unless otherwise stated, all inference models were symmetric to their corresponding generative models, with the addition of "highway" connections between hidden layers. Though not essential, we found that these connections improved stability and performance. Iterative inference models for all experiments were implemented as feed-forward networks to make comparison with standard inference models easier. See appendix B for further details. To visualize the optimization surface and trajectories of latent Gaussian models, we trained models with 2 latent dimensions and a point estimate approximate posterior. That is, q(z|x) = δ(z = µ q) is a Dirac delta function at the point µ q = (µ 1, µ 2). We used a 2D point estimate approximate posterior instead of a 1D Gaussian density because it in more variety in the optimization surface, making it easier to visualize the optimization. We trained these models on binarized MNIST due to the data set's relatively low complexity, meaning that 2 latent dimensions can reasonably capture the relevant information specific to a data example. The generative models consisted of a neural network with 2 hidden layers, each with 512 units. The output of the generative model was the mean of a Bernoulli distribution, and log p θ (x|z) was evaluated using binary cross-entropy. KL-divergences were estimated using 1 sample of z ∼ q(z|x). The optimization surface of each model was evaluated on a grid with range [-5, 5] in increments of 0.05 for each latent variable. To approximate the MAP estimate, we up-sampled the optimization surface using a cubic interpolation scheme. FIG0 visualizes the ELBO optimization surface after training for 80 epochs. Figure 3 visualizes the ELBO optimization surface after training (by encoding x, ε x, and ε z) for 50 epochs. For the qualitative shown in figure 4, we trained iterative inference models on MNIST, Omniglot, and Street View House Numbers by encoding approximate posterior gradients (∇ λ L) for 16 inference iterations. For CIFAR-10, we had difficulty in obtaining sharp reconstructions in a reasonable number of inference iterations, so we trained an iterative inference model by encoding errors for 10 inference iterations. For binarized MNIST and Omniglot, we used a generative model architecture with 2 hidden layers, each with 512 units, a latent space of size 64, and a symmetric iterative inference model, with the addition of highway connections at each layer. For Street View House Numbers and CIFAR-10, we used 3 hidden layers in the iterative inference and 1 in the generative model, with 2048 units at each hidden layer and a latent space of size 1024. We used the same architecture of 2 hidden layers, each with 512 units, for the output model and inference models. The latent variables consisted of 64 dimensions. Each model was trained by drawing the corresponding number of samples from the approximate posterior distribution using the reparameterization trick, yielding lower variance ELBO estimates and gradients. Iterative inference models were trained by encoding the data (x) and the approximate posterior gradients (∇ λ L) for 5 inference iterations. All models were trained for 1,500 epochs. The model architecture for all encoding schemes was identical to that used in the previous section. All models were trained by evaluating the ELBO with a single approximate posterior sample. We trained all models for 1,500 epochs. We were unable to run multiple trials for each experimental set-up, but on a subset of runs for standard and iterative inference models, we observed that final performance had a standard deviation less than 0.1 nats, below the difference in performance between models trained with different numbers of inference iterations. Directly comparing inference optimization performance between inference techniques is difficult; inference estimates affect learning, ing in models that are better suited to the inference scheme. Instead, to quantitatively compare the performance between standard and iterative inference models, we trained models with the same architecture using each inference model form. We trained both one-level and hierarchical models on MNIST and one-level models on CIFAR-10. In each case, iterative inference models were trained by encoding the data and errors for 5 inference iterations. We estimated marginal log-likelihoods for each model using 5,000 importance weighted samples per data example. C.5.1 MNIST For MNIST, one-level models consisted of a latent variable of size 64, and the inference and generative networks both consisted of 2 hidden layers, each with 512 units. Hierarchical models consisted of 2 levels with latent variables of size 64 and 32 in hierarchically ascending order. At each level, the inference and generative networks consisted of 2 hidden layers, with 512 units at the first level and 256 units at the second level. At the first level of latent variables, we also used a set of deterministic units, also of size 64, in both the inference and generative networks. Hierarchical models included batch normalization layers at each hidden layer of the inference and generative networks; we found this beneficial for training both standard and iterative inference models. Both encoder and decoder networks in the hierarchical model utilized highway skip connections at each layer at both levels. For CIFAR-10, models consisted of a latent variable of size 1024, an encoder network with 3 hidden layers of 2048 units with highway connections, and a decoder network with 1 hidden layer with 2048 units. The variance of the output Gaussian distribution was a global variable for this model. We note that the reported in table 1 are significantly worse than those typically reported in the literature, however these are for relatively small fully-connected networks rather than larger convolutional networks. We also experimented with hierarchical iterative inference models on CIFAR-10, but found these models more difficult to train without running into numerical instabilities. C.6 COMPARISON WITH VARIATIONAL EM Variational EM is not typically used in practice, as it does not scale well with large models or large data sets. However, because iterative inference models iteratively optimize the approximate posterior parameters, we felt it would be beneficial to provide a comparison of inference optimization performance between iterative inference models and expectation steps from variational EM. We used one-level latent Gaussian models trained with iterative inference models on MNIST for 16 iterations. We compared against vanilla SGD, SGD with momentum, RMSProp, and AdaM, trying learning rates in {0.5, 0.4, 0.3, 0.2, 0.1, 0.01, 0.001}. In all comparisons, we found that iterative inference models outperformed conventional optimization techniques by large margins. Figure 6 shows the optimization performance on the test set for all optimizers and an iterative inference model trained by encoding the approximate posterior gradients. The iterative inference model quickly arrives at a stable approximate posterior estimate, outperforming all optimizers. It is important to note that the iterative inference model here actually has less derivative information than the adaptive optimizers; it only has access to the local gradient. Also, despite only being trained using 16 iterations, the iterative inference remains stable for hundreds of iterations. We also compared the optimization techniques on the basis of wall clock time: FIG0 reproduces the from figure 6. We observe that, despite requiring more time per inference iteration, the iterative inference model still outperforms the conventional optimization techniques. Concurrent with our work, propose closing the amortization gap by performing inference optimization steps after initially encoding the data with a standard inference model, reporting substantial gains on sparse, high-dimensional data, such as text and ratings. We observe similar findings and present a confirmatory experimental on the RCV1 data set , which consists of 10,000 dimensions containing word counts. We follow the same processing procedure as , encoding data using normalized TF-IDF features and modeling the data using a multinomial distribution. For encoder and decoder, we use 2-layer networks, each with 512 units and ELU non-linearities. We use a latent variable of size 512 as well. The iterative inference model was trained by encoding gradients for 16 steps. We evaluate the models by reporting (an upper bound on) perplexity on the test set TAB1. Perplexity, P, is defined as DISPLAYFORM0 where N is the number of examples and N i is the total number of word counts in example i. We evaluate perplexity by estimating each log p(x i) with 5,000 importance weighted samples. We observe that iterative inference models outperform standard inference models on this data set by a similar margin reported by. Note, however, that iterative inference models here have substantially fewer input parameters than standard inference models (2,048 vs. 10,000). We also run a single optimization procedure for an order of magnitude fewer steps than that of.In FIG0, we further illustrate the optimization capabilities of the iterative inference model used here. Plotting the average gradient magnitude of the approximate posterior for inference iterations in FIG0, we see that over successive iterations, the magnitude decreases. This implies that the model is capable of arriving at near-optimal estimates, where the gradient is close to zero. In FIG0, we plot the average relative improvement in the ELBO over inference iterations. We see that the model is quickly able to improve its inference estimates, eventually reaching a relative improvement of roughly 25%.

Represent the short summary for finding the corresponding scientific article (long)

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: In artificial neural networks trained with gradient descent, the weights used for processing stimuli are also used during backward passes to calculate gradients. For the real brain to approximate gradients, gradient information would have to be propagated separately, such that one set of synaptic weights is used for processing and another set is used for backward passes. This produces the so-called "weight transport problem" for biological models of learning, where the backward weights used to calculate gradients need to mirror the forward weights used to process stimuli. This weight transport problem has been considered so hard that popular proposals for biological learning assume that the backward weights are simply random, as in the feedback alignment algorithm. However, such random weights do not appear to work well for large networks. Here we show how the discontinuity introduced in a spiking system can lead to a solution to this problem. The ing algorithm is a special case of an estimator used for causal inference in econometrics, regression discontinuity design. We show empirically that this algorithm rapidly makes the backward weights approximate the forward weights. As the backward weights become correct, this improves learning performance over feedback alignment on tasks such as Fashion-MNIST and CIFAR-10. Our demonstrate that a simple learning rule in a spiking network can allow neurons to produce the right backward connections and thus solve the weight transport problem. Any learning system that makes small changes to its parameters will only improve if the changes are correlated to the gradient of the loss function. Given that people and animals can also show clear behavioral improvements on specific tasks , however the brain determines its synaptic updates, on average, the changes in must also correlate with the gradients of some loss function related to the task . As such, the brain may have some way of calculating at least an estimator of gradients. To-date, the bulk of models for how the brain may estimate gradients are framed in terms of setting up a system where there are both bottom-up, feedforward and top-down, feedback connections. The feedback connections are used for propagating activity that can be used to estimate a gradient (; ; ; ; ; ;). In all such models, the gradient estimator is less biased the more the feedback connections mirror the feedforward weights. For example, in the REINFORCE algorithm , and related algorithms like AGREL , learning is optimal when the feedforward and feedback connections are perfectly symmetric, such that for any two neurons i and j the synaptic weight from i to j equals the weight from j to i, e.g. W ji = W ij (Figure 1). Some algorithms simply assume weight symmetry, such as Equilibrium Propagation . The requirement for synaptic weight symmetry is sometimes referred to as the "weight transport problem", since it seems to mandate that the values of the feedforward synaptic weights are somehow transported into the feedback weights, which is not biologically realistic (-01-12;). Solving the weight transport problem is crucial to biologically realistic gradient estimation algorithms , and is thus an important topic of study. Several solutions to the weight transport problem have been proposed for biological models, including hard-wired sign symmetry , random fixed feedback weights , and learning to make the feedback weights symmetric (; ; ;). Learning to make the weights symmetric is promising because it is both more biologically feasible than hard-wired sign symmetry and it leads to less bias in the gradient estimator (and thereby, better training ) than using fixed random feedback weights . However, of the current proposals for learning weight symmetry some do not actually work well in practice and others still rely on some biologically unrealistic assumptions, including scalar value activation functions (as opposed to all-or-none spikes) and separate error feedback pathways with one-to-one matching between processing neurons for the forward pass and error propagation neurons for the backward pass;. Interestingly, learning weight symmetry is implicitly a causal inference problem-the feedback weights need to represent the causal influence of the upstream neuron on its downstream partners. As such, we may look to the causal infererence literature to develop better, more biologically realistic algorithms for learning weight symmetry. In econometrics, which focuses on quasi-experiments, researchers have developed various means of estimating causality without the need to actually randomize and control the variables in question;. Among such quasi-experimental methods, regression discontinuity design (RDD) is particularly promising. It uses the discontinuity introduced by a threshold to estimate causal effects. For example, RDD can be used to estimate the causal impact of getting into a particular school (which is a discontinuous, all-or-none variable) on later earning power. RDD is also potentially promising for estimating causal impact in biological neural networks, because real neurons communicate with discontinuous, all-or-none spikes. Indeed, it has been shown that the RDD approach can produce unbiased estimators of causal effects in a system of spiking neurons. Given that learning weight symmetry is fundamentally a causal estimation problem, we hypothesized that RDD could be used to solve the weight transport problem in biologically realistic, spiking neural networks. Here, we present a learning rule for feedback synaptic weights that is a special case of the RDD algorithm previously developed for spiking neural networks . Our algorithm takes advantage of a neuron's spiking discontinuity to infer the causal effect of its spiking on the activity of downstream neurons. Since this causal effect is proportional to the feedforward synaptic weight between the two neurons, by estimating it, feedback synapses can align their weights to be symmetric with the reciprocal feedforward weights, thereby overcoming the weight transport problem. We demonstrate that this leads to the reduction of a cost function which measures the weight symmetry (or the lack thereof), that it can lead to better weight symmetry in spiking neural networks than other algorithms for weight alignment and it leads to better learning in deep neural networks in comparison to the use of fixed feedback weights . Altogether, these demonstrate a novel algorithm for solving the weight transport problem that takes advantage of discontinuous spiking, and which could be used in future models of biologically plausible gradient estimation. Previous work has shown that even when feedback weights in a neural network are initialized randomly and remain fixed throughout training, the feedforward weights learn to partially align themselves to the feedback weights, an algorithm known as feedback alignment . While feedback alignment is successful at matching the learning performance of true gradient descent in relatively shallow networks, it does not scale well to deeper networks and performs poorly on difficult computer vision tasks . The gap in learning performance between feedback alignment and gradient descent can be overcome if feedback weights are continually updated to match the sign of the reciprocal feedforward weights . Furthermore, learning the feedback weights in order to make them more symmetric to the feedforward weights has been shown to improve learning over feedback alignment . To understand the underlying dynamics of learning weight symmetry, define the symmetric alignment cost function, R SA, as one possible cost function that, when minimized, leads to weight symmetry: where W are feedforward weights and Y are feedback weights. The first two terms are simply weight regularization terms that can be minimized using techniques like weight decay. But, the third term is the critical one for ensuring weight alignment. In this paper we present a biologically plausible method of minimizing the third term. This method is based on the work of , who demonstrated that neurons can estimate their causal effect on a global reward signal using the discontinuity introduced by spiking. This is accomplished using RDD, wherein a piecewise linear model is fit around a discontinuity, and the differences in the regression intercepts indicates the causal impact of the discontinuous variable. , neurons learn a piece-wise linear model of a reward signal as a function of their input drive, and estimate the causal effect of spiking by looking at the discontinuity at the spike threshold. Here, we modify this technique to perform causal inference on the effect of spiking on downstream neurons, rather than a reward signal. We leverage this to develop a learning rule for feedback weights that induces weight symmetry and improves training. The primary contributions of this paper are as follows: • We demonstrate that spiking neurons can accurately estimate the causal effect of their spiking on downstream neurons by using a piece-wise linear model of the feedback as a function of the input drive to the neuron. • We present a learning rule for feedback weights that uses the causal effect estimator to encourage weight symmetry. We show that when feedback weights update using this algorithm it minimizes the symmetric alignment cost function, R SA. • We demonstrate that this learning weight symmetry rule improves training and test accuracy over feedback alignment, approaching gradient-descent-level performance on Fashion-MNIST, SVHN, CIFAR-10 and VOC in deeper networks. In this work, we utilize a spiking neural network model for aligning feedforward and feedback weights. However, due to the intense computational demands of spiking neural networks, we only use spikes for the RDD algorithm. We then use the feedback weights learned by the RDD algorithm for training a non-spiking convolutional neural network. We do this because the goal of our work here is to develop an algorithm for aligning feedback weights in spiking networks, not for training feedforward weights in spiking networks on other tasks. Hence, in the interest of computational expediency, we only used spiking neurons when learning to align the weights. Additional details on this procedure are given below. At the start of every training epoch of a convolutional neural network, we use an RDD feedback weight training phase, during which all fully-connected sets of feedback weights in the network are updated. To perform these updates, we simulate a separate network of leaky integrate-and-fire (LIF) neurons. LIF neurons incorporate key elements of real neurons such as voltages, spiking thresholds and refractory periods. Each epoch, we begin by training the feedback weights in the LIF network. These weights are then transferred to the convolutional network, which is used for training the feedforward weights. The new feedforward weights are then transferred to the LIF net, and another feedback training phase with the LIF net starts the next epoch (Figure 2A). During the feedback training phase, the LIF network undergoes a training phase lasting 90 s of simulated time (30 s per set of feedback weights) (Figure 2B). We find that the spiking network used for RDD feedback training and the convolutional neural network are very closely matched in the activity of the units (Figure S1), which gives us confidence that this approach of using a separate non-spiking network for training the feedforward weights is legitimate. During the feedback training phase, a small subset of neurons in the first layer receive driving input that causes them to spike, while other neurons in this layer receive no input (see Appendix A.2). The subset of neurons that receive driving input is randomly selected every 100 ms of simulated time. This continues for 30 s in simulated time, after which the same process occurs for the subsequent hidden layers in the network. This protocol enforces sparse, de-correlated firing patterns that improve the causal inference procedure of RDD. During the RDD feedback training phase, each unit in the network is simulated as a leaky integrateand-fire neuron. Spiking inputs from the previous layer arrive at feedforward synapses, where they are convolved with a temporal exponential kernel to simulate post-synaptic spike responses p = [p 1, p 2, ..., p m] (see Appendix A.1). The neurons can also receive driving inputp i, instead of synaptic inputs. The total feedforward input to neuron i is thus defined as: where W ij is the feedforward weight to neuron i from neuron j in the previous layer, and ω is a hyperparameter. The voltage of the neuron, v i, evolves as: where g L and g D are leak and dendritic conductance constants, respectively. The input drive to the neuron, u i, is similarly modeled: If the voltage v i passes a spiking threshold θ, the neuron spikes and the voltage is reset to a value v reset = −1 (Figure 2C). Note that the input drive does not reset. This helps us to perform regressions both above and below the spike threshold. In addition to feedforward inputs, spiking inputs from the downstream layer arrive at feedback synapses, where they create post-synaptic spike responses q = [q 1, q 2, ..., q n]. These responses are used in the causal effect estimation (see below). Whenever the voltage approaches the threshold θ (ie. |v i − θ| < α where α is a constant), an RDD window is initiated, lasting T = 30 ms in simulated time (Figure 2C). At the end of this time window, at each feedback synapse, the maximum input drive during the RDD window, u provides a measure of how strongly neuron i was driven by its inputs (and whether or not it passed the spiking threshold θ), while ∆q avg k is a measure of how the input received as feedback from neuron k changed after neuron i was driven close to its spiking threshold. These two values are then used to fit a piece-wise linear model of ∆q avg k as a function of u max i (Figure 2D). This piece-wise linear model is defined as: The parameters c ik are updated to perform linear regression using gradient descent: An estimate of the causal effect of neuron i spiking on the activity of neuron k, β ik, is then defined as the difference in the two sides of the piece-wise linear function at the spiking threshold: Finally, the weight at the feedback synapse, Y ik, is updated to be a scaled version of β ik: where γ is a hyperparameter and σ 2 β is the standard deviation of β values for all feedback synapses in the layer. This ensures that the scale of the full set of feedback weights between two layers in the network remains stable during training. To measure how well the causal effect estimate at each feedback synapse, β ik, and thus the feedback weight Y ik, reflects the reciprocal feedforward weight W ki, we can measure the percentage of feedback weights that have the same sign as the reciprocal feedforward weights (Figure 3A). When training on CIFAR-10 with no RDD feedback training phase (ie. feedback weights remain fixed throughout training), the feedback alignment effect somewhat increases the sign alignment during training, but it is ineffective at aligning the signs of weights in earlier layers in the network. Compared to feedback alignment, the addition of an RDD feedback training phase greatly increases the sign aligmnent between feedback and feedforward weights for all layers in the network, especially at earlier layers. In addition, the RDD algorithm increases sign alignment throughout the hierarchy more than the current state-of-the-art algorithm for weight alignment introduced recently by (Figure 3A). Furthermore, RDD feedback training changes feedback weights to not only match the sign but also the magnitude of the reciprocal feedforward weights (Figure 3B), which makes it better for weight alignment than hard-wired sign symmetry . Figure 3: A. Evolution of sign alignment (the percent of feedforward and feedback weights that have the same sign) for each fully-connected layer in the network when trained on CIFAR-10 using RDD feedback training (blue), using the algorithm proposed by (purple), and using feedback alignment (red). B. Feedforward vs. feedback weights for each fully-connected layer at the end of training, with RDD feedback training (blue), the Akrout algorithm (purple), and feedback alignment (red). The symmetric alignment cost function (Equation 1) can be broken down as: where we define R decay and R self as: R decay is simply a weight regularization term that can be minimized using techniques like weight decay. R self, in contrast, measures how well aligned in direction the two matrices are. Our learning rule for feedback weights minimizes the R self term for weights throughout the network (Figure 4). By comparison, feedback alignment decreases R self to a smaller extent, and its ability to do so diminishes at earlier layers in the network. This helps to explain why our algorithm induces weight alignment, and can improve training performance (see below). Figure 4: Evolution of Rself for each fully-connected layer in the network when trained on CIFAR-10 using RDD feedback training (solid lines) and using feedback alignment (dashed lines). RDD feedback training dramatically decreases this loss compared to feedback alignment, especially in earlier layers. We trained the same network architecture (see Appendix A.3) on the Fashion-MNIST, SVHN, CIFAR-10 and VOC datasets using standard autograd techniques (backprop), feedback alignment and our RDD feedback training phase. RDD feedback training substantially improved the network's performance over feedback alignment, and led to backprop-level accuracy on the train and test sets (Figure 5). In order to understand how the brain learns complex tasks that require coordinated plasticity across many layers of synaptic connections, it is important to consider the weight transport problem. Here, we presented an algorithm for updating feedback weights in a network of spiking neurons that takes advantage of the spiking discontinuity to estimate the causal effect between two neurons (Figure 2). We showed that this algorithm enforces weight alignment (Figure 3), and identified a loss function, R self, that is minimized by our algorithm (Figure 4). Finally, we demonstrated that our algorithm allows deep neural networks to achieve better learning performance than feedback alignment on Fashion-MNIST and CIFAR-10 (Figure 5). These demonstrate the potential power of RDD as a means for solving the weight transport problem in biologically plausible deep learning models. One aspect of our algorithm that is still biologically implausible is that it does not adhere to Dale's principle, which states that a neuron performs the same action on all of its target cells (Strata & Harvey). This means that a neuron's outgoing connections cannot include both positive and negative weights. However, even under this constraint, a neuron can have an excitatory effect on one downstream target and an inhibitory effect on another, by activating intermediary inhibitory interneurons. Because our algorithm provides a causal estimate of one neuron's impact on another, theoretically, it could capture such polysynaptic effects. Therefore, this algorithm is in theory compatible with Dale's principle. Future work should test the effects of this algorithm when implemented in a network of neurons that are explicitly excitatory or inhibitory. A APPENDIX Post-synaptic spike responses at feedforward synapses, p, were calculated from pre-synaptic binary spikes using an exponential kernel function κ: wheret jk is the k th spike time of input neuron j and κ is given by: where τ s = 0.003 s and τ L = 0.01 s represent short and long time constants, and Θ is the Heaviside step function. Post-synaptic spike responses at feedback synapses, q, were computed in the same way. A.2.1 WEIGHT SCALING Weights were shared between the convolutional network and the network of LIF neurons, but feedforward weights in the LIF network were scaled versions of the convolutional network weights: where W Conv is a feedforward weight matrix in the convolutional network, W LIF is the corresponding weight matrix in the LIF network, m is the number of units in the upstream layer (ie. the number of columns in W Conv), σ 2 W Conv is the standard deviation of W Conv and ψ is a hyperparameter. This rescaling ensures that spike rates in the LIF network stay within an optimal range for the RDD algorithm to converge quickly, even if the scale of the feedforward weights in the convolutional network changes during training. This avoids situations where the scale of feedforward weights is so small that little or no spiking occurs in the LIF neurons. The RDD feedback training paradigm is implemented as follows. We start by providing driving input to the first layer in the network of LIF neurons. To create this driving input, we choose a subset of 20% of the neurons in that layer, and create a unique input spike train for each of these neurons using a Poisson process with a rate of 200 Hz. All other neurons in the layer receive no driving input. Every 100 ms, a new set of neurons to receive driving input is randomly chosen. After 30 s, this layer stops receiving driving input, and the process repeats for the next layer in the network. The network architectures used to train on Fashion-MNIST and CIFAR-10 are described in Table 1. Inputs were randomly cropped and flipped during training, and batch normalization was used at each layer. Networks were trained using a minibatch size of 32. A.4 AKROUT ALGORITHM IMPLEMENTATION In experiments that compared sign alignment using our RDD algorithm with the algorithm, we kept the same RDD feedback training paradigm (ie. layers were sequentially driven, and a small subset of neurons in each layer was active at once). However, rather than updating feedback weights using RDD, we recorded the mean firing rates of the active neurons in the upstream layer, r l, and the mean firing rates in the downstream layer, r l+1. We then used the following feedback weight update rule: where Y are the feedback weights between layers l + 1 and l, and η and λ WD are learning rate and weight decay hyperparameters, respectively. Figure S1: Comparison of average spike rates in the fully-connected layers of the LIF network vs. activities of the same layers in the convolutional network, when both sets of layers were fed the same input. Spike rates in the LIF network are largely correlated with activities of units in the convolutional network.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a learning rule for feedback weights in a spiking neural network that addresses the weight transport problem.

We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.