Question:
represent the text to find a very short summary (1-2 sentences max)

Examples:
'We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the ing movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video (https://youtu.be/CaDEf-QcKwA) summarizing our . A broad challenge in machine learning for control and robotics is to produce policies capable of general, flexible, and adaptive behavior of complex, physical bodies. To build policies that can effectively control simulated humanoid bodies, researchers must simultaneously overcome foundational challenges related to high-dimensional control, body balance, and locomotion. Recent progress in deep reinforcement learning has raised hopes that such behaviors can be learned end-to-end with minimal manual intervention. Yet, even though significant progress has been made thanks to better algorithms, training regimes, and computational infrastructure, the ing behaviors still tend to exhibit significant idiosyncrasies (e.g. BID2 .One advantage of working with humanoids in this context is that motion capture data is widely available and can serve to help design controllers that produce apparently humanlike movement. Indeed, recent developments are now allowing for the production of highly specialized expert policies which robustly, albeit narrowly, reproduce single motion capture clips (e.g. BID18 ; BID30).A remaining challenge on the way to truly flexible and general purpose control is to be able to sequence and generalize individual movements or "skills" in a task-directed manner. Achieving this goal requires not just the ability to acquire individual skills in the first place, but also an architecture and associated training procedure that supports representation, recruitment, and composition of a large number of skills. This paper presents a step in this direction. Specifically, the setting we focus on will be one in which we have a large number of robust experts that perform single skills well and we wish to transfer these skills into a shared policy that can do what each expert does as well as the expert, while also generalizing to unseen behaviors within the distribution of skills. To this end we design a system that performs one-shot imitation as well as permits straightforward reuse (or transfer) of skills. We require our approach to scale to a very large number of individual skills while also keeping manual intervention and oversight to a minimum. Our primary contribution is the development of a neural network architecture that can represent and generate many motor behaviors, which we refer to as neural probabilistic motor primitives. This architecture is designed to perform one-shot imitation, while learning a dense embedding space of a large number of individual motor skills. Once trained, this module does not just reproduce individual behaviors in the training data, but can sequence and compose these behaviors in a controlled fashion as well as synthesize novel movements consistent with the training data distribution. Empirically, we also find that training controllers to reuse this learned motor primitive module for new tasks generates surprisingly human-like movement and the behavior generated seems to interpolate the space of behaviors well. In order to facilitate transfer and compression of expert skills at the scale of thousands of behaviors, we wish to avoid closed-loop RL training. We call the general, offline, functional transfer of policy content policy transfer or policy cloning and consider two approaches. The natural baseline approach involves the application of behavioral cloning to data gathered by executing experts many times, with noise, and logging intended expert actions, resembling the approach of BID16. This works well, as it ensures the student behaves like the expert not only along nominal expert rollouts but also at points arrived at by perturbing the expert. However, this approach may require many rollouts, which can be costly to obtain in many settings. As a more efficient alternative we therefore consider a second solution that operates by comprehensively transferring the functional properties of an expert to a student policy by matching the local noise-feedback properties along one or a small number of representative expert reference trajectories. We call this specific proposal linear feedback policy cloning (LFPC), and we demonstrate that it is competitive with behavioral cloning from many more rollouts in our setting. Recent efforts in RL for humanoid control build on a large body of research in robotics and animation. While contemporary for learning from scratch BID35 can be impressive the behaviors are not consistently human-like. Learning from motion capture (mocap) can provide strong constraints, especially for running BID29. Several recent approaches have demonstrated that it is possible to acquire specific behavioral skills, possibly jointly with external RL objectives BID30 BID17. At present, the policies produced tend to be restricted to single skills/behaviors and can require very large quantities of environment interactions, motivating us to seek methods which reuse existing single-skill expert policies. Knowledge transfer refers to the broad class of approaches which transfer the input-output functional mapping, to some extent or another, from a teacher (or expert) to a student BID12 BID36 BID8. Distillation connotes the transfer of function from one or more expert systems into a single student system often with the goal of compression or of combining multiple experts qualities BID12 BID28 BID33 BID40. Imitation learning is the control-specific term for the production of a student policy from either an expert policy or the behavioral demonstrations of an expert. One basic algorithm is behavioral cloning, which refers to supervised training of the policy from state-action pairs. In the most simple case it only requires examples from the expert. A broader setting is that in which more liberal queries to the expert are permitted; e.g. for the online-imitation setting as in DAGGER BID32. This setting is often satisfied e.g. if we wish to combine behavior from multiple experts. One-shot imitation is a concept which means that a trained system, at test time, can watch an example behavior and imitate it, as, for instance, in BID7. More similar to our work is the setting examined by, in which full-body humanoid movements were studied. Compared with this latter work, we will employ an architecture here that encourages imitation of motor details, rather than overall movement type, and we scale our approach to more expert demonstrations. The most similar work also demonstrates large-scale one-shot humanoid tracking and was contemporaneously published BID4; the approach they described involves direct tracking as well as failure recovery, but relative to our work the authors do not consider skill reuse. The notion of motor primitives is widespread in neuroscience, where there is evidence that lower dimensional control signals can selectively coordinate and blend behaviors produced by spinal circuits BID3, and that the cortex organizes the space of primitive motor behaviors BID9. In our setting, motor primitives refer to the reusable embedding space learned from many related behaviors and the associated context-modulable policy capable of generating sensory-feedback-stabilized motor behavior when executed in an environment. The particular architecture we consider is inspired by the formalization presented in BID41, which places a probabilistic latent bottleneck on the sensory-motor mapping. In the robotics literature, there is a rich line of research into various parameterizations of motion trajectories used for robot control. A class of these are referred to as "movement primitives" (e.g. BID34, including the "probabilistic movement primitives" of BID27 (see also e.g. BID26 . These approaches can be seen as specific implementation choices for a certain notion of motor primitive, which emphasize the parameterization and learning of movement trajectories from repeated demonstrations BID27 BID20, rather than learning the actuation/stabilization element, which is often handled by a prespecified PID controller. It has previously been recognized that linear-feedback policies can work well around optimal trajectories or limit cycles even for high DoF bodies. These can be obtained by sample-based optimization (e.g. BID6) or by differential dynamic programming BID24. For linear-quadratic-Gaussian control BID1 or differential dynamic programming BID19 BID13, we obtain feedback policies where the feedback terms are computed from the value function, amounting effectively to feedbackstabilized plans. Work by BID23 has shown that linear-feedback policies ing from trajectory optimization can be used to train neural networks. We employ a similar idea to transfer optimal behavior from an existing policy, observing that an optimal policy implicitly reflects the structure of the (local) value landscape and appropriately functions as a feedback controller. Figure 1: Examples of representative experts learned from motion capture. From top to bottom, these are "run and dodge", "cartwheel", "backflip", and "twist". See accompanying video. Note that these four behaviors will be used as representative examples for validation in single-skill transfer experiments. In this section, we will first briefly describe the expert policies used in this work (Sec. 2.1). We then describe the Neural Probabilistic Motor Primitive architecture and objective (Sec. 2.2). We then describe two approaches for training the module offline (Sec. 2.3). In order to study how to transfer and consolidate experts, we must be able to generate adequate quantities of expert data. For this work, we use expert policies trained to reproduce motion capture clips. The approach we use for producing experts is detailed more fully in BID22 and largely follows BID30. It yields time-indexed neural network policies that are robust to moderate amounts of action noise (see appendix A for additional details on the training procedure). Some examples of the ing single-skill time-indexed policies that are obtained from this procedure are depicted in FIG4. All our experts were trained in MuJoCo environments. Data We use the CMU Mocap database 1, which contains more than 2000 clips of varying lengths from more than 100 subjects. The motions in this dataset are quite varied, including many clips of walking, turning, running, jumping, dancing, various hand movements, and many more idiosyncratic behaviors. From this, we selected various clips of generic whole-body movements -any clips longer than 6 seconds were cut into smaller pieces yielding approximately 3000, roughly 2-6 second snippets. Just over half of these are generic locomotion such as walking, running, jumping and turning. The rest of the clips mostly contained diverse hand movements while standing. We trained one expert policy per selected snippet, yielding 2707 expert policies in our training set. Our goal is to obtain a motor primitive module that can flexibly and robustly deploy, sequence, and interpolate a diverse set of skills from a large database of reference trajectories without any manual alignment or other processing of the raw experts. This requires a representation that does not just reliably encode all behavioral modes but also allows effective indexing of behaviors for recall. To ensure plausible and reliable transitions it is further desirable that the encoding of similar behaviors should be close in some sense in the representation space. Compression of many expert skills via a latent variable inverse model We achieve this goal by training an autoregressive latent variable model of the state-conditional action sequence which, at training time, is conditioned on short look-ahead snippets of the nominal/reference trajectory (see FIG0). This architecture has the general structure of an inverse model, which produces actions based on the current state and a target. The architecture and training scheme are designed for the embedding space to reflect short-term motor behavior. As we demonstrate below, this allows for the selective execution of particular behavioral modes and also admits one-shot imitation via the trajectory encoder. We use a model with a latent variable z t at each time step, modelling the state conditional action distribution. The encoder and decoder are distributions q(z t |z t−1, x t) and π(a t |z t, s t) where s t is the state as in preceding sections and x t is concatenation of a small number of future states x t = [s t, ..., s t+K]. The encoder and decoder are MLPs with two and three layers, respectively. For architecture and experimental details see appendix B. The generative part of the model is given by: DISPLAYFORM0 Temporally nearby trajectory snippets should have a similar representation in the latent space. To implement this intuition, we choose an AR process as a weak prior: DISPLAYFORM1 where σ = √ 1 − α 2, ensuring that marginally z t ∼ N (0, I), and set α = 0.95 in experiments unless otherwise stated. In subsequent efforts, it may be interesting to investigate different values of α and learnable priors. In order to train this model, we consider the evidence lower bound (ELBO): DISPLAYFORM2 with a β parameter to tune the weight of the prior. For β = 1 this objective forms the well-known variational lower bound to log p(a 1:T |s 1:T). This objective can be optimized using supervised learning (i.e. behavioral cloning from noisy rollouts) offline. Note we chose not to condition the encoder on actions, since we are interested in one-shot imitation in settings where actions are unobserved. We experimented with different values of K and obtained similar performance. All the reported in this paper use K = 5. Our architecture effectively implements a conditional information bottleneck between the desired future trajectory x t and the action a t given the past latent state z t−1 (similar to Alemi et al. FORMULA0). As discussed above the auto-correlated prior encourages an encoding in which temporally nearby latent states from the same trajectory tend to be close in the latent space, and the information bottleneck more generally encourages a limited dependence on x t with z t forming a compressed representation of the future trajectory as required for the action choice. When transferring knowledge from an expert policy to a student we would like the student to replicate the expert's behavior in the full set of states plausibly visited by the expert. In our case, experts trained to reproduce single clips can be conceptualized as nonlinear feedback controllers around a nominal trajectory, and the manifold of states visited by experts can be thought of as a tube around that reference. We require the student to be able to operate successfully in and remain close to this tube even in the face of small perturbations. Formally, to ensure that the student retains expert robustness, we would like expert actions µ E (s) and student actions µ θ (s) to be close under a plausible (noisy) expert state distribution ρ E. A surrogate loss used in imitation learning as well as knowledge transfer is the quadratic loss between actions (or activations BID36). DISPLAYFORM0 Behavioral cloning can refer to optimization of this objective, where ρ E is replaced with an empirical distribution of a set of state-action pairs S. This works well if S adequately covers the state distribution later experienced by the student. Anticipating and generating an appropriate set of states on which to train the student typically requires many rollouts and can thus be expensive. Since we are aiming to compress the behavior of thousands of experts we desire a computationally efficient method. We investigate two schemes that allow us to record the experts' state-action mappings on a small-sample estimate of the experts' state distributions and to then train the student via supervised learning. Both schemes are convenient to implement in a regular supervised learning pipeline and require neither querying many experts simultaneously (which limits scalability when dealing with thousands of experts) nor execution of the student at training time. Behavioral cloning from noisy rollouts The first approach amounts to simply gathering a number of noisy trajectories from the expert (either under a stochastic policy or with noise injection) while logging the optimal/mean action of the expert instead of the noisy action actually executed. A version of this is equivalent to the DART algorithm of BID16. We then perform behavioral cloning from that data. Specifically, given an expert policy π E, let µ E (s) be the mean action of the expert in state s. To obtain noisy rollouts, we run π η E, the expert with moderate action noise (η) to obtain a set of data {s DISPLAYFORM1 And we optimize the policy according to Eqn. 4, with the expectation over s ∼ ρ E being approximated by a sum over the set of state and expert-actions collected. While we expect this approach can work well, we do not expect it to be particularly efficient insofar as the expert may need to be executed for many rollouts. Linear-feedback policy cloning (LFPC) The second approach, which we refer to as linearfeedback policy cloning (LFPC), logs the action-state Jacobian as well as the expert action along a single nominal trajectory. The Jacobian can be used to construct a linear feedback controller which gives target actions in nearby perturbed states during training (described below). This approach is not intended to outperform behavioral cloning, as this should not be possible for arbitrary quantities of expert rollout data. Instead the motivation for LFPC is to do as well as behavioral cloning while using considerably fewer expert rollouts. As pointed out above, experts trained to reproduce single clips robustly can be thought of as nonlinear feedback controllers around this nominal trajectory. The nominal trajectory refers to the sequence of nominal state-action pairs {s t, a t} 1...T obtained by executing µ E (s) recursively from an initial point s 0. Since expert behavior in our setting is well characterized by single nominal trajectories, we expect we can capture the relevant behavior of the expert by a linearization around the nominal trajectory 3.Let δs be a small perturbation of the state and let J = dµ E (s) ds | s=s be the Jacobian. Then DISPLAYFORM2 This linearization induces a linear-feedback-stabilized policy that at each time-step has a nominal action a t, but also expects to be in state s t, and correspondingly adjusts the nominal action with a linear correction based on discrepancy between the nominal and actual state at time t: DISPLAYFORM3 We empirically validated that a linear feedback policy about the nominal trajectory of the expert can approximate the expert behavior reasonably well for clips we examine (see Fig. 3).Above we presented the expert as a feedback controller operating in a tube around some nominal trajectory with states s 1,..., s T, actions a 1,..., a T, and Jacobians J 1,..., J T. We approximate ρ E with the distribution of states introduced by state perturbations around this nominal trajectory: DISPLAYFORM4 However, this objective still requires expert evaluations at the perturbed states. Using the linearization described above we can replace the expert action µ E (s + δs) with the Jacobian-based linearfeedback policy µ F B (s + δs), which is available offline. This yields the LFPC objective: DISPLAYFORM5 One potentially important choice is the perturbation distribution ∆(s). Ideally, we would like ∆(s) to be the state-dependent distribution induced by physically plausible transitions, but estimating this distribution may require potentially expensive rollouts which we are trying to avoid. A cheaper object to estimate is the stationary transition noise distribution induced by noisy actions, which can be efficiently approximated from a small number of trajectories. Empirically, we found the objective 8 to be relatively robust to some variations in ∆, and we use a fixed marginal distribution for all clips. Objective 8 bears interesting similarities to approaches such as denoising autoencoders BID43, where networks can learn to ignore local noise perturbations on inputs sampled from a high-dimensional noise distribution. Further, BID23 successfully distill feedback policies obtained from a planner. One question left open by this latter work is that of how much data might be required. Empirically we show in the experiments below that the augmented objective 8 can produce the desired robustness even from a very limited set of states. Figure 3: Comparisons of trajectory rollouts for 4 reference behaviors for the nominal trajectory and at varying noise levels. Note that the score is determined by similarity to motion-capture reference and the expert may be slightly suboptimal so slight improvements on the expert may arise by chance. There are multiple, relevant perspectives on LFPC. From one perspective, LFPC amounts to a data augmentation method. From another vantage, the approach attempts to match the mean action as well as the Jacobian at the set of relevant behavioral states, here sampled along the nominal trajectory. In settings where expert behavior is more diverse or multimodal, LFPC should be applied to states which representatively cover relevant behavioral modes or perhaps are expanded backwards from goal states (roughly similar to the procedure used to expand LQR-trees by Tedrake 2009). Explicit Jacobian matching has been proposed elsewhere, for example in. See appendix C for further disambiguation relative to other approaches. To train our Neural Probabilistic Motor Primitive architecture using LFPC we can adapt the objective in Eqn. 3 as follows: DISPLAYFORM6 log π(a t + J t δs t |s t + δs t, z t) + β log p z (z t |z t−1) − log q(z t |z t−1, x t + δx t), where δs t are i.i.d. perturbations drawn from suitable perturbation distribution ∆ and δx t is the concatenation of [δs t, δs t+1, ..., δs t+K]. To ground our in a simple setting, we begin with transfer of a single-skill, time-indexed policy from one network to another. We compare the performance of various time-indexed policies for each of the experts depicted in FIG4 Fig. 3 ). For additional validation, see appendix D.Figure 4: Performance relative to expert policies for trained neural probabilistic motor primitive models. Performance of model variations are compared on training and testing data. We compare models trained using cloning with 100 trajectories per expert for different levels of regularization, using a smaller latent space of dimension 20 rather than 60 in all other experiments, as well as LFPC. Having validated that single skills can be transferred, we next consider how well we can compress behaviors of the 2707 experts in our training set into the neural probabilistic motor primitive architecture. Assessing the models using the action-reconstruction loss is not very intuitive since it does not capture model behavior in the environment. Instead we report a more relevant measure based on expert imitation. Here we encode an expert trajectory into a sequence of latent variables and then execute the policy in the environment conditioned on this sequence. Note that this approach is openloop with respect to the latents while being closed-loop with respect to state. We can then compare the performance of the trained system against experts on training and held-out clips according to the tracking reward used to train the experts originally. To account for different expert reward scales we report performance relative to the expert policy. Importantly, that this approach works is itself a partial validation of the premise of this work, insofar as open-loop execution of action sequences usually trivially fails with minor perturbations. The trained neural probabilistic motor primitive system can execute behaviors conditioned on an open-loop noisy latent variable trajectory, implying that the decoder has learned to stabilize the body during latent-conditioned behavior. There are a few key takeaways from the comparisons we have run (see Fig. 4). Most saliently cloning based on 100 trajectories from each expert with a medium regularization value (β = 0.1) works best. LFPC with comparable parameters works less well here, but has qualitatively fairly similar performance. Our ablations show that regularization and a large latent space are important for good . We also set the autoregressive parameter α = 0 (.95 in other runs), making the latent variables i.i.d.. This hurts performance, validating our choice of prior. We have no expectation that trajectories well outside the training distribution are likely to be either representable by the encoder or executable by the decoder. Nevertheless, when one-shot imitation of a trajectory fails, a natural question is whether the decoder is incapable of expressing the desired actions, or the encoder fails to encode the trajectory in such a way that the decoder will produce it. Figure 5: These panels consist of visualizations of the PCA latent space with comparisons in this space between one-shot latent-variable sequences and optimized latent variable sequences for various behaviors: A. Run B. Backwards walking C. Jumping. Running executes well based on the one-shot trajectory so serves as a reference for which optimization is not noticeably different. Walking backwards and jumping one-shot imitations fail, but are noticeably improved by optimization. We propose an analysis to distinguish this for borderline cases. For held out trajectories that yield unsatisfying performance on one-shot imitation, we can simply optimize directly: DISPLAYFORM0 where µ θ is the decoder mean. Empirically we see that this optimization meaningfully improves the executed behavior, and we visualize the shift in a three-dimensional space given by the first three principal components in Fig. 5.We exhibit three examples where we visualize the original latent trajectory as well as the optimized latent trajectory. Performance is significantly improved (see supplementary video), showing the latent space can represent behaviors for which one-shot imitation fails. However execution remains imperfect suggesting that while much of the fault may lie with the encoder, the decoder still may be slightly undertrained on these relatively rare behavior categories. Quantitatively, among a larger set of clips with less than 50% relative expert performance for one-shot imitation we found that optimization as described above improved median relative expert performance from 43% to 78%.Other exploratory probes of the module suggest that it is possible in certain cases to obtain seamless transitioning between behaviors by concatenating latent-variable trajectories and running the policy conditioned on this sequence (e.g. in order to perform a sequence of turns). See additional supplementary video. Reuse of motor primitive module Finally, we experimented with reuse of the decoder as a motor primitive module. We treat the latent space as a new custom action space and train a new high-level (HL) policy to operate in this space. At each time-step the high-level policy outputs a latent-variable (a) Median return value across 10 seeds for the goto-target task vs learner steps. Compared to a very weakly regularized module (β = 0.001), more regularized motor primitives modules both trained faster and achieved higher final performance.(b) Our model is able to track the target speed accurately. Shown here are target speed and actual speed in the egocentric forward direction for three episodes. The reward function is a Gaussian centered at the target speed. The shaded region corresponds to ± one standard deviation. z t. The actual action is then given by the motor primitive module p(a t |s t, z t). For training we used SVG BID10 with the Retrace off-policy correction for learning the Q-function BID25.A natural locomotion task that can challenge the motor module is a task which requires abrupt, frequently redirected movement with sharp turns and changes of speed. To implement this we provide the higher-level controller with a target that is constant until the humanoid is near it for a few timesteps at which point it randomly moves to another nearby location. While no single task will comprehensively probe the module, performing well in this task demands a wide range of quick locomotion behavior. With only a sparse task reward, the HL-controller can learn to control the body through the learned primitive space, and it produces rather humanlike task-directed movement. We observed that more regularized motor primitive modules had more stable initial behavior when connected to the untrained high-level controller (i.e. were less likely to fall at the beginning of training). Compared to a very weakly regularized module (β = 0.001), more regularized motor primitives modules both trained faster and achieved higher final performance (see FIG2). We also investigated a go-to-target task with bumpy terrain that is unobserved by the agent. The fact that our model can learn to solve this task demonstrates its robustness to unseen perturbations for which the motor primitive module was not explicitly trained. In another experiment we investigated a task in which the agent has to move at a random, changing target speed. This requires transitions between qualitatively different locomotion behavior such as walking, jogging, and running (see FIG2).See an extended video of these experiments. In a final reuse experiment, we consider an obstacle course requiring the agent to jump across gaps (as in BID22). We were able to solve this challenging task with a high-level controller that operated using egocentric visual inputs (see the main supplementary video).We emphasize a few points about these to impact their importance: Using a pretrained neural probabilistic motor primitives module, new controllers can be trained effectively from scratch on sparse reward tasks, the ing movements are visually rather humanlike without additional constraints implying that the learned embedding space is well structured, and the module enables fairly comprehensive and smooth coverage for the purposes of physics-based control. In this paper we have described approaches for transfer and compression of control policies. We have exhibited a motor primitive module that learns to represent and execute motor behaviors for control of a simulated humanoid body. Using either a variant of behavioral cloning or linear feedback policy cloning we can train the neural probabilistic motor primitive sytem to perform robust one-shotimitation, and with the latter we can use relatively restricted data consisting of only single rollouts from each expert. While LFPC did not work quite as well in the full-scale model as cloning from noisy rollouts, we consider it remarkable that it is possible in our setting to transfer expert behavior using a single rollout. We believe LFPC holds promise insofar as it may be useful in settings where rollouts are costly to obtain (e.g. adapted to real-world robotic applications), and there is room for further improvement as we did not carefully tune certain parameters, most saliently the marginal noise distribution ∆.The ing neural probabilistic motor primitive module is interpretable and reusable. We are optimistic that this kind of architecture could serve as a basis for further continual learning of motor skills. This work has been restricted to motor behaviors which do not involve interactions with objects and where a full set a of behaviors are available in advance. Meaningful extensions of this work may attempt to greatly enrich the space of behaviors or demonstrate how to perform continual learning and reuse of new skills. The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.We used the reparametrization trick BID15 BID31 to train the model and used stochastic gradient descent with ADAM BID14 with a learning rate of 0.0001. In the case of models trained on 100 trajectories per expert we used minibatches of 512 subsequences of length 30.For LFPC we sampled 32 subsequences of length 30 and produced 5 perturbed state sequences per subsequence. In preliminary experiments the length of the subsequences did not have a major impact on model performance. Firstly, we note that the emphasis of the proposal in this work is to match the responsivity of the expert policy in a neighborhood around each state. This is distinct from activation matching or KL matching where the emphasis is on matching the action/activation distribution for a particular state BID33 BID40. Secondly, we emphasize that the kind of robust knowledge transfer we discuss here is distinct from that which is seen to be important in other settings. For example BID36 provide a line of reasoning that involves training a student system to match the exact activations of a teacher in the presence of perturbations on the student inputs. This logic is sound in the setting of large-scale vision systems. However in the context of control policies, this would look like: DISPLAYFORM0 This essentially means that the student policy is learning to "blindly" reproduce the action of the expert exactly, despite input perturbations. While this is well motivated if the noise is thought to be orthogonal to the proper functioning of the system, this is a very bad idea for control, where you need to pay close attention to small input perturbations. Technically, this amounts to setting the local feedback to zero, and behaving in a sort of open-loop-like fashion. Locomotion behavior is, at least in the simplest case roughly a limit cycle. In an additional experiment to test LFPC we gathered three gait cycles of running behavior and performed LFPC. Note that here the student policy need not be time-indexed even when the demonstrations were time-indexed. This restricted case shows striking generalization in the presence of noise (see The limited reference data originating from a time-indexed policy has been projected into the same space (green). Observe that the rollouts are considerably noisier and consistently deviate from the reference trajectory, nevertheless the cloned-policy trajectories return to the limit cycle.' == 'Neural Probabilistic Motor Primitives compress motion capture tracking policies into one flexible model capable of one-shot imitation and reuse as a low-level controller.' != 'We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.'

Question: The goal of standard compressive sensing is to estimate an unknown vector from linear measurements under the assumption of sparsity in some basis. Recently, it has been shown that significantly fewer measurements may be required if the sparsity assumption is replaced by the assumption that the unknown vector lies near the range of a suitably-chosen generative model. In particular, in (Bora {\em et al.}, 2017) it was shown that roughly $O(k\log L)$ random Gaussian measurements suffice for accurate recovery when the $k$-input generative model is bounded and $L$-Lipschitz, and that $O(kd \log w)$ measurements suffice for $k$-input ReLU networks with depth $d$ and width $w$. In this paper, we establish corresponding algorithm-independent lower bounds on the sample complexity using tools from minimax statistical analysis. In accordance with the above upper bounds, our are summarized as follows: (i) We construct an $L$-Lipschitz generative model capable of generating group-sparse signals, and show that the ing necessary number of measurements is $\Omega(k \log L)$; (ii) Using similar ideas, we construct two-layer ReLU networks of high width requiring $\Omega(k \log w)$ measurements, as well as lower-width deep ReLU networks requiring $\Omega(k d)$ measurements. As a , we establish that the scaling laws derived in (Bora {\em et al.}, 2017) are optimal or near-optimal in the absence of further assumptions. The problem of sparse estimation via linear measurements (commonly referred to as compressive sensing) is well-understood, with theoretical developments including sharp performance bounds for both practical algorithms and (potentially intractable) information-theoretically optimal algorithms. Following the tremendous success of deep generative models in a variety of applications, a new perspective on compressive sensing was recently introduced, in which the sparsity assumption is replaced by the assumption of the underlying signal being well-modeled by a generative model (typically corresponding to a deep neural network). This approach was seen to exhibit impressive performance in experiments, with reductions in the number of measurements by large factors such as 5 to 10 compared to sparsity-based methods. In addition, provided theoretical guarantees on their proposed algorithm, essentially showing that an L-Lipschitz generative model with bounded k-dimensional inputs leads to reliable recovery with m = O(k log L) random Gaussian measurements (see Section 2 for a precise statement). Moreover, for a ReLU network generative model from R k to R n with width w and depth d, it suffices to have m = O(kd log w). A variety of follow-up works provided additional theoretical guarantees (e.g., for more specific optimization algorithms, more general models, or under random neural network weights ) for compressive sensing with generative models, but the main of are by far the most relevant to ours. In this paper, we address a prominent gap in the existing literature by establishing algorithmindependent lower bounds on the number of measurements needed (e.g., this is explicitly posed as an open problem in ). Using tools from minimax statistical analysis, we show that for generative models satisfying the assumptions of, the above-mentioned dependencies m = O(k log L) and m = O(kd log w) cannot be improved (or in the latter case, cannot be improved by more than a log n factor) without further assumptions. Our argument is essentially based on a reduction to compressive sensing with a group sparsity model (e.g., see ), i.e., forming a neural network that is capable of producing such signals. The proofs are presented in the full paper. We begin by stating a simple corollary of a main of Bora et al.. As we show in, this is obtained by extending [10, Thm. 1.2] from spherical to rectangular domains, and then converting the high-probability bound to an average one. In, we also handle the case of spherical domains. for a universal constant C, we have for a universal constant C that In the following, we construct a Lipschitz-continuous generative model that can generate bounded kgroup-sparse vectors. Then, by making use of minimax statistical analysis for group-sparse recovery, we provide an information-theoretic lower bounds that matches the upper bound in Corollary 1. More precisely, we say that a signal in R n is k-group-sparse if, when divided into k blocks of size n k, each block contains at most one non-zero entry. We define Our construction of the generative function G: is given as follows:... figure shows the mapping from z1 → (x1, . . ., x n/k), and the same relation holds for z2 → (x n/k+1, . . ., x 2n/k), etc. up to z k → (x n−k+1, . . ., xn). • The output x ∈ R n is divided into k sub-sequences of length is only a function of the corresponding input z i, for i = 1,..., k. • The mapping from z i to x (i) is as shown in Figure 1. The interval [−r, r] is divided into n k intervals of length 2rk n, and the jth entry of x (i) can only be nonzero if z i takes a value in the j-th interval. Within that interval, the mapping takes a "doubletriangular" shape -the endpoints and midpoint are mapped to zero, the points It is easy to show that the generative model G: kr. In addition, using similar steps to the case of k-sparse recovery, we are able to obtain a minimax lower bound for k-group-sparse recovery, which holds when x max is not too small. Based on these , the following sample complexity lower bound is proved in. n (and associated output dimension n) such that, for any A ∈ R m×n satisfying A 2 F = C A n, any algorithm that produces somex satisfying sup with a sufficiently large implied constant, which is a very mild assumption since for fixed r and α, the right-hand side tends to zero as k grows large (whereas typical Lipschitz constants are at least equal to one, if not much higher). In this section, as opposed to considering general Lipschitz-continuous generative models, we focus on generative models given by neural networks with ReLU activations. Similar to the derivation of Corollary 1, we have the following corollary for ReLU-based networks from [10, Thm. 2 such that for a universal constant C, sup Note that this holds even when the domain D = R k, so we do not need to distinguish between the rectangular and spherical domains. Moreover, this makes no assumptions about the neural network weights (nor domain size), but rather, only the input size, width, and depth. Thus far, we have considered forming a generative model G: R k → R n capable of producing k-group-sparse signals, which leads to a lower bound of m = Ω(k log n). While this precise approach does not appear to be suited to properly understanding the dependence on width and depth in Corollary 2, we now show that a simple variant indeed suffices: We form a wide and/or deep ReLU network G: R k → R n capable of producing all (kk 0)-group-sparse signals having non-zero entries ±ξ, where k 0 is a certain positive integer that may be much larger than one. The idea of the construction is illustrated in Figure 2, which shows the mappings for k = 1 (the general case simply repeats this structure in parallel to get an output dimension n = n 0 k). Note also that we need to replace the rectangular shapes by trapeziums (with high-gradient diagonals) to make them implementable with a ReLU network. Again using the minimax lower bound for group-sparse recovery and a suitable choice of ξ, the following is proved in. Theorem 2. Fix C 1, C A > 0, and consider the problem of compressive sensing with generative models under i.i.d. N 0, α m noise, a measurement matrix A ∈ R m×n satisfying A 2 F = C A n, and the above-described generative model G: R k → R n with parameters k, k 0, n 0, and ξ. Then, if n 0 ≥ C 0 k 0 for an absolute constant C 0, then there exists a constant C 2 = Θ such that the choice ξ = C2α k yields the following: • Any algorithm producing somex satisfying sup x * ∈Range(G) E x − x * 2 2 ≤ C 1 α must also have m = Ω kk 0 log n kk0 (or equivalently m = Ω kk 0 log n0 k0, since n = n 0 k). • The generative function G can be implemented as a ReLU network with a single hidden layer (i.e., d = 2) of width at most w = O(k( n0 k0) k0 ). • Alternatively, if n0 k0 is an integer power of two, the generative function G can be implemented as a ReLU network with depth d = O k 0 log n0 k0 and width w = O(n). In the settings described in the second and third dot points, the sample complexity from Corollary 2 behaves as kd log w = O kk 0 log n0 k0 + k log k and kd log w = O kk 0 · log n0 k0 · log n respectively. While we do not claim a lower bound for every possible combination of depth and width, the final statement of Theorem 2 reveals that the upper and lower bounds match up to a constant factor (high-width case with log k ≤ O k 0 log n0 k0) or up to a log n factor (high-depth case). The proof of the claim for the high-width case is based on the fact that in Figure 2, upon replacing the rectangles by trapeziums, each mapping is piecewise linear, and at the -th scale the number of pieces is O n0 k0 −1, which we sum over = 1,..., k 0 to get the overall width. In the high-depth case, we exploit the periodic nature of the signals in Figure 2, and use the fact that depth-d neural networks can be used to produce periodic signals with O(2 d) repetitions. In our case, the maximum number of repetitions is O n0 k0 k0 (at the finest scale in Figure 2). We have established, to our knowledge, the first lower bounds on the sample complexity for compressive sensing with generative models. To achieve these, we constructed generative models capable of producing group-sparse signals, and then applied a minimax lower bound for group-sparse recovery. For bounded Lipschitz-continuous generative models we matched the O(k log L) scaling law derived in, and for ReLU-based generative models, we showed that the dependence of the O(kd log w) bound from has an optimal or near-optimal dependence on both the width and depth. A possible direction for future research is to understand what additional assumptions could be placed on the generative model to further reduce the sample complexity.

Represent the short summary for finding the corresponding scientific article (long)!

We establish that the scaling laws derived in (Bora et al., 2017) are optimal or near-optimal in the absence of further assumptions.

We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.


Question: Represent text

Question: Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether modern deep reinforcement learning can be used to train agents to perform causal reasoning. We adopt a meta-learning approach, where the agent learns a policy for conducting experiments via causal interventions, in order to support a subsequent task which rewards making accurate causal inferences. We also found the agent could make sophisticated counterfactual predictions, as well as learn to draw causal inferences from purely observational data. Though powerful formalisms for causal reasoning have been developed, applying them in real-world domains can be difficult because fitting to large amounts of high dimensional data often requires making idealized assumptions. Our suggest that causal reasoning in complex settings may benefit from powerful learning-based approaches. More generally, this work may offer new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform—and interpret—experiments. Many machine learning algorithms are rooted in discovering patterns of correlation in data. While this has been sufficient to excel in several areas BID20 BID7, sometimes the problems we are interested in are fundamentally causal. Answering questions such as "Does smoking cause cancer?" or "Was this person denied a job due to racial discrimination?" or "Did this marketing campaign cause sales to go up?" all require an ability to reason about causes and effects and cannot be achieved by purely associative inference. Even for problems that are not obviously causal, like image classification, it has been suggested that some failure modes emerge from lack of causal understanding. Causal reasoning may be an essential component of natural intelligence and is present in human babies, rats and even birds BID23 BID14 BID15 BID5 BID21. There is a rich literature on formal approaches for defining and performing causal reasoning BID29 BID33 BID8 BID30 ).Here we investigate whether procedures for learning and using causal structure can be produced by meta-learning. The approach of meta-learning is to learn the learning (or inference) procedure itself, directly from data. We adopt the specific method of BID9 and BID35, training a recurrent neural network (RNN) through model-free reinforcement learning. We train on a large family of tasks, each underpinned by a different causal structure. The use of meta-learning avoids the need to manually implement explicit causal reasoning methods in an algorithm, offers advantages of scalability by amortizing computations, and allows automatic incorporation of complex prior knowledge BID1 BID35 BID11. Additionally, by learning end-to-end, the algorithm has the potential to find the internal representations of causal structure best suited for the types of causal inference required. This work probed how an agent could learn to perform causal reasoning in three distinct settingsobservational, interventional, and counterfactual -corresponding to different types of data available to the agent during the first phase of an episode. In the observational setting (Experiment 1), the agent could only obtain passive observations from the environment. This type of data allows an agent to infer associations (associative reasoning) and, when the structure of the underlying causal model permits it, to estimate the effect that changing a variable in the environment has on another variable, namely to estimate causal effects (cause-effect reasoning).In the interventional setting (Experiment 2), the agent could directly set the values of some variables in the environment. This type of data in principle allows an agent to estimate causal effects for any underlying causal model. In the counterfactual setting (Experiment 3), the agent first had an opportunity to learn about the causal graph through interventions. At the last step of the episode, it was asked a counterfactual question of the form "What would have happened if a different intervention had been made in the previous time-step?".Next we will formalize these three settings and patterns of reasoning possible in each, using the graphical model framework BID29 BID33 BID8 1, and introduce the meta-learning methods that we will use to train agents that are capable of such reasoning. Causal relationships among random variables can be expressed using causal directed acyclic graphs (DAGs) (see Appendix). A causal DAG is a graphical model that captures both independence and causal relations. Each node X i corresponds to a random variable, and the joint distribution p(X 1, ..., X N) is given by the product of conditional distributions of each node X i given its parent nodes pa(DISPLAYFORM0 Edges carry causal semantics: if there exists a directed path from X i to X j, then X i is a potential cause of X j . Directed paths are also called causal paths. The causal effect of X i on X j is the conditional distribution of X j given X i restricted to only causal paths. DISPLAYFORM1 An example causal DAG G is given in the figure on the left, where E represents hours of exercise in a week, H cardiac health, and A age. The causal effect of E on H is the conditional distribution restricted to the path E → H, i.e. excluding the path E ← A → H. The variable A is called a confounder, as it confounds the causal effect with non-causal statistical influence. Simply observing cardiac health conditioning on exercise level from p(H|E) (associative reasoning) cannot answer if change in exercise levels cause changes in cardiac health (cause-effect reasoning), since there is always the possibility that correlation between the two is because of the common confounder of age. Cause-effect Reasoning. The causal effect can be seen as the conditional distribution p →E=e (H|E = e) 2 on the graph G →E=e above (right), ing from intervening on E by replacing p(E|A) with a delta distribution δ E=e (thereby removing the link from A to E) and leaving the remaining conditional distributions p(H|E,A) and p(A) unaltered. The rules of do-calculus BID29 BID30 tell us how to compute p →E=e (H|E = e) using observations from G. In this case p →E=e (H|E = e) = Counterfactual Reasoning. Cause-effect reasoning can be used to correctly answer predictive questions of the type "Does exercising improve cardiac health?" by accounting for causal structure and confounding. However, it cannot answer retrospective questions about what would have happened. For example, given an individual i who has died of a heart attack, this method would not be able to answer questions of the type "What would the cardiac health of this individual have been had they done more exercise?". This type of question requires estimating unobserved sources of noise and then reasoning about the effects of this noise under a graph conditioned on a different intervention. Meta-learning refers to a broad range of approaches in which aspects of the learning algorithm itself are learned from the data. Many individual components of deep learning algorithms have been successfully meta-learned, including the optimizer BID1, initial parameter settings BID11, a metric space BID34, and use of external memory BID31.Following the approach of BID9 BID35, we parameterize the entire learning algorithm as a recurrent neural network (RNN), and we train the weights of the RNN with model-free reinforcement learning (RL). The RNN is trained on a broad distribution of problems which each require learning. When trained in this way, the RNN is able to implement a learning algorithm capable of efficiently solving novel learning problems in or near the training distribution. Learning the weights of the RNN by model-free RL can be thought of as the "outer loop" of learning. The outer loop shapes the weights of the RNN into an "inner loop" learning algorithm. This inner loop algorithm plays out in the activation dynamics of the RNN and can continue learning even when the weights of the network are frozen. The inner loop algorithm can also have very different properties from the outer loop algorithm used to train it. For example, in previous work this approach was used to negotiate the exploration-exploitation tradeoff in multi-armed bandits BID9 and learn algorithms which dynamically adjust their own learning rates BID35. In the present work we explore the possibility of obtaining a causally-aware inner-loop learning algorithm. See the Appendix for a more formal approach to meta-learning. In the experiments, in each episode the agent interacted with a different causal DAG G. G was drawn randomly from the space of possible DAGs under the constraints given in the next paragraph. Each episode consisted of T steps, and was divided into two phases: information and quiz. The information phase, corresponding to the first T −1 steps, allowed the agent to collect information by interacting with or passively observing samples from G. The agent could potentially use this information to infer the connectivity and weights of G. The quiz phase, corresponding to the final step T, required the agent to exploit the causal knowledge it collected in the information phase, to select the node with the highest value under a random external intervention. Causal graphs, observations, and actions. We generated all graphs on N =5 nodes, with edges only in the upper triangular of the adjacency matrix (this guarantees that all the graphs obtained are DAGs), with edge weights, w ji ∈{−1,0,1} (uniformly sampled), and removed 300 for held-out testing. The remaining 58749 (or 3 N(N−1)/2 − 300) were used as the training set. Each node's value, X i ∈ R, was Gaussiandistributed. The values of parentless nodes were drawn from N (µ = 0.0,σ = 0.1). The conditional probability of a node with parents was p(X i |pa(X i)) = N (µ = j w ji X j,σ = 0.1), where pa(X i) represents the parents of node X i in G. The values of the 4 observable nodes (the root node, was always hidden), were concatenated to create v t and provided to the agent in its observation vector, o t =[v t,m t], where m t is a one-hot vector indicating external intervention during the quiz phase (explained below).Information phase. In the information phase, an information action, a t, caused an intervention on the a t -th node, setting its value to X at = 5. We choose an intervention value outside the likely range of sampled observations, to facilitate learning of the causal graph. The observation from the intervened graph, G →Xa t =5, was sampled similarly to G, except the incoming edges to X at were severed, and its intervened value was used for conditioning its children's values. The node values in G →Xa t =5 were distributed as p →Xi=5 (X 1:N\i |X i = 5). If a quiz action was chosen during the information phase, it was ignored, the G values were sampled as if no intervention had been made, and the agent was given a penalty of r t =−5 in order to encourage it to take quiz actions at only during quiz phase. After the action was selected, an observation was provided to the agent. The default length of this phase was fixed to T = N = 5 since in the noise-free limit, a minimum of T −1=4 interventions are required in general to resolve the causal structure, and score perfectly on the test phase. Quiz phase. In the quiz phase, one non-hidden node was selected at random to be intervened on externally, X j, and its value was set to −5. We chose an intervention value of −5 never previously observed by the agent in that episode, thus disallowing the agent from memorizing the of interventions in the information phase to perform well on the quiz phase. The agent was informed of this by the observed m T −1 (a one-hot vector which indicated which node would be intervened on), from the final pre-quiz phase time-step, T −1. Note, m t was set to a zero-vector for steps t < T −1. A quiz action, a T, chosen by the agent indicated the node whose value would be given to the agent as a reward. In other words, the agent would receive reward, r T =X a T −(N−1). Again, if a quiz action was chosen during the information phase, the node values were not sampled and the agent was simply given a penalty of r T =−5.Active vs passive agents. Our agents had to perform two distinct tasks during the information phase: a) actively choose which nodes to set values on, and b) infer the causal DAG from its observations. We refer to this setup as the "active" condition. To control for (a), we created the "passive" condition, where the agent's information phase actions are not learned. To provide a benchmark for how well the active agent can perform task (a), we fixed the passive agent's intervention policy to be an exhaustive sweep through all observable nodes. This is close to optimal for this domain -in fact it is the optimal policy for noise-free conditional node values. We also compared the active agent's performance to a baseline agent whose policy is to intervene randomly on the observable nodes in the information phase, in the Appendix. Two kinds of learning The "inner loop" of learning (see Section 2.2) occurs within each episode where the agent is learning from the evidence it gathers during the information phase in order to perform well in the quiz phase. The same agent then enters a new episode, where it has to repeat the task on a different DAG. Test performance is reported on DAGs that the agent has never previously seen, after all the weights of the RNN have been fixed. Hence, the only transfer from training to test (or the "outer loop" of learning) is the ability to discover causal dependencies based on observations in the information phase, and to perform causal inference in the quiz phase. We used a long short-term memory (LSTM) network BID18 ) (with 96 hidden units) that, at each time-step t, receives a concatenated vector containing [o t,a t−1,r t−1] as input, where o t is the observation 5, a t−1 is the previous action (as a one-hot vector) and r t−1 the reward (as a single real-value) 6. The outputs, calculated as linear projections of the LSTM's hidden state, are a set of policy logits (with dimensionality equal to the number of available actions), plus a scalar baseline. The policy logits are transformed by a softmax function, and then sampled to give a selected action. Learning was by asynchronous advantage actor-critic BID24. In this framework, the loss function consists of three terms -the policy gradient, the baseline cost and an entropy cost. The baseline cost was weighted by 0.05 relative to the policy gradient cost. The weighting of the entropy cost was annealed over the course of training from 0.05 to 0. Optimization was done by RMSProp with =10 −5, momentum = 0.9 and decay = 0.95. Learning rate was annealed from 3×10 −6 to 0. For all experiments, after training, the agent was tested with the learning rate set to zero, on a held-out test set. Our three experiments (observational, interventional, and counterfactual) differed in the properties of the v t that was observed by the agent during the information phase, and thereby limited the extent of causal reasoning possible within each data setting. Our measure of performance is the reward earned in the quiz phase for held-out DAGs. Choosing a random node node in the quiz phase in a reward of −5/4=−1.25, since one node (the externally intervened node) always has value −5 and the others have on average 0 value. By learning to simply avoid the externally intervened node, the agent can earn on average 0 reward. Consistently picking the node with the highest value in the quiz phase requires the agent to perform causal reasoning. For each agent, we take the average reward earned across 1200 episodes (300 held-out test DAGs, with 4 possible external interventions). We train 12 copies of each agent and report the average reward earned by these, with error bars showing 95% confidence intervals. In Experiment 1, the agent could neither intervene to set the value of variables in the environment, nor observe any external interventions. In other words, it only received observations from G, not G →Xj (where X j is a node that has been intervened on). This limits the extent of causal inference possible. In this experiment, we tested six agents, four of which were learned: "Observational", "Long Observational", "Active Conditional", "Passive Conditional", "Observational MAP Baseline"(not learned) and the "Optimal Associative Baseline" (not learned). We also ran two other standard RL baselines-see the Appendix for details. Observational Agents: In the information phase, the actions of the agent were ignored 7, and the observational agent always received the values of the observable nodes as sampled from the joint distribution associated with G. In addition to the default T =5 episode length, we also trained this agent with 4× longer episode length (Long Observational Agent), to measure performance increase with more observational data. Conditional Agents: The information phase actions corresponded to observing a world in which the selected node X j is equal to X j =5, and the remaining nodes are sampled from the conditional distribution p(X 1:N\j |X j =5), where X 1:N\j indicates the set of all nodes except X j. This differs from intervening on the variable X j by setting it to the value X j =5, since here we take a conditional sample from G rather than from G →Xj=5 (i.e. from p →Xj=5 (X 1:N\j |X j = 5)), and inference about the corresponding node's parents is possible. Therefore, this agent still has access to only observational data, as with the observational agents. However, on average it receives more diagnostic information about the relation between the random variables in G, since it can observe samples where a node takes a value far outside the likely range of sampled observations. We run active and passive versions of this agent as described in Section 3 Optimal Associative Baseline: This baseline receives the true joint distribution p(X 1:N) implied by the DAG in that episode, therefore it has full knowledge of the correlation structure of the environment 8. It can therefore do exact associative reasoning of the form p(X j |X i = x), but cannot do any cause-effect reasoning of the form p →Xi=x (X j |X i = x). In the quiz phase, this baseline chooses the node that has the maximum value according to the true p(X j |X i =x) in that episode, where X i is the node externally intervened upon, and x=−5.Observational MAP Baseline: This baseline follows the traditional method of separating causal induction and causal inference. We first carry out exact maximum a posteriori (MAP) inference over the space of DAGs in each episode (i.e. causal induction) by selecting the DAG (G MAP) of the 59049 unique possibilities that maximizes the likelihood of the data observed, v 1:T, by the Observational Agent in that episode. This is equivalent to maximizing the posterior probability since the prior over graphs is uniform. We focus on three key questions in this experiment: (i) Can our agents learn to do associative reasoning with observational data?, (ii) Can they learn to do cause-effect reasoning from observational data?, and (iii) In addition to making causal inferences, can our agent also choose good actions in the information phase to generate the data it observes? For (i), we see that the Observational Agents achieve reward above the random baseline (see the Appendix), and that more observations (Long Observational Agent) lead to better performance FIG0, indicating that the agent is indeed learning the statistical dependencies between the nodes. We see that the performance of the Passive-Conditional Agent is better than either of the Observational Agents, since the data it observes is very informative about the statistical dependencies in the environment. Finally, we see that the PassiveConditional Agent's performance is comparable (in fact surpasses as discussed below) the performance of the Optimal Associative Baseline, indicating that it is able to do perfect associative inference. For (ii), we see the crucial that the Passive-Conditional Agent's performance is significantly above the Optimal Associative Baseline, i.e. it performs better than what is possible using only correlations. We compare their performances, split by whether or the node that was intervened on in the quiz phase of the episode has a parent FIG0. If the intervened node X j has no parents, then G =G →Xj, and there is no advantage to being able to do cause-effect reasoning. We see indeed that the Passive-Conditional agent performs better than the Optimal Associative Baseline only when the intervened node has parents (denoted by hatched bars in FIG0), indicating that this agent is able to carry out some cause-effect reasoning, despite access to only observational data -i.e. it learns some form of do-calculus. We show the quiz phase for an example test DAG in FIG0, seeing that the Optimal Associative Baseline chooses according to the node values predicted by G whereas the Passive-Conditional Agent chooses according the node values predicted by G →Xj.For (iii), we see FIG0 that the Active-Conditional Agent's performance is only marginally below the performance of the Passive-Conditional Agent, indicating that when the agent is allowed to choose its actions, it makes reasonable choices that allow good performance. In Experiment 2, the agent receives interventional data in the information phase -it can choose to intervene on any observable node, X j, and observe a sample from the ing graph G →Xj. As discussed in Section 2.1, access to intervention data permits cause-effect reasoning even in the presence of unobserved confounders, a feat which is in general impossible with access only to observational data. In this experiment, we test four new agents, two of which were learned: "Active Interventional", "Passive Interventional", "Interventional MAP Baseline"(not learned), and "Optimal Cause-Effect Baseline" (not learned).Interventional Agents: The information phase actions correspond to performing an intervention on the selected node X j and sampling from G →Xj (see Section 3 for details). We run active and passive versions of this agent as described in Section 3. We see that the Passive-Int. Agent's choice is consistent with maximizing on these (correct) node value.each node according to G MAP →Xj where X j is the node externally intervened upon (i.e. causal inference), and choose the node with the highest value. Optimal Cause-Effect Baseline: This baseline receives the true DAG, G. In the quiz phase, it chooses the node that has the maximum value according to G →Xj, where X j is the node externally intervened upon. We focus on three key questions in this experiment: (i) Can our agents learn to do cause-effect reasoning from interventional data?, (ii) How does the cause-effect reasoning in our agents which have access to interventional data differ from the cause-effect reasoning measured in Experiment 1 (in agents that have access only to observational data)? (iii) In addition to making causal inferences, can our agent also choose good actions in the information phase to generate the data it observes?For (i) we see in FIG2 that the Passive-Interventional Agent's performance is comparable to the Optimal Cause-Effect Baseline, indicating that it is able to do close to perfect cause-effect reasoning in this domain. For (ii) we see in FIG2 the crucial that the Passive-Interventional Agent's performance is significantly better than the Passive-Conditional Agent. We compare the performances of these two agents, split by whether the node that was intervened on in the quiz phase of the episode had unobserved confounders with other variables in the graph FIG2. In confounded cases, as described in Section 2.1, cause-effect reasoning is impossible with only observational data. We see that the performance of the Passive-Interventional Agent does not vary significantly with confoundedness, whereas the performance of the Passive-Conditional Agent is significantly lower in the confounded cases. This indicates that the improvement in the performance of the agent that has access to interventional data (as compared to the agents that had access to only observational data) is largely driven by its ability to also do cause-effect reasoning in the presence of confounders. This is highlighted by FIG2, which shows the quiz phase for an example DAG, where the Passive-Conditional agent is unable to resolve the confounder, but the Passive-Interventional agent can. For (iii), we see in FIG3 that the Active-Interventional Agent's performance is only marginally below the performance of the near optimal Passive-Interventional Agent, indicating that when the agent is allowed to choose its actions, it makes reasonable choices that allow good performance. In Experiment 3, the agent was again allowed to make interventions as in Experiment 2, but in this case the quiz phase task entailed answering a counterfactual question. We explain here what a counterfactual question in this domain looks like. Consider the conditional distribution p(X i |pa(X i))=N (j w ji X j,0.1) as described in Section 3 as X i = j w ji X j + where is distributed as N (0.0,0.1), and represents the specific randomness introduced when taking one sample from the DAG. After observing the nodes FIG0 for a legend. Here, the left panel shows G →Xj=−5 and the nodes taking the mean values prescribed by p →Xj=−5 (X 1:N\j |X j =−5). We see that the Passive-Int. Agent's choice is consistent with maximizing on these node values, where it makes a random choice between two nodes with the same value. The right panel panel shows G →Xj=−5 and the nodes taking the exact values prescribed by the means of p →Xj=−5 (X 1:N\j |X j =−5), combined with the specific randomness inferred from the previous time step. As a of accounting for the randomness, the two previously degenerate maximum values are now distinct. We see that the Passive-CF. agent's choice is consistent with maximizing on these node values.in the DAG in one sample, we can infer this specific randomness i for each node X i (i.e. abduction as described in the Appendix) and answer counterfactual questions like "What would the values of the nodes be, had X j in that particular sample taken on a different value than what we observed?", for any of the nodes X j. We test 2 new learned agents: "Active Counterfactual" and "Passive Counterfactual".Counterfactual Agents: This agent is exactly analogous to the Interventional agent, with the addition that the exogenous noise in the last information phase step t=T −1 (where say X p =+5), is stored and the same noise is used in the quiz phase step t=T (where say X f =−5). While the question our agents have had to answer correctly so far in order to maximize their reward in the quiz phase was "Which of the nodes X 1:N\j will have the highest value when X f is set to −5? ", in this setting, we ask "Which of the nodes X 1:N\j would have had the highest value in the last step of the information phase, if instead of having X p =+5, we had X f =−5? ". We run active and passive versions of this agent as described in Section 3.Optimal Counterfactual Baseline: This baseline receives the true DAG and does exact abduction based on the exogenous noise observed in the penultimate step of the information phase, and combines this correctly with the appropriate interventional inference on the true DAG in the quiz phase. We focus on two key questions in this experiment: (i) Can our agents learn to do counterfactual reasoning?, (ii) In addition to making causal inferences, can our agent also choose good actions in the information phase to generate the data it observes?For (i), we see that the Passive-Counterfactual Agent achieves higher reward than the Passive-Interventional Agent and the Optimal Cause-Effect Baseline. To evaluate whether this difference from the agent's use of abduction (see the Appendix for details), we split the test set into two groups, depending on whether or not the decision for which node will have the highest value in the quiz phase is affected by exogenous noise, i.e. whether or not the node with the maximum value in the quiz phase changes if the noise is resampled. This is most prevalent in cases where the maximum expected reward is degenerate, i.e. where several nodes give the same maximum reward (denoted by hatched bars in FIG4). Here, agents with no access to the noise have no basis for choosing one over the other, but different noise samples can give rise to significant differences in the actual values that these degenerate nodes have. We see indeed that there is no difference in the rewards received by the Passive-Counterfactual and Passive-Interventional Agents in the cases where the maximum values are distinct, however the Passive-Counterfactual Agent significantly outperforms the Passive-Interventional Agent in cases where there are degenerate maximum values. For (ii), we see in FIG5 that the Active-Counterfactual Agent's performance is only marginally below the performance of the Passive-Counterfactual agent, indicating that when the agent is allowed to choose its actions, it makes reasonable choices that allow good performance. We introduced and tested a framework for learning causal reasoning in various data settings-observational, interventional, and counterfactual-using deep meta-RL. Crucially, our approach did not require explicit encoding of formal principles of causal inference. Rather, by optimizing an agent to perform a task that depended on causal structure, the agent learned implicit strategies to use the available data for causal reasoning, including drawing inferences from passive observation, actively intervening, and making counterfactual predictions. Below, we summarize the keys from each of the three experiments. In Section 4.1 and FIG0 show that the agent learns to perform do-calculus. In FIG0 we see that, compared to the highest possible reward achievable without causal knowledge, the trained agent received more reward. This observation is corroborated by FIG0 which shows that performance increased selectively in cases where do-calculus made a prediction distinguishable from the predictions based on correlations. These are situations where the externally intervened node had a parent -meaning that the intervention ed in a different graph. In Section 4.2 and FIG2, we show that the agent learns to resolve unobserved confounders using interventions (a feat impossible with only observational data). In FIG2 we see that the agent with access to interventional data performs better than an agent with access to only observational data. FIG2 shows that the performance increase is greater in cases where the intervened node shared an unobserved parent (a confounder) with other variables in the graph. In this section we also compare the agent's performance to a MAP estimate of the causal structure and find that the agent's performance matches it, indicating that the agent is indeed doing close to optimal causal inference. In Section 4.3 and FIG4, we show that the agent learns to use counterfactuals. In FIG4 we see that the agent with additional access to the specific randomness in the test phase performs better than an agent with access to only interventional data. In FIG4, we find that the increased performance is observed only in cases where the maximum mean value in the graph is degenerate, and optimal choice is affected by the exogenous noise -i.e. where multiple nodes have the same value on average and the specific randomness can be used to distinguish their actual values in that specific case. This work is the first demonstration that causal reasoning can arise out of model-free reinforcement learning. This opens up the possibility of leveraging powerful learning-based methods for causal inference in complex settings. Traditional formal approaches usually decouple the two problems of causal induction (i.e. inferring the structure of the underlying model) and causal inference (i.e. estimating causal effects and answering counterfactual questions), and despite advances in both BID26 BID6 BID27 BID32 BID12 BID22, inducing models often requires assumptions that are difficult to fit to complex real-world conditions. By learning these end-to-end, our method can potentially find representations of causal structure best tuned to the specific causal inferences required. Another key advantage of our meta-RL approach is that it allows the agent to learn to interact with the environment in order to acquire necessary observations in the service of its task-i.e. to perform active learning. In our experimental domain, our agents' active intervention policy was close to optimal, which demonstrates the promise of agents that can learn to experiment on their environment and perform rich causal reasoning on the observations. Future work should explore agents that perform experiments to support structured exploration in RL, and optimal experiment design in complex domains where large numbers of blind interventions are prohibitive. To this end, follow-up work should focus on scaling up our approach to larger environments, with more complex causal structure and a more diverse range of tasks. Though the here are a first step in this direction which use relatively standard deep RL components, our approach will likely benefit from more advanced architectures (e.g. BID16 BID17 that allow longer more complex episodes, as well as models which are more explicitly compositional (e.g. BID3 BID0 or have richer semantics (e.g. BID13, that more explicitly leverage symmetries like equivalance classes in the environment. We can also compare the performance of these agents to two standard model-free RL baselines. The Q-total agent learns a Q-value for each action across all steps for all the episodes. The Q-episode agent learns a Q-value for each action conditioned on the input at each time step [o t,a t−1,r t−1], but with no LSTM memory to store previous actions and observations. Since the relationship between action and reward is random between episodes, Q-total was equivalent to selecting actions randomly, ing in a considerably negative reward. The Q-episode agent essentially makes sure to not choose the arm that is indicated by m t to be the external intervention (which is assured to be equal to −5), and essentially chooses randomly otherwise, giving an average reward of 0. Consider a distribution D over Markov Decision Processes (MDPs). We train an agent with memory (in our case an RNN-based agent) on this distribution. In each episode, we sample a task m ∼ D. At each step t within an episode, the agent sees an observation o t, executes an action a t, and receives a reward r t. Both a t−1 and r t−1 are given as additional inputs to the network. Thus, via the recurrence of the network, each action is a function of the entire trajectory H t = {o 0,a 0,r 0,...,o t−1,a t−1,r t−1,o t} of the episode. Because this function is parameterized by the neural network, its complexity is limited only by the size of the network. BID30's "abduction-action-prediction" method prescribes one method for answering counterfactual queries, by estimating the specific unobserved makeup of individual i and by transferring it to the counterfactual world. Assume, for example, the following model for G of Section 2.1: E = w AE A+η, H =w AH A+w EH E+, where the weights w ij represent the known causal effects in G and and η are terms of (e.g.) Gaussian noise that represent the unobserved randomness in the makeup of each individual 9. Suppose that for individual i we observe: A = a i, E = e i, H = h i. We can answer the counterfactual question of "What if individual i had done more exercise, i.e. E =e, instead?" by: a) Abduction: estimate the individual's specific makeup with i =h i −w AH a i −w EH e i, b) Action: set E to more exercise e, c) Prediction: predict a new value for cardiac health as h =w AH a i +w EH e + i. Observational. The purview of the previous experiments was to show a proof of concept on a simple tractable system, demonstrating that causal induction and inference can be learned and implemented via a meta-learned agent. In this experiment, we generalize some of the to nonlinear, non-Gaussian causal graphs which are more typical of real-world causal graphs and to demonstrate that our hold without loss of generality on such systems. Here we investigate causal DAGs with a quadratic dependence on the parents by changing the conditional distribution to p(X i |pa( DISPLAYFORM0 Here, although each node is normally distributed given its parents, the joint distribution is not multivariate Gaussian due to the non-linearity in how the means are determined. We find that the Long-Observational achieves more reward than the Observational agent indicating that the agent is in fact learning the statistical dependencies between the nodes, within an episode. We also find that although the Active-Interventional agent is not far behind the performance of the MAP baseline, and achieves reward well above the Long-Observational 10 The fact that the MAP baseline gets so close to the Optimal Cause-Effect baseline indicates that the Active agent is choosing close to optimal actions. In the experiments reported in the main paper, the test set was a random subset of all graphs, and training examples were generated randomly subject to the constraint that they not be in the test set. However, this raised the possibility that any test graph might have an equivalent graph in the training set, which could in a type of overfitting. We therefore ran a new set of experiments where the entire equivalence class of each test graph was held out from the training set 11 . Performance on the test set therefore indicates generalization of the inference procedures learned to previously unseen equivalence classes of causal DAGs. For these experiments, we used graphs with N =6 nodes, because 5-node graphs have too few equivalence classes to partition in this way. All other details were the same as in the main paper. We see in FIG9 that the agents learn to generalize well to these held out examples, and we find the same pattern of behavior noted in the main text where the rewards earned are ordered such that Observational agent < Passive-Conditional agent < Passive-Interventional agent < Passive-Counterfactual agent. We see additionally in FIG9 that the Active-Interventional agent performs at par with the Passive-Interventional agent (which is allowed to see the of interventions on all nodes) and significantly better than an additional baseline we use here of the Random-Interventional agent whose information phase policy is to intervene on nodes at random, indicating that the intervention policy learned by the Active agent is good. Graphical models BID28 BID4 BID19 BID2 BID25 are a marriage between graph and probability theory that allows to graphically represent and assess statistical dependence. In the following sections, we give some basic definitions and describe a method (d-separation) for graphically assessing statistical independence in belief networks. Figure 10: (a): Directed acyclic graph. The node X 3 is a collider on the path X 1 →X 3 ←X 2 and a non-collider on the path X 2 →X 3 →X 4. (b): Cyclic graph obtained from (a) by adding a link from X 4 to X 1.A graph is a collection of nodes and links connecting pairs of nodes. The links may be directed or undirected, giving rise to directed or undirected graphs respectively. A path from node X i to node X j is a sequence of linked nodes starting at X i and ending at X j. A directed path is a path whose links are directed and pointing from preceding towards following nodes in the sequence. 10 The conditional distribution p(X 1:N\j |Xj =5), and therefore Conditional agents, were non-trivial to calculate for the quadratic case.11 The hidden node was guaranteed to be a root node by rejecting all DAGs where the hidden node has parents A directed acyclic graph (DAG) is a directed graph with no directed paths starting and ending at the same node. For example, the directed graph in FIG1 is acyclic. The addition of a link from X 4 to X 1 gives rise to a cyclic graph (FIG1).A node X i with a directed link to X j is called parent of X j. In this case, X j is called child of X i.A node is a collider on a specified path if it has (at least) two parents on that path. Notice that a node can be a collider on a path and a non-collider on another path. For example, in FIG1 (a) X 3 is a collider on the path X 1 →X 3 ←X 2 and a non-collider on the path X 2 →X 3 →X 4.A node X i is an ancestor of a node X j if there exists a directed path from X i to X j. In this case, X j is a descendant of X i.A graphical model is a graph in which nodes represent random variables and links express statistical relationships between the variables. A belief network is a directed acyclic graphical model in which each node X i is associated with the conditional distribution p(X i |pa(X i)), where pa(X i) indicates the parents of X i. The joint distribution of all nodes in the graph, p(X 1:N), is given by the product of all conditional distributions, i.e. DISPLAYFORM0 p(X i |pa(X i)). Given the sets of random variables X,Y and Z, X and Y are statistically independent given Z (X ⊥ ⊥Y|Z) if all paths from any element of X to any element of Y are closed (or blocked). A path is closed if at least one of the following conditions is satisfied:(Ia) There is a non-collider on the path which belongs to the conditioning set Z.(Ib) There is a collider on the path such that neither the collider nor any of its descendants belong to the conditioning set Z.

Represent the TLDR for finding the corresponding scientific article

meta-learn a learning algorithm capable of causal reasoning

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.