Question: Represent the text to find a very short summary (1-2 sentences max).

Question: We aim to build complex humanoid agents that integrate perception, motor control, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies. The ing system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. Supplementary video link: https://youtu.be/fBoir7PNxPk In reinforcement learning (RL), a major challenge is to simultaneously cope with high-dimensional input and high-dimensional action spaces. As techniques have matured, it is now possible to train high-dimensional vision-based policies from scratch to generate a range of interesting behaviors ranging from game-playing to navigation BID17 BID32 BID41. Likewise, for controlling bodies with a large number of degrees of freedom (DoFs), in simulation, reinforcement learning methods are beginning to surpass optimal control techniques. Here, we try to synthesize this progress and tackle high-dimensional input and output at the same time. We evaluate the feasibility of full-body visuomotor control by comparing several strategies for humanoid control from vision. Both to simplify the engineering of a visuomotor system and to reduce the complexity of taskdirected exploration, we construct modular agents in which a high-level system possessing egocentric vision and memory is coupled to a low-level, reactive motor control system. We build on recent advances in imitation learning to make flexible low-level motor controllers for high-DoF humanoids. The motor skills embodied by the low-level controllers are coordinated and sequenced by the high-level system, which is trained to maximize sparse task reward. Our approach is inspired by themes from neuroscience as well as ideas developed and made concrete algorithmically in the animation and robotics literatures. In motor neuroscience, studies of spinal reflexes in animals ranging from frogs to cats have led to the view that locomotion and reaching are highly prestructured, enabling subcortical structures such as the basal ganglia to coordinate a motor repertoire; and cortical systems with access to visual input can send low complexity signals to motor systems in order to evoke elaborate movements BID7 BID1 BID9.The study of "movement primitives" for robotics descends from the work of BID16. Subsequent research has focused on innovations for learning or constructing primitives for control of movments BID15 BID20 ), deploying and sequencing them to solve tasks BID36 BID19 BID22, and increasing the complexity of the control inputs to the primitives BID31. Particularly relevant to our cause is the work of BID21 in which primitives were coupled by reinforcement learning to external perceptual inputs. Research in the animation literature has also sought to produce physically simulated characters capable of distinct movements that can be flexibly sequenced. This ambition can be traced to the virtual stuntman BID6 a) and has been advanced markedly in the work of Liu BID27. Further recent work has relied on reinforcement learning to schedule control policies known as "control fragments", each one able to carry out only a specialized short movement segment BID24. In work to date, such control fragments have yet to be coupled to visual input as we will pursue here. From the perspective of the RL literature BID38, motor primitives and control fragments may be considered specialized instantiations of "option" sub-policies. Our work aims to contribute to this multi-disciplinary literature by demonstrating concretely how control-fragment-like low-level movements can be coupled to and controlled by a vision and memory-based high-level controller to solve tasks. Furthermore, we demonstrate the scalability of the approach to greater number of control fragments than previous works. Taken together, we demonstrate progress towards the goal of integrated agents with vision, memory, and motor control. We present a system capable of solving tasks from vision by switching among low-level motor controllers for the humanoid body. This scheme involves a general separation of control where a low-level controller handles motor coordination and a high-level controller signals/selects lowlevel behavior based on task context (see also BID12 BID33 . In the present work, the low-level motor controllers operate using proprioceptive observations, and the high-level controller operate using proprioception along with first-person/egocentric vision. We first describe the procedure for creating low-level controllers from motion capture data, then describe and contrast multiple approaches for interfacing the high-and low-level controllers. For simulated character control, there has been a line of research extracting humanoid behavior from motion capture ("mocap") data. The SAMCON algorithm is a forward sampling approach that converts a possibly noisy, kinematic pose sequence into a physical trajectory. It relies on a beamsearch-like planning algorithm BID26 that infers an action sequence corresponding to the pose sequence. In subsequent work, these behaviors have been adapted into policies BID27 BID2. More recently, RL has also been used to produce time-indexed policies which serve as robust tracking controllers BID34. While the ing time-indexed policies are somewhat less general as a , time-indexing or phase-variables are common in the animation literature and also employed in kinematic control of characters BID14. We likewise use mocap trajectories as reference data, from which we derive policies that are single purpose -that is, each policy robustly tracks a short motion capture reference motion (2-6 sec), but that is all each policy is capable of. Humanoid body We use a 56 degree-of-freedom (DoF) humanoid body that was developed in previous work, a version of which is available with motion-capture playback in the DeepMind control suite BID39. Here, we actuate the joints with position-control: each joint is given an actuation range in [−1, 1], and this is mapped to the angular range of that joint. Single-clip tracking policies For each clip, we train a policy π θ (a|s, t) with parameters θ such that it maximizes a discounted sum of rewards, r t, where the reward at each step comes from a custom scoring function (see eqns. 1, 2 defined immediately below). This tracking approach most closely follows BID34. Note that here the state optionally includes a normalized time t that goes from 0 at the beginning of the clip to 1 at the end of the clip. For cyclical behaviors like locomotion, a gait cycle can be isolated manually and kinematically blended circularly by weighted Illustration of tracking-based RL training. Training iteratively refines a policy to robustly track the reference trajectory as well as physically feasible.linear interpolation of the poses to produce a repeating walk. The time input is reset each gaitcycle (i.e. it follows a sawtooth function). As proposed in BID34, episodes are initialized along the motion capture trajectory; and episodes can be terminated when it is determined that the behavior has failed significantly or irrecoverably. Our specific termination condition triggers if parts of the body other than hands or feet make contact with the ground. See FIG0 for a schematic. We first define an energy function most similar to SAMCON's BID26: DISPLAYFORM0 where E qpos is a energy defined on all joint angles, E qvel on joint velocities, E ori on the body root (global-space) quaternion, E ee on egocentric vectors between the root and the end-effectors (see), E vel on the (global-space) translational velocities, and E gyro on the body root rotational velocities. More specifically: DISPLAYFORM1 where q represents the pose and q represents the reference pose. In this work, we used coefficients w qpos = 5, w qvel = 1, w ori = 20, w gyro = 1, w vel = 1, w ee = 2. We tuned these by sweeping over parameters in a custom implementation of SAMCON (not detailed here), and we have found these coefficients tend to work fairly well across a wide range of movements for this body. From the energy, we write the reward function: DISPLAYFORM2 where w total is the sum of the per energy-term weights and β is a sharpness parameter (β = 10 throughout). Since all terms in the energy are non-negative, the reward is normalized r t ∈ with perfect tracking giving a reward of 1 and large deviations tending toward 0.Acquiring reference data features for some quantities required setting the body to the pose specified by the joint angles: e.g., setting x pos, q pos, and q ori to compute the end-effector vectors q ee. Joint angle velocities, root rotational velocities, and translational velocities (q vel, q gyro, x vel) were derived from the motion capture data by finite difference calculations on the corresponding positions. Note that the reward function here was not restricted to egocentric features -indeed, the velocity and quaternion were non-egocentric. Importantly, however, the policy received exclusively egocentric observations, so that, for example, rotating the initial pose of the humanoid would not affect the policy's ability to execute the behavior. The full set of proprioceptive features we provided the policy consists of joint angles (q pos) and velocities (q vel), root-to-end-effector vectors (q ee), rootframe velocimeter (q veloc), rotational velocity (q gyro), root-frame accelerometers (q accel), and 3D orientation relative to the z-axis (r z : functionally a gravity sensor).Low-level controller reinforcement learning details Because the body is position-controlled, (a t has the same dimension and semantics as a subset of the body pose), we can pre-train the policy to produce target poses by supervised learning max θ t log π(q * pos,t+1 |s * t, t). This produces very poor control but facilitates the subsequent stage of RL-based imitation learning. We generally found that training with some pretraining considerably shortened the time the training took to converge and improved the ing policies. For RL, we performed off-policy training using a distributed actor-critic implementation, closest to that used in BID10. This implementation used a replay buffer and target networks as done in previous work. The Q-function was learned off-policy using TD-learning using importance-weighted Retrace BID30, and the actor was learned off-policy using SVG. This is to say that we learned the policy by taking gradients with respect to the Q function (target networks were updated every 500 learning steps). Gradient updates to the policy were performed using short time windows, {s τ, a τ} τ =1...T, sampled from replay: DISPLAYFORM3 where η was fixed in our experiments. While the general details of the RL algorithm are not pertinent to the success of this approach (e.g. Peng et al. FORMULA0 used on-policy RL), we found two details to be critical, and both were consistent with the reported in BID34. Policy updates needed to be performed conservatively with the update including a term which restricts BID35. Secondly, we found that attempting to learn the variance of the policy actions tended to in premature convergence, so best were obtained using a stochastic policy with fixed noise (we used noise with σ = .1). DISPLAYFORM4 We next consider how to design low-level motor controllers derived from motion capture trajectories. Broadly, existing approaches fall into two categories: structured and cold-switching controllers. In structured controllers, there is a hand-designed relationship between "skill-selection" variables and the generated behavior. Recent work by BID34 explored specific handdesigned, structured controllers. While parameterized skill-selection coupled with manual curation and preprocessing of motion capture data can produce artistically satisfying , the range of behavior has been limited and implementation requires considerable expertise and animation skill. By contrast, an approach in which behaviors are combined by a more automatic procedure promises to ultimately scale to a wider range of behaviors. Below, we describe some specific choices for both structured and cold-switching controllers. For structured control schemes, we consider: a steerable controller that produces running behavior with a controllable turning radius, and a switching controller that is a single policy that can switch between the behaviors learned from multiple mocap clips, with switch points allowed at the end of gait cycles. The allowed transitions were defined by a transition graph. For cold switching, we will not explicitly train transitions between behaviors. Steerable controller Following up on the ability to track a single cyclical behavior like locomotion described above, we can introduce the ability to parametrically turn. To do this we distorted the reference trajectory accordingly and trained the policy to track the reference with the turning radius as additional input. Each gait cycle we picked a random turning radius parameter and in that gaitcyle we rotate the reference clip heading (q ori) at that constant rate (with appropriate bookkeeping for other positions and velocities). The was a policy that, using only one gait cycle clip as input, could turn with a specified rate of turning. Switching controller An alternative to a single behavior with a single continuously controllable parameter is a single policy that is capable of switching among a discrete set of behaviors based on a 1-of-k input. Training consisted of randomly starting in a pose sampled from a random mocap clip and transitioning among clips according to a graph of permitted transitions. Given a small, discrete set of clips that were manually "cut" to begin and end at similar points in a gait cycle, we initialized a discrete Markov process among clips with some initial distribution over clips and transitioned between clips that were compatible (walk forward to turn left, etc.) FIG1 ). Cold-switching of behaviors and control fragments We can also leave the task of sequencing behaviors to the high-level controller, instead of building structured low-level policies with explicit, designed transitions. Here, we did not attempt to combine the clips into a single policy; instead, we cut behaviors into short micro-behaviors of roughly 0.1 to 0.3 seconds, which we refer to as control fragments BID24. Compared to switching using the complete behaviors, the micro-behaviors, or control fragments, allow for better transitions and more flexible locomotion. Additionally, we can easily scale to many clips without manual intervention. For example, clip 1 would generate a list of fragments: DISPLAYFORM0 10 (a|s t, τ). When fragment 1 was chosen, τ the time-indexing variable was set to τ = 0 initially and ticked until, say, τ = 0.1. Choosing fragment 2, π 1 2, would likewise send a signal to the clip 1 policy starting from τ = 0.1, etc. Whereas we have to specify a small set of consistent behaviors for the other lowlevel controller models, we could easily construct hundreds (or possibly more) control fragments cheaply and without significant curatorial attention. Since the control fragments were not trained with switching behavior, we refer to the random access switching among fragments by the highlevel controller as "cold-switching" (Fig. 3).Figure 3: Cold-switching among a set of behaviors (A) only at end of clips to form a trajectory composed of sequentially activation of the policies (B). Alternatively, policies are fragmented at a pre-specified set of times, cutting the policy into sub-policies (C), which serve as control fragments, enabling sequencing at a higher frequency (D). We integrated the low-level controllers into an agent architecture with vision and and an LSTM memory in order to apply it to tasks including directed movements to target locations, a running course with wall or gap obstacles, a foraging task for "balls", and a simple memory task involving detecting and memorizing the reward value of the balls. The interface between the high-level controller and the low-level depends on the type of low-level controller: for the steerable controller, the high-level produces a one-dimensional output; for the switching and control fragment controllers, the high-level produces a 1-of-K index to select the lowlevel policies. The high-level policies are trained off-policy using data from a replay buffer. The replay buffer contains data generated from distributed actors, and in general the learner processes the same replay data multiple times. The high-level controller senses inputs from proprioceptive data and, for visual tasks, an egocentric camera mounted at the root of the body FIG2. A noteworthy challenge arises due to the movement of the camera itself during locomotion. The proprioceptive inputs are encoded by a single linear layer, and the image is encoded by a ResNet (see Appendix A). The separate inputs streams are then flattened, concatenated, and passed to an LSTM, enabling temporally integrated decisions, with a stochastic policy and a value function head. The high-level controller receives inputs at each time step even though it may only act when the previous behavior (gait cycle or control fragment) has terminated. Importantly, while the low-level skills used exclusively egocentric proprioceptive input, the highlevel controller used vision to select from or modulate them, enabling the system as a whole to effect visuomotor computations. High-level controller reinforcement learning details For the steerable controller, the policy was a parameterized Gaussian distribution that produces the steering angle a s ∈ [−1.5, 1.5]. The mean of Gaussian was constrained via a tanh and sampled actions were clipped to the steering angle range. The steering angle was held constant for a full gait cycle. The policy was trained as previously described by learning a state-action value function off-policy using TD-learning with Retrace BID30 with the policy trained using SVG.For the switching controller and the discrete control fragments approach, the policy was a multinomial over the discrete set of behaviors. In either case, the high-level controller would trigger the behavior for its period T (a gait cycle or a fragment length). To train these discrete controllers, we fit the state-value baseline V -function using V-Trace and update the policy according to the method in BID4. While we provided a target for the value function loss at each time step, Figure 5: A. Go-to-target: in this task, the agent moves on an open plane to a target provided in egocentric coordinates. B. Walls: The agent runs forward while avoiding solid walls using vision. C. Gaps: The agent runs forward and must jump between platforms to advance. D. Forage: Using vision, the agent roams in a procedurally-generated maze to collect balls, which provide sparse rewards. E. Heterogeneous Forage: The agent must probe and remember rewards that are randomly assigned to the balls in each episode.the policy gradient loss for the high-level was non-zero only when a new action was sampled (every T steps).Query-based control fragment selection We considered an alternative family of ideas to interface with control fragments based on producing a Gaussian policy search query to be compared against a feature-key for each control fragment. We then selected the control fragment whose key was nearest the query-action. Our method was based on the Wolpertinger approach introduced in BID3. Here, the Q-function was evaluated for each of k nearest neighbors to the query-action, and the control fragment were selected with Boltzmann exploration, i.e. p(a DISPLAYFORM0, where h is the output of the LSTM. See Appendix A.3.3 for more details. The intuition was that this would allow the high-level policy to be less precise as the Qfunction could assist it in selecting good actions. However, this approach under-performed relative to discrete action selection as we show in our . We compared the various approaches on a variety of tasks implemented in MuJoCo BID40 . The core tasks we considered for the main comparisons were Go-to-target, wall navigation (Walls), running on gapped platforms (Gaps), foraging for colored ball rewards (Forage), and a foraging task requiring the agent to remember the reward value of the different colored balls (Heterogeneous Forage) (see Fig. 5). In Go-to-target, the agent received a sparse reward of 1 for each time step it was within a proximity radius of the target. For Walls and Gaps, adapted from to operate from vision, the agent received a reward proportional to its forward velocity. Forage was broadly similar to explore object locations in the DeepMind Lab task suite BID0 (with a humanoid body) while Heterogeneous Forage was a simplified version of explore object rewards. In all tasks, the body was initialized to a random pose from a subset of the reference motion capture data. For all tasks, other than Go-to-target, the high-level agent received a 64x64 image from the camera attached to the root of the body, in addition to the proprioceptive information. We compared the agents on our core set of tasks. Our overall best were achieved using control fragments with discrete selection (Fig. 6). Additional training details are provided in Appendix A. For comparison, we also include the control experiment of training a policy to control the humanoid from scratch (without low-level controllers) as well as training a simple rolling ball body. The Figure 6: Performance of various approaches on each core task. Of the approaches we compared, discrete switching among control fragments performed the best. Plots show the mean and standard error over multiple runs.performance of the rolling ball is not directly comparable because its velocity differs from that of the humanoid, but isolates the task complexity from the challenge of motor control of the humanoid body. The switching controllers selected between a base set of four policies: stand, run, left and right turn. For the control fragments approach we were able to augment this set as described in TAB2.The end-to-end approach (described in Appendix A.4) succeeded at only Go-to-target, however the ing visual appearance was jarring. In the more complex Forage task, the end-to-end approach failed entirely. The steering controller was also able to perform the Go-to-target task, but a fixed turning radius meant that it was unable to make a direct approach the target, ing in a long travel time to the target and lower score. Both the steering controller and switching controller were able to reach the end of the course in the Walls task, but only the control fragments approach allowed for sharper turns and quicker adjustments for agent to achieve a higher velocity. Generally, the switching controller with transitions started to learn faster and appeared the most graceful because of its predefined, smooth transitions, but its comparative lack of flexibility meant that its asymptotic task performance was relatively low. In the Forage task, where a score of > 150 means the agent is able to move around the maze and 600 is maximum collection of reward, the switching controller with transitions was able to traverse the maze but unable to adjust to the layout of the maze to make sharper turns to collect all objects. The control fragments approach was able to construct rotations and abrupt turns to collect the objects in each room. In the Gaps task, we were able to use the control fragments approach with 12 single-clip policies, where it would be laborious to pretrain transitions for each of these. In this task, the high-level controller selected between the 4 original stand, run and turn policies as well as 8 additional jumps, ing in 359 fragments, and was able to synthesize them to move forward along the separated platforms. In the final Heterogeneous Forage task, we confirmed that the agent, equipped with an LSTM in the high-level controller, was capable of memory-dependent control behavior. See our Extended Video 2 for a comprehensive presentation of the controllers. All control fragment comparisons above used control fragments of 3 time steps (0.09s). To further understand the performance of the control fragment approach, we did a more exhaustive comparison of performance on Go-to-target of the effect of fragment length, number of fragments, as well as Figure 7: Example agent-view frames and corresponding visuomotor salience visualizations. Note that the ball is more sharply emphasized, suggesting the selected actions were influenced by the affordance of tacking toward the ball.introduction of redundant clips (see appendix B). We saw benefits in early exploration due to using fragments for more than one time step but lower ultimate performance. Adding more fragments was helpful when those fragments were functionally similar to the standard set and the high-level controller was able to robustly handle those that involved extraneous movements unrelated to locomotion. While the query-based approaches did not outperform the discrete control fragment selection (Fig. 6), we include a representative visualization in Appendix A.3 to help clarify why this approach may not have worked well. In the present setting, it appears that the proposal distribution over queries generated by the high-level policy was high variance and did not learn to index the fragments precisely. On Forage, the high-level controller with discrete selection of control fragments generated structured transitions between fragments (Appendix C). Largely, movements remained within clip or behavior type. The high-level controller ignored some fragments involving transitions from standing to running and left-right turns to use fast-walk-and-turn movements. To assess the visual features that drove movements, we computed saliency maps BID37 showing the intensity of the gradient of the selected action's log-probability with respect to each pixel: DISPLAYFORM0 c |∇ Ix,y,c log π(a HL t |h t)|) with normalization Z and clipping g (Fig. 7). Consistently, action selection was sensitive to the borders of the balls as well as to the walls. The visual features that this analysis identifies correspond roughly to sensorimotor affordances BID8; the agent's perceptual representations were shaped by goals and action. In this work we explored the problem of learning to reuse motor skills to solve whole body humanoid tasks from egocentric camera observations. We compared a range of approaches for reusing lowlevel motor skills that were obtained from motion capture data, including variations related to those presented in BID24 BID34. To date, there is limited learning-based work on humanoids in simulation reusing motor skills to solve new tasks, and much of what does exist is in the animation literature. A technical contribution of the present work was to move past hand-designed observation features (as used in BID34) towards a more ecological observation setting: using a front-facing camera is more similar to the kinds of observations a real-world, embodied agent would have. We also show that hierarchical motor skill reuse allowed us to solve tasks that we could not with a flat policy. For the walls and go-to-target tasks, learning from scratch was slower and produced less robust behavior. For the forage tasks, learning from scratch failed completely. Finally, the heterogeneous forage is an example of task that integrates memory and perception. There are some other very clear continuities between what we present here and previous work. For learning low-level tracking policies from motion capture data, we employed a manually specified similarity measure against motion capture reference trajectories, consistent with previous work BID26 BID34. Additionally, the low-level policies were time-indexed: they operated over only a certain temporal duration and received time or phase as input. Considerably less research has focused on learning imitation policies either without a pre-specified scoring function or without time-indexing (but see e.g.). Compared to previous work using control fragments BID24, our low-level controllers were built without a sampling-based planner and were parameterized as neural networks rather than linear-feedback policies. We also want to make clear that the graph-transition and steerable structured low-level control approaches require significant manual curation and design: motion capture clips must be segmented by hand, possibly manipulated by blending/smoothing clips from the end of one clip to the beginning of another. This labor intensive process requires considerable skill as an animator; in some sense this almost treats humanoid control as a computer-aided animation problem, whereas we aim to treat humanoid motor control as an automated and data-driven machine learning problem. We acknowledge that relative to previous work aimed at graphics and animation, our controllers are less graceful. Each approach involving motion capture data can suffer from distinct artifacts, especially without detailed manual editing -the hand-designed controllers have artifacts at transitions due to imprecise kinematic blending but are smooth within a behavior, whereas the control fragments have a lesser but consistent level of jitter throughout due to frequent switching. Methods to automatically (i.e. without human labor) reduce movement artifacts when dealing with large movement repertoires would be interesting to pursue. Moreover, we wish to emphasize that due to the human-intensive components of training structured low-level controllers, fully objective algorithm comparison with previous work can be somewhat difficult. This will remain an issue so long as human editing is a significant component of the dominant solutions. Here, we focused on building movement behaviors with minimal curation, at scale, that can be recruited to solve tasks. Specifically, we presented two methods that do not require curation and can re-use low-level skills with cold-switching. Additionally, these methods can scale to a large number of different behaviors without further intervention. We view this work as an important step toward the flexible use of motor skills in an integrated visuomotor agent that is able to cope with tasks that pose simultaneous perceptual, memory, and motor challenges to the agent. Future work will necessarily involve refining the naturalness of the motor skills to enable more general environment interactions and to subserve more complicated, compositional tasks. A , all training was done using a distributed actor-learner architecture. Many asynchronous actors interact with the environment to produce trajectories of (s t, a t, r t, s t+1) tuples of a fixed rollout length, N. In contrast to BID4, each trajectory was stored in a replay buffer. The learner sampled trajectories of length N at random and performed updates. Each actor retrieved parameters from the learner at a fixed time interval. The learner ran on a single Pascal 100 or Volta 100 GPU. The plots presented use the steps processed by the learner on the x-axis. This is the number of transition retrieved from the replay buffer, which is equivalent to the number of gradient updates x batch size x rollout length. We performed all optimization with Adam and used hyperparameter sweeps to select learning rates and batch sizes. For the switching controller and control fragments approach we used a standard set of four policies trained from motion capture which imitated stand, run, left and right turn behaviors. In the switching controller, pretrained transitions were created in the reference data. For the control fragments approach, we were able to augment the set without any additional work and the selected policies are described in TAB2 In the heterogeneous forage task, the humanoid is spawned in a room with 6 balls, 3 colored red and 3 colored green. Each episode, one color is selected at random and assigned a positive value (+30) or a negative value (-10) and the agent must sample a ball and then only collect the positive ones. The architecture of the high-level controller consisted of proprioceptive encoder and an optional image encoder which, along with prior reward and action, were passed to an LSTM. This encoding core was shared with both the actor and critic. The details of the encoder are depicted in Fig. A. 1. The policy was trained with a SVG update. A state-action value / Q function was implemented as an MLP with dimensions in Table 1 and trained with a Retrace target. Target networks were used for Q and updated every 100 training iterations. The policy was also updated by an additional entropy cost at each time step, which was added to the policy update with a weight of 1e −5. The switching controller policy head took as input the outputs of the LSTM in Fig. A. 1. The policy head was an LSTM, with a state size of 128, followed by a linear layer to produce the logits of the multinomial distribution. The policy was updated with a policy gradient using N -step empirical returns with bootstrapping to compute an advantage, where N was equivalent to the rollout length in Table 1. The value-function (trained via V-Trace) was used as a baseline. The policy was also updated by an additional entropy cost at each time step, which was added to the policy update with a weight of.01. We train a policy to produce a continuous feature vector (i.e. the query-action), so the selector is parameterized by a diagonal multivariate Gaussian action model. The semantics of the query-action will correspond to the features in the control fragment feature-key vectors, which were partial state observations (velocity, orientation, and end-effector relative positions) of the control fragment's nominal start or end pose. Figure A.2: Illustration of query-based control fragment selection in which a query feature vector is produced, compared with key feature vectors for all control fragments, and the Q-value of selecting each control fragment in the current state is used to determine which control fragment is executed. In this approach, the Q function was trained with 1 step returns. So, for samples (s t, a t, r t, s t+1) from replay: DISPLAYFORM0 The total loss is summed across minibatches sampled from replay. Note that for the query-based approach we have query actions which are in the same space as but generally distinct from the reference keys of the control fragments which are selected after the sampling procedure. The high-level policy emits query actions, which are rectified to reference keys by a nearest lookup (i.e. the selected actions). This leads to two, slightly different high-level actions in the same space. This leads to the question of what is the appropriate action on which to perform both policy updates and value function updates. To handle this, it proved most stable to compute targets and loss terms for both the query-actions and selected actions for each state from replay. In this way, a single Q function represented the value of query-actions and selected actions. This was technically an approximation as the training of the Q-function pools these two kinds of action input. Finally, the policy update used the SVG-style update: DISPLAYFORM1 Figure A.3: Visualization (using PCA) of the actions produced by the trained query-based policy (on go-to-target). Query actions are the continuous actions generated by the policy. Reference keys are the feature vectors associated with the control fragments (here, features of the final state of the nominal trajectory of the fragment). Selected actions are the actions produced by the sampling mechanism and they are overlain to emphasize the control reference keys actually selected. The most prominent feature of this visualization is the lack of precision in the query actions. Note that 45/105 control fragments were selected by the trained policy. We hypothesize that the limited success of this approach is perhaps partly due to the impreciseness in their selectivity (see Fig A. 3). After finding these approaches were not working very well, an additional analysis of a trained discrete-selection agent, not shown here, found that the second most preferred control fragment (in a given state) was not usually the control fragment with the most similar reference key. This implies that the premise of the query-based approach, namely that similar fragments should be preferentially confused/explored may not be as well-justified in this case as we speculated, when we initially conceived of trying it. That analysis notwithstanding, we remain optimistic this approach may end up being more useful than we found it to be here. End-to-end training on each task was performed in a similar fashion to training the low-level controllers. Using the same architecture as in A.1, the policy was trained to output the 56-dimensional action for the position-controlled humanoid. As in the low-level training, the policy was trained with a SVG update and Q was trained with a Retrace target. The episode was terminated when the humanoid was in an irrecoverable state and when the head fell below a fixed height. We compared performance as a function of the number of fragments as a well as the length of fragments. If using a fixed number of behaviors and cutting them into control fragments of various lengths, two features are coupled: the length of the fragments vs. how many fragments there are. One can imagine a trade-off -more fragments might make exploration harder, but shorter temporal commitment to a fragment may ultimately lead to more precise control. To partially decouple the number of fragments from their length, we also compared performance with functionally redundant but larger sets of control fragments. Ultimately, it appears that from a strict task-performance perspective, shorter control fragments tend to perform best as they allow greatest responsiveness. That being said, the visual appearance of the behavior tends to be smoother for longer control fragments. Control fragments of length 3 (.09 sec) seemed to trade-off behavioral coherence against performance favorably. Hypothetically, longer control fragments might also shape the action-space and exploration distribution favorably. We see a suggestion of this with longer-fragment curves ascending earlier. Figure A.5: Transition density between control fragments for a trained agent on Forage. The colors reflect density of transitions within a clip/behavior class (darker is denser), and single fragment transition densities are overlain as circles where size indicates the density of that particular transition. Figure A.6: We depict a timeseries of the behavior of a trained high-level policy on Forage. In this particular trained agent, it frequently transitions to a particular stand fragment which it has learned to rely on. Green vertical lines depict reward acquisition.

represent this

------

Examples:
Provided: "In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation." Match: "Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods may have large bias. In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation. The proposed method can generate low-biased samples by increasing the length of MCMC simulation and optimising the MCMC hyper-parameters, which offers attractive balance between approximation bias and computational efficiency. We show that our method produces promising on popular benchmarks when compared to recent hybrid methods of MCMC and VI. Statistical inference methods in machine learning are dominated by two approaches: simulation and optimisation. Markov chain Monte Carlo (MCMC) is a well-known simulation-based method, which promises asymptotically unbiased samples from arbitrary distributions at the cost of expensive Markov simulations. Variational inference (VI) is a well-known method using optimisation, which fits a parametric approximation to the target distribution. VI is biased but offers a computationally efficient generation of approximate samples. There is a recent trend of hybrid methods of MCMC and VI to achieve a better balance between computational efficiency and bias. Hybrid methods often use MCMC or VI as an algorithmic component of the other. In particular, proposed a promising modified VI method that reduces approximation bias by using MCMC transition kernels. Another technique reduces the computational complexity of MCMC by initialising the Markov simulation from a pretrained variational approximation . proposed to improve MCMC using flexible non-linear transformations given by neural networks and gradientbased auto-tuning strategies. In this work, we propose a novel hybrid method, called ergodic inference (EI). EI improves over both MCMC and VI by tuning the hyper-parameters of a flexible finite-step MCMC chain so that its last state sampling distribution converges fast to a target distribution. EI optimises a tractable objective function which only requires to evaluate the logarithm of the unnormalized target density. Furthermore, unlike in traditional MCMC methods, the samples generated by EI from the last state of the MCMC chain are independent and have no correlations. EI offers an appealing option to balance computational complexity vs. bias on popular benchmarks in machine learning. Compared with previous hybrid methods, EI has following advantages: • EI's hyperparameter tuning produces sampling distributions with lower approximation bias. • The bias is guaranteed to decrease as the length of the MCMC chain increases. • By stopping gradient computations, EI has less computational cost than related baselines. We also state some disadvantages of our method: • The initial state distribution in EI's MCMC chain has to have higher entropy than the target. • The computational complexity per simulated sample of EI is in general higher than in VI. Monte Carlo (MC) statistical inference approximates expectations under a given distribution using simulated samples. Given a target distribution π, MC estimations of an expectation E π [f (x)] are defined as empirical average of the evaluation of f on samples from π. To generate samples from π, we assume that the unnormalized density function π * (x) can be easily computed. In a Bayesian setting we typically work with π * (x|y) given by the product of the prior p(x) and the likelihood p(y|x), where y denotes observed variables and x denotes the model parameters specifying p(y|x). Markov chain Monte Carlo (MCMC) casts inference as simulation of ergodic Markov chains that converge to the target π. The MCMC kernel M (x |x) is characterised by the detailed balance (DB) property: π(x)M (x |x) = π(x)M (x|x). Given an unnormalised target density π *, an MCMC kernel can be constructed in three steps: first, sample an auxiliary random variable r from an auxiliary distribution q φ1 with parameters φ 1; second, create a new candidate sample as (x, r) = f φ2 (x t−1, r), where f φ2 is a deterministic function with parameters φ 2; finally, accept the proposal as x t = x with probability p MH = min {0, π * (x)q φ1 (r)/[π * (x t−1)q φ1 (r)]}, otherwise duplicate the previous sample as x t = x t−1. The last step is well known in the literature as the Metropolis-Hastings (M-H) correction step and it in MCMC kernels that satisfy the DB condition. In the following, we denote the joint MCMC parameters (φ 1, φ 2) by φ. If f φ2 does not preserve volume, then it requires a Jacobian correction factor in the ratio in p M H. Hamiltonian Monte Carlo (HMC) is a successful MCMC method which has drawn great attention. A few recent works based on this method are;;. In HMC, the auxiliary distribution q φ1 is often chosen to be Gaussian with zero-mean and a constant diagonal covariance matrix specified by φ 1. The most common f φ2 in HMC is a numeric integrator called the leapfrog algorithm, which simulates Hamiltonian dynamics defined by log π * . The leapfrog integrator requires the gradient of log π * and a step size parameter given by φ 2. Given any initial state x 0, MCMC can generate asymptotically unbiased samples x 1:n. For this, MCMC iteratively simulates the next sample x t through the application of the MCMC transition kernel to the previous sample x t−1. It is well known in the literature that MCMC is computationally demanding . In particular, it is often necessary to run sufficiently long burn-in MCMC simulations to reduce simulation bias. Another drawback of MCMC is sample correlation, which increases the variance of the MC estimator . To avoid strong sample correlation, the common practice in MCMC to tune hyper-parameters manually using sample quality metrics like effective sample size, , which has been developed into automated gradient-based tuning strategies in recent work . Variational inference (VI) is a popular alternative to MCMC for generating approximate samples from π. Unlike MCMC reducing sample bias by long burn-in simulation, VI casts the sample bias reduction as an optimisation problem, where a parametric approximate sampling distribution P is fit to the target π. In particular, VI optimises the evidence lower bound (ELBO) given by where, also known as the entropy, must be tractable to compute. L ELBO (P T π *) is a lower bound on the log normalising constant log Z = log π * (x) dx. This bound is tight when P = π. therefore, the approximation bias in VI can be defined as the gap between L ELBO (P π *) and log Z, that is, where D KL (P π) denotes the Kullback-Leibler (KL) divergence. Variational approximations often belong to simple parametric families like the multivariate Gaussian distribution with diagonal covariance matrix. This in computationally efficient algorithms for bias reduction and sample generation, but may also produce highly biased samples in cases of over-simplified approximation that ignores correlation. Designing variational approximation to achieve low bias under the constraint of tractable entropy and efficient sampling procedures is possible using flexible distributions parameterised by neural networks (NNs) . However, how to design such NNs for VI is still a research challenge. The balance between computational efficiency and bias is a challenge at the heart of all inference methods. MCMC represents a family of simulation-based methods that guarantee low-bias samples at cost of expensive simulations; VI represents a family of optimisation-based methods that generate high-bias samples at a low computational cost. Many recent works seek a better balance between efficiency and bias by combining MCMC and VI. proposed to reduce variational bias by optimising an ELBO specified in terms of the tractable joint density of short MCMC chains. The idea seems initially promising, but the proposed ELBO becomes looser and looser as the chain grows longer. construct an alternative ELBO for HMC that still has problems since the auxiliary momentum variables are sampled only once at the beginning of the chain, which reduces the empirical performance of HMC. Inspired by contrastive divergence, proposed a novel variational objective function to optimise variational parameters by adding additional term that minimise the KL between a MCMC distribution and variational approximation to reduce variational bias. and proposed to replace expensive burn-in simulations in MCMC with samples from pre-trained variational approximations. This approach is effective at finding good initial proposal distributions. However, it does not offer a solution for tuning HMC parameters , which are critical for good empirical performance. Another line of research has focused on improving inference using flexible distributions, which are transformed from simple parametric distributions by non-linear non-volume preserving (NVP) functions. proposed to tune NVP parameterised MCMC w.r.t. a variant of the expected squared jumped distance (ESJD) loss proposed by. proposed a similar auto-tuning for NVP parameterised MCMC using an adversarial loss. Ergodic inference (EI) is motivated by the well-known convergence of MCMC chains : MCMC chains converge in terms of the total variation (TV) distance between the marginal distribution of the MCMC chain and the target π. Inspired by the convergence property of MCMC chains, we define an ergodic approximation P T to π with T MCMC steps as following. Given a parametric distribution P 0 with tractable density p 0 (x 0 ; φ 0) parameterlized by φ 0 and an MCMC kernel M (x |x; φ) constructed using the unnormalised target density π * and with MCMC hyperparameter φ, an ergodic approximation of π is the marginal distribution of the final state of an T -step MCMC chain initialized from P 0: We call φ 0 and φ the ergodic parameters of P T. Well known in MCMC literature like , the ergodic approximation p T converges to π after every MCMC transition and with sufficiently long chain p T is guaranteed to be arbitrarily close to π with arbitrary φ and φ 0. It is important to clarify that ergodic approximation is different from the modified variational methods like which only optimise the variational parameters φ 0, but the optimisation objective functions involve MCMC similation. In the following section, we show how EI can tune the ergodic parameters to minimise the bias of P T as an approximation to the target π with finite T. To reduce the KL divergence D KL (P T π), one could tune the burn-in parameter φ 0 and the MCMC parameter φ by minimizing equation 2. However, this is infeasible because we cannot analytically evaluate p T in equation 3. Instead, we exploit the convergence of ergodic Markov chains and propose to optimise an alternative objective as the following constrained optimisation problem: max where h is a hyperparameter that should be close to the entropy of the target, that is, h ≈ H(π). We call the objective in equation 4 the ergodic modified lower bound (EMLBO), denoted by L(φ 0, φ, π *). Note that the EMLBO is similar to L ELBO (P T π *), with the intractable entropy H(P T) replaced by the tractable L ELBO (P 0 π *). We now give some motivation for this constrained objective. First, we explain the inclusion of the term L ELBO (P 0 π *) in equation 4 and its connection to H(P T). If we maximised only the first term E p [log π * (x)] with respect to a fully flexible distribution P, the would be a point probability mass at the mode of the target π. This degenerate solution is avoided in VI by optimising the sum of E p [log π * (x)] and the entropy term H(P), which enforces P to move away from a point probability mass. However, H(P T) is intractable in ergodic approximation. Fotunately, we notice that maximising the term L ELBO (P 0 π *) = E p0 [log π * (x)] + H(P 0) has similar effect of maximising H(P T) for preventing P 0 from collapsing to the mode of π. It is easy to show that P T cannot be a delta unless H(P T) = −∞, which also implies L ELBO (P T π *) does not exist. Since the KL divergence D KL (P t π) never increases after each MCMC transition step , The constraint in equation 5 is necessary to eliminate the following pathology. If P 0 does not satisfy, we will favor P T to stay close to P 0 instead of making it converge to π faster. This is illustrated by the plot in the right part of Figure 2. To avoid this pathological case, note that It is interesting to compare the EMLBO with the objective function optimised by , that is, the ELBO given by where p(x 0:T −1 |x T) denotes the conditional density of the first T states of the MCMC chain given the last one x T and r(x 0:T −1 |x T) is an auxiliary variational distribution that approximates p(x 0:T −1 |x T). Note that the negative KL term in equation 6 will increase as T increases. This makes the ELBO in equation 6 become looser and looser as the chain length increases. In this case, the optimisation of equation 6 in an MCMC sampler that fits well the biased inverse model r(x 0:T −1 |x T) but whose marginal distribution for x T does not approximate π well. This limits the effectiveness of this method in chains with multiple MCMC transitions. By contrast, the EMLBO does not have this problem and its optimisation will produce a more and more accurate P T as T increases. EI combines the benefits of MCMC and VI and avoids their drawbacks, as shown in Table 1. In particular, the bias in EI is reduced by using longer chains, as in MCMC, and EI generates independent samples, as in VI. Futhermore, EI optimises an objective that directly quantifies the bias of the generated samples, as in VI. Methods for tuning MCMC do not satisfy the latter and optimise instead indirect proxies for mixing speed, e.g. expected squared jumped distance . Importantly, EI can use gradients to tune different MCMC parameters at each step of the chain, as suggested by. This gives EI an extra flexibility which existing MCMC methods do not have. Finally, EI is different from parallel-chain MCMC: while EI generates independent samples, parallel-chain MCMC draws correlated samples from several chains running in parallel. We now show how to maximise the ergodic objective using gradient-based optimisation. The gradient ∂ φ0,φ L(φ 0, φ, π *) is equal to the sum of two gradient terms. The first one ∂ φ0 L ELBO (P 0 π *) is affected by the constraint H(P 0) > h, while the second term is not. If we ignore the constraint, the first gradient term can be estimated by Monte Carlo using the reparameterization trick proposed in (D.P. ;): where f φ0 (·) is a deterministic function that maps the random variable i sampled from a simple distribution, e.g. a factorized standard Gaussian, into the random variable x i 0 sampled from p 0 (·; φ 0). To guarantee that our gradient-based optimiser yields a solution satisfying the constraint, we first initialize φ 0 so that H(P 0) > h and, afterwards, we force the gradient descent optimiser to leave φ 0 unchanged if H(P 0) is to get lower than h during the optimisation process. The Monte Carlo estimation of ∂ φ E p T [log π * (x)] can also be computed using the reparameterization trick. For this, the Metropolis-Hastings (M-H) correction step in the MCMC transitions, as described in Section 2.2, can be reformulated as applying the following transformation to x t−1: where is an indicator function that takes value one if p MH > u and zero otherwise. In Hamiltonian Monte Carlo (HMC), f φ is the leapfrog integrator of Hamiltonian dynamics with the leapfrog step size φ. We define the T -times composition of g φ, given in equation 8, as the transformation x T = g T φ (x 0, r 1:T ; u 1:T). Then, the second gradient term can be estimated by Monte Carlo as follows: T t=1 q(r t)Unif(u t ; 0, 1). Note that the gradient term equation 9 is correct under the assumption f φ is volume-preserving in the joint space of (x t−1, r t), otherwise additional gradient term of the Jacobian of f φ w.r.t. φ is required. However, it is not a concern for many popular MCMC kernels. For example, the leapfrog integrator in HMC f φ guarantees the preservation of volume as shown in . It is worth to mention that the indicator function in equation 8 is not continuous but differentiable almost everywhere. Therefore, the gradient in equation 9 can be computed conveniently using standard autodifferentiation tools. The gradient in equation 9 requires computing ∂ φ g T φ (x 0, r 1:T ; u 1:T), which can be done easily by using auto-differentiation and gradient backpropagation through the transfromations g φ (·, r t ; u t) with t = T,..., 1. However, backpropagation in deep compositions can be computationally demanding. We discovered a trick to accelerate the gradient computation by stopping the backpropagation of the gradient at the input x t−1 of g φ (x t−1, r t ; u t), for t = 1,..., T. Empirically this trick has almost no impact on the convergence speed of the ergodic approximation, as shown in Figure 2. 3.3 THE ENTROPY CONSTRAINT AND HYPERPARAMETER TUNING As mentioned previously, ignoring the constraint H(P 0) > H(π) may lead to pathological when optimising the ergodic objective. To illustrate this, we consider fitting an ergodic approximation given by a Hamilton Monte Carlo (HMC) transition kernel with T = 9. P 9 denotes the initial ergodic approximation before traing and P * 9 denotes the same approximation after training. The target distribution is a correlated bivariate Gaussian given by π = N (0, (2.0, 1.5; 1.5; 1.6)). Samples from this distribution are shown in plot (a) in Figure 1. We optimise different a separate HMC parameter φ t, as described in Section 2.2, for each HMC step t. We consider two initial distributions. The first one is P 0 = N (0, 3I) which satisfies the assumption H(P 0) > H(π). The second one is P 0 = N (0, I) with the entropy H(P 0) < H(π), which violates the assumption. In this latter case, we perform the unconstrained optimisation of equation 4. Plots (b) and (c) in Figure 1 show samples from P 9 and P * 9 for the valid P 0. In this first example, maximising the ergodic objective under equation 5 significantly accelerates the chain convergence as further shown by the left plot in Figure 2. Plots (d) and (e) in Figure 1 show samples from P 9 and P * 9 for the invalid initial distribution P 0. In the second example, E p0 [log π * (x)] is higher than E π [log π * (x)] and, consequently, maximising the unconstrained ergodic objective actually deteriorates the quality of the ing approximation. This is further illustrated by the right plot in Figure 2 which shows how the convergence of E pt [log π * (x)] to E π [log π * (x)] is significantly slowed down by the optimisation under the invalid P 0. Fortunately, it is straightforward to prevent this type of failure cases by appropriately tuning the scalar hyperparameter h in equation 5. A value of h that is too low may in higher bias of P T after optimisation as illustrated by the convergence of E pt [log π * (x)] in the blue and orange curves in Plot (b) in Figure 2. Furthermore, in many cases, estimating an upper bound on H(π) is feasible. For example, in Bayesian inference, the entropy of the prior distribution p(x) is often higher than the entropy of the posterior p(x|y). Therefore, the prior entropy can be used as a reference for tuning h. Figure 1: Histograms of samples from ergodic inference using HMC transition kernels. P 9 denotes the ergodic approximation before traing; P * 9 denotes the ergodic approximation after training. ] as a function of the length of the chain T using 10000 samples: Left: with the valid P 0 as H(P 0) > H(π); Right: with invalid P 0 as H(P 0) < H(π). SG training means the stop gradient is applied to the x from previous HMC step in equation 9. We first describe the general configuration of the ergodic inference method used in our experiments. Our ergodic approximation is constructed using HMC, one of the most successful MCMC methods in machine learning literature. We use T HMC transitions, each one involving 5 steps of the vanilla leapfrog integrator which was implemented following. The leapfrog pseudocode can be found in the appendix. In each HMC transition, the auxiliary variables are sampled from a zero-mean Gaussian distribution with diagonal covariance matrix. We tune the following HMC parameters: the variance of the auxiliary variables and the leapfrog step size, as mentioned in Section 2.2. We use and optimise a different value of the HMC parameters for each of the T HMC transitions considered. We call our ergodic inference method Hamiltonian ergodic inference (HEI). The burn-in model P 0 is factorized Gaussian. The initial entropy of P 0 is chosen to be the same as the entropy of the prior. The stocastic optimisation algorithm is Adam with TensorFlow implemtation and the optimiser hyperparameter setting is (β 1 = 0.9, β 2 = 0.999, = 10 −8). The initial HMC leapfrog step sizes are sampled uniformly between 0.01 and 0.025. Additional experiment on Bayesian neural networks is included in Appendix 6.3. We first compare Hamiltonian ergodic inference (HEI) with previous related methods on 6 synthetic bivariate benchmark distributions. Histograms of ground truth samples from each target distribution using rejection sampling are shown in Figure 3. The baselines considered include: 1) Hamiltonian variational inference (HVI) ; 2) generalized Hamiltonian Monte Carlo (GHMC) using an NVP parameterized HMC kernel and gradient-based auto-tuning of MCMC parameters w.r.t. sample correlation loss ; 3) Hamiltonian annealed importance sampling (HAIS) . It is worth to mention that we do not consider other hybrid inference methods like in our experiment, because these methods only combines MCMC simulation with VI but not optimise the parameters of MCMC kernel using the gradient-based approach like EI. HVI is the most similar method to HEI among all three baselines, because both HEI and HVI methods generate samples from the last state of MCMC chains and use gradient-based MCMC hyperparameter tuning to reduce bias. For a fair comparison between HVI and HEI, we consider the HMC chains with exactly the same setting in both methods: the initial state follows a standard Gaussian distribution and the length of HMC chain is T = 10. The key difference between HVI and HEI is the hyperparameter tuning objective, as mentioned in Section 3.1. We trained HVI for 1000 iterations and verified the ELBO converges to a (local) minimum (plots of the training ELBO values are included in Appendix 6.2). We trained HEI for 50 iterations. Following the setting of HAIS by, we used 1,000 intermediate distributions with 5 leapfrog steps per HMC transition and manually tuned the HMC parameters to have acceptance rate around 70%. GHMC 1 was run using 100 parallel chains with 5 leapfrog steps per GHMC transition, 100 burn-in steps and 1000 auto-tuned training iterations. The verification of the convergence of E p T [log π * (x)] to E π [log π * (x)] for HEI is shown in plot (a) of Figure 5. We generate 100,000 samples with each method and evaluate sample quality using two metrics: 1) the histogram of simulated samples for visual inspection; 2) the MC estimation of E π [log π * (x)]. Effective sample size (ESS) is a popular sample correlation based evaluation metric in recent MCMC literature . However, we do not consider ESS in this experiment, because GHMC is the only method among all methods generating correlated samples. Therefore, the ESS of GHMC is guaranteed to be lower than HVI and HEI. To generate ground truth samples from benchmark distributions, we use. The ing sample histograms of the ground truth using rejection sampling are shown in figures 3 and considered approximated sampling methods are shown in 4. Table 2 shows the ing estimates of −E π [log π * (x)] together with the wall-clock simulation time for generating 100,000 samples. The left part of Table 3 shows the training time of the MCMC parameter optimisation for all methods except HAIS, which does not support gradient-based HMC hyperparameter tuning. HEI is faster than HVI and GHMC. Note, however, that the acceleration of HEI over HSVI is due to the stopping gradient trick described in Section 3.2. The histograms and the estimates of −E π [log π * (x)] generated by HEI are consistent with the of the more expensive unbiased samplers GHMC and HAIS, which are close to the ground truth. By contrast, HVI exhibits a clear bias in all benchmarks. Regarding the sampling time, HVI and HEI simulate HMC chains with the same length and, consequently, perform similarly in this case while sample simulation from HAIS and GHMC is much more expensive. 6.00 3000 -83.57 50 HVI(T =1, 16LF, n h =800, ConvNet encoder) 6.00 360 -83.68 48 HVAE(T =1, 16LF, n h =500, ConvNet encoder) 6.00 360 -84.22 48 HEI(T =30, 5LF, n h =500, no neural net encoder) 1.65 54 -83.17 48 HEI(T =30, 5LF, n h =500, no neural net encoder) 3.00 100 -82.76 46 HEI(T =30, 5LF, n h =500, no neural net encoder) 6.00 200 -82.65 45 HEI(T =30, 5LF, n h =500, no neural net encoder) 12.00 400 -81.43 38 HEI(T =15, 5LF, n h =500, no neural net encoder) 8.00 540 -83.30 48 Table 4: Comparisons in terms of compuational efficiency and test log-likelihood in the training of deep generative models on the MNIST dataset. We implemented the deconvolutional decoder network in to test HVI. , the test likelihood is estimated using importence-weighted samples from the encoder network. In our experiment, we use Hamiltonian annealled importance sampling and report the effective sample size (ESS). We now evaluate HEI in the task of training deep generative models. MNIST is a standard benchmark problem in this case with 60,000 grey level 28 × 28 images of handwritten digits. For fair comparison with previous works, we use the 10,000 prebinarised MNIST test images 2 used by. The architecture of the generative model considered follows the deconvolutional network from. In particular, the unnormalised target p θ (x, y) consists of 32 dimensional latent variables x with Gaussian prior p(x) = N (0, I) and a deconvolutional network p θ (y|x) from top to bottom including a single fully-connected layer with 500 RELU hidden units, then three deconvolutional layers with 5 × 5 filters, feature maps, RELU activations and a logistic output layer. We consider a baseline given by a standard VAE with a factorised Gaussian approximate 4 Table 3: Left. The training time of MCMC parameter optimisation in seconds for 100 iterations for all candidate methods to produce the in Figure 4. The training time of HEI is lower than HVI because of the stop gradient trick mentioned in Section 3.2. We do not report the training time for HAIS, because HAIS requires manual tuning of MCMC hyperparameters which is not directly comparable to the gradient-based autotuning used by the other methods. Right. The training time in seconds per epoch for the experiments with deep generative models (DGM). posterior generated by an encoder network q(x|y) which mirrors the architecture of the decoder . The code for is not publicly available. Nevertheless, we reimplemented their convolutional VAE and were able to reproduce the marginal likelihood reported by , as shown in Table 4. This verifies that our implementation of the generation network is correct. We implemented HVI in using an auxiliary reverse model in the ELBO parameterized by a single hidden layer network with 640 hidden units and RELU activations. We also implemented the Hamiltonian variational encoder (HVAE) method , which is similar to HVI but without the reverse model. Unlike in the original HVAE, our implementation does not use tempering but still produces similar to those from. For the HEI encoder, we use T = 30 HMC steps, each with 5 leapfrog steps. The initial approximation P 0 is kept fixed to be the prior p(x). We optimise the decoder and the HEI encoder jointly using Adam. Table 4 shows the marginal test log-likelihood for HEI and the other methods, as estimated with 1,000 HAIS samples . , we also include the effective sample size (ESS) of HAIS samples for the purpose of verifying the reliability of the reported test log-likelihoods. Overall, HEI outperforms HVI, HVAE and the standard VAE in test log-likelihood when the training time of all methods is fixed to be 6 hours. HEI still produces significant gains when the training time is extended to 12 hours and, with only 1.6 hours of training, HEI can already outperform the convolutional VAE of with 6 hours of training. To verify the convergence of HEI, we show in plot (b) of Figure 5 estimates of.., 10 on five randomly chosen test images, where the ground truth E π [log π * (x)] is estimated by HAIS, after HMC hyper-parameter tuning in HEI (blue) and without hyper-parameter tuning in HEI (green), i.e. just using the initial hyper-parameter values. Plot (c) in Figure 5 shows similar , but using the maximum mean discrepancy (MMD) score to quantify the similarity of samples from p T to samples from π, where the latter ground truth samples are generated by HAIS. These plots suggests that shortening the HEI chain to T = 10 HMC steps will have a negligible effect on final simulation accuracy. Finally, the right part of Table 3 shows the training time of HEI with and without the stopping gradient trick. These resuls show that the former method is up to 5 times faster.: a: the targets are 2D benchmarks with the ground truth of E π [log π * (x)]; b: the target π is the VAE posterior p(x|y) each curve represents one random chosen test MNIST image y with the ground truth of E π [log π * (x)] estimated by HAIS using 100 samples; c: MMD score between HEI samples and HAIS samples. We have proposed Ergodic Inference (EI), a novel hybrid inference method that bridges MCMC and VI. EI a) reduces the approximation bias by increasing the number of MCMC steps, b) generates independent samples and c) tunes MCMC hyperparameters by optimising an objective function that directly quantifies the bias of the ing samples. The effectiveness of EI was verified on synthetic examples and on popular benchmarks for deep generative models. We have shown that we can generate samples much closer to a gold standard sampling method than similar hybrid inference methods and at a low computational cost. However, one disadvantage of EI is that it requires the entropy of the first MCMC step to be larger than the entropy of the target distribution. Here is the code for the vanilla leapfrog algorithm we used in HVI, HEI and HAIS. Algorithm 1: Leapfrog Input: x: state, r: momenta, φ 1: r variance, φ 2: step size, m: number of steps Result: x: new state, r: new momentum x = x; r = r; for t ← 1 to m dō r =r − 0.5φ 2 ∂ x U (x); x =x + φ 2 /φ 1r; r =r − 0.5φ 2 ∂ x U (x); end x =x; r =r; return x and r; The plots in Figure 6 show training loss (negative ELBO) of HVI and the training expected log likelihood E p T [log π * (x)] with T = 10 HMC steps with Adam with hyperparameter setting described in Section 4. It is clear that HVI is well trained but the approximation is biased, because E p T [log π * (x)] does not converge to the true loss (the red line on the right plots). In comparison, in Figure 6 (Left) in our paper, E p T [log π * (x)] of HEI converges to the ground true by optimising our ergodic loss. In this additional experiment we approximate the posterior distribution of Bayesian neural networks with standard Gaussian priors. We consider four UCI datasets and compare HEI with the stochastic gradient Hamilton Monte Carlo (SGHMC) method from. The networks used in this experiment have 50 hidden layers and 1 real valued output unit, as stated in. The HEI chain contains 50 HMC transformation with 3 Leapfrog steps each. The initial proposal distribution P 0 is a factorised Gaussian distribution with mean values obtained by running standard mean-field VI using Adam for 200 iterations. We do not use in P 0 the variance values returned by VI because these are unlikely to in higher entropy than the exact posterior since VI tends to understimate uncertainty. Instead, we choose the marginal variances to be n −0.5 where n is the number of inputs to the neural network layer for the weight. To reduce computational cost, we use in this case stochastic gradients in the leapfrog integrator. For this, we split the training data into 19 mini-batches and only use one random sampled mini-batch for computing the gradient in each leapfrog iteration. We train our HEI for 10 epochs and the stationary distribution is chosen as approximate posterior on a random sampled mini-batch. The ing test log-likelihoods are shown in Table 5. Overall, HEI produce significantly better than SGHMC. We also show in the right plot of Figure 7 estimates of E pt [log p(x, y)] for t = 1,..., 50 after HMC hyper-parameter tuning and without hyper-parameter tuning."

Solve tasks involving vision-guided humanoid locomotion, reusing locomotion behavior from motion capture data.

represent the short summary for finding the corresponding scientific article (long)

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.