title: Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
authors: Tianyu Li Roberto Calandra Deepak Pathak Yuandong Tian Franziska Meier Akshara Rai
text: 
  
  
  
  
  
  
  IEEE Robotics and Automation Letters. Preprint Version. Accepted Feb, 2021
  Li et al.: Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
  
  
  
  
  
  Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
  
      Tianyu Li^1, Roberto Calandra^1, Deepak Pathak^2, Yuandong Tian^1, Franziska Meier^1, Akshara Rai^1
  
   Manuscript received: Oct, 15, 2020; Revised: Dec, 17, 2020; Accepted: Feb, 17, 2021.
  This paper was recommended for publication by
  Editor Abderrahmane Kheddar upon evaluation of the Associate Editor and Reviewers’ comments. 
  ^1FAIR, Menlo Park, CA, 94025, USA, 
          {tianyul,rcalandra, yuandong,fmeier,akshararai}@fb.com
  ^2 Robotics Institute, Carnegie Mellon University,
          Pittsburgh, PA, 15213, USA,
          dpathak@cs.cmu.edu
          
  Supplementary Video Link: www.youtube.com/watch?v=yopO26MF-HU
  Digital Object Identifier (DOI): see top of this page.
  
  
      December 30, 2023
  ================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
  
  
  
  
  
  Hierarchical learning has been successful at learning generalizable locomotion skills on walking robots in a sample-efficient manner. However, the low-dimensional “latent” action used to communicate between two layers of the hierarchy is typically user-designed. In this work, we present a fully-learned hierarchical framework, that is capable of jointly learning the low-level controller and the high-level latent action space. Once this latent space is learned, we plan over continuous latent actions in a model-predictive control fashion, using a learned high-level dynamics model. This framework generalizes to multiple robots, and we present results on a Daisy hexapod simulation, A1 quadruped simulation, and Daisy robot hardware. We compare a range of learned hierarchical approaches from literature, and show that our framework outperforms baselines on multiple tasks and two simulations. In addition to learning approaches, we also compare to inverse-kinematics (IK) acting on desired robot motion, and show that our fully-learned framework outperforms IK in adverse settings on both A1 and Daisy simulations. On hardware, we show the Daisy hexapod achieve multiple locomotion tasks, in an unstructured outdoor setting, with only 2000 hardware samples, reinforcing the robustness and sample-efficiency of our approach. 
  
  
  
  
  
  
  
  
  
  
  § INTRODUCTION
  
  Traditional control techniques used in legged locomotion, like inverse dynamics make assumptions about the dynamics of the robot, and can lead to poor performance when these assumptions are violated. In contrast, learning-based approaches do not make strict assumptions about dynamics, but are expensive to train. 
  
  
  
  
  Learning locomotion skills can be made scalable to real robots by leveraging a two-layer hierarchical control structure <cit.>.
  Typically in hierarchical control literature, the action space used by the high-level controller to interact with the low-level controller is user-defined <cit.>.
  Such a user-defined action space can potentially be too restrictive for some tasks. For example, <cit.> constrain the latent space to choose from a pre-defined set of primitives, limiting performance to the quality and number of primitives; <cit.> constrain the latent space to a footstep pattern, and learn a conservative walking pattern, while a different gait might move faster.
  
  
  
  
  
  In this work, we introduce a generalizable, fully-learned, hierarchical control framework that eliminates the need for pre-defined action spaces for the high-level controller. We start by jointly learning a low-level policy and a high-level latent action space using imitation learning to reproduce a set of experts in simulation. This step transforms the initially discrete space of expert primitives into a continuous space, allowing us to go beyond a finite number of primitives, while learning a suitable high-level latent action space. For the high level controller, we propose a model-based planner, and plan a sequence of learned latent actions to achieve a desired goal. We learn a `coarse' dynamics model over one cycle of the low-level policy given a latent action input, and use it for model-predictive control (MPC). This results in reactive planning in learned latent action spaces, allowing continuous modulation of robot motion to achieve changing targets, respond to disturbances, and generalize to new tasks.
  
  
  
  
  
  
  Using the simulation of a Daisy hexapod and A1 quadruped (Figure <ref>), we compare our approach against different learned hierarchical approaches from literature and find that our approach outperforms baselines in all evaluation tasks, while being more sample-efficient. This includes a comparison between MPC on our learned latent action space and MPC on a library of experts that were used for learning our low-level, signifying the importance of jointly learning a high-level latent action space and a low-level policy. 
  Furthermore, we compare our method with an informed baseline of inverse-kinematics (IK) acting on desired center of mass (CoM) motion. The IK baseline can be seen as the best-case low-level policy for a fixed action space, making it a highly competitive baseline rarely addressed in hierarchical learning literature. Our approach does not require any prior knowledge about the robot, unlike IK, but performs comparably to IK in normal conditions, and outperforms it in adverse settings. This experiment strongly reinforces the advantages of learning a high-level latent action space. Finally, we demonstrate that our approach can be used to solve complex locomotion tasks in a sample-efficient manner in the real world with a Daisy hexapod robot, with only 2000 hardware samples. 
  Together, our experiments establish that our proposed framework generalizes to hardware, multiple robot simulations, and multiple tasks with no re-training per task, significantly improving the state-of-the-art of learned hierarchical controllers for locomotion. 
  
  The main contributions of this work are: 1) Present a sample-efficient fully-learned hierarchical framework for locomotion and deploy it on a legged-robot. 2) Demonstrate a model-based planner to plan over learned high-level latent actions. 3) Extensively compare and analyze different learned as well as traditional hierarchical control schemes from literature. To the best of our knowledge, this is the first fully-learned model-based hierarchical framework demonstrated on a robot hardware. We present experiments on multiple robot simulations, further reinforcing the significance of our results. The generalizability and sample-efficiency of our approach makes it suitable for solving long-horizon legged locomotion problems, such as indoor navigation (Figure <ref>).
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § BACKGROUND AND RELATED WORK
  
  
  
    
  Model predictive control
  We consider a Markov Decision Process with actions 𝐚 and states 𝐬, cost c(𝐬_t,𝐚_t)  and transition dynamics 𝐬_t+1 = f_dyn(𝐬_t, 𝐚_t). 
  The objective of model-predictive control (MPC) is to minimize the long term cost of a trajectory τ: J = ∑_t=0^T 𝔼_τ [c(𝐬_t,𝐚_t)] 
  with respect to the actions 𝐚_0:T, given the dynamics f_dyn(𝐬_t, 𝐚_t), starting from an initial state 𝐬_0. 
  MPC plans a sequence of actions over a horizon H 𝐚_0:H = _𝐚_0:H∑_h=0^H c(𝐬_h, 𝐚_h), given 𝐬_h+1 = f_dyn(𝐬_h, 𝐚_h) and 𝐬_0 = 𝐬_curr, starting from current state 𝐬_curr.
  The first action a_0 is applied, and the process repeats, starting with the new current state. MPC has been applied to locomotion control in <cit.>, <cit.>, <cit.>.
  In this work, we learn the transition dynamics f_dyn from data, over a temporally-extended action sequence 𝐚_1:N of length N, i.e., 𝐬_t+N = f_dyn(𝐬_t, 𝐚_t:t+N). Similar dynamics models are used in <cit.> over Poincare sections of a bipedal gait. 
  
  
  
    
  Learning for locomotion
  Classical locomotion controllers like <cit.> require known dynamics, while machine learning approaches can learn locomotion without dynamics <cit.>. Successful learning for locomotion has been demonstrated on robots using dynamics randomization <cit.>, and other sim2real approaches like domain-specific features <cit.>. <cit.> iteratively learn a low-level policy and a latent input in simulation, and optimize the latent input on hardware using Bayesian optimization, or other sampling-based optimization approaches. While these approaches can learn to walk, they cannot directly learn skills that can generalize to new tasks, like multiple goals and different desired velocities. 
  
  
  
  
  In contrast, hierarchical decomposition of control holds the promise of solving complex locomotion tasks in a general manner. For instance, <cit.> decompose the problem of controlling a humanoid robot to center of mass planning, followed by model-based controllers. Also for learned controllers, <cit.> demonstrate the efficacy of hierarchical structure for solving locomotion tasks. <cit.> jointly learn a discrete low-level policy and a switching high-level policy in simulation. <cit.> use a learned high-level controller to decide a footstep location for a learned low-level policy in simulation. On hardware, <cit.> use a high-level controller to sequence a set of pre-learned primitives on a hexapod robot and <cit.> use the high-level policy to define sub-goals for the low-level policy for a quadruped. However, these works assume a known high-level action space that communicates between the two levels of the hierarchy – either as footsteps, or a discrete set of primitives. This can be restrictive if the pre-defined action space is not complex enough to represent a task. In contrast, we present a framework that can learn a continuous space of low-level primitives along with a learned high-level action space, and combine this with MPC to achieve multiple locomotion tasks.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
    
  Latent space learning for control
  There has been a lot of interest in the robotics community in learning latent representations of high-dimensional state like images, and using them for control, such as <cit.> <cit.>, <cit.>, <cit.>  and <cit.>. 
  However, there are fewer works that deal with latent actions, or latent inputs to a policy (without transition dynamics). Most closely related to our work is the work by <cit.>, <cit.> and <cit.>. 
  
  These works iteratively learn a policy and a latent input to the policy on a large range of environments in simulation, and fine-tune the latent input on hardware. 
  In contrast to these approaches, we jointly learn a high-level latent action space, and a low-level policy in a hierarchical setting. We also learn a dynamics model over this learned latent space and use MPC to plan over continuous latent actions, allowing us to change the latent action input to our low-level policy on the fly. As a result, each hardware trajectory in our approach can have different latent actions per step, as needed by the task, while the latent input is kept fixed in <cit.> for each hardware rollout. This allows us to respond to online disturbances and reach changing targets.
  
  
  
  
  § PLANNING IN A LEARNED LATENT ACTION SPACE
  
  
  Our hierarchical learning framework, illustrated in Figure <ref>, consists of three steps: 1) Jointly optimize a low-level policy π_θ, and a latent action . 2) Learn a dynamics model given the learned low-level policy π_θ(·,) over randomly sampled . 3) Plan over  using MPC and learned dynamics. This method can generalize to multiple robots and locomotion tasks while remaining sample-efficient, without re-training per task. 
  
  
  
  
  
   §.§ Hierarchical planning for locomotion
  
  We divide the control of legged robots into a low-level policy and a high-level model-based planner. The low-level policy π_θ runs at a frequency of 100Hz and its behaviour is modulated by a latent input  from the high-level planner, which runs at about 1Hz. Different  result in different behaviors on the robot. One popular example of this architecture is designing the low-level policy with inverse kinematics, and choosing high-level actions as CoM velocity. In contrast, we present a framework to jointly learn the  low-level policy, and latent action space, eliminating the need for user-defined action spaces. We also compare our method to the popular inverse kinematics setting in Section <ref>. 
  
  
        
          
  
  
      
  
  
  
  
  
    §.§.§ Low-level policy
  
  Our low-level policy is parametrized by a neural network that takes a phase variable t ∈ (0,1] as input, along with a latent action  from the high-level planner.
  𝐪_des,t = π_θ(t, 𝐳) ,    t = n/N
  where n is the current time step, which goes up to the primitive length N, at which point it is reset to 0 again. All primitives are assumed to be the same pre-determined length N. 𝐪_des,t are the desired joint-angles sent to the robot. 
  The phase t linearly increases to N, and ensures a cyclic nature of the low-level policy. For a fixed , the policy generates the same joint angle pattern every N time steps in a periodic manner. 
  
  
  
    §.§.§ High-level planner
  
  
  We use a model-based high-level planner that plans a latent action sequence ^1:H for a planning horizon H using MPC. The dynamics used in this planning are learned over temporally-extended low-level action sequences, rather than per time-step transitions. 
  
  Starting from the current center of mass (CoM) position and orientation s_curr = (𝐱_com, θ_com), our high level planner does a search over the possible sequences of latent actions to find the optimal sequence over a horizon H: 
  
  
  
  
      𝐳^*_1:H = _𝐳_1:H∑_h=1^H c_hl(𝐬_h, 𝐳_h) 
      s.t.  𝐬_h+1 = f_dyn(𝐬_h, 𝐳_h),   𝐬_0=𝐬_curr
  
  c_hl is the high-level cost function; h is a robot step, consisting of N time steps. The first action ^*_1 is applied on the robot, and the optimization is repeated, starting from the new state. We optimize Equation <ref> using random shooting with 8000 samples, over a horizon of H=1 and pick the best action. The cost is designed by the user depending on the task at hand, described in Section <ref>. Using a learned dynamics model, MPC can generalize to multiple tasks without any re-training, hence significantly improving the sample-efficiency.
  
  
  
  
  
  
   §.§ Jointly learning low-level policy and latent actions
  
  We propose a framework for jointly learning a low-level policy and high-level latent actions, starting with G expert demonstrations. 
  
  We use supervised learning to jointly optimize a low-level policy π_θ and a latent action  input into π_θ that leads to a desired expert behavior π^exp.  Specifically, we optimize θ and one latent action per expert _g, such that the optimal policy π_θ^*(·, ^*_g) matches the expert g: 
  
  
  
      π_θ^*(t, 𝐳^*_g) = π^exp_g(t)   for   g = 1 … G
  
  
  This ensures that, if trained properly, π_θ^* is capable of generating at least G experts, given different ^*_g per expert. This can be seen as a bottle-neck formulation where we want to jointly learn a policy parametrization θ^* and high-level latent action space  such that a large variety of gaits can be produced by the policy π_θ^* for different inputs . The loss for each gait g becomes ℒ_g = π_θ(t, 𝐳_g) - π^exp_g(t)^2. θ and _1:G can now be optimized together to reduce the total loss
  
  
  
      θ^*, 𝐳^*_1:G = _θ, 𝐳_1:G∑_g=1^G π_θ(t, 𝐳_g) - π^exp_g(t) ^2.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
      
  
  
  
  The partial derivatives of the total loss ℒ with respect to latent variable _g and the policy weights θ are:
  
  
  
      ∂ℒ/∂θ = ∂/∂θ∑_g=1^G π_θ(t, 𝐳_g) - π^exp_g(t)^2,
        ∂ℒ/∂𝐳_g = ∂/∂𝐳_gπ_θ(t, 𝐳_g) - π^exp_g(t) ^2.
  
  This naturally leads to a formulation where the update to each latent input _g is only affected by the supervised learning loss of the expert g that it is trying to imitate. On the other hand, updates to low-level policy parameters θ are optimized by reducing the loss over all G experts. 
  This is in contrast to <cit.> that iteratively optimize the latent action and policy which can lead to unstable training. Moreover, unlike <cit.> we learn a unified low-level policy across all experts, and not separate policies for different experts.
  
  
  The expert data for supervised learning is collected using a model-based hierarchical controller proposed in <cit.>. An expert-designed feedback law converts desired CoM velocities to footsteps, which are followed using a low-level inverse kinematics (IK) policy. We randomly sample 50 desired CoM velocities, and use the generated joint angle trajectories as the experts used in training the low-level policy π_θ and latent action space . Once π_θ is learned, we are no longer limited to the G experts that were used during training. By sampling in the space of latent actions , we can generate new controllers that interpolate, and extrapolate from the training experts. As illustrated in Figure <ref>, different  lead to different CoM trajectories. Close-by samples in the latent action space usually lead to continuous behaviors in the CoM space, showing that our learned latent space is structured. 
  
  
  
   §.§ Learning center of mass dynamics
  
  Once the policy π_θ and latent space  is learned, we can plan over  to accomplish a variety of locomotion tasks, like reaching goals and trajectory tracking. To do this, we learn a temporally-extended dynamics model of the center of mass (CoM), learned by sampling a series of random latent actions  and measuring the resultant CoM state after one cycle of the low-level policy π_θ(·, ). 
  
  This leads to a `coarse' dynamics model where the next state s_t+N = f_dyn(s_t, _t) is the state after executing the low-level policy for N time steps, starting from s_t and using latent action _t.
  
  In a similar spirit, previous works like <cit.> have used approximate CoM dynamics models such as a linear inverted pendulum (LIPM). Unlike LIPM, our CoM dynamics model is learned directly from robot data, and hence adheres to the dynamics of the robot. For example, the dynamics learn that certain latent actions result in slipping on the robot, and hence lead to slower CoM movement. As a result, slipping and other unobserved factors are directly absorbed in our dynamics, eliminating the need for no-slip constraints like friction cones, as in <cit.>. Moreover, we learn dynamics in a learned latent action space, and are not restricted to planning in a user-defined space like CoM velocity. 
  
  Formally, we represent current state of the CoM by s_curr = (x_curr, y_curr, γ_curr, ẋ_curr, ẏ_curr), where (x_curr, y_curr) is the current horizontal position, (ẋ_curr, ẏ_curr) is the current horizontal velocity and γ_curr is the current yaw. The learned dynamics represent a transition from the current state to the next state s_next = (x_next, y_next, γ_next, ẋ_next, ẏ_next), using the low-level controller π_θ with latent action . To simplify the learning and improve generalization, we learn a delta dynamics model in the current local CoM frame
  
  
  
      Δ x, Δ y, Δγ, ẋ_next, ẏ_next = f_dyn(ẋ_curr, ẏ_curr, z)
  
  where Δ x = x_next - x_curr , Δ y = y_next - y_curr  and Δγ = γ_next - γ_curr. The delta predictions and measurements of s_curr can be used to predict the next state s_next.
  
  
  A learned dynamics model for high-level planning has several advantages, over learning a model-free high-level policy: 1) It is much more sample-efficient  2) It generalizes to new tasks, and needs no additional re-training for new tasks. In our experiments in Section <ref>, we show that this model-based planner outperforms model-free approaches, like <cit.>, in sample-efficiency and overall performance.
  
  We do not explicitly model the CoM dynamics in the vertical plane, or the roll and pitch of the robot, to improve sample-efficiency of dynamics learning. Implicitly, controllers with lower CoM height, or high roll and pitch lead to slower gaits, which is learned by our dynamics.
  
  
  
  § EXPERIMENTS
  
  
  We evaluate our proposed framework on two robot simulations (Daisy and A1) and one real robot (Daisy). We provide extensive comparisons to prior learning-based methods, as well as, a traditional control approach.
  
  
  Our results demonstrate that our approach outperforms current state-of-the-art in learned hierarchical locomotion, generalizes to multiple robots and solves real-world locomotion tasks. 
  
  
  
  
   §.§ Experimental Setup
  
  
  We use Pybullet <cit.> as our physics simulator, and build models for two robots: Daisy hexapod robot from Hebi robotics <cit.> and A1 quadruped from Unitree Robotics <cit.>, shown in Figure <ref> and <ref> respectively.
  
  Daisy is a 6-legged robot, with 3 motors in each leg, weighing 21kg and 1.1m by 1.1m wide in its nominal stance. A1 is a 4-legged robot, with 3 motors in each leg, weighing 12kg and 0.5m by 0.3m in dimensions. They have very different robot morphologies (hexapedal vs. quadrupedal) and significantly different mechanics and dynamics. 
  On hardware, we use the Daisy robot with a Realsense camera <cit.> to measure the position, orientation and velocity of the robot in an unstructured outdoor space. The position and velocity feedback gains for the base, shoulder and elbow were [2.0,3.0,4.0] and [0.2, 0.1, 0.15] for all legs in all hardware experiments.
  
  
  
  To evaluate our method, we create a series of tasks that are relevant to real-world locomotion. 
  
  
  
  
  
  
  
  
  Velocity tracking: This task measures the robot's ability to track a desired CoM velocity 𝐯_tgt and orientation γ_tgt, leading to cost c_1 = w_1||𝐯_tgt -  𝐯_curr || + w_2 ||γ_tgt -  γ_curr ||, where [w_1, w_2] = [2, 1]. The target velocity is varied throughout the experiment and the controller has to adapt to new targets. Target velocities were [0.0,0.2]m/s, [0.2,0.0]m/s, [0.0,-0.2]m/s, [-0.2, 0.0]m/s where each variable indicates desired velocity in x and y directions respectively.
  
  Goal reaching: Some real-world locomotion tasks involve reaching a goal in space, either as a long-distance goal, or an intermediate waypoint. This tasks requires the robot to reach a range of CoM targets s_tgt = (x_tgt, y_tgt) and orientation γ_tgt using cost c_2 = w_1||s_tgt -  s_curr || + w_2||γ_tgt -  γ_curr ||, where [w_1, w_2] = [2, 1]. Eight desired goals were uniformly distributed on a circle of radius 2m, with target orientation always pointing towards y axis.
  
  Trajectory tracking:  For navigating cluttered spaces with obstacles, or controlled environments, like a road, it is important to closely follow a CoM trajectory designed by a planner. We track an S-shaped desired trajectory consisting of target CoM positions s_tgt, t and orientations γ_tgt, t that change with time t, leading to cost c_3 = w_1||s_tgt, t -  s_curr || + w_2||γ_tgt, t -  γ_curr ||, where [w_1, w_2] = [2, 1]. 
  
  This is the first detailed study of hierarchical learning for locomotion on a rich set of tasks, including demonstration on hardware. While <cit.> have been shown to achieve interesting results in simulation, they have not been demonstrated on hardware, or extensively analyzed in simulation. On the other hand, <cit.> only show results on a subset of our designed tasks on hardware. Our analysis highlights the relative advantages and disadvantages of the hierarchical choices made in literature and helps make significant, scientific conclusions about the different approaches. By experimenting on two different robot designs, we further reinforce the statistical significance and generality of our results.
  
  
  
  
  
  
   §.§ Comparison with hierarchical baselines
  
  
  To compare our framework against these prior approaches, we create an ablation experiment in simulation. We characterize all hierarchical approaches by the choice of their low-level and high-level policies, and sweep through the different choices made in literature. Finally we test all the approaches on A1 and Daisy robot simulations and tasks described in Section <ref>. Note that all model-free approaches had to be re-trained for new tasks, and hence, effectively take 3 times as much data as model-based approaches.
  The approaches compared are shown in Figure <ref> and can be broadly categorized into the following groups:
  
  A library of experts (LIB): <cit.> use a library of low-level experts, and learn a high-level policy that chooses between the experts. We consider the same library of 50 experts that was used for supervised learning of our low-level policy, and used model-based planning for the high-level. This comparison highlights the importance of a learned high-level action space versus a discrete set of expert primitives.
  
  
  Model-free high-level policy (SAC):  An alternative to model-based planning is to learn a model-free high-level policy, as in  <cit.>, <cit.>. Assuming the same learned low-level policy as our approach, we use goal-conditioned SAC <cit.> to learn a high-level policy. This setting is also similar to <cit.> and <cit.>, though these approaches cannot directly generalize to multiple targets. This motivates the importance of model-based planning in learned latent action spaces for generalization versus learning a model-free policy.
  
  Model-free learned low-level and high-level policy (SAC-SAC):   <cit.> use model-free RL for training a low-level policy designed to reach sub-goals, and train a model-free high-level policy to define sub-goals for the pre-trained low-level policy. We use goal-conditioned SAC to learn both levels of this hierarchy, first training the low-level, then training the high-level, keeping the low-level fixed. This experiment compares our framework with model-free frameworks like <cit.> on a wide-range of locomotion tasks. It highlights the importance of jointly learning a high-level action space with a low-level policy, followed by model-based planning.
  
  The summary of these comparisons is shown in Figure <ref> and  <ref> for Daisy and A1 robot simulations. We used 10,000 samples for training the high-level dynamics and policies with an Adam optimizer, 1e-3 as learning rate and 512 as batch size. All model-free approaches are trained on half of the targets, and tested on all targets. For example, goal-conditioned SAC is trained to reach 4 goals, and tested on all 8. Both SAC and SAC-SAC are re-trained for new tasks, hence using 3 times as much data as our approach (LAT). SAC and SAC-SAC fail to generalize to new targets and perform worse than our approach on all tasks. On A1, SAC incurs a cost of 0.57 ± 0.06 at goal reaching, and SAC-SAC incurs a cost of 0.92 ± 0.04, much higher than LAT with 0.34 ± 0.02. This is not surprising as model-free approaches are known to be less sample-efficient and poorer at generalizing than model-based approaches <cit.>. SAC-SAC also has a user-defined high-level action space that performs poorly than the learned action space used in both LAT and SAC. This experiment reinforces our claim that model-based planning in learned action spaces improves performance and generalization.
  
  The baseline LIB uses a discrete set of primitives with a model-based high-level controller, and generalizes well to goal reaching. For velocity tracking, however, the low-level primitives in LIB do not include experts that can reach the target velocity, and hence LIB incurs a high cost of 0.58 ± 0.01 on A1. On the other hand, LAT can plan in the learned latent action space and interpolate between the discrete set of primitives, incurring a much lower cost of 0.16 ± 0.10 on A1. This experiment proves our hypothesis that a jointly learned continuous high-level action space and low-level policy can outperform a discrete set of low-level primitives. Our approach combines the robustness of model-based planning with learned high-level and low-level controllers, and can successfully solve a large range of tasks sample-efficiently, without re-training per task. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Comparisons with a Inverse Kinematics in simulation
  
  
  Next, we present a comparison with a structured model-based hierarchical control scheme, where the high-level controller plans over desired CoM velocities, which are converted into desired footsteps using an expert-designed feedback law, and followed using IK, as proposed in <cit.>. Similar control schemes have been successful at solving a large range of locomotion tasks <cit.>. This is an easy change for our setup, where the low-level policy π_θ is replaced by IK, and the desired CoM velocities works as an `oracle' high level latent space. 
  A high-level model-based planner uses a learned CoM transition dynamics model, similar to Section <ref>, with desired CoM velocities as the `latent' action. IK represents the best-case low-level policy for a fixed action space, and hence makes a highly challenging comparison for our approach. Unlike IK, our approach does not use kinematic knowledge of the robot, and can outperform IK when kinematic assumptions are violated. 
  
  
  With this in mind, we run comparison experiments with IK in a normal setting, as well as, an adverse setting where two hind-legs are disabled by fixing their joint angles in simulation. The high-level CoM dynamics model is re-learned for both LAT and IK with broken legs, but the low-level policies are the same as those of the normal setting. The results are summarized in Figure <ref> and <ref> for Daisy and A1 simulations, and show that our fully-learned approach is close to IK in the normal setting. However, when the two hind-legs are disabled, IK's performance deteriorates significantly to 0.91 ± 0.06 at velocity tracking on A1, while our learned latent space compensates for this adverse setting, incurring a cost of 0.62 ± 0.06. This experiment highlights that a learned action space adds additional robustness to hierarchical control, including when the robot is damaged and other adverse scenarios.
  
  
  
  
  
  
  
  
  
   §.§ Hardware experiments on Daisy Hexapod
  
  
  
  Transfer to hardware is straight-forward for our approach, once the latent action space and low-level policy have been learned in simulation. We keep the low-level policy and latent action space from simulation fixed on hardware, and learn the CoM dynamics by randomly sampling latent actions on the Daisy robot hardware (Figure <ref>). Once the dynamics models is learned, it is kept fixed, as we conduct hardware experiments, which span multiple days. Since our hardware experiments are conducted outdoors, in an uncontrolled environment, they naturally include noise due to tracking errors by the Realsense camera, slipping on gravel, joint tracking and ground height disturbances.
  We use 2000 hardware samples for learning the dynamics model, and 20 simulation expert controllers for learning the low-level policy. The performance of our approach on hardware is shown in Figure <ref>. The behavior of the robot during these experiments can be seen in the supplementary video. 
  For goal reaching, our approach could reach all goals in 16.0 ± 1.4 steps, averaged over 4 goals and 3 trials. The velocity tracking error on hardware was 0.56 ± 0.015 and trajectory tracking error was 0.57 ± 0.05 (Figure <ref>).
  
  Our approach was successfully able to track the desired trajectory, and reach desired goals on the Daisy hardware. Its performance was worse than in simulation because: 1) The number of samples used for learning dynamics on hardware were 1/5 of the samples used in simulation. 2) Learning dynamics on hardware was harder due to noise from poor joint tracking, slipping and noise in CoM estimation. 3) The robot was unable to achieve some high velocity targets, as the low-level controllers that could achieve high velocities in simulation did not transfer well to hardware.
  The first two issues can be alleviated by using more hardware samples for learning the CoM dynamics model, or starting from a simulation dynamics model and fine-tuning. The third issue was caused because the simulation does not sufficiently capture the motor bandwidth of the robot, which can be solved by using dynamics-randomization, or higher-fidelity simulators. We leave this to future work.
  
  
  
  
  
  
  
   §.§ Ablation study of hyperparameters
  
  
  Lastly, we  conduct an ablation experiment where we change the dimension of our latent space, number of expert controllers and number of samples used in training the dynamics model in Daisy simulation. We vary the latent action dimensions to be 2, 3, 4, and find that there is no significant effect on the performance. Next, we use different numbers of expert controllers – 10, 30, 50 experts – for training the low level policy. As the number of experts increases the performance on each tasks improves, though the difference from 30 to 50 is minor. This shows that a low-level policy learned with 30 experts can generalize well, and perform similar to a low-level policy with 50 experts. Lastly, we vary the number of samples used for learning the high-level dynamics, using 1000, 2000 and 5000 datapoints. We observe a significant improvement in performance when the dynamics is learned from 2000 data points versus 1000, showing that a better dynamics model leads to better performance. However, there is no significant difference in performance between 2000 and 5000 data points, signifying that our approach is not sensitive to inaccurate dynamics model, and can correct for small dynamics errors. The result are summarized in Figure <ref>, and each test setting is averaged over 10 independent trials. These results further reinforce the robustness and sample-efficiency of our proposed approach.
  
  
  
  
  
  
  § DISCUSSION AND FUTURE WORK
  
  
  
  
  
  Gait transitions in learned latent space: When the low-level policy is learned with experts spanning multiple gaits, the high-level latent action space can be used to transit between gaits, if needed by the task. We observe this on both robot simulations, as well as hardware, shown in Figure <ref>. Exploring gait transitions for achieving complex tasks is a promising extension to our current results that we aim to explore in the future.
  
  Diversity of expert gaits for low-level training: For our hardware experiments, we used 20 experts from simulation to learn the low-level policy. The richness of the expert library directly affects the performance of the low-level controller, and since some of the low-level controllers did not transfer to hardware, the performance of our approach suffered. One of the challenges of our approach is to build a sufficiently diverse low-level policy, that can transfer to hardware. A combination of curiosity-driven learning, and dynamics randomization <cit.> can be used.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § CONCLUSIONS
  
  In this paper, we present a hierarchical control framework for planning a sequence of learned latent actions from a high-level controller to a low-level policy. This framework allows us to accomplish several real-world locomotion tasks, such as goal-reaching, trajectory and velocity tracking with only 2000 samples on hardware. We present comparisons of our approach to other state-of-the-art hierarchical control approaches from literature – including both learned and expert-designed approaches, on Daisy and A1 robot simulations. Our approach outperforms learning-based baselines on all tasks and performs similar to the expert-designed approach in normal settings while outperforming it when two hind legs are disabled in simulation. This work demonstrates the efficacy of a fully-learned hierarchical framework at achieving various locomotion tasks, and solving real-world problems like navigation on two very different robot platforms. It removes the need for user-designed action spaces, and opens up avenues for further research such as discovering new gaits, enabling sim-to-real transfer without performance loss, and navigating unknown terrains autonomously. 
  
  
  
  
  
  
  IEEEtran
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  License for dataset used Gibson Database of Spaces. License at <http://svl.stanford.edu/gibson2/assets/GDS_agreement.pdf>
  
  
  
  