Preference-Based Learning for Exoskeleton Gait Optimization
Maegan Tucker Ellen Novoseller Claudia Kann Yanan Sui Yisong Yue Joel Burdick Aaron D. Ames


J0453+1559: a neutron star–white dwarf binary from a thermonuclear electron-capture supernova?
    [
    December 30, 2023
==============================================================================================

empty
empty




This paper presents a personalized gait optimization framework for lower-body exoskeletons. Rather than optimizing numerical objectives such as the mechanical cost of transport, our approach directly learns from user preferences, e.g., for comfort. Building upon work in preference-based interactive learning, we present the  algorithm.  prompts the user to give pairwise preferences between trials and suggest improvements; as exoskeleton walking is a non-intuitive behavior, users can provide preferences more easily and reliably than numerical feedback. We show that  performs competitively in simulation and demonstrate a prototype implementation of  on a lower-body exoskeleton to optimize human walking trajectory features. In the experiments,  consistently found user-preferred parameters of the exoskeleton's walking gait, which suggests that it is a promising starting point for adapting and personalizing exoskeletons (or other assistive devices) to individual users.








§ INTRODUCTION



The field of human-robot interaction is receiving increasing attention in many application domains, from mobility assistance to autonomous driving, and from education to dialog systems. In many such domains, for a robotic system to interact optimally with a human user, it must adapt to user feedback. In particular, learning from user feedback could help to improve robotic assistive devices.

This work focuses on optimizing walking gaits for a lower-body exoskeleton, Atalante, to maximize user comfort. Atalante, developed by Wandercraft <cit.>, uses 12 actuated joints to restore mobility to individuals with lower-limb mobility impairments, which could potentially benefit approximately 6.4 million people in the United States alone <cit.>. Existing work with Atalante has demonstrated dynamically-stable walking using the method of partial hybrid zero dynamics (PHZD), originally designed for bipedal robots <cit.>. While this method generates stable bipedal locomotion, there is no current framework to optimize for  comfort; yet, user comfort should be a critical objective of gait optimization for exoskeleton walking. While existing methods <cit.> can generate human-like walking gaits for bipedal robots, it is unlikely that these methods fulfill the preferences of individuals using robotic assistance.



Existing human-in-the-loop algorithms optimize quantitative metrics such as metabolic expenditure <cit.>; however, since the goal of this work is to optimize for user comfort, the presented learning approach uses user preferences obtained from sequential gait trials. By directly incorporating personalized feedback, we avoid making overly-strong assumptions about gait preference, or optimizing for a numerical quantity not aligned to personalized comfort.


For exoskeleton gait generation, as in many real-world settings involving people <cit.>, it is challenging for people to reliably specify numerical scores or provide demonstrations. In such cases, the users' relative preferences measure their comfort more reliably. Previous studies have found preferences to be more reliable than numerical scores in a range of domains, including information retrieval <cit.> and autonomous driving <cit.>.




Building upon techniques from  dueling bandits <cit.> and coactive learning <cit.>, we propose the  algorithm to learn user-preferred exoskeleton gaits.  is a mixed-initiative approach, which both queries the user for preferences and allows the user to suggest improvements. 



We also validate  in simulation and human experiments, in which  finds user-preferred gaits within a gait library. This procedure not only identifies users' preferred walking trajectories, but also provides insights into the users' preferences for certain gaits.
            


§ GAIT GENERATION FOR BIPEDAL ROBOTS


Many existing lower-body exoskeletons either require the use of arm-crutches <cit.> or use slow static gaits with speeds around 0.05 m/s <cit.>. Using the PHZD method, dynamic crutchless exoskeleton walking has been demonstrated to generate dynamically-stable gaits. We briefly explain this method to illustrate how it can be adapted based on user preferences; for more details, refer to <cit.>.

Partial Hybrid Zero Dynamics Method Systems with impulse effects, such as ground impacts, can be represented as hybrid control systems <cit.>. Summarizing from <cit.>, the natural system dynamics can then be represented on an invariant reduced-dimensional surface, termed the zero dynamics surface <cit.>, by appropriately defining the virtual constraints and using a feedback-linearizing controller to drive them to zero. Since the exoskeleton's desired forward hip velocity is constant and its actual velocity experiences a jump at impact, the partial zero dynamics surface is considered. The virtual constraints are defined as the difference between the actual and desired outputs:

    y_1(q,q̇)    = y_1^a(q,q̇) - v_d 
     
        y_2(q, α)    = y_2^a(q) - y_2^d(τ(q), α) ,

where the actual outputs y_1^a and y_2^a are velocity-regulating and position-modulating terms, respectively. The output y_1^a is driven to a constant desired velocity v_d, while y_2^a is driven to a vector of desired trajectories, y_2^d. The trajectories y_2^d are represented using a Bézier polynomial with coefficients α and state-based timing variable τ(q). 

According to Theorem 2 in <cit.>, if there exist virtual constraints that yield an impact-invariant periodic orbit on the partial zero dynamics surface, then these outputs, when properly controlled on the exoskeleton, yield stable periodic walking. The orbit is impact-invariant if it returns to the partial zero dynamics surface 𝒫𝒵_α after an impact event. To find the polynomials α that yield an impact-invariant periodic orbit on the reduced-order manifold, we formulate an optimization problem of the form: 

    α^∗ = αargmin       𝒥(α), 
    s.t.       Δ(𝒮∩𝒫𝒵_α) ⊂𝒫𝒵_α, 
       𝒲_ix ≤ b_i,

where 𝒥(α) is a user-determined cost, (<ref>) is the impact invariance condition, (<ref>) are other physical constraints, 𝒮 is the guard defining the conditions under which impulsive behavior occurs, and Δ is the reset map governing the system's dynamical response to hitting the guard. 

The optimization in (<ref>)-(<ref>) produces a gait that can be altered by varying the cost function 𝒥(α) and/or adding physical constraints. In bipedal walking, this cost is frequently the mechanical cost of transport (COT) defined by Eqs. (17)-(18) in <cit.>. To create the desired motion, one must add physical constraints such as step length and foot height.


as:

    𝒥 := 1/mgl_step∫_t+^t_-∑_j=1^m ||u_j(t)·q̇_j(t)||dt

where t_+ and t_- are the initial and final time of the gait, mg is the weight of the robot, l_step is the step length of one gait cycle, and P(q̇(t),u(t)) is the 2-norm sum of the mechanical power with u_j(t) and q̇_j(t) is the computed torque and joint velocity of each actuated joint j. 


Gait Generation Applied to Lower-Body Exoskeletons To translate gait generation to lower-body exoskeletons, one must choose the optimization cost function and physical constraints to obtain user-preferred gaits. While it is possible to optimize generated gaits for mechanical properties such as COT, there is currently no well-understood relationship between the parameters of the optimization problem and user preferences. Additionally, due to the time-consuming nature of gait generation—both the time required to tune the optimization problem's constraints and the time required to run the program—the issue of generating human-preferred dynamically-stable walking gaits remains largely unexplored.

Gait Library It has become increasingly common to pre-compute a set of nominal walking gaits over a grid of various parameters <cit.>. These pre-computed gaits are combined to form a “gait library,” through which gaits can be selected and executed immediately. For the purpose of exoskeleton walking, a gait library allows the operator to select a gait that is comfortable for the patient; however, it is not yet clear how to select an appropriate walking gait to optimize user comfort and preference. Thus, we consider learning from the user's preferences, as discussed below.            


§ PREFERENCE-BASED LEARNING ALGORITHM


We leverage preference-based learning (e.g., does the user prefer gait A over gait B?) to determine the gait parameters most preferred by the user <cit.>, since preference feedback has been shown to be much more reliable than absolute feedback when learning from subjective human responses <cit.>.  Thus, our goal to personalize the exoskeleton's gait can be framed as dueling bandit  <cit.> and coactive learning <cit.> problems.




Our work builds upon the Self-Sparring algorithm, a Bayesian dueling bandits approach that enjoys both competitive theoretical convergence guarantees and empirical performance <cit.>. 

Self-Sparring learns a Bayesian posterior over each action's utility to the user and draws multiple samples from the model's posterior to “duel” or “spar” via preference elicitation. The Self-Sparring algorithm iteratively: a) draws multiple samples from the posterior model of the actions' utilities; b) for each sampled model, executes the action with the highest sampled utility; c) queries for preference feedback between the executed actions; and d) updates the posterior according to the acquired preference data.

To collect more feedback beyond just one bit per preference, we also 



allow the user to suggest improvements during their trials. This approach resembles the coactive learning framework <cit.>, 

in which the user identifies an improved action as feedback to each presented action. Coactive learning has been applied to robot trajectory planning <cit.>, but has not, to our knowledge, yet been applied to robotic gait generation or in concert with preference learning. 



The  Algorithm To optimize an exoskeleton's gait within the gait library (Section <ref>), we propose the  algorithm, a mixed-initiative learning approach <cit.> which extends the Self-Sparring algorithm to incorporate coactive feedback. Similarly to Self-Sparring,  maintains a Bayesian preference relation function over the possible actions, which is fitted to observed preference feedback. 

 updates this model with user feedback and uses it to select actions for new trials and to elicit feedback. We first define the Bayesian preference model, and then detail the steps of Algorithm <ref>.


Modeling Utilities from Preference Data 

We adopt the preference-based Gaussian process model of <cit.>.
Let 𝒜⊂ℝ^d be the finite set of available actions with cardinality A = |𝒜|. 
At any point in time,  has collected a preference feedback dataset D = {x_k1≻x_k2  |   k = 1, …, N} consisting of N preferences, where x_k1≻x_k2 indicates that the user prefers action x_k1∈𝒜 to action x_k2∈𝒜 in preference k. Furthermore, we assume each action x_i has a latent, underlying utility to the user, f(x_i). For finite action spaces, the utilities can be written in vector form: f := [f(x_1), f(x_2), …, f(x_A)]^T. Given preference data D, we are interested in the posterior probability of f:

    P(f | D) ∝ P(D | f)P(f).

We define a Gaussian prior over f:


    P(f) = 1/(2π)^A/2 |Σ|^1/2exp(-1/2f^T Σ^-1f),

where Σ∈ℝ^A × A, [Σ]_ij = 𝒦(x_i, x_j), and 𝒦 is a kernel, for instance the squared exponential kernel.

For computing the likelihood P(D | f), we assume  feedback may be corrupted by i.i.d. Gaussian noise:  when presented with action x_i, the user determines  her internal valuation y(x_i) = f(x_i) + ε_i, where ε_i ∼𝒩(0, σ^2). Then,

P(x_k1 ≻x_k2 | f)    = P(y(x_k1) > y(x_k2) | f(x_k1), f(x_k2)) 
    = Φ[f(x_k1) - f(x_k2)/√(2)σ ],

where Φ is the standard normal cumulative distribution function, and y(x_kj) = f(x_kj) + ε_kj, j ∈{1, 2}. Thus, the full expression for the likelihood is:

    P(D | f) = ∏_k = 1^N Φ[f(x_k1) - f(x_k2)/√(2)σ].


The posterior P(f | D) can be estimated via the Laplace approximation as a multivariate Gaussian distribution; see <cit.> for details. Finally, in formulating the posterior, preferences can be weighted relatively to one another if some are thought to be noisier than others. This is accomplished by changing σ to σ_k in (<ref>) to model differing values of the preference noise parameter among the data points, and is analogous to weighted Gaussian process regression <cit.>.



The Learning Algorithm Let (Σ, σ) represent the prior parameters of the Bayesian preference model, as outlined above. From these parameters, one obtains the prior mean and covariance, (μ_0, Σ_0) (Line <ref> in Alg. <ref>). In each iteration,  updates the utility model (Line <ref>) via the Laplace approximation to the posterior in (<ref>) to obtain 𝒩(μ_t, Σ_t).




To select actions in the tth iteration (Lines <ref>-<ref>), the algorithm first draws n samples from the posterior, 𝒩(μ_t - 1, Σ_t - 1). Each of these is a utility function f_j, giving a utility value for each action in 𝒜. The corresponding selected action is simply the one maximizing f_j (Line <ref>).

The n actions are executed (Line <ref>), and the user provides pairwise preference feedback between pairs of actions (the user can always state “no preference”). We extend Self-Sparring <cit.> to extract more preference comparisons from the available trials by assuming that the user can remember the b actions preceding the current n actions. The user thus provides preferences between any combination of the current n actions and the previous b actions. 
For instance, for n = 1, b > 0, one can interpret b as a buffer of previous trials that the user remembers. 
For n = b = 1, the user can report preferences between any pair of two consecutive trials, i.e., the user is asked, “Did you like this trial more or less than the last trial?” 



We expect that setting n = 1 while increasing b to as many trials as the user can accurately remember would minimize the trials required to reach a preferred gait. In Line <ref>, the pairwise preferences from iteration t form a matrix R ∈ℝ^n × (n + b), where r_jk∈{0, 1, ∅}; the values 0 and 1 express preference information, while ∅ denotes the lack of a preference between the actions concerned. 

Finally, the user can suggest improvements in the form of coactive feedback (Line <ref>). For example, the user could request a longer or shorter step length. In Line <ref>, ∅ indicates that no coactive feedback was provided. Otherwise, the user's suggestion is appended to the data D as preferred to the previously-tested action. In learning the model posterior, one can assign the coactive preferences a smaller weight relative to pairwise preferences via the input parameter β > 0.


















      


§ SIMULATION RESULTS


The performance of  is evaluated in two sets of simulations: (1) the compass-gait (CG) biped's COT,[Bayesian model's kernel: squared exponential with lengthscale = 0.025, signal variance = 0.0001, noise variance = 1e-8; preference noise (σ) = 0.01] and (2) a set of synthetic optimization objective functions.[Kernel: squared exponential with lengthscale = [0.15, 0.15], signal variance = 0.0001, noise variance = 1e-5; preference noise (σ) = 0.01] In both cases,  efficiently converges to the optimum.

Optimizing the Compass-Gait Biped's Cost of Transport We first evaluate our approach with a simulated CG biped, optimizing its COT over the step length via preference feedback (Fig. <ref>). Preferences are determined by comparing COT values, calculated by simulating gaits for multiple step lengths, each at a fixed forward hip velocity of 0.2 m/s. These simulated gaits were synthesized via a single-point shooting partial hybrid zero dynamics method <cit.>.



We use  to minimize the one-dimensional objective function in Fig. <ref>, using pairwise preferences obtained by comparing COT values for the different step lengths. Here, we use   with n = 2, b = 0, and without coactive feedback. Note that without a buffer or coactive feedback,  reduces to  Self-Sparring <cit.>. At each iteration, two new samples are drawn from the Bayesian posterior, and the resultant two step lengths are compared to elicit a preference. Using the new preferences,  updates its posterior over the utility of each step length.

Fig. <ref> depicts the evolution of the posterior preference model, where each iteration corresponds to a preference between two new trials. With more preference data, the posterior utility increasingly peaks at the point of lowest COT.  These results suggest that  can efficiently identify high-utility actions from preference feedback alone.








Optimizing Synthetic Two-Dimensional Functions We next test   on synthetic 2D utility functions, such as the one shown in Fig. <ref>. Each utility function was generated from a Gaussian process prior on a 30-by-30 grid.  These experiments evaluate the potential to scale  to higher dimensions and the advantages of coactive feedback.

We compare three settings for 's (n,b) parameters: (2,0), (3,0), (1,1) as explained in Sec. <ref>. For each setting—as well as with and without coactive feedback—we simulate  on each of the 100 random objective functions. In each case, the number of objective function evaluations, or experimental trials, was held constant at 150.




Coactive feedback is simulated using a 2nd-order differencing approximation of the objective function's gradient. 

If  selects a point at which both gradient components have magnitudes below their respective 50th percentile thresholds, then no coactive feedback is given. Otherwise, we consider the higher-magnitude gradient component, and depending on the highest threshold that it exceeds (50th or 75th), simulate coactive feedback as either a 5% or 10% increase in the appropriate direction and dimension. 



Fig. <ref> shows the simulation results. In each case, the mixed-initiative simulations involving coactive feedback improve upon those with only preferences. Learning is slowest for n = 2, b = 0 (Fig. <ref>), since that case elicits the fewest preferences. Fig. <ref> depicts the utility model's posterior mean for the objective function in Fig. <ref>, learned in the simulation with n = 1, b = 1, and mixed-initiative feedback. In comparing Fig. <ref> to Fig. <ref>, we see that  learns a sharp peak around the optimum, as it is designed to converge to sampling preferred regions, rather than giving the user undesirable options by exploring elsewhere.
    



                  


§ HUMAN SUBJECT EXPERIMENTS


After its validation in simulation,  was deployed on a lower-body exoskeleton, Atalante, in two personalized gait optimization experiments with human subjects (video: <cit.>). Both experiments aimed to determine gait parameter values that maximize user comfort, as captured by preference and coactive feedback. 


The first experiment,[Kernel: squared exponential with lengthscale = 0.03, signal variance = 0.005, noise variance = 1e-7; preference noise (σ) = 0.02 ] repeated for three able-bodied subjects, used  to determine the user's preferred step length, i.e., optimizing over a one-dimensional feature space. The second experiment[Same parameters as in <ref> except for step duration lengthscale = 0.08 and step width lengthscale = 0.03] demonstrates 's effectiveness in two-dimensional feature spaces, and  optimizes simultaneously over two different gait feature pairs. Importantly,  operates independently of the choice of gait features. The subjects' metabolic expenditure was also recorded via direct calorimetry as shown in Fig. <ref>, but this data was uninformative of user preferences, as users are not required to expend effort toward walking.



Learning Preferences between Step Lengths In the first experiment, all three subjects walked inside the Atalante exoskeleton, with  selecting the gaits. We considered 15 equally-spaced step lengths between 0.08 and 0.18 meters, each with a precomputed gait from the gait library. Feature discretization was based on users' ability to distinguish nearby values. The users decided when to end each trial, so as to be comfortable providing feedback. Since users have difficulty remembering more than two trials at once, we used  with n = 1 and b = 1, which corresponds to asking the user to compare each current trial with the preceding one. Additionally, we query the user for coactive feedback: after each trial, the user can suggest a longer or shorter step length (±20% of the range), a slightly longer or shorter step length (±10%), or no feedback. 

Each participant completed 20 gait trials, providing preference and coactive feedback after each trial. Fig. <ref> illustrates the posterior's evolution over the experiment. After only five exoskeleton trials,  was already able to identify a relatively-compact preferred step length subregion. After the 20 trials, three points along the utility model's posterior mean were selected: the maximum, mean, and  minimum. The user walked in the exoskeleton with each of these step lengths in a randomized ordering, and gave a blind ranking of the three, as shown in Fig. <ref>. For each subject, the blind rankings match the preference posterior obtained by , indicating effective learning of individual user preferences.

Learning Preferences over Multiple Features We further demonstrate 's practicality to personalize over multiple features, by optimizing over two different feature pairs: 1) step length and step duration and 2) step length and step width. The protocol of the 1D experiment was repeated for subject 1, with step lengths discretized as before, step duration discretized into 10 equally-spaced values between 0.85 and 1.15 seconds (with 10% and 20% modifications under coactive feedback), and step width into 6 values between 0.25 and 0.30 meters (20% and 40%). After each trial, the user was queried for both a pairwise preference and coactive feedback. Fig. <ref> shows the results for both feature spaces. The estimated preference values were consistent with a 3-sample blind ranking evaluation, suggesting that  successfully identified user-preferred parameters. Fig. <ref> displays phase diagrams of the gaits with minimum, mean, and maximum posterior utility values to illustrate the difference between preferred and non-preferred gaits.                  


§ CONCLUSIONS


This work develops and demonstrates (video: <cit.>) the  interactive learning framework for optimizing gaits with respect to user comfort, using human preferences as feedback. We demonstrate the algorithm in simulation, showing that it efficiently learns to select optimal actions. We next apply  in a user study with the Atalante lower-body exoskeleton, demonstrating the first application of preference-based learning for optimizing dynamic crutchless walking.  successfully models the users' preferences, identifying compact subregions of preferred gaits.

In the future, we plan to apply  toward optimizing over larger sets of gait parameters; this will likely require integrating the algorithm with techniques for learning over high-dimensional feature spaces <cit.>. The method could also be extended beyond working with precomputed gait libraries to generating entirely new gaits  or controller designs (e.g., via preference-based reinforcement learning <cit.>).              






§ ACKNOWLEDGMENTS

The authors would like to thank the volunteers who participated in the experiments, as well as the entire Wandercraft team that designed Atalante and continues to provide technical support for this project.

IEEEtran


