Style-based inverse kinematics.
This paper presents an inverse kinematics system based on a learned model of human poses. Given a set of constraints, our system can produce the most likely pose satisfying those constraints, in real-time. Training the model on different input data leads to different styles of IK. The model is represented as a probability distribution over the space of all possible poses. This means that our IK system can generate any pose, but prefers poses that are most similar to the space of poses in the training data. We represent the probability with a novel model called a Scaled Gaussian Process Latent Variable Model. The parameters of the model are all learned automatically; no manual tuning is required for the learning component of the system. We additionally describe a novel procedure for interpolating between styles.Our style-based IK can replace conventional IK, wherever it is used in computer animation and computer vision. We demonstrate our system in the context of a number of applications: interactive character posing, trajectory keyframing, real-time motion capture with missing markers, and posing from a 2D image.
Overview

The main idea of our work is to learn a probability distribution function
(PDF) over character poses from motion data, and then use
this to select new poses during IK. We represent each pose with
a 42-dimensional vector q, which consists of joint angles, and the
position and orientation of the root of the kinematic chain. Our
approach consists of the following steps:
Feature vectors. In order to provide meaningful features for
IK, we convert each pose vector to a feature representation y that
represents the character pose and velocity in a local coordinate
frame. Each motion capture pose qi has a corresponding feature
vector yi, where i is an index over the training poses. These features
include joint angles, velocity, and vertical orientation, and are
described in detail in Section 4.
SGPLVM learning. We model the likelihood of motion capture
poses using a novel model called a Scaled Gaussian Process Latent
Variable Model (SGPLVM). Given the features {yi} a set of motion
capture poses, we learn the parameters of an SGPLVM, as described
in Section 5. The SGPLVM de?nes a low-dimensional representation
of the original data: every pose qi has a corresponding vector
xi, usually in a 3-dimensional space. The low-dimensional space
of xi values is called the latent space. In the learning process, we
estimate the {xi} parameters for each input pose, along with the
parameters of the SGPLVM model (denoted ?, ?, ?, and {wk}).
This learning process entails numerical optimization of an objective
function LGP. The likelihood of new poses is then described by
the original poses and the model parameters. In order to keep the
model ef?cient, the algorithm selects a subset of the original poses
to keep, called the active set.
Pose synthesis. To generate new poses, we optimize an objective
function LIK(x,y(q)), which is derived from the SGPLVM
model. This function describes the likelihood of new poses, given
the original poses and the learned model parameters. For each new
pose, we also optimize the low-dimensional vector x. Several different
applications are supported, as described in Section 7.
4 Character model
In this section, we de?ne the parameterization we use for charac-
ters, as well as the features that we use for learning. We describe
the 3D pose of a character with a vector q that consists of the global
position and orientation of the root of the kinematic chain, plus all
of the joint angles in the body. The root orientation is represented
as a quaternion, and the joint angles are represented as exponential
maps. The joint parameterizations are rotated so that the space of
natural motions does not include singularities in the parameterization

For each pose, we additionally de?ne a corresponding D-dimensional
feature vector y. This feature vector selects the features
of character poses that we wish the learning algorithm to be
sensitive to. This vector includes the following features:
Joint angles: All of the joint angles from q are included. We
omit the global position and orientation, as we do not want
the learning to be sensitive to them.
Vertical orientation: We include a feature that measures the
global orientation of the character with respect to the ?up di-
rection,? (along the Z-axis) de?ned as follows. Let R be a
rotation matrix that maps a vector in the character's local co-ordinate
frame to the world coordinate frame. We take the
three canonical basis vectors in the local coordinate frame,
rotate them by this matrix, and take their Z-components, to
get an estimate to the degree that the character is leaning forward
and to the side. This reduces to simply taking the third
row of R.
Velocity and acceleration: In animations, we would like the
new pose to be sensitive to the pose in the previous time frame.
Hence, we use velocity and acceleration vectors for each of
the above features. For a feature vector at time t, the velociy
and acceleration are given by yt ?yt?1 and yt ?2yt?1 +yt?2,
respectively.
The features for a pose may be computed from the current frame
and the previous frame. We write this as a function y(q). We omit
the previous frames from the notation, as they are always constant
in our applications. All vectors in this paper are column vectors.
5 Learning a model of poses
In this section, we describe the Scaled Gaussian Process Latent
Variable Model (SGPLVM), and a procedure for learning the model
parameters from training poses. The model is based on the Gaussian
Process (GP) model, which describes the mapping from x values
to y values. GPs for interpolation were introduced by O'Hagan
[1978], Neal [1996] and Williams and Rasmussen [1996]. For
a detailed tutorial on GPs, see [MacKay 1998]. We additionally
build upon the Gaussian Process Latent Variable Model, recently
poposed by Lawrence [2004]. Although the mathematical
background for GPs is somewhat involved, the implementation is
straightforward.Kernel function. Before describing the learning algorithm, we
?rst de?ne the parameters of the GP model. A GP model describes
the mapping between x values and y values: given some training
data {xi,yi}, the GP predicts the likelihood of a new y given a new
x. A key ingredient of the GP model is the de?nition of a kernel
function that measures the similarity between two points x and x in
the input space:

are the same point, and 0
otherwise, so that k(x,x)=? + ??1 and the ?x,x term vanishes
whenever the similarity is measured between two distinct variables.
The kernel function tells us how correlated two data values y and
y are, based on their corresponding x and x values. The parameter
? tells us the ?spread? of the similarity function, ? tells us how
correlated pairs of points are in general, and ? tells us how much
noise there is in predictions. For a set of N input vectors {xi},we
de?ne the N ?N kernel matrix K, in which
The different data dimensions have different intrinsic scales (or,
equivalently, different levels of variance): a small change in global
rotation of the character affects the pose much more than a small
change in the wrist angle; similarly, orientations vary much more
than their velocities. Hence, we will need to estimate a separate
scaling wk for each dimension. This scaling is collected in a diagonal
this matrix is used to rescale
features as Wy.
Learning. We now describe the process of learning an SG-
PLVM, from a set of N training data points {yi}. We ?rst compute
the mean of the training set: We then collect the k-th
component of every feature vector into a vector Yk and subtract
the means (so that Yk =[y1,k ? ?k,.,yN,k ? ?k]T ). The SGPLVM
model parameters are learned by minimizing the following objective
with respect to the unknowns {xi},?,?,? and {wk}. This objective
function is derived from the Gaussian Process model (Appendix
A). Formally, LGP is the negative log-posterior of the model pa-
rameters. Once we have optimized these parameters, the SGPLVM
provides a likelihood function for use in real-time IK, based on the
training data and the model parameters.
Intuitively, minimizing this objective function arranges the xi
values in the latent space so that similar poses are nearby and the
dissimilar poses are far apart, and learns the smoothness of the
space of poses. More generally, we are trying to adjust all unknown
parameters so that the kernel matrix K matches the correlations
in the original y's (Appendix A). Learning in the SGPLVM
model generalizes conventional PCA [Lawrence 2004], which corresponds
to ?xing using a linear kernel. As
described below, the SGPLVM also generalizes Radial Basis Function
(RBF) interpolation, providing a method for learning all RBF
parameters and for constrained pose optimization.
The simplest way to minimize LGP is with numerical optimization
methods such as L-BFGS [Nocedal and Wright 1999]. How-
ever, in order for the real-time system to be ef?cient, we would
like to discard some of the training data; the training points that
are kept are called the active set. Once we have optimized the un-
knowns, we use a heuristic [Lawrence et al. 2003] to determine
the active set. Moreover, the optimization itself may be inef?cient
for large datasets, and so we use a heuristic optimization based on
Lawrence's [2004] in order to ef?ciently learn the model parameters
and to select the active set. This algorithm alternates between

Figure

1: SGPLVM latent spaces learned from different motion capture sequences: a walk cycle, a jump shot, and a baseball pitch. Points:
The learning process estimates a 2D position x associated with every training pose; plus signs (+) indicate positions of the original training
points in the 2D space. Red points indicate training poses included in the training set. Poses: Some of the original poses are shown along
with the plots, connected to their 2D positions by orange lines. Additionally, some novel poses are shown, connected by green lines to their
positions in the 2D plot. Note that the new poses extrapolate from the original poses in a sensible way, and that the original poses have been
arranged so that similar poses are nearby in the 2D space. Likelihood plot: The grayscale plot visualizes ?D ln?2(x) ? 1 ||x||2 for each
position x. This component of the inverse kinematics likelihood LIK measures how ?good? x is. Observe that points are more likely if they
lie near or between similar training poses.
optimizing the model parameters, optimizing the latent variables,
and selecting the active set. These algorithms and their tradeoffs are
described in Appendix B. We require that the user specify the size
M of the active set, although this could also be speci?ed in terms
of an error tolerance. Choosing a larger active set yields a better
model, whereas a smaller active set will lead to faster performance
during both learning and synthesis.
New poses. Once the parameters have been learned, we have a
general-purpose probability distribution for new poses. The objective
function for a new pose parameterized by x and y is:
where
and K is the kernel matrix for the active set, Y =[y1 ? ?,.,yM ?
?]T is the matrix of active set points (mean-subtracted), and k(x) is
a vector in which the i-th entry contains k(x,xi), i.e., the similarity
between x and the i-th point in the active set. The vector f(x) is the
pose that the model would predict for a given x; this is equivalent to
RBF interpolation of the training poses. The variance ?2(x) indicates
the uncertainty of this prediction; the certainty is greatest near
the training data. The derivation of LIK is given in Appendix A.
The objective function LIK can be interpreted as follows. Optimization
of a (x,y) pair tries to simultaneously keep the y close
to the corresponding prediction f(x) (due to the ||W(y ? f(x))||2
while keeping the x value close to the training data (due tothe ln?2(x) term), since this is where the prediction is most reli-
able. The 1 ||x||2 term has very little effect on this process, and isincluded mainly for consistency with learning.
6 Pose synthesis
We now describe novel algorithms for performing IK with SG-
PLVMs. Given a set of motion capture poses {qi}, we compute the
corresponding feature vectors yi (as described in Section 4), and
then learn an SGPLVM from them as described in the previous sec-
tion. Learning gives us a latent space coordinate xi for each pose yi,
as well as the parameters of the SGPLVM (?, ?, ?, and {wk}). In

Figure

1, we show SGPLVM likelihood functions learned from different
training sequences. These visualizations illustrate the power
of the SGPLVM to learn a good arrangement of the training poses
in the latent space, while also learning a smooth likelihood function
near the spaces occupied by the data. Note that the PDF is not
simply a matter of, for example, Gaussian distributions centered at
each training data point, since the spaces inbetween data points are
more likely than spaces equidistant but outside of the training data.
The objective function is smooth but multimodal.
Over?tting is a signi?cant problem for many popular PDF mod-
els, particularly for small datasets without redundancy (such as the
ones shown here). The SGPLVM avoids over?tting and yields
smooth objective functions both for large and for small data sets
(the technical reason for this is that it marginalizes over the space
of model representations [MacKay 1998], which properly takes into
account uncertainty in the model). In Figure 2, we compare with another
common PDF model, a mixtures-of-Gaussians (MoG) model
[Bishop 1995; Redner and Walker 1984], which exhibits problems
with both over?tting and local minima during learning1. In addi-
1The MoG model is similar to what has been used previously for learning
in motion capture. Roughly speaking, both the SHMM [Brand and Hertzmann
2000] and SLDS [Li et al. 2002] reduce to MoGs in synthesis, if we
Gaussian components Log-likelihood

Figure

2: Mixtures-of-Gaussians (MoG). We applied conventional
PCA to reduce the baseball pitch data to 2D, then ?t an MoG model
with EM. Although it assigns highest probability near the data set,
the log-likelihood exhibits a number of undesirable artifacts, such
as long-and-skinny Gaussians which assign very high probabilities
to very small regions and create a very bumpy objective function.
In contrast, the likelihood functions shown in Figure 1 are much
smoother and more appropriate for the data. In general, we ?nd
that 10D PCA is required to yield a reasonable model, and MoG
artifacts are much worse in higher dimensions.
tion, using an MoG requires dimension reduction (such as PCA) as
a preprocess, both of which have parameters that need to be tuned.
There are principled ways to estimate these parameters, but they are
dif?cult to work with in practice. We have been able to get reason-able
results using MoGs on small data-sets, but only with the help
of heuristics and manual tweaking of model parameters.
6.1 Synthesis
New poses q are created by optimizing LIK(x,y(q)) with respect to
the unknowns x and q. Examples of learned models are illustrated
in

Figure

1. There are a number of different scenarios for synthesizing
poses; we ?rst describe these cases and how to state them as
optimization problems. Optimization techniques are described in
Section 6.2.
The general setting for pose synthesis is to optimize q given
some constraints. In order to get a good estimate for q, we also
must estimate an associated x. The general problem statement is:
s.t. C(q)=0 (8)
for some constraints C(q)=0.
The most common case is when only a set of handle constraints
C(q)=0 are speci?ed; these handle constraints may come from a
user in an interactive session, or from a mocap system.
Our system also provides a 2D visualization of the latent space,
and allows the user to drag the mouse in this window, in order to
view the space of poses in this model. Each point in the window
corresponds to a speci?c value of x; we compute the corresponding
pose by maximizing LIK with respect to q. A third case occurs when
the user speci?es handle constraints and then drags the mouse in
the latent space. In this case, q is optimized during dragging. This
provides an alternative way for the user to ?nd a point in the space
that works well with the given constraints.
6.1.1 Model smoothing
Our method produces an objective function that is, locally, very
smooth, and thus well-suited for local optimization methods. How-
view a single frame of a sequence in isolation. The SHMM's entropic prior
helps smooth the model, but at the expense of overly-smooth motions.
Figure

3: Annealing SGPLVMs. Top row: The left-most plot shows
the ?unannealed? original model, trained on the baseball pitch. The
plot on the right shows the model retrained with noisy data. The
middle plot shows an interpolation between the parameters of the
outer models. Bottom row: The same plots visualized in 3D.
ever, distributions over likely poses must necessarily have many
local minima, and a gradient-based numerical optimizer can easily
get trapped in a poor minima when optimizing LIK.Wenow
describe a new procedure for smoothing an SGPLVM model that
can be used in an annealing-like procedure, in which we search in
smoother versions of the model before the ?nal optimization. Given
training data and a learned SGPLVM, our goal is to create smoothed
(or ?annealed?) versions of this SGPLVM. We have found that the
simplest annealing strategy of scaling the individual model parameters
(for example, halving the value of ?) does not work well, since
the scales of the three ?, ?, and ? parameters are closely intertwined

Instead, we use the following strategy to produce a smoother
model. We ?rst learn a normal (unannealed) SGPLVM as described
in Section 5. We then create a noisy version of the training set, by
adding zero-mean Gaussian noise to all of the {yi} values in the
active set. We then learn new values ?, ?, and ? using the same
algorithm as before, but while holding {xi} and {wk} ?xed. This
gives us new ?annealed? parameters ?, ?, ?. The variance of the
noise added to data determines how smooth the model becomes.
Given this annealed model, we can generate a range of models by
linear interpolation between the parameters of the normal SGPLVM
and the annealed SGPLVM. An example of this range of annealed
models is shown in Figure 3.
6.2 Real-time optimization algorithm
Our system optimizes LIK using gradient-based optimization meth-
ods; we have experimented with Sequential Quadratic Programming
(SQP) and L-BFGS [Nocedal and Wright 1999]. SQP allows
the use of hard constraints on the pose. However, hard constraints
can only be used for underconstrained IK, otherwise the system
quickly becomes infeasible and the solver fails. The more general
solution we use is to convert the constraints into soft constraints by
adding a term ||C(q)||2 to the objective function with a large weight.
A more desirable approach would be to enforce hard constraints as
much as possible, but convert some constraints to soft constraints
when necessary [Yamane and Nakamura 2003].
Because the LIK objective is rarely unimodal, we use an
annealing-like scheme to prevent the pose synthesis algorithm from
getting stuck in local minima. During the learning phase, we pre-compute
an annealed model as described in the previous section. In
our tests, we set the noise variance to .05 for smaller data sets and
for larger data sets. During synthesis, we ?rst run a few steps of
optimization using the smoothed model (?, ?, ?), as described in
the previous section. We then run additional steps on an intermediate
model, with parameters interpolated as ?1 ? +(1 ? ?1 )?.
The same interpolation is applied to ? and ?. We then ?nish the
optimization with respect to the original model (?, ?, ?). During
interactive editing, there may not be enough time to fully optimize
between dragging steps, in which case the optimization is only up-dated
with respect to the smoothest model; in this case, the ?ner
models are only used when dragging stops.
6.3 Style interpolation
We now describe a simple new approach to interpolating between
two styles represented by SGPLVMs. Our goal is to generate a new
style-speci?c SGPLVM that interpolates two existing SGPLVMs
LIK0 and LIK1. Given an interpolation parameter s, the new objective
function is:
Generating new poses entails optimizing Ls with respect to the pose
q as well a latent variables x0 and x1 (one for each of the original
styles).
We can place this interpolation scheme in the context of the following
novel method for interpolating style-speci?c PDFs. Given
two or more pose styles ? represented by PDFs over possible poses
our goal is to produce a new PDF representing a style that is ?in
between? the input poses. Given two PDFs over poses p(y|?0) and
p(y|?1), where ?0 and ?1 describe the parameters of these styles,
and an interpolation parameter s, we form the interpolated style
PDF as
New poses are created by maximizing ps(y(q)). In the SGPLVM
case, we have ln p(y|?0)=?LIK0 and ln p(y|?0)=?LIK1.We
discuss the motivation for this approach in Appendix C.
Applications
In order to explore the effectiveness of the style-based IK, we tested
it on a few applications: interactive character posing, trajectory
realtime motion capture with missing markers, and determining
human pose from 2D image correspondences. Examples
of all these applications are shown in the accompanying video.
7.1 Interactive character posing
One of the most basic ? and powerful ? applications of our system
is for interactive character posing, in which an animator can
interactively de?ne a character pose by moving handle constraints
in real-time. In our experience, posing this way is substantially
faster and more intuitive than posing without an objective function.
7.2 Trajectory keyframing
We developed a test animation system aimed at rapid-prototyping
of character animations. In this system, the animator creates an animation
by constraining a small set of points on the character. Each
constrained point is controlled by modifying a trajectory curve. The
animation is played back in realtime so that the animator can immediately
view the effects of path modi?cations on the resulting
motion. Since the animator constrains only a minimal set of points,
the rest of the pose for each time frame is automatically synthesized
using style-based IK. The user can use different styles for different
Figure

4: Trajectory keyframing, using a style learned from the
baseball pitch data. Top row: A baseball pitch. Bottom row: A
side-arm pitch. In each case, the feet and one arm were keyframed;
no other constraints were used. The side-arm contains poses very
different from those in the original data.
parts of the animation, by smoothly blending from one style to an-
other. An example of creating a motion by keyframing is shown in

Figure

4, using three keyframed markers.
7.3 Real-time motion capture with missing marker

In optical motion capture systems, the tracked markers often disappear
due to occlusion, resulting in inaccurate reconstructions and
noticeable glitches. Existing joint reconstruction methods quickly
fail if several markers go missing, or they are missing for an extended
period of time. Furthermore, once the a set of missing markers
reappears, it is hard to relabel each one of them so that they
correspond to the correct points on the body.
We designed a real-time motion reconstruction system based on
style-based IK that ?lls in missing markers. We learn the style from
the initial portion of the motion capture sequence, and use that style
to estimate the character pose. In our experiments, this approach
can faithfully reconstruct poses even with more than 50% of the
markers missing.
We expect that our method could be used to provide a metric
for marker matching as well. Of course, the effectiveness of style-based
degrades if the new motion diverges from the learned
style. This could potentially be addressed by incrementally relearning
the style as the new pose samples are processed.
7.4 Posing from 2D images
We can also use our IK system to reconstruct the most likely pose
from a 2D image of a person. Given a photograph of a person, a user
interactively speci?es 2D projections (i.e., image coordinates) of a
few character handles. For example, the user might specify the location
of the hands and feet. Each of these 2D positions establishes
a constraint that the selected handle project to the 2D position indicated
by the user, or, in other words, that the 3D handle lie on the
line containing the camera center and the projected position. The
3D pose is then estimated by minimizing LIK subject to these 2D
constraints. With only three or four established correspondences
between the 2D image points and character handles, we can reconstruct
the most likely pose; with a little additional effort, the pose
can be ?ne-tuned. Several examples are shown in Figure 5. In
the baseball example (bottom row of the ?gure) the system obtains
a plausible pose from six projection constraints, but the depth of
from having any feasible solution.
There are many possible improvements to the SGPLVM learning
algorithm, such as experimenting with other kernels, or selecting
automatically based on the data set. Additionally, the
current optimization algorithm employs some heuristics for convenience
and speed; it would be desirable to have a more principled
and ef?cient method for optimization. We ?nd that the annealing
heuristic for real-time synthesis requires some tuning, and it would
be desirable to ?nd a better procedure for real-time optimization.

Acknowledgements

Frontal view Side view

Figure

5: 3D posing from a 2D image. Yellow circles in the front
view correspond to user-placed 2D constraints; these 2D constraints
appear as ?line constraints? from a side view.
the right hand does not match the image. This could be ?xed by
one more constraint, e.g., from another viewpoint or from temporal
coherence.
8 Discussion and future work
We have presented an inverse kinematics system based on a learned
probability model of human poses. Given a set of arbitrary algebraic
constraints, our system can produce the most likely pose satisfying
those constraints, in real-time. We demonstrated this system
in the context of several applications, and we expect that style-based
IK can be used effectively for any problem where it is necessary to
restrict the space of valid poses, including problems in computer
vision as well as animation. For example, the SGPLVM could be
used as a replacement for PCA and for RBFs in example-based animation
methods.
Additionally, there are a number of potential applications for
games, in which it is necessary that the motions of character both
look realistic and satisfy very speci?c constraints (e.g., catching a
ball or reaching a base) in real-time. This would require not only
real-time posing, but, potentially, some sort of planning ahead. We
are encouraged by the fact that a leading game developer licensed
an early version of our system for the purpose of rapid content development

There are some limitations in our system that could be addressed
in future work. For example, our system does not model dynam-
ics, and does not take into account the constraints that produced the
original motion capture. It would also be interesting to incorporate
style-based IK more closely into an animation pipeline. For ex-
ample, our approach may be thought of as automating the process
of ?rigging,? i.e., determining high-level controls for a character.
In a production environment, a rigging designer might want to design
some of the character controls in a speci?c way, while using
an automatic procedure for other controls. It would also be useful
to have a more principled method for balancing hard and soft
constraints in real-time, perhaps similar to [Yamane and Nakamura
2003], because too many hard constraints can prevent the problemMany thanks to Neil Lawrence for detailed discussions and for placing
his source code online. We are indebted to Colin Zheng for
creating the 2D posing application, and to Jia-Chu Wu for for last-minute
image and video production. David Hsu and Eugene Hsu
implemented the ?rst prototypes of this system. This work was supported
in part by UW Animation Research Labs, NSF grants EIA-
0121326, CCR-0092970, IIS-0113007, CCR-0098005, an NSERC
Discovery Grant, the Connaught fund, Alfred P. Sloan Fellowship,
Electronic Arts, Sony, and Microsoft Research.
A Background on Gaussian Processes
In this section, we brie?y describe the likelihood function used in
this paper. Gaussian Processes (GPs) for learning were originally
developed in the context of classi?cation and regression problems
[Neal 1996; O'Hagan 1978; Williams and Rasmussen 1996]. For
detailed background on Gaussian Processes, see [MacKay 1998].
Scaled Gaussian Processes. The general setting for regression
is as follows: we are given a collection of training pairs
{xi,yi}, where each element xi and yi is a vector, and we wish to
learn a mapping Typically, this is done by least-squared
?tting of a parametric function, such as a B-spline basis or a neural
network. This ?tting procedure is sensitive to a number of important
choices, e.g., the number of basis functions and smooth-
ness/regularization assumptions; if these choices are not made care-
fully, over- or under-?tting results. However, from a Bayesian point
of view, we should never estimate a speci?c function f during re-
gression. Instead, we should marginalize over all possible choices
of f when computing new points ? in doing so, we can avoid over-
?tting and under?tting, and can additionally learn the smoothness
parameters and noise parameters. Remarkably, it turns out that,
for a wide variety of types of function f (including polynomials,
splines, single-hidden-layer neural networks, and Gaussian RBFs),
marginalization over all possible values of f yields a Gaussian Process
model of the data. For a GP model of a single output dimension
k, the likelihood of the outputs given the inputs is:
using the variables de?ned in Section 5.
In this paper, we generalize GP models to account for different
variances in the output dimensions, by introducing scaling parameters
wk for each output dimension. This is equivalent to de?ning
a separate kernel function k(x,x)/w2 for each output dimension2;
plugging this into the GP likelihood for dimension k yields:
2Alternatively, we can derive this model as a Warped GP [Snelson et al.
2004], in which the warping function rescales the features as wkYk
The complete joint likelihood of all data dimensions is
p({yi}|{xi},?,?,?,{wk})=?k p({yi,k}|{xi},?,?,?,wk).
SGPLVMs. The Scaled Gaussian Process Latent Variable Model
(SGPLVM) is a general technique for learning PDFs, based on recent
work Lawrence [2004]. Given a set of data points {yi},we
model the likelihood of these points with a scaled GP as above,
in which the corresponding values {xi} are initially unknown ?
we must now learn the xi as well as the model parameters. We
also place priors on the unknowns: p(x)=N (0;I), p(?,?,?) ?
??1??1??1.
In order to learn the SGPLVM from training data {yi},we
need to maximize the posterior p({xi},?,?,?,{wk}|{yi}). This
is equivalent to minimizing the negative log-posterior
with respect to the unknowns (constant terms have been dropped
from these expressions).
One way to interpret this objective function as follows. Suppose
we ignore the priors p(x) and p(?,?,?), and just optimize LGP
with respect to an xi value. The optima should occur when
?LGP One condition for this to occur is similarly,
this would make LGP optimal with respect to all {xi} values and the
?, ?, and ? parameters. If we solve Equation 15), we
obtain a system of equations of the form
The right-hand side of this expression will be large when the two
poses are very similar, and negative when they are very different.
This means that we try to arrange the x's so that xi and xj are nearby
if and only if yi and yj are similar. More generally, the kernel matrix
sho?uld match the covariance matrix of the original data rescaled
by W/ D. The prior terms p(x) and p(?,?,?) help prevent over-
?tting on small training sets.
Once the parameters have been learned, we have a general-purpose
probability distribution for new poses. In order to de?ne
this probability, we augment the data with a new pose (x,y),in
which one or both of (x,y) are unknown. Adding this new pose
to LGP, rearranging terms, and dropping constants yields the log-posterior
(Equation 3).
Learning algorithm
We tested two different algorithms for optimizing LGP. The ?rst
directly optimizes the objective function, and then selects an active
set (i.e., a reduced set of example poses) from the training data.
The second is a heuristic described below. Based on preliminary
tests, it appears that there are a few tradeoffs between the two al-
gorithms. The heuristic algorithm is much faster, but more tied to
the initialization for small data sets, often producing x values that
are very close to the PCA initialization. The full optimization algorithm
produces better arrangements of the latent space x values ?
especially for larger data sets ? but may require higher latent dimensionality
(3D instead of 2D in our tests). However, because the
full optimization optimizes all points, it can get by with less active
set points, making it more ef?cient at run-time. Nonetheless, both
algorithms work well, and we used the heuristic algorithm for all
examples shown in this paper and the video.Active set selection. We ?rst outline the greedy algorithm for
selecting the active set, given a learned model. The active set initially
contains one training pose. Then the algorithm repeatedly
determines which of the points not in the active set has the highest
prediction variance ?2(x) (Equation 5). This point is added to the
active set, and the algorithm repeats until there are M points in the
active set (where M is a limit predetermined by a user). For ef?-
ciency, the variances are computed incrementally as described by
Lawrence et al. [2003].
Heuristic optimization algorithm. For all examples in this
paper, we used the following procedure for optimizing LGP, based
on the one proposed by Lawrence [2004], but modi?ed3 to learn
{wk}. The algorithm alternates between updating the active set,
and the following steps: First, the algorithm optimizes the model
parameters, ?, ?, and ? by numerical optimization of LGP (Equa-
tion 2); however, LGP is modi?ed so that only the active set are
included in LGP. Next, the algorithm optimizes the latent variables
xi for points that are not included in the active set; this is done by
numerical optimization of LIK (Equation 3). Finally, the scaling is
updated by closed-form optimization of LGP with respect to {wk}.
Numerical optimization is performed using the Scaled Conjugate
Gradients algorithm, although other search algorithms could also
be used. After each of these steps, the active set is recomputed.
The algorithm may be summarized as follows. See [Lawrence
2004] for further details.
function LEARNSGPLVM({yi})
initialize {x} with conventional PCA applied to {yi}
for
Select new active set
Minimize LGP (over the active set) with respect to ?, ?, ?
Select new active set
for each point i not in the active set do
Minimize LIK(xi,yi) with respect to xi.
end for
Select new active set
for each data dimension d do

end for
end for
return {xi},?,?,?,{wk}
Parameters. The active set size and latent dimensionality trade-off
run-time speed versus quality. We typically used 50 active set
points for small data sets and 100 for large data sets. Using a long
walking sequence (of about 500 frames) as training, 100 active set
points and a 3-dimensional latent space gave 23 frames-per-second
synthesis on a 2.8 GHz Pentium 4; increasing the active set size
slows performance without noticably improving quality. We found
that, in all cases, a 3D latent space gave as good or better quality
than a 2D latent space. We use higher dimensionality when multiple
distinct motions were included in the training set.
C Style interpolation
Although we have no formal justi?cation for our interpolation
method in Section 6.3 (e.g., as maximizing a known likelihood
function), we can motivate it as follows. In general, there is no
reason to believe the interpolation of two objective functions gives
a reasonable interpolation of their styles. For example, suppose we
3We adapted the source code available from
http://www.dcs.shef.ac.uk/ neil/gplvm/
represent styles as Gaussian distributions p(y|?0)=N (y|?0;?2)
and p(y|?1)=N (y|?1;?2) where ?0 and ?1 are the means of the
Gaussians, and ?2 is the variance. If we simply interpolate these
PDFs, i.e., ps(y)=?(1 ?s)exp(?||y ? ?0||2/?2)?sexp(?||y ?
?1||2/2?2), then the interpolated function is not Gaussian ? for
most values of s, it has two minima (near ?0 and ?1). Howver,
using the log-space interpolation scheme, we get an intuitive re-
sult: the interpolated style ps(y) is also a Gaussian, with mean
variance ?2. In other words, the mean linearly
interpolates the means of the input Gaussians, and the variance
is unchanged. A similarly-intuitive interpolation results when
the Gaussians have different covariances. While analyzing the SG-
PLVM case is more dif?cult, we ?nd that in practice this scheme
works quite well. Moreover, it should be straightforward to interpolate
any two likelihood models (e.g., interpolate an SGPLVM with
an MoG), which would be dif?cult to achieve otherwise.
Gradients
The gradients of LIK and LGP may be computed with the help of the
following derivatives, along with the chain rule:
?y
?x ?x

?x ?x
?x ?x
?x
containing the mean-
subtracted training data.


--R

Synthesizing Constrained Motions from Examples.
Motion Synthesis From Annotations.
Neural Networks for Pattern Recognition.
The process of motion capture
Style machines.
Shadow Puppetry.
Motion signal pro- cessing
Handrix: Animating the Human Hand.
Computational Modeling for the Computer Animation of Legged Figures.
Believable Automatically Synthesized Motion by Knowledge-Enhanced Motion Transformation
Inferring 3D Structure with a Statistical Image-Based Shape Model

Bayesian Reconstructions of 3D Human Motion from Single-Camera Video
Automated Extraction and Parameterization of Motions in Large Data Sets.
Motion Graphs.
Fast Sparse Gaussian Process Methods: The Informative Vector Ma- chine
Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data.
Interactive Control of Avatars Animated With Human Motion Data.
Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis
Introduction to Gaussian processes.
Bayesian Learning for Neural Networks.
Numerical Optimization.


Motion Capture Assisted Animation: Texturing and Synthesis.
Automatic annotation of everyday movements.
Mixture Densities
Learning Body Pose Via Specialized Maps.
Verbs and Adverbs: Multidimensional Motion Interpolation.

Implicit probabilistic models of human motion for synthesis and tracking.
Warped Gaussian Processes.
Reconstruction of Articulated Objects from Point Correspondences in a Single Image.
Inverse Kinematics and Geometric Constraints for Articulated Figure Manipulation.
Interpolation Synthesis of Articulated Figure Motion.
Gaussian Processes for Regression.
Parametric Hidden Markov Models for Gesture Recognition.

Natural motion animation through constraining and deconstraining at will.
Gesticulation Behaviors for Virtual Humans.
--TR
Motion signal processing
Motion warping
Physically based motion transformation
Parametric Hidden Markov Models for Gesture Recognition
Computational modeling for the computer animation of legged figures
Style machines
Bayesian Learning for Neural Networks
Neural Networks for Pattern Recognition
Motion texture
Motion graphs
Interactive control of avatars animated with human motion data
Motion capture assisted animation
Interpolation Synthesis of Articulated Figure Motion
Verbs and Adverbs
Implicit Probabilistic Models of Human Motion for Synthesis and Tracking
Gesticulation Behaviors for Virtual Humans
Handrix
Shadow Puppetry
Motion synthesis from annotations
Believable automatically synthesized motion by knowledge-enhanced motion transformation
Inferring 3D Structure with a Statistical Image-Based Shape Model
Automated extraction and parameterization of motions in large data sets

--CTR
Neil D. Lawrence , Joaquin Quionero-Candela, Local distance preservation in the GP-LVM through back constraints, Proceedings of the 23rd international conference on Machine learning, p.513-520, June 25-29, 2006, Pittsburgh, Pennsylvania
Edmond S. L. Ho , Taku Komura , Rynson W. H. Lau, Computing inverse kinematics with linear programming, Proceedings of the ACM symposium on Virtual reality software and technology, November 07-09, 2005, Monterey, CA, USA
Lionel Reveret , Laurent Favreau , Christine Depraz , Marie-Paule Cani, Morphable model of quadrupeds skeletons for animating 3D animals, Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 29-31, 2005, Los Angeles, California
Guodong Liu , Leonard McMillan, Segment-based human motion compression, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
Jack M. Wang , David J. Fleet , Aaron Hertzmann, Multifactor Gaussian process models for style-content separation, Proceedings of the 24th international conference on Machine learning, p.975-982, June 20-24, 2007, Corvalis, Oregon
Edwin Chang , Odest Chadwicke Jenkins, Sketching articulation and pose for facial animation, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
Kevin G. Der , Robert W. Sumner , Jovan Popovi, Inverse kinematics for reduced deformable models, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
J. P. Lewis , Jonathan Mooser , Zhigang Deng , Ulrich Neumann, Reducing blendshape interference by selected motion attenuation, Proceedings of the 2005 symposium on Interactive 3D graphics and games, April 03-06, 2005, Washington, District of Columbia
Neil D. Lawrence , Andrew J. Moore, Hierarchical Gaussian process latent variable models, Proceedings of the 24th international conference on Machine learning, p.481-488, June 20-24, 2007, Corvalis, Oregon
A. Majkowska , V. B. Zordan , P. Faloutsos, Automatic splicing for hand and body animations, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
T. Igarashi , T. Moscovich , J. F. Hughes, Spatial keyframing for performance-driven animation, ACM SIGGRAPH 2006 Courses, July 30-August 03, 2006, Boston, Massachusetts
T. Igarashi , T. Moscovich , J. F. Hughes, Spatial keyframing for performance-driven animation, Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 29-31, 2005, Los Angeles, California
Yeuhi Abe , Jovan Popovi, Interactive animation of dynamic manipulation, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
Michael Neff , Eugene Fiume, Methods for exploring expressive stance, Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, August 27-29, 2004, Grenoble, France
Neil Lawrence, Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models, The Journal of Machine Learning Research, 6, p.1783-1816, 12/1/2005
Chen Mao , Sheng Feng Qin , David K. Wright, Sketching-out virtual humans: from 2D storyboarding to immediate 3D character animation, Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, June 14-16, 2006, Hollywood, California
Tomohiko Mukai , Shigeru Kuriyama, Geostatistical motion interpolation, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Philippe Beaudoin , Pierre Poulin , Michiel van de Panne, Adapting wavelet compression to human motion capture clips, Proceedings of Graphics Interface 2007, May 28-30, 2007, Montreal, Canada
Yu-Chi Lai , Stephen Chenney , ShaoHua Fan, Group motion graphs, Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 29-31, 2005, Los Angeles, California
Chen Mao , Sheng Feng Qin , David K. Wright, Sketching-out virtual humans: from 2D storyboarding to immediate 3D character animation, Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, June 14-16, 2006, Hollywood, California
C. Karen Liu , Aaron Hertzmann , Zoran Popovi, Composition of complex optimal multi-character motions, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
Kang Hoon Lee , Myung Geol Choi , Jehee Lee, Motion patches: building blocks for virtual environments annotated with motion data, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
Robert W. Sumner , Matthias Zwicker , Craig Gotsman , Jovan Popovi, Mesh-based inverse kinematics, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Jackie Assa , Yaron Caspi , Daniel Cohen-Or, Action synopsis: pose selection and illustration, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Franck Multon , Richard Kulpa , Benoit Bideau, Mkm: A global framework for animating humans in virtual reality applications, Presence: Teleoperators and Virtual Environments, v.17 n.1, p.17-28, February 2008
Lucas Kovar , Michael Gleicher, Automated extraction and parameterization of motions in large data sets, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
Michael Neff , Eugene Fiume, Methods for exploring expressive stance, Graphical Models, v.68 n.2, p.133-157, March 2006
Eugene Hsu , Kari Pulli , Jovan Popovi, Style translation for human motion, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Zonghua Zhang , Nikolaus F. Troje, 3D Periodic Human Motion Reconstruction from 2D Motion Sequences, Neural Computation, v.19 n.5, p.1400-1421, May 2007
Jinxiang Chai , Jessica K. Hodgins, Performance animation from low-dimensional control signals, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Dragomir Anguelov , Praveen Srinivasan , Daphne Koller , Sebastian Thrun , Jim Rodgers , James Davis, SCAPE: shape completion and animation of people, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
C. Karen Liu , Aaron Hertzmann , Zoran Popovi, Learning physics-based motion style with nonlinear inverse optimization, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Thomas B. Moeslund , Adrian Hilton , Volker Krger, A survey of advances in vision-based human motion capture and analysis, Computer Vision and Image Understanding, v.104 n.2, p.90-126, November 2006
David A. Forsyth , Okan Arikan , Leslie Ikemoto , James O'Brien , Deva Ramanan, Computational studies of human motion: part 1, tracking and motion synthesis, Foundations and Trends in Computer Graphics and Vision, v.1 n.2, p.77-254, July 2006
gaussian processes;inverse kinematics;motion style;non-linear dimensionality reduction;character animation;style interpolation;machine learning