Taming Diffusion Probabilistic Models for Character Control
Rui Chen Mingyi Shi Shaoli Huang Ping Tan Taku Komura Xuelin Chen










riorui@foxmail.com
0009-0003-7122-5207

  Hong Kong University of Science and Technology
  Hong Kong

Joint first authors and work done during an internship at Tencent AI Lab



myshi@cs.hku.hk
0000-0002-5180-600X

  The University of Hong Kong
  Hong Kong

[1]


shaolihuang@tencent.com
0000-0002-1445-3196

  Tencent AI Lab
  China



pingtan@ust.hk
0000-0002-4506-6973

  Hong Kong University of Science and Technology
  Hong Kong



taku@cs.hku.hk
0000-0002-2729-5860

  The University of Hong Kong
  Hong Kong



xuelin.chen.3d@gmail.com
0009-0007-0158-9469

  Tencent AI Lab
  China

Corresponding author



















We present a novel character control framework that effectively utilizes motion diffusion probabilistic models to generate high-quality and diverse character animations, responding in real-time to a variety of dynamic user-supplied control signals. 

At the heart of our method lies a transformer-based Conditional Autoregressive Motion Diffusion Model (), which takes as input the character’s historical motion and can generate a range of diverse potential future motions conditioned on high-level, coarse user control.

To meet the demands for diversity, controllability, and computational efficiency required by a real-time controller,
we incorporate several key algorithmic designs. 
These include separate condition tokenization, classifier-free guidance on past motion, and heuristic future trajectory extension, all designed to address the challenges associated with taming motion diffusion probabilistic models for character control.

As a result,
our work represents the first model
that enables real-time generation of high-quality, diverse character animations based on user interactive control, supporting animating the character in multiple styles with a single
unified model. 

We evaluate our method on a diverse set of locomotion skills, demonstrating the merits of our method over existing character controllers.







<ccs2012>
   <concept>
       <concept_id>10010147.10010371.10010352.10010380</concept_id>
       <concept_desc>Computing methodologies Motion processing</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Motion processing









    < g r a p h i c s >

  
  
  Real-time character control powered by our Conditional Autoregressive Motion Diffusion Model (). 
  Top: Our method produces diverse outputs for character animation even given identical input control signals.
  Bottom: Our method can generate seamless transitions between different styles, even in cases where the transition data is absent from the dataset.
  All achieved with a single unified model, in real time.
  
  
  TBA.
  
  



acmlicensed
TOG
2023 42 4  8 15.0010.1145/3592395

Taming Diffusion Probabilistic Models for Character Control
    Xuelin Chen
    April 29, 2024
===========================================================







§ INTRODUCTION





Real-time character control is the holy grail in computer animation, playing a pivotal role in creating immersive and interactive experiences. 

It involves generating natural and expressive motions of virtual characters, that align well to the high-level specifications of the user interactions in a dynamic and real-time environment.

The evolution of this field has been driven by remarkable advancements in data-driven techniques <cit.> and the growing demand for realistic and interactive content in various sectors, including gaming and virtual reality. 




Despite significant progress, real-time character control continues to present several key challenges.

These include
generating visually realistic motions of high quality and diversity, 
ensuring controllability of the generation, and 
attaining a delicate balance between computational efficiency and visual realism.

More specifically,
diversity can refer to variations of different granularity, including intra-style diversity, which requires modeling the variation of joint curves within a particular style, and inter-style diversity, which refers to the variation between different styles. 

While much progress has been made in this field, almost all existing approaches struggle to address one or more of these challenges, consequently impeding their widespread adoption and implementation.



Recent advancements in deep learning, particularly diffusion-based generative models, have greatly impacted the motion generation field, showing more promise in potentially resolving these challenges.

As one of the pioneering works in this direction,
Phase-Functioned Neural Network (PFNN) <cit.>, along with the following studies <cit.>, learns a deep neural network to predict the next posture of the character given user control signals, in an autoregressive manner.

However, 
defining the future posture can be vague based on high-level, sketchy user inputs.
Therefore, the deterministic models employed in their approach suffer from regression to the mean pose, resulting in artifacts (e.g., sliding feet and reduced range of motions), despite the inclusion of heuristically designed or learned features for disambiguation. 
Moreover, the lack of motion diversity leads to repetitive and visually monotonous characters, reducing the visual realism in real-time and interactive applications.

More recently,
motion diffusion probabilistic models have shown the capability of capturing complex data distribution in large motion datasets and then generating high-quality and diverse samples in highly controllable ways through a long Markov denoising chain  <cit.>.

Now, the question arises as to how diffusion-based generative models can be effectively employed in real-time character controllers, producing high-quality and diverse outputs in response to user interactions.






In this paper, 
we introduce a novel character controller, which effectively harnesses motion diffusion probabilistic models for producing character animations that appear natural and respond in real-time to various, varying user-supplied control signals.
These control parameters specify various aspects to be expected from the generated motions, including the style/gait, moving speed, facing direction, future trajectory on the ground, etc.

Our method builds upon the autoregressive framework introduced in <cit.>,

but the core of our method is a transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM, pronounced cam-dem), 
which takes as input the character's history motions and can generate a range of diverse potential future motions conditioned on high-level, coarse control signals.

Particularly,
to achieve the controllability, diversity, and computational efficiency for real-time character controllers as aforementioned,
several algorithmic designs are incorporated to address challenges that arise from working with motion diffusion models.


First,
in contrast to previous approaches that represent various conditions as a single feature vector <cit.>,
which can lead to unstable control due to certain feature components dominating unexpectedly,
our model learns a separate token for each condition. 
By leveraging the attention mechanism of the transformer, our model enhances the effectiveness of each condition within the network, resulting in stable control.

Second,
regarding the diversity,
on the one hand,
we found training the model to forecast long future motions and applying as many frames as possible to the character (only when the user inputs remain unchanged) significantly improved the intra-style diversity of the autoregression.
Nonetheless, during runtime this strategy causes missing model-predicted future trajectories, which are supposed to be blended with user-supplied synthetic future trajectories for balancing the motion naturalness and trajectory alignment of the next prediction <cit.>,
leading to abrupt jittering between the past and predicted motions.
We then devise a simple yet effective module, which recycles the last small fraction of the previous model-predicted future trajectory to obtain a heuristic extension beyond the end of the model-predicted future
trajectory. 

On the other hand,
transitions between different styles are generally lacking in the mocap data, which poses challenges to controlling the style and thus obtaining inter-style diversity in the autoregression.
Hence, we propose to train the autoregressive motion diffusion with classifier-free guidance <cit.> on the conditional token of history motions, instead of on the token of style labels ­– an intuitive choice but fails in our experiments.
We show that this design significantly helps transition between different styles in the autoregression, yielding natural transitional motions.

Last,
our model uses only 8 denoising diffusion steps, which is significantly less than the 1000 steps in other literature, and hence can respond to the control in real-time.


To the best of our knowledge, our work represents the first model that enables real-time generation of high-quality and diverse character animations based on user interactive control, supporting animating the character in multiple styles with a single unified model.
0

We demonstrate our method on several publicly available locomotion mocap datasets, as locomotion skills are important ingredients of character animations in many downstream applications that involve user interactive controls.
These mocap datasets, namely 100STYLE <cit.> and a quadruped locomotion dataset <cit.>, 
feature an array of diverse locomotion skills and radically different morphologies (i.e., humans and dogs).


We demonstrate our method on a large publicly available mocap dataset featuring a diverse set of locomotion skills, as locomotion skills are important ingredients of character animations in many applications that involve user interactive controls.

Our extensive experimental results show
that our method is more than competent in producing high-quality and diverse character animations in real-time response to user-supplied control signals. 

Important algorithmic designs are also validated in ablation studies.



§ RELATED WORK

In this section,
we first review deep learning-based approaches for real-time locomotion generation based on user-supplied control parameters. 
Next, we review recent literature for synthesizing natural and expressive motions using diffusion probabilistic models.



  
DL-based Real-time Character Controllers
Real-time character control plays an important role in character animation in gaming.
Generally, state machines and motion matching are commonly used techniques in the industry.

The former defines a set of states characters can be in, such as idle, walking, running, jumping, attacking, etc. Each state represents a specific animation and behavior. 

Transitions between states are triggered by certain user inputs. 

Motion matching, on the other hand, searches and blends motion data in a large dataset to create smooth and realistic character animations based on local control context.

While these techniques have allowed for managing character behaviors and interactions,
they face challenges in delivering high-quality and diverse animations, and in scaling up to large-scale mocap data for producing much more complex character animations in controllable ways.

Hence, research efforts have taken various approaches to apply deep neural networks to large-scale mocap data, due to their scalability and high run-time performance <cit.>.


Training neural networks to predict future postures of the character based on user inputs is still challenging,
as the high-level, sketchy user inputs do not encompass every fine detail, implying ambiguities in the final definition of the expected motion.
Hence, simply training the network with the MSE minimization objective would result in obvious artifacts in the character animation, e.g., foot skating.  
Some propose to overcome this issue with the help of additional features crafted from the motion data.

<cit.>
propose the phase-functioned neural network, which learns and predicts the phase of the character's feet, significantly disambiguating the future posture prediction in locomotion.

<cit.> extend the concept of motion phase to complex motion by assigning each bone motion in a character with a separate motion phase. 

<cit.> propose a method that utilizes autoregressive and control information to modify network weights, enabling the generation of controllable quadruped motion. They demonstrate the successful generation of both cyclic motion, such as gait, and simple non-cyclic motion like jumping.

<cit.> extract a deep multi-dimensional phase space from full-body motion data to achieve better temporal and spatial alignment and demonstrate the merit of the learned deep features in character animation.

However, 
their character control framework is based on deterministic models, which inherently face challenges in generating realistic motions, especially when aiming for high diversity. This struggle becomes more pronounced when the available predictive input information is inadequate for effective disambiguation.


Probabilistic models offer an alternative to prevent collapsing into a mean pose. 

By training models to learn all plausible poses based on history motions and control inputs, the generated motion should exhibit convincing and realistic motion while following the user's specifications.
<cit.> first train a Variational Auto-Encoder (VAE) to learn a latent motion space,
with which a reinforcement learning policy network learns to steer the latent variable to follow control inputs.
However, their method often encounters significant errors in accurately tracking input trajectories, primarily due to the limitations of VAEs in effectively modeling complex data distributions.  

While the probabilistic normalizing flow-based method introduced in <cit.> can model conditional motion distributions and thus accommodate user-supplied control parameters,  its real-time performance is limited, implying it is less suitable for interactive applications like video games.



Another line of literature focuses on producing physically plausible motions,
where typically deep reinforcement learning (deep RL) policy networks are trained to control characters in simulated environments.



<cit.> demonstrate the potential of deep RL in animating characters within physically simulated environments by training an RL policy to adhere to a reference kinematic motion.

Subsequent works propose controlling physically simulated characters via RL policies trained to track a wide range of reference motions produced by kinematic motion generation modules <cit.>.

However, these methods typically rely heavily on the kinematic motion generation module, leading to challenges in composing diverse motions that surpass the capabilities of the kinematic generation.


Moreover, probabilistic models have been leveraged to enhance the diversity and generalizability of physically simulated characters <cit.>, enabling diverse generation of physically plausible motions given only a limited set of kinematic motions.

Although some methods in this domain have also been introduced to learn character controllers from extensive datasets <cit.>, 
these techniques still face difficulties in meeting all the essential criteria of a real-time character controller, namely realism, controllability, and computational efficiency as mentioned above.







  
Motion Diffusion Probabilistic Models
Recently, diffusion-based generative models have revolutionized natural image generation by learning to denoise <cit.>. As a result, there has been a surge of subsequent works adopting these models for motion generation <cit.>, leading to state-of-the-art performance in generating diverse and natural human motions conditioned on arbitrary textual prompts.

Although significant progress has been made in these endeavors, the language description alone has inherent limitations in providing more precise control over the generated motion. 
Hence, a new line of research has emerged, focusing on incorporating user-provided spatial constraints to enhance control over motion generation <cit.>.

Our work also builds upon the powerful diffusion probabilistic model for conditional motion generation,
but differs significantly from these existing models by the particular emphasis on learning a conditional autoregressive motion diffusion model for real-time character control.

Another related work <cit.> has showcased the utilization of an autoregressive motion diffusion model to generate future character postures based on joystick control signals. However, they have not demonstrated the ability to support multiple styles and their approach achieves controllability by training a dedicated policy network through reinforcement learning to guide the motion diffusion synthesis process.

In contrast,
our method directly models conditional motion distributions, and so has controllability built in.


§ METHOD


Given a large-scale locomotion dataset ,
our method first trains a Conditional Autoregressive Motion Diffusion Model (CAMDM), which
takes as input the past motion  of  frames of the character, and user control parameters , and then learns to capture the conditional distribution of the future motions  (of  frames).

During runtime, is applied at each frame with the on-the-fly collected character's historical poses, user control inputs, and randomly sampled Gaussian noise, and then sample from the conditional motion distribution, obtaining a sequence of realistic future postures to be displayed.

 
The character animated using this autoregressive generation approach can exhibit coherent and diverse motions while adhering to user inputs. 
This is achieved because is trained to capture all possible future motions under different conditions, yielding a wide range of plausible animations.





 §.§ Motion Diffusion Model


Following the recent success of motion generative models, 
our method employs the diffusion probabilistic model, which learns 
to generate by learning to denoise, to model the complex motion data distribution in a large-scale mocap dataset.

Given a clean motion sequence sample _0, the forward diffusion process is a Markov chain that progressively adds noise to _0 in T steps, producing an array of noisy samples _1, ..., _T: q_θ(_t|_0) = 𝒩(_t; √(α̅_t)_0,(1 - α̅) 𝐈), where α̅_t = ∏_s=1^t(1-β_s) and β_t is the variance scheduler of the added noise.

Then a model parameterized by a deep neural network  is trained to learn the reverse process in another Markov chain, i.e., learning the mapping p(_t-1|_t), to denoise the noisy sample sequentially in  steps. 


Instead of predicting the added noise as in <cit.>, 
we directly predict the clean sample itself at each denoising step: _0 = (_t, t),
which is a common practice in various diffusion models <cit.> and has been shown to
achieve slightly better generative performance by providing additional geometric losses on the predicted results.

Once trained,
 can transform randomly sampled Gaussian noise sequences to generate high-quality and diverse motion samples through  denoising steps.






  
Motion Representation
A motion sample  to be predicted by the model is defined by a temporal set of  poses
that each ^1:F consists of root joint displacements ∈^× 3 and joint rotations ∈^×,
where  is the number of joints and  is the number of rotation features.

Instead of directly using the global root displacements , 
we convert  to local root displacements 
by subtracting the displacement of the first pose from the sequential displacements.

The joint rotations are defined in the coordinate frame of their parent in the kinematic chain,
and we use the 6D rotation representation (i.e., = 6) proposed by <cit.>.











 §.§ Conditional Autoregressive Motion Diffusion Model

Our central objective is to learn a motion diffusion probabilistic model that can take as input the historical poses of the character and output a diverse set of plausible future poses based on user inputs,
so during runtime, the character can be naturally animated to follow control specifications in an autoregressive manner.

To this end,
we further condition the motion diffusion model on additional inputs, including the character's past motion  and user-supplied control parameters , and train it to generate plausible future motions  of  frames.

More specifically,
the control parameters supported in our method contain the style/gait label , future root displacements , and orientations  projected onto the ground.

At each denoising step, the model takes as input a noisy motion sample _t, diffusion step t, along with various conditions,
then learns to predict the original clean _0 (See Figure <ref>):


    _0 = (_t, t; , , , ).






  
Training Losses
The denoising objective is to minimize the MSE loss between the predicted x̂_0 and the clean sample x_0:


    = 𝔼
        
        ||_0 - _0 
     
     ||_2^2.
  
As mentioned in <cit.>, an advantage of predicting the signal itself is that we can leverage geometric losses. 

During training, we employ differentiable forward kinematics functions FK(·) to transform local joint rotations into global joint positions,
based on which we can calculate 3D joint position loss:

    =
     
     FK(
        _0,
        
        )-
        FK(
        _0,
        
        )_2^2,

where  is the rest skeleton used in the mocap data and remains unchanged throughout the training. 

Similarly, we can also compute the foot contact loss  and velocity loss  during the training (see  <cit.> for more details).

Finally, the total training loss is written as:

    = 
     +  +
     + 
    ,
 
with λ weights controlling the importance of each loss.

Once trained, this conditional autoregressive model can be used in the runtime animation system for character control.



Nonetheless, 
the above description only provides a conceptually valid and general formulation for learning a conditional motion distribution. 
We further elaborate on
algorithmic designs that are crucial for taming the motion diffusion probabilistic model for real-time character control.



  
Separate Condition Tokenization (SCT)
We found that learning to map various conditions into a single feature vector as in <cit.> or a single token as in  <cit.> can lead to highly unstable control due to certain feature components unexpectedly dominating the regression process. 
For instance, the style label may fail to effectively control the gait of the motion, or the character may not correctly follow the input future trajectory.

Hence,  we choose to use separate tokenizers to learn individual tokens for each input condition and add all these conditional tokens at the beginning of the noisy motion sequence as input to the denoising transformer network.

Then by leveraging the attention mechanism of the transformer, our model enhances the effectiveness of each condition within the network, resulting in a stable control.








  
Classifier-free Guidance on Past Motion (CFG-PM)
As mentioned earlier, generating transitional motions between different styles is challenging due to the lack of transitional motions in the mocap dataset.

While one workaround is to utilize the classifier-free guidance (CFG) technique with style label conditions <cit.>, this strategy fails to effectively transition between different styles in our tasks. 

We discovered that the key lies in controlling the influence of the past motion in conditional autoregressive generation, which led us to apply CFG to the conditional past motion. 

Specifically, during training, we randomly set the conditional past motion token to nil with a probability of 0.15.

At runtime,
when the style label is changed to a new one by the user input,
the future motion is sampled with a guidance scale of γ:

    (_t, t; , , , ) = 
        (_t, t; =∅, , , ) 
    
        + γ( 
        (_t, t; , , , ) - (_t, t; =∅, , , ) 
        ).

Commonly, CFG is typically utilized with γ>1 to amplify the conditioning. However, in this paper, we employ γ<1 to mitigate the influence of past motion while achieving smooth transitions between different styles. 
Note that CFG-PM is NOT used when the style label remains unchanged. Refer to section <ref> for more details.





  
Heuristic Future Trajectory Extension (HFTE)
Regarding the intra-style motion diversity mentioned earlier,


we found the lazy trigger of the next autoregression significantly improved the intra-style diversity of the autoregression.
The lazy trigger means that we train the model to forecast a long future motion, and then during runtime apply as many frames as possible to the character when the user-supplied control remains unchanged.




Specifically, our method is based on the character control framework in <cit.>, where the model-predicted future trajectory (in our case the floor projection of the root joint of generated motions) is blended with the user-supplied synthetic future trajectory,
which is synthesized by the critical spring damper function <cit.>,
for obtaining the final controlling trajectory input to the network. This strategy offers a good balance between the motion smoothness and trajectory alignment of the next prediction.

Previous methods <cit.> adopting this strategy usually generate and apply only one posture to the character at a time, whereas we generate and apply  postures from the generated motion before the next autoregression.

Obviously,
this lazy trigger strategy can deplete the model-predicted future trajectory.
This can result in a blending of two trajectories of different lengths, consequently causing sudden jittering between the current posture and the next autoregressed posture if not properly dressed (see the top of Figure <ref>.



We then devise a simple yet effective module, which
heuristically constructs a smoothly extended trajectory beyond the end of the model-predicted future trajectory,
by
recycling the last  points of the model-predicted future trajectory backward and forward multiple times, if necessary, until the length matches that of the user-supplied synthetic future trajectory. See the bottom of Figure <ref>.











  
Real-time Generation.
Our task requires the model to respond to the user control in real time.

Although there is extensive research in speeding up the diffusion model  <cit.>, we observed that their performance on our conditional autoregressive motion diffusion resulted in degraded generation results.
Similar to the finding in <cit.> we empirically discovered that using only 8 diffusion steps during both training and runtime surprisingly leads to convincing performance.






 §.§ Implementation Details





  
Training.
The mocap data utilized for training our model is at a frame rate of 30 frames per second.
By default, our model is conditioned on = 10 frames of the past motion and learns to generate = 45 frames for the future motion.

We set =  =  =  = 1 when training the denoising model.

We implement the denoising network with a straightforward transformer encoder-only architecture and follow the hyperparameters described in <cit.> for training. Our model is trained on a single A100 GPU for about 20 hours, and the character controller animates the character in real time on a RTX 3060 GPU.



  
Runtime.
Our runtime system is based on the open-source system AI4Animation <cit.>, and we deploy the trained model using the third-party library ONNX <cit.> for real-time inference.

The file containing the model weights is highly compact, with a size of just 20MB.
Even without any optimization, our implemented prototype character control system runs smoothly at over 60 frames per second.
During runtime, 

when the user changes the style label, the classifier-free guidance on past motion (CFG-PM) is triggered, and we set the guidance scale γ to 0.7.  When the style label remains unchanged, we experimented with various γ and found values between 1 and 2 yield similar best results.
Therefore, we set γ to 1, which is equivalent to using only the fully conditioned model (_t, t; , , , ) to minimize the computational cost while preserving the highest quality. 
We apply = 15 generated postures to the character when the user-supplied signals remain unchanged.

For the heuristic future trajectory extension, we recycle K = 15 samples of the model-predicted future trajectory. 



§ EXPERIMENTS

In this section, we evaluate the effectiveness of our method for real-time multi-style character control, compare it to other real-time character controllers, and validate the important design choices through ablation studies.

video for qualitative evaluation.



 §.§ Evaluation Setting



  
Datasets.

We conduct experiments on 100STYLE dataset  <cit.>, a publicly available mocap dataset featuring a diverse set of locomotion skills.
The data comprises over 4 million frames of motion capture data encompassing 100 different styles. Our experiments involved training the model using all styles and subsequently testing it within a real-time character control system, where varying control signals are provided.

0
(ii) The Quadruped Motion Dataset <cit.> offers a diverse range of dog behaviors, including idle, locomotion, jumping, and sitting, among others. We successfully trained our model using this dataset without requiring any additional labeling.





  
Metrics.


To evaluate our method, we assess the quality and diversity of the motion it generates, the alignment of its output with user-supplied control signals, and its responsiveness to changes in those inputs.

While no single metric can perfectly evaluate the generation, we utilize a combination of the following metrics that are well-established and commonly used in character animation.

To evaluate the motion quality, we use
(1) Fréchet Inception Distance (FID), which is a widely used metric in generative models. It measures the distance between the distribution of the generated motion and the distribution of the training data, offering insights into the fidelity and similarity between the generated and real motion data.
In our evaluation, we calculate it for each style and then report the average value;

(2) Foot Sliding Distance, which is crucial for evaluating the realism and plausibility of the generated motion. It quantifies the moving distance (in meters) of the character's toes when the height of the joint is below a certain threshold (e.g., 1cm).


(3) Acceleration, the mean per-joint acceleration (in centimeter per second) is calculated to assess the smoothness of the generated motion;

(4) Diversity (intra-style): Given the motions generated with the same control signals (the style and trajectory), this metric computes
the variance of each joint's spatial locations over time, and the average over all joints.


To evaluate the alignment between the generated motion and the control signals, we use 
(5) Trajectory Error, which is measured as the angular difference (in degrees) between the expected root moving direction for the subsequent frame and the actual root moving direction of that frame.


(6) Orientation Error, which is the difference (in degrees) between the expected and actual generated orientations of the root;

(7) Style Accuracy, each set of generation is divided into 60 frames clips, and we report the percentage of these clips that are consistent with the input style label. 

0
(5) Condition Alignment: In the animation system, control signals such as target direction, velocity, and desired motion style are established. We measure the deviation between the anticipated and actual generated positions, including metrics like trajectory position distance, directional angle offset, and style classification accuracy.



Last, to evaluate the performance of each model in transitioning between different styles, we use
(8) Transition Duration, which records the number of frames needed for successful transitions;
(9) Transition Success Rate, for which we ask human users to determine if the transition is successfully made within a time window (4s). We report the percentage of successful transitions to a target style.


0
(6) Transition response speed and success rate: When new styles are introduced, systems vary in the number of frames they require to achieve a successful transition. We record the number of frames needed for these transitions as well as the corresponding success rates.

align with the tables.




  
Baseline methods.
We compare our method with several state-of-the-art DL-based character controllers.
Motion matching is the de facto standard character controller in the industry. 
The default algorithm utilizes pose and velocity features to search the best match from the database; we enhance the algorithm by incorporating the DeepPhase features introduced in <cit.>, denoted as MM-DP. 

We assign a weight of 1 to the phase vector to maintain a balanced influence in our feature set.
Due to the high memory-consumption nature of motion matching,
we can only apply it to 10 styles to avoid memory overflow during the construction of KD trees. 

Then we compare against
Local Motion Phase (LMP) <cit.> and a variant of combining the Mixture of Experts in MANN <cit.> with DeepPhase features <cit.> (denoted as MANN-DP), which have shown state-of-the-art performance in recent literature <cit.>.


Furthermore,
we compare our method against another generative method ­– MoGlow <cit.>. 

Similar to training our model, we apply this method to the entire 100STYLE dataset, and then deploy them to produce motions by concatenating the style label as an extra dimension with others control signals for direct comparisons.


Note that the results of MoGlow are generated offline as it does not support real-time character control.







 





 §.§ Results

We evaluate the performance of each method with two different settings, namely single-style character control and multi-style character control.


In the single-style character control test,
for each style,
we preset a sequence of varying control trajectories, input it to all the competing models with the same style label, and record the rolled-out motions multiple times for evaluating the generated motion in terms of the quality, intra-style diversity, and control alignment.



To evaluate the ability to deliver inter-style diversity, we conduct experiments using a multi-style control setting. In this setting, we pre-determined a path for the character to follow, updated the style label in the middle, and recorded the transition motions between different styles for evaluation. Specifically, we used each of the 100 styles as the starting style and randomly sampled five styles as the target style for transitioning.







  
Qualitative Evaluation.





Snapshots of the single-style character control experiments are shown in Figure <ref>;
the trajectories of the joints produced in real-time using our method and the other baselines based on the same user inputs are presented.






As illustrated in Figure <ref>, our character produces much more diversified motions yet consistent with the control inputs.

Baseline methods tend to be trapped in periodic states, resulting in repetitive motions and larger deviations from the control trajectory. 

While MoGlow seems to produce smooth movements, it often results in numerous frames sticking, which hampers the generation of high-quality movements(e.g., severe foot sliding during side walking). 

In addition, MoGlow is unable to arbitrarily switch between different styles.







In the multi-style control transition experiments,
our method can produce natural and agile motions for transitioning between different styles,
even when the transition motions between styles do not exist in the dataset.

In particular,
the neural models have difficulties in switching to a target style that has mirrored motions of those of the original style, as their motion features are symmetric and there is absolutely no transition motions in the dataset.

In Figure <ref>, 
we select a pair of such styles, namely LeftHop and RightHop, for visual comparisons of all controllers. 
It is evident that the baseline methods fail to switch between these styles, whereas our method successfully completed the switch. 
This can be attributed to our proposed sampling strategy, where we apply CFG on the past motion, which facilitates rapid and seamless style switching.








More qualitative results can be found in the supplementary video.



  
Quantitative Evaluation.





We present the quantitative results of single-style character control experiments in Table <ref>, and multi-style transition experiments in Table <ref>.

From Table <ref>, we can see that
our method outperforms all baseline methods in terms of the motion quality and condition alignment.

We observed that
while motion matching has fewer artifacts, it occasionally fails to find good matches that respond perfectly to the given control signal, causing obvious misalignments.

LMP and MANN share the same autoregressive architecture, allowing them to follow the specified trajectory more 
closely. 
However, their input features are concatenated to form a single embedding, the balance between various signals might be biased under different situations, leading to unstable control, e.g., the style label lost control and the style of the motion is dominated by that of the past motion. 
More specifically, we observed that some styles, such as LeftHop, RightHop, etc., are challenging to transition to from different styles.
Additionally, the generated motion

rh!0.3 
    
        Div. ↑       
 MM-DP     0.414   
 
    LMP       0.426   

    MANN-DP     0.353   
 Ours       0.486     
        


sometimes loses its distinct characteristics, leading to lower FID scores. 


The competing generative method, MoGlow, struggles to produce high-quality motions and follow the style specification correctly,
as manifested by worse FID and style accuracy scores.








Note the intra-style diversity score is calculated only based on the results obtained from the "waving" and "drunk" styles, as they display a high degree of diversity, 
unlike other styles.

The numerical results are presented in the inset table, where our method has the best diversity.

For multi-style transition experiments, Table <ref> shows that our method reliably and efficiently transitions to new target styles with fewer transitional frames and a high success rate. 
In contrast to our approach, other baselines frequently struggle to transition to the target style or require many more frames for successful transitions.
































 §.§ Ablation Study
 
To show the importance of various algorithmic designs, we compare our full model to several  variants:

  
  * Ours w/o SCT, which removes the separate condition tokenization and instead learns to map all conditions into a single token as done in <cit.>;
  
  * Ours w/ CFG-S, which uses the CFG on the style label
  
  for generating transition motions between different styles;
  
  * Ours w/o HFTE, which removes the heuristic future trajectory extension module;
  
  * Ours w/ MLP, which uses a multilayer perception (MLP) network (8 layers) instead of the transformer.
  
  













Table <ref> shows that our full model performs best over the ablated versions.
More specifically, 
removing the separate condition tokenization (Ours w/o SCT) and instead representing various control signals as a single toke leads to degraded motion quality (overly smooth motions in particular) and control alignment.
Disabling the heuristic future trajectory extension (Ours w/o HFTE) results in sudden jitters in runtime, which is reflected in the significantly increased Acceleration score.

Using an MLP as the network backbone (Ours w/ MLP) results in a deterioration of network performance, as evidenced by a lower FID and higher sliding error. 




CFG on the style label (CFG-S), is a straightforward way of generating transitional motions between two different styles. Similar to our CFG-PM, CFG-S is implemented by randomly setting the style vector to nil while keeping all other conditions during training, so we can apply CFG technique on the style label during runtime. 
However, Table <ref> shows that applying CFG on the style label leads to failure in transitioning between different styles, whereas our method can achieve remarkable results in quickly and accurately transitioning between different styles of motions, even with the absence of transitional data between distinct styles. 


We speculate that during the training phase, past motions contain
rich stylized features that could influence the autoregressive generation of the future motion and prevent it from switching to a new style in the absence of transitional data.

Therefore, by applying the CFG technique to the past motion with a scale factor, we can 
guide the model to rely less on the past motion and instead focus more on the style label, leading to successful transitions.












  
Effect of different diffusion steps.
In Table <ref>, We demonstrate the impact of using different diffusion steps during training and sampling.
It indicates that the 8-step diffusion model is an ideal choice for our real-time character control task, striking a good balance between quality and execution time.
Indeed, this design choice was shaped by the distillation technique inherent in diffusion models.
However, during our initial development, we discovered that training a 1000-step CAMDM first and then distilling it into a fewer-step model led to highly unstable results, whereas directly training fewer-step diffusion models yielded significantly better results. This empirical observation aligns with the finding of MDM <cit.> as demonstrated in Table 6 in the supplementary and that of AMDM <cit.>, where a 1000-step diffusion model does not consistently produce optimal results.


We also conduct experiments to show the effect of changing the number of frames applied to the character during runtime on the motion diversity (see the supplementary video).












0






0








0



§ APPLICATIONS


TBA.

Random motion generation

conditional generation

Target position following


§ CONCLUSION


We have presented a novel character controller
that enables real-time generation of high-quality, diverse character animations based on user interactive control, supporting multiple styles with a single unified model. 

This is achieved by a transformer-based
Conditional Autoregressive Motion Diffusion Model, which takes as input
the past motion and can generate a range of diverse potential
future motions conditioned on high-level, coarse user control.

Despite the success demonstrated,
we note a few shortcomings of our method.

 

While our method can animate characters in real-time using an RTX 3060 GPU, 
it is worth noting that the 8-step diffusion model used in our method is still inherently computationally demanding. 
This presents a significant challenge for its potential adoption on mobile devices, due to their limited computational capabilities.

Hence, it would be worth exploring a more advanced way of accelerating diffusion generation, such as the consistency model <cit.> and latent consistency model <cit.> that are commonly adopted in image diffusion models.

Moreover, we have demonstrated on mocap data containing only human interactions with flat ground.

It would be intriguing to explore whether the proposed model can effectively process and understand more complex sensory data from the surrounding environment, and then generate natural, adaptive motions in real time that are responsive to changes in the scene.

Lastly, the joystick-based controller, due to its limited expressiveness, may struggle to accurately represent more complex desired motions.
Looking ahead, we aim to explore the potential of developing new control interfaces that integrate signals from various modalities. These could include text prompts, audio instructions, and brain electroencephalogram signals, among others, to enhance the system's ability to generate intricate movements.




Taku Komura and Mingyi Shi
are partly supported by Technology Commission
(Ref:ITS/319/21FP) and Research Grant Council (Ref:
17210222), Hong Kong.




ACM-Reference-Format


































