GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting
Kyusun Cho Joungbin Lee Heeji Yoon Yeobin Hong Jaehoon Ko Sangjun Ahn Seungryong Kim

empty

GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting]GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting








Cho et al.]Kyusun Cho^*1, Joungbin Lee^*1, Heeji Yoon^*1, Yeobin Hong^1, 
 Jaehoon Ko^1, Sangjun Ahn^2, and Seungryong Kim^† 1  

  

  ^1 Korea University  ^2 NCSOFT
    

    <https://ku-cvlab.github.io/GaussianTalker/>



















We propose GaussianTalker, a novel framework for real-time generation of pose-controllable talking heads. It leverages the fast rendering capabilities of 3D Gaussian Splatting (3DGS) while addressing the challenges of directly controlling 3DGS with speech audio. GaussianTalker constructs a canonical 3DGS representation of the head and deforms it in sync with the audio. A key insight is to encode the 3D Gaussian attributes into a shared implicit feature representation, where it is merged with audio features to manipulate each Gaussian attribute. This design exploits the spatial-aware features and enforces interactions between neighboring points. The feature embeddings are then fed to a spatial-audio attention module, which predicts frame-wise offsets for the attributes of each Gaussian. It is more stable than previous concatenation or multiplication approaches for manipulating the numerous Gaussians and their intricate parameters. Experimental results showcase GaussianTalker’s superiority in facial fidelity, lip synchronization accuracy, and rendering speed compared to previous methods. Specifically, GaussianTalker achieves a remarkable rendering speed up to 120 FPS, surpassing previous benchmarks. The code is made public in <https://ku-cvlab.github.io/GaussianTalker/>.





<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010245.10010254</concept_id>
       <concept_desc>Computing methodologies Reconstruction</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10002951.10003227.10003251.10003256</concept_id>
       <concept_desc>Information systems Multimedia content creation</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224.10010226.10010239</concept_id>
       <concept_desc>Computing methodologies 3D imaging</concept_desc>
       <concept_significance>100</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Reconstruction
[300]Information systems Multimedia content creation
[100]Computing methodologies 3D imaging





[
    [
    April 29, 2024
==================

* Contributed equally to this research.
† Corresponding author.







§ INTRODUCTION

Generating a talking head video driven by arbitrary speech audio is a popular task that has various uses, including the generation of digital humans, virtual avatars, movie production, and teleconferencing <cit.>.
While various works <cit.> have successfully attempted to solve this task using generative models, they do not focus on controlling head poses, limiting their realism and applicability. 
Recently, numerous studies <cit.> have applied neural radiance fields (NeRF) <cit.> for the creation of pose controllable talking portraits. By directly conditioning audio features in the multi-layer perceptron (MLP) of NeRF, these methods can synthesize view-consistent 3D head structure with its lips synced to the input audio. 
Although these NeRF-based techniques achieve high-quality and consistent visual outputs, their slow inference speed limits their practicality. 
Despite recent advancements <cit.> achieving rendering speeds up to 30 frames per second (fps) at 512 × 512 resolution, computational bottlenecks must be overcome to be applied in real-world scenarios.

Addressing this limitation, an intuitive solution is to leverage the fast rendering capabilities of 3D Gaussian Splatting (3DGS) <cit.>. Recently recognized as a viable alternative to NeRF, 3DGS offers comparable rendering quality while significantly improving inference speeds. 
Although 3DGS was initially proposed for reconstructing static 3D scenes, subsequent works have extended it to dynamic scenes <cit.>. However, there has been little research on leveraging 3DGS to create dynamic 3D scenes with controllable inputs, most of which focused on using an intermediate mesh representation to drive the 3D Gaussians <cit.>. 
However, relying on an intermediate 3D mesh representation, such as FLAME <cit.>, for deformation often lacks fine details in hair and facial wrinkles. 


We identify two major challenges in directly mapping the speech audio to the deformation of 3D Gaussians. First, the 3DGS representation lacks shared spatial information among the adjacent points, complicating its manipulation. The optimization process of 3DGS does not consider relationships between neighboring Gaussians, crucial for maintaining facial region cohesion during deformation. Secondly, the extensive parameter space and a substantial number of Gaussians pose a challenge to their manipulation.
Unlike controllable NeRF representations where the position and the number of sampling points are fixed, the position, shape, and appearance attributes of numerous Gaussian points need to be deformed per frame, while also preserving the intricate facial details. 


In this paper, we present GaussianTalker, a novel framework for real-time pose-controllable talking head synthesis. 
For the first time, we leverage the 3D Gaussian representation to exploit its fast scene modeling capability for audio-driven dynamic facial animation. We construct a static 3DGS representation of the canonical head shape and deform this in sync with the audio. 
Specifically, we employ a multi-resolution triplane to extract feature embeddings for each 3D Gaussian position, from which each Gaussian attribute is directly estimated. This design ensures that the triplane learns the spatial and semantic information of the 3D head, while the interpolation mechanism of the 2D feature grids efficiently enforces interactions between neighboring points. 
The feature embeddings are subsequently fed to the proposed spatial-audio attention module, where they are merged with the audio features to predict the frame-wise offsets for the attributes of each Gaussian. This module successfully models the relevance between audio features and the motions for each Gaussian primitive. The cross attention offers a more stable approach of manipulating the substantial number of Gaussians and their intricate parameter space, compared to concatenation <cit.> or multiplication <cit.> as in previous works.  
Qualitative and quantitative experiments demonstrate GaussianTalker's superiority in facial fidelity, lip synchronization accuracy, and rendering speed compared to previous methods. Additionally, we conduct ablation studies to verify the effectiveness of individual design choices within our model.

Our main contributions are summarized as follows:

    
  * For the first time, we present a novel audio-conditioned 3D Gaussian Splatting framework real-time 3D-aware talking head synthesis.
    
  * We reformulate the 3D Gaussian representation with a feature volume representation in order to enforce spatial consistency among adjacent Gaussians. 
    
  * We integrate cross-attention mechanisms between audio and spatial features to improve stability and ensure region-specific deformation across a significant number of Gaussians.




§ RELATED WORK




 §.§ Audio-driven talking portrait synthesis

Audio-driven talking portrait synthesis aims to create realistic facial animations with accurate lip movements based on audio input. Early 2D GAN-based methods <cit.> achieved photorealism but lacked control over head pose due to the absence of 3D geometry. In order to control the head poses, some works <cit.> utilize model-based methods, where facial landmarks and 3D morphable models reinforce the lip sync model with the ability to adjust the orientation of the head. However, these approaches lead to new problems such as extra errors from the intermediate representations, and inaccuracies in identity preservation and realism. 

Recently, Neural Radiance Fields (NeRF) <cit.> have been explored for talking portraits due to their ability to capture complex scenes. AD-NeRF <cit.> pioneered using NeRF's implicit representation for conditional audio input, but separate networks for head and torso limited its flexibility. Subsequent NeRF-based methods <cit.> achieved high quality but suffered from slow rendering speeds. While RAD-NeRF <cit.> and ER-NeRF <cit.> improved efficiency and quality with grid-based NeRF <cit.>, real-time rendering of pose-controllable 3D talking head remains challenging.





 §.§ 3D Gaussian splatting

3DGS <cit.> is a pioneering technique in point cloud rendering that utilizes a multitude of ellipsoidal, anisotropic balls to precisely represent a scene. Each point embodies a 3D Gaussian distribution, with its mean, covariance, opacity, and spherical harmonics parameters optimized to accurately capture the scene's shapes and appearances. This approach effectively resolves common issues in point rendering, such as output gaps. Furthermore, combined with a tile-based rasterization algorithm, it facilitates expedited training and real-time rendering capabilities. Recently, 3DGS has gained widespread application in 3D vision tasks such as object manipulation <cit.>, reconstruction <cit.>, and perception <cit.>  within 3D environments. 







 §.§ Facial animation with 3DGS

Previous methods for facial reconstruction and animation primarily relied on 3D Morphable Models(3DMM) <cit.> or utilized neural implicit representations <cit.>. Recent 
approaches <cit.> have shifted towards adopting the 3DGS representation, aiming to leverage the benefits of rapid training and rendering while still achieving competitive levels of photorealism. GaussianAvatars <cit.> reconstructed head avatars by rigging 3D Gaussians on FLAME <cit.> mesh. MonoGaussianAvatar <cit.> learned explicit head avatars by shifting the mean position of 3D Gaussians from canonical to deformed space using Linear Blend Skinning (LBS) and simultaneously adjusts other Gaussian parameters through a deformation field. GaussianHead <cit.> adopted a motion deformation field to adapt to facial movements while preserving head geometry and separately utilized a tri-plane to retain the appearance information of individual 3D Gaussians. 
However, the aforementioned methods tend to depend on parametric models for facial animation. In contrast to previous works, our audio-driven method is not only free from the need for data beyond the speech sequence for facial reenactment but also is readily applicable to novel audio. 





§ PRELIMINARY: 3D GAUSSIAN SPLATTING


3D Gaussian splatting (3DGS) <cit.> employs anisotropioc 3D Gaussians as geometric primitives for learning an explicit 3D representation. Each 3D Gaussian is defined by a center mean μ∈ℝ^3 and covariance matrix Σ∈ℝ^3 × 3 in the 3D coordinate as follows:

    g(x) = exp(-1/2 (x-μ)^TΣ ^-1(x-μ)),

for a 3D coordinate x ∈ℝ^3.
The covariance matrix Σ is further decomposed into Σ = RSS^TR^T with a scaling matrix S and a rotation matrix R, defined by a scaling factor s ∈ℝ^3 and a learnable quaternion r ∈ℝ^4, respectively. 
Additionally, to encode the appearance information, each 3D Gaussian contains a set of spherical harmonics with degree k such that SH ∈ℝ^3(k+1)(k+1), along with an opacity value α∈ℝ.
In summary, 3DGS represents a 3D scene with a set of 3D Gaussians parameters, defined as:

    𝒢 ={μ, r, s, SH, α},

Given a novel viewing direction π, a 2D image Î is rendered as:

    Î = ℛ(𝒢 ; π),

where ℛ(·) is the differentiable rasterizer. 

More specifically, for ℛ(·), 3DGS employs differential splatting <cit.> during novel view rendering.
In order to project 3D Gaussians to 2D for rendering, the covariance matrix in the 2D space, Σ ^ '∈ℝ^2 × 2, is calculated by viewing transform W and the Jacobian J of the affine approximation of the projective transformation <cit.>, such as:

    Σ ^ ' = JWΣ W^TJ^T.

Subsequently, the color of each pixel is computed by blending all Gaussians that overlap the pixel and ordered by their depths as follows:

    C = ∑_i = 1c_i  α^'_i ∏_j=1^i-1 (1-α^'_j),

where c_i is the color of each point determined using the SH coefficient with view direction, and α_i^' is computed by the multiplication of the opacity α of the 3D Gaussian and its projected covariance Σ ^ '.







§ METHODOLOGY





 §.§ Problem formulation and Overview

In this section, we describe the main components of GaussianTalker, designed for the real-time synthesis of high-fidelity, pose-controllable talking head images driven by audio input. 
Our model is trained on a talking portrait video 𝒱={I_n} consisting of N number of image frames for an identity. 
Our objective is to reconstruct a set of canonical 3D Gaussians that represent the mean shape of the talking head, and learn a deformation module that deforms the 3D Gaussians according to corresponding input audio. 
During inference, for the input audio a_n, the deformation module predicts the offsets of each Gaussian attribute, and the deformed Gaussians are rasterized at the viewing point π_n to output the novel image Î_n. 


An overview of our proposed method is depicted in fig:main_figure.
We first introduce the multi-resolution tri-plane that encodes the low-dimensional features of the 3D Gaussians to represent the static mean shape of the canonical head in subsec:spatial_encoding.
In subsec:attention, we introduce the speech-motion cross-attention module that fuses 3D Gaussians features and audio features to accurately model facial motion driven by input audio.
Finally, subsec:training_details describes the stage-wise training strategy and the utilized loss functions.





 §.§ Learning canonical 3D Gaussians with triplane representation


In this section, we introduce the details of learning the canonical shape of the talking head with 3D Gaussian representation. The vanilla implementation of 3DGS <cit.> does not inherently capture the spatial relationships between neighboring and distant 3D Gaussians. 
However, an ideal feature representation for a dynamic 3D head should be analogous for proximal facial regions and distinct for separated ones, as the close facial primitives would likely move to the same direction.

To realize this, we modify the 3D Gaussian representation by learning a low-dimensional feature representation, which can be later merged with the audio features for per-Gaussian deformation. We formulate the embedding space to encode information of the attributes of the 3D Gaussians, in order to take into account the shape and appearance of each Gaussian when predicting its deformation offsets. More specifically, we adopt a hybrid 3D representation that utilizes the explicit 3D representation of 3DGS, while also taking advantage of the encoded spatial information of implicit neural radiance fields <cit.>.
For each of the canoncial 3D positions μ_c, we extract feature embeddings f(μ_c) from a multi-resolution triplane representation <cit.>. 
These feature embeddings are utilized to calculate the scale s_c, rotation r_c, spherical harmonics SH_c, and opacity α_c of each point. 
These computed attributes make up the canonical 3D Gaussian of the talking head, denoted as:

    𝒢_can ={μ_c, r_c, s_c, SH_c, α_c}.

During training, instead of directly updating the 3D Gaussian attributes, the feature grids of the triplane and the attribute prediction networks are optimized.
This allows for the feature embedding f(μ_c) to store the region-specific facial information of the canonical 3D head, while also enforcing spatial relationships between neighboring Gaussians. 
In the following, we introduce the formulation of each module in detail. 







  §.§.§ Triplane representation for 3D Gaussian



In order to encode the spatial information of the canonical 3D head, we adopt a multi-resolution triplane representation, constructed by three orthogonal 2D feature grids, P
= { P^xy, P^yz, P^zx}. 
Each of these planes has shape H× R × R, where H stands for the hidden dimension of features, and R denotes the resolution of each dimension. 
For individual 3D Gaussian with position μ, each of its coordinate values is normalized between [0, R), and its corresponding features are computed by interpolating the point into a regularly spaced 2D grid for each plane. These features are combined using the Hadamard product ∏ for each plane, followed by concatenation ⋃ along the different dimensions, to produce a final feature vector f(μ) of length H for each of the canonical Gaussian position μ_c, such as:

    f(μ) =⋃∏_p∈ Pinterp(p, ζ_p(μ_c)),

where ζ_p(μ) denotes a projection of μ onto the p'th plane and `interp' denotes bilinear interpolation of a point into the regularly spaced 2D grid.
The visualization of features in our multi-resolution triplane is depicted in fig:triplane_viz.




  §.§.§ Attribute prediction of canonical 3D Gaussians

Unlike the original 3DGS implementation shown in (<ref>), we do not explicitly store the shape information r and s, and the appearance information SH and α.
Instead, these attributes are obtained from the corresponding feature representation f(μ). 
Specifically, we employ a set of MLP layers, denoted as ℱ_can(·), to map the feature to the mean scale s_c, mean rotation r_c, mean spherical harmonics SH_c, and mean opacity value α_c from f(μ), such as:

    {s_c, r_c, SH_c, α_c}  = ℱ_can(f(μ)).

Compared to the original 3DGS <cit.> where each Gaussian is optimized independently, our hybrid representation conditioned on an implicit feature volume enforces shared facial information between adjacent points. 





 §.§ Learning audio-driven deformation of 3D Gaussians


Previous works <cit.> employ a conditional NeRF representation, wherein the 3D coordinates of the sampling point along each ray remain fixed, with only color and density conditioned to input audio. However, in order to fully benefit from the explicit representation of 3DGS, we choose to deform the 3D Gaussians, where we manipulate not only the appearance information but also the spatial positions and shape of each Gaussian primitive. 
While this can more accurately capture the constantly fluctuating 3D shape of the talking head, deformation of 3D Gaussians is a much more complex task compared to controlling a NeRF representation. 
The intricate nature of Gaussian primitives, coupled with their sheer quantity, presents significant challenges for deformation due to the extensive parameter space of 3D Gaussians. 
In addition, input audio does not impact the whole facial image uniformly, making it vital for the deformation module to understand how varying facial regions respond to audio conditions for authentic facial animation.


In order to model the relations between the dynamic features and the vast amount of 3D Gaussians, we fuse the input speech audio a_n with the encoded feature f(μ_c) in an attention mechanism, in order to produce the audio-aware feature h_n for the n-th image frame. The deformation offsets of each Gaussian attribute for subsequent frames are directly conditioned on the feature h_n. 
Finally, the deformed set of 3D Gaussian for the n-th image frame is defined as:

    𝒢_deform,n ={μ_c + Δμ_n, r_c + Δ r_n, s_c + Δ s_n , SH_c + Δ SH_n, α_c + Δα_n },

where Δμ_n, Δs_n, Δr_n, ΔSH_n, Δα_n are the deformation offsets at n-th frame for 3D position, scale, rotation, spherical harmonics parameters and opacity, respectively. 
The details of each module is introduced in the following paragraphs. 



  §.§.§ Spatial-audio cross attention
 

Previous approaches to implement region-aware audio, like ER-NeRF <cit.>, simply adjust the weights for the audio features at each 3D point through elementwise multiplication. However, it encounters a challenge in that, regardless of the diverse audio inputs in a dynamic scene, a particular static 3D point consistently maintains the same audio weight.
This fails to acknowledge that a fixed 3D coordinate may not consistently correspond to the same facial region as the scene progresses. 
To address this issue and enhance the extraction of spatial-audio features, we introduce spatial-audio attention module, a cross-attention mechanism that merges spatial feature embedding f(μ_c) of the canonical 3D Gaussians with subsequent audio features, capturing how the input speech audio affects the movement of the 3D Gaussians. The spatial-audio attention module comprises L sets of cross-attention layer 𝒯_CA(·) and feed-forward layer FFN(·), each interconnected with skip connections. The module is formulated as:

    z_n^0 =f(μ_c),


    z'_n^l = 𝒯_CA(z^l-1_n, a_n)+z^l-1_n,   l=1...L ,


    z^l_n = FFN(z'_n^l)+ z'_n^l,   l=1...L ,

whereby the cross-attention between the spatial feature f and the audio feature a_n of the n-th image frame is computed. As a result, the output feature z^L_n successfully amalgamates audio features with the rich facial details captured by each 3D Gaussian.
This cross-attention module offers a more nuanced and stable method of feature combination than simple concatenation or multiplication, as the module reforms the spatial-aware facial features with respect to the subsequent audio features, taking into account the dynamic variability inherent in each 3D Gaussian. 








  §.§.§ Disentanglement of speech-related motion.



When synthesizing a talking head, the corresponding speech audio does not account for all the intricate and diverse facial movements. Subtle expressions like eye blinks and facial wrinkles, along with external factors such as hair movement and variations in lighting, do not directly correlate with input speech audio. Thereby, it is crucial to separate the non-verbal motions and scene variations when mapping speech audio to the 3D Gaussian deformation. In this section, we address this challenge by introducing additional input conditions that capture non-verbal motions, allowing us to disentangle speech-related motion from the monocular video.

Following previous works <cit.>, we first apply explicit eye blinking control with the eye feature e. Specifically, we employ AU45 from the Facial Action Coding System <cit.> to describe the degree of the eye blink, and utilize a sinusoidal positional encoding in order to match the input dimensions.
Additionally, we integrate the camera viewpoint as an auxiliary input to disentangle non-verbal scene variations. While we formulate the framewise camera π_n as facial viewpoints, the typical video is recorded with a static camera while the head undergoes continuous movement. Consequently, variations in the portrait image, such as hair displacement and lighting changes, occur independently of the speech audio. Hence, we employ a facial viewpoint embedding υ as an additional input condition to disentangle these non-auditory scene fluctuations. υ_n is an embedding vector obtained by mapping the extrinsic camera pose π_n to a small MLP to have the same dimensionality as the other inputs. 
Finally, we discovered that using a single null-vector () for all frames promotes consistency as a global feature across video frames. We incorporate this null-vector as an additional input for our cross-attention network.
Thus, we reformulate (<ref>) as:

    z'_n^l = 𝒯_CA(z^l-1_n,{a_n,e_n,υ_n, })+z^l-1_n,    l=1...L .

In fig:attention_viz, we visualize the attention scores for each input in order to demonstrate the efficacy of disentangling audio-related motion. 
Further details on the network structure and visualization procedure are provided in the supplementary file.











  §.§.§ Audio-conditioned deformation of 3D Gaussian


The final deformation network takes the spatially-aware audio features encoded in each 3D Gaussians in order to compute the deformation of position, rotation, and scaling.
We define the set of MLP regressors ℱ_deform(·) in order to predict the offsets of each Gaussian attributes, such as:

    {Δμ_n, Δs_n, Δr_n, ΔSH_n, Δα_n}  = ℱ_deform(z^L_n).









 
 
 








 §.§ Training
 




  §.§.§ Stage-wise optimization

3DGS <cit.> showed that the quality of reconstruction is influenced by the initialization of 3D Gaussians. Similarly, the training of the deformation field should also be conducted using a proper initialization of the canonical facial shape. 
To this end, we employ a two-stage training approach.

In the first stage, canonical stage, we first reconstruct the mean shape of the talking face, by optimizing the positions of 3D Gaussians and the multi-resolution triplane. 
Instead of the conventional initialization using structure from motion (SFM) points, we opt to utilize the 3D coordinates of the mesh vertices from fitting 3D morphable models. Note that the 3DMM fitting of each frame involves no extra preprocessing, as this is a necessary part of obtaining the camera parameters of the talking face and is widely adopted in NeRF-based talking face synthesis works <cit.>. 
The static image of the canonical talking head is rasterized via:

    Î_can = R(𝒢_can; π_n).


This is followed by the deformation stage, where we optimize the whole network, from which we learn the cross-attention deformation network. 
For each frame, the dynamic talking head video frame can be rendered as:

    Î_n = R(𝒢_deform,n; π_n).






  §.§.§ Loss Functions

For the canonical stage for a static shape of talking head, we follow the original 3DGS implementation <cit.> and utilize a combination of L1 color loss ℒ_1 and a D-SSIM term ℒ_D-SSIM.
Following previous audio-driven NeRF works <cit.>, we also utilize LPIPS <cit.> loss ℒ_lpips to capture sharp details. 
For a given input frame I, the overall loss function of the canonical stage is denoted as ℒ_can=  ℒ_L1 + λ_lpipsℒ_lpips
    + λ_D-SSIMℒ_D-SSIM.
During the deformation stage, we employ an additional loss function on the lip area of the talking head.
Specifically, we apply a reconstruction loss for the image patch obtained by cropping where the lips are located based on the facial landmarks <cit.>.
Thus, the total loss function for the deformation stage can be formulated as ℒ_deform = ℒ_can
     + λ_lipℒ_lip.
Note that the deformed 3D Gaussians are directly splatted onto the combined background and torso image, in order to render the head with the background and torso, a common technique that prevents noise around the facial contours <cit.>. A more detailed explanation of this technique can be found in the supplementary file. 



§ EXPERIMENTS







 §.§ Experimental Settings



  §.§.§ Dataset and pre-processing. 

For each target subject, we require several minutes of talking portrait video with a corresponding audio track for training. Specifically, the datasets are obtained from publicly-released video datasets utilized in previous NeRF-based works <cit.>, averaging 6,000 frames for each video at 25 fps. We also perform experiments on selected video clips sourced from the HDTF dataset. <cit.>.
Each portrait video is cropped and resized to 512 × 512, apart from the Obama video, which is of the resolution 450 × 450.  We split each video into train and test sets at a ratio of 10:1, following the pre-processing steps introduced in AD-NeRF <cit.>.




  §.§.§ Comparison baselines.

We comparatively evaluate our proposed GaussianTalker framework against recent NeRF-based approaches tackling the same task. We introduce two variants of our method: the full model GaussianTalker with L=2 cross-attention layers and a lightweight version, GaussianTalker^*, with L=1 layer. 
Our method is compared with the recent NeRF-based approaches that address the same problem settings. We utilize three models as baselines: AD-NeRF <cit.>, RAD-NeRF <cit.>, and ER-NeRF <cit.>. For fair comparison, we implement each method by utilizing the torso part from the ground-truth frames. 
Additionally, we include a comparison with one-shot 2D talking head models, such as Wav2Lip <cit.> and PC-AVS <cit.>, to provide a wide range of comparisons.





 §.§ Quantitative Evaluation



  §.§.§ Comparison settings and metrics. 

Following previous NeRF-based works <cit.>, our comparisons are structured into two distinct settings: self-driven and cross-driven. In the self-driven setting, we evaluate the accuracy of head reconstruction for a particular identity using the test subset. We employ several reconstruction metrics including peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS). Notably, these metrics are exclusively measured on the facial region. We also measure realism of the reconstructed face using Fréchet Inception Distance (FID) <cit.> and identity preservation of the animated video using Cosine Similarity of Identity Embedding (CSIM) <cit.>. 

For the cross-driven setting, all methods are driven by entirely unrelated audio tracks to evaluate lip synchronization. 
The audio clips used in this setup were extracted from demos of SynObama <cit.>.
Due to the absence of ground-truth images, we assess lip sync accuracy with landmark distance (LMD) and SyncNet confidence score (Sync). We also employ action units error (AUE) to measure the precision of facial movements. Finally, we compare the training time and frames-per-second (FPS) as measures to evaluate the efficiency of each method. 



  §.§.§ Self-driven evaluation. 
 
The self-driven evaluation results are presented in tab:selfdriven. Note that Wav2Lip <cit.> scores for PSNR, SSIM and LPIPS are not valid as it takes ground truth images as input.
While the one-shot 2D-based methods, Wav2Lip and PC-AVS generate results with high synchronization scores, they fall short in the faithful reconstruction, showing low PSNR and LPIPS scores. 
Benefiting from the 3DGS representation, GaussianTalker achieves comparable image fidelity with significantly faster rendering speeds (over 120 fps for GaussianTalker*). Our method also shows the best scores in most metrics while reaching higher score than other NeRF-based baselines in Sync scores. The results show that our method can synthesize high lip-sync accurate 3D heads in real time rendering speeds. 



  §.§.§ Cross-driven evaluation. 

Results in Table <ref> showcase successful lip movement synthesis with general audio input.
GaussianTalker consistently exhibits the highest Sync score among NeRF-based methods, demonstrating its effectiveness in handling unseen audio for lip synchronization. 
These results highlight GaussianTalker's ability to generate high-fidelity 3D heads with real-time rendering speeds and accurate lip synchronization even with diverse audio inputs.







 §.§ Qualitative Evaluation

In fig:full_qual, we showcase results from self-driven and cross-driven experiments. We choose four key frames from each of the two experiment settings to compare the reconstruction quality and lip-sync accuracy.
While 2D-based methods (Wav2Lip, PC-AVS) excel in lip synchronization, they for short of generating a faithful and consistent face when the head is rotated. 
AD-NeRF suffers from blurry reconstructions due to its lack of eye blink control. RAD-NeRF and ER-NeRF, while demonstrating improved facial consistency, can exhibit discrepancies in lip synchronization and fail to capture hair movement during head rotations. 

In contrast, GaussianTalker generates photorealistic images with intricate details in non-rigid regions like eyes and wrinkles. Our spatial-audio attention module effectively disentangles audio-driven motions from scene variations, enabling precise control of mouth movements. This capability allows our model to capture hair movement realistically when the head rotates, leading to superior overall head reconstruction fidelity.
In order to comprehensively visualize the efficacy of our proposed method, we provide the rendered videos in the supplementary file. The provided supplementary video demonstrates impressive lip synchronization capabilities and high fidelity head reconstruction with realistic motion. 







 §.§ Ablation Study

In this section, we provide ablation studies to validate the efficacy of the design choices of our model.
We also show detailed visualizations of the generated results in the supplementary material for better comparison. 








  §.§.§ Attribute conditions for triplane

Our proposed triplane encodes the facial information of the canonical 3D head learned by 3D Gaussians. The mechanism also enforces spatial relationships between Gaussians for better deformation. 
In tab:canonical_attribute_selection, we demonstrate the effectiveness of this approach by conducting quantitative ablation on the selection of attributes that are conditioned on the embedding f(μ_c). 
We also provide results where all attributes are optimized separately following the original implementation, and the triplane is trained in the deformation stage. 
Utilizing only subsets of the Gaussian attributes show lower performance in lip synchronization and precision. Removing the attribute conditions during training leads to loss of spatial information embedded in the triplane embeddings, leading to a lack of facial cohesion during inference time. 

 
 
 







  §.§.§ Selection of deformed attributes.

A major challenge of manipulating the Gaussians is the magnitude of the parameters that need to be controlled. While estimating offsets for only a subset of attributes could reduce computational load, it may compromise overall fidelity due to the lack of control. To address this, in tab:deformed_attribute_selection, we investigate different selections of Gaussian attributes for deformation. 
Controlling only SH and α makes the formulation similar to conditional NeRF-based works <cit.>. 
Because 3DGS is an explicit representation that specifies the 3D positions and shapes, only controlling the appearance attributes leads to loss of overall fidelity. 
However, only controlling attributes that make up the position and shape of 3D Gaussians show lower reconstruction accuracy. Deformation of all Gaussian attribute is crucial for the highest fidelity and superior lip synchronization. 





  §.§.§ Disentanglement of audio-unrelated motion.

We also investigate the significance of using augmented conditions, such as eye blink, facial viewpoint, and null-vector. We evaluate the influence of additional conditions on image fidelity and lip synchronization by selectively removing them during training (Table <ref>). The lower reconstruction scores are attributed to the low lip-sync accuracy due to entanglement of verbal motion and scene variations unrelated to audio. In the supplementary material, we also visualize the attention scores of each comparison experiment for detailed analysis. 




  §.§.§ Stagewise optimization
 
In tab:ablation_stagewise, we investigate the importance of employing a separate canonical stage. We opt to optimize the whole architecture by training each of the module simultaneously from scratch. 
While the final generated results show similar performance, optimizing the coarse facial geometry before training the deformation network results in faster optimization of the whole methodology. 








§ CONCLUSION

In this work, we have proposed GaussianTalker,  a novel framework for real-time pose-controllable 3D talking head synthesis, leveraging the 3D Gaussians for the head representation. Our method enables precise control over Gaussian primitives by conditioning features extracted from a multi-resolution triplane. Additionally, the integration of a spatial-audio attention module facilitates the dynamic deformation of facial regions, allowing for nuanced adjustments based on audio cues and enhancing verbal motion disentanglement.
Our method is distinguished from prior NeRF-based methods by its superior inference speed and high-fidelity results for out-of-domain audio tracks.
The efficacy of our approach is validated by quantitative and qualitative analyses. We look forward to enriched user experiences, particularly in video game development, where real-time rendering capabilities of GaussianTalker promise to enhance interactive digital environments.



ACM-Reference-Format


Appendix



In the following, we describe the implementation details and further analyses of GaussianTalker. 
Specifically, we first introduce the details of our network design and hyperparameter settings in supsec:implementation_details. We also provide details of our analysis on the proposed method that was conducted in the main paper in sup:analysis. In sup:additional_exp, we validate our methodology with more qualitative results from our experiments, and also conduct a user study. Then, more ablation studies are conducted in sup:ablation_studies.
To further demonstrate the robustness and effectiveness of our framework, we also provide a supplementary video (sup:video). Finally, we discuss the limitations and ethical considerations of our research in sup:ethical_considerations.












§ IMPLEMENTATION DETAILS




 §.§ Network architecture



  §.§.§ Multi-resolution Triplane.

Our multi-resolution triplane consists of three orthogonal grids, with the hidden feature dimension of H=64, and its base resolution of R=64, which is further upsampled by 2.  




  §.§.§ Canonical 3D Gaussian attribute predictor.

The employed network that predicts the attributes of canoncial 3D Gaussians is made up of MLPs, such as: ℱ_can = {ϕ_shared, ϕ_r, ϕ_s, ϕ_SH, ϕ_α}. Specifically, a tiny MLP ϕ_shared encodes the triplane embedding f(μ_c) and outputs a shared feature κ for all attributes. The following MLP regressors maps this feature to each 3D Gaussian attribute such as:

    κ = ϕ_shared(f(μ)), 
    
        r_c = ϕ_r(κ),  s_c = ϕ_s(κ),  SH_c =ϕ_SH(κ), α_c = ϕ_α(κ).




  §.§.§ Deformation offset predictor.

Similar to ℱ_can, the deformation prediction network, ℱ_can = {ψ_μ, ψ_r, ψ_s, ψ_SH, ψ_α}, that estimates the deformation offsets of each Gaussian attribute for each frame consists of several small MLP regressors. For the n-th frame, the final output embedding from the cross-attention module, z^L_n, is mapped to each attribute offset such that 

    Δμ_n = ψ_μ(z^L_n),  Δr_n = ψ_r(z^L_n), Δs_n = ψ_s(z^L_n), 
    ΔSH_n = ψ_SH(z^L_n),  Δα_n = ψ_α(z^L_n).







 §.§ Hyperparameter Configuration

During the canonical stage, we conduct training over 8,000 iterations for a specific identity. We set the weights for the loss functions as follows: λ_1 = 0.8, λ_lpips = 0.01, and λ_D-SSIM = 0.2. The initial learning rate for the multi-resolution triplane is set to 0.0016, gradually decaying to 0.00016. Similarly, the learning rate for ℱ_can starts at 0.0001 and diminishes to 0.00001. We cap the maximum number of 3D Gaussians at 50,000, and we abstain from utilizing the opacity reset operation from the original implementation <cit.>, as we found it does not yield discernible benefits in our experiments.

Subsequently, in the deformation stage, we proceed with training the network for 8,000 iterations. We maintain the same weighting scheme for the loss functions: λ_1 = 0.8, λ_lpips = 0.01, λ_D-SSIM = 0.2, and λ_lip = 0.8. All modules are trained with an initial learning rate of 0.0001, gradually decreasing to 0.00001.

While our spatial-audio attention module primarily employs L=2 cross-attention layers, our modified GaussianTalker^* with L=1 can achieve comparable results with even faster inference speeds.







§ ADDITIONAL ANALYSIS



 §.§ Splatting on the background image



Initially, our research followed the method outlined in the original implementation <cit.>, where faces were generated on a white background. However, we encountered limitations with this approach. To render images containing only faces on a white background, corresponding ground truth images with similar characteristics were required, necessitating the use of a segmentation model. However, due to the inherent inaccuracies of the segmentation model, the obtained facial masks tended to encompass larger areas, including the background. Additionally, the disproportionate emphasis of loss terms such as SSIM and perceptual loss on imperfect facial contours relative to mouth and eye movements hindered the learning process.

As a solution, we opted to generate faces  GT backgrounds instead. This approach allowed for the accurate learning of Gaussian presence boundaries by distributing loss across the entire image. Similar to preprocessing techniques employed in previous NeRF-based works <cit.>, we interpolated the human form from the background image to create an image with the person removed. Subsequently, faces were directly rendered using Gaussian methods, enabling comparisons with GT videos. By adopting this strategy, our GaussianTalker is trained without the need for facial mask, facilitating the faithful representation of intricate details such as hair.









 §.§ Visualization of Attention


In our spatial-audio attention module, the computation of the attention score is formalized by the following equation:

    A_n^l = softmax(q k^⊺_n)^l/√(d_k),

where l denotes the index of {a_n,e_n,υ_n, } and A_n^l corresponds to its calculated attention score. A_n denotes the concatenation of all A_n^l, resulting in a shape of B×H×N×d_k, which respectively indicate batch size, number of heads, number of Gaussians, and number of features per Gaussian.



For each attention score A_n^l, we visualize the attention by assigning the score to RGB values. Thereby we obtain attention visualization colors c_att for each Gaussian. The overall visualization of attention is then calculated with the typical 3DGS rendering process following (<ref>).
This formulation allows us to visually interpret the model’s focus within the generated representations, effectively highlighting the areas of greatest feature impact.






 §.§ Visualization of triplane

Fig. 3 of the main paper visualizes the PCA analysis result of our multi-resolution triplane, showing the efficacy of using triplane to embed Gaussian features. We perform PCA on each triplane with dimensions H× R × R, linearly transforming the first dimension down to three principal components, resulting in dimensions 3× R × R. 
Subsequently, the values of the first dimension are normalized between [0, 255] to denote RGB values. As a result, in all xy, yz, and zx triplanes, semantically close facial regions are consistently represented with similar colorations.



§ ADDITIONAL EXPERIMENTS
 


 §.§ Additional qualitative experiments.

We present additional visualization of generated keyframes from comparison experiments in the self-driven setting and the cross-driven setting in supfig:self-driven and supfig:cross-driven respectively. These experiments showcase the stability of our method and its applicability to various identities. 




 §.§ User study

Following previous works <cit.>, we conducted a user study in order to better judge the visual quality of the generated talking head videos.
21 participants with an age range of 20-40 years old were solicited to evaluate the rendered results in the head reconstruction setting. 
For accurate judgments, we combine all generated videos into a single high-resolution video, enabling simultaneous observation of all movements by the participants. To ensure fairness in the comparison process, we assign a number to each generated result instead of identifying them by their method. Participants were asked to evaluate the three perspectives of the generated portraits: (1) Lip-sync Accuracy; (2) Video Realness; and (3) Image Quality. The results are shown in tab:user_study. 






































§ ABLATION STUDIES
 


 §.§ Initialization of μ_c

Our study explores the impact of initialization on canonical 3D Gaussian optimization. In the default setting, we leverage a pre-optimized Basel Face Model <cit.> to obtain camera parameters during preprocessing. These optimized mesh vertices are used to initialize the 3D positions, μ_c of the 3D Gaussians.

To investigate the impact of the proposed 3DMM-based initialization, we conduct an ablation study by comparing it to random initialization from a sphere. In supfig:ablation_init, we visually analyze the optimization process of the canonical stage under both initialization settings. 
Our experiments demonstrate that utilizing 3DMM-based initialization leads to faster convergence, attributed to the facial depth information encoded in the initialized points.





 §.§ Selection of attributes inferred for triplane embeddings.

In supfig:ablation_triplane_conditioned_attributes, we support the quantitative comparison in the main paper by presenting key frames of the rendered results. 
Conditioning the triplane embeddings on the structure information such as r and s tends to show less accurate facial details such as wrinkles in facial muscle.
In contrast, while conditioning on appearance information SH and α produce accurate reconstructions of the canonical head, the facial motion appears less dynamic compared to the ground truth, and does not correlate well with input speech audio.




 §.§ Selection of deformed attributes.

We also provide qualitative comparisons from our ablation study on selection of Gaussian attributes to be deformed. Utilizing the same comparison settings from Sec.5.4.2, we visualize the rendered results in supfig:ablation_features. Only deforming SH and α show blurry results with unrealistic deformations, while only manipulating 




 §.§ Disentanglement of audio-unrelated motion

Finally, we reinforce the insights drawn from the quantitative analysis in Section 5.4.3. 
We elucidate the disentanglement of speech-related motion in supfig:ablation_attention_viz by presenting visualizations of the attention scores for the input conditions across the ablation experiment settings. Notably, the attention scores of the input speech audio become more widely distributed across other facial regions, indicating inadequate disentanglement of speech-related motion when solely provided with speech as the input condition.

















§ SUPPLEMENTARY VIDEO

To comprehensively visualize the efficacy of our proposed method in the domain of talking facial video synthesis, we prepared a supplementary video. 
This video encompasses the results and analysis of our experiments presented in the main paper and the supplementary document. 
We showcase talking head videos generated under both the self-driven and cross-driven settings and compare them with previous NeRF-based works <cit.>. We also demonstrate the effectiveness of our spatial-audio attention module by showing how the attention scores of each condition evolve as the scene progresses. 
Lastly, the video includes a set of ablation studies that systematically examine the impact of each component of our proposed method.






§ FURTHER DISCUSSIONS




 §.§ Ethical Considerations

Our goal with GaussianTalker is to create realistic talking 3D heads for practical real-world applications like digital assistants and video production.  However, its photorealism raises ethical concerns, as it's difficult to distinguish real from synthetic videos. This can be used to create deepfakes, which are manipulated videos that can be used to spread misinformation or damage someone's reputation. 
To address this, we propose several measures:  1) informing users about video authenticity, 2) sharing our results with deepfake detection communities to improve detection algorithms, and 3) advocating for digital watermarks in real videos to deter misuse. Finally, we believe responsible use requires clear regulations to govern deepfakes on social media, protecting users from potential manipulation.



 §.§ Limitations and future work

GaussianTalker shares a common limitation with previous NeRF-based talking head synthesis methods: per-identity training. This restricts the model's ability to generalize to new identities, making data preparation for audio and eye features time-consuming. Additionally, free-viewpoint rendering remains a challenge due to the lack of multi-view training data. While the deformation stage achieves high fidelity and generalizes well to out-of-domain audio, it struggles with extreme viewpoints. Our current approach uses limited canonical training for coarse structure, leading to inconsistencies when synthesizing from very different angles.

Future work will focus on overcoming these limitations. We aim to explore techniques for multi-identity training and efficient data pre-processing. Additionally, we will investigate methods for free-viewpoint rendering using techniques like multi-view data acquisition or neural rendering approaches.






































 


























