TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting
Jiahe Li Jiawei Zhang Xiao Bai Jin Zheng Xin Ning Jun Zhou Lin Gu




 



TalkingGaussian






J. Li et al.




School of Computer Science and Engineering, State Key Laboratory of 
 Complex & Critical Software Environment, Jiangxi Research Institute, 
 Beihang University Institute of Semiconductors, Chinese Academy of Sciences School of Information and Communication Technology, Griffith University RIKEN AIP The University of Tokyo


TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting
    Jiahe Li1 Jiawei Zhang1 Xiao Bai1Corresponding author: Xiao Bai (baixiao@buaa.edu.cn). Jin Zheng1 Xin Ning2 Jun Zhou3 Lin Gu4,5 


    Received xx; accepted xx
======================================================================================================================================






Radiance fields have demonstrated impressive performance in synthesizing lifelike 3D talking heads. However, due to the difficulty in fitting steep appearance changes, the prevailing paradigm that presents facial motions by directly modifying point appearance may lead to distortions in dynamic regions. To tackle this challenge, we introduce TalkingGaussian, a deformation-based radiance fields framework for high-fidelity talking head synthesis. Leveraging the point-based Gaussian Splatting, facial motions can be represented in our method by applying smooth and continuous deformations to persistent Gaussian primitives, without requiring to learn the difficult appearance change like previous methods. Due to this simplification, precise facial motions can be synthesized while keeping a highly intact facial feature. Under such a deformation paradigm, we further identify a face-mouth motion inconsistency that would affect the learning of detailed speaking motions. To address this conflict, we decompose the model into two branches separately for the face and inside mouth areas, therefore simplifying the learning tasks to help reconstruct more accurate motion and structure of the mouth region. 
Extensive experiments demonstrate that our method renders high-quality lip-synchronized talking head videos, with better facial fidelity and higher efficiency compared with previous methods.



  





§ INTRODUCTION


Synthesizing audio-driven talking head videos is valuable to a wide range of digital applications such as virtual reality, film-making, and human-computer interaction. Recently,  radiance fields like Neural Radiance Fields (NeRF) <cit.> have been adopted by many methods <cit.> to improve the stability of 3D head structure while providing photo-realistic rendering, which has achieved great success in synthesizing high-fidelity talking head videos.

Most of these NeRF-based approaches <cit.> synthesize different face motions by directly modifying color and density with neural networks, predicting a temporary condition-dependent appearance for each spatial point in the radiance fields whenever receiving a condition feature. This appearance-modification paradigm enables previous methods to achieve dynamic lip-audio synchronization in a fixed space representation. However, since even neighbor regions can also show significantly different colors and various structures on a human face, it's challenging for these continuous and smooth neural fields to accurately fit the rapidly changing appearance to represent facial motions, which may lead to some heavy distortions on the facial features like a messy mouth and transparent eyelids, as shown in Fig. <ref>.

In this paper, we propose TalkingGaussian, a deformation-based talking head synthesis framework, that attempts to utilize the recent 3D Gaussian Splatting (3DGS) <cit.> to address the facial distortion problem in existing radiance-fields-based methods. 
The core idea of our method is to represent complex and fine-grained facial motions with several individual smooth deformations to simplify the learning task. 
To achieve this goal, we first obtain a persistent head structure that keeps an unchangeable appearance and stable geometry with 3DGS. Then, motions can be precisely represented just by the deformation applied to the head structure, therefore eliminating distortions produced from inaccurately predicted appearance, and leading to better facial fidelity while synthesizing high-quality talking heads.

Specifically, we represent the dynamic talking head with a 3DGS-based Deformable Gaussian Field, consisting of a static Persistent Gaussian Field and a neural Grid-based Motion Field to decouple the persistent head structure and dynamic facial motions. 
Unlike previous continuous neural-based backbones <cit.>, 3DGS provides an explicit space representation by a definite set of Gaussian primitives, enabling us to obtain a more stable head structure and accurate control of spatial points. 
Based on this, we apply a point-wise deformation, which changes the position and shape of each primitive while persisting its color and opacity, to represent facial motions via the motion fields. Then the deformed primitives are input into the 3DGS rasterizer to render the target images. To facilitate the smooth learning for a target facial motion, we introduce an incremental sampling strategy that utilizes face action priors to schedule the optimization process of deformation.

In the Deformable Gaussian Fields, we further decompose the entire head as a face branch and an inside mouth branch to solve the motion inconsistency between these two regions, which hugely improves the synthesis quality both in static structure and dynamic performance. Since the motions of the face and inside mouth are not related totally in tight and may be much different sometimes, it is hard to accurately represent these delicate but conflicted motions with just one single motion field. To simplify the learning of both these two distinct motions, we divide these two regions in 2D input images with a semantic mask, and build two model branches to represent them individually. As the motion in each branch has been simplified to become smooth, our method can achieve better visual-audio synchronization and reconstruct a more accurate mouth structure.

The main contributions of our paper are summarized as follows:

    
  * We present a novel deformation-based framework that synthesizes talking heads by applying deformations to a persistent head structure, to escape an inherent facial distortion problem from the inaccurate prediction of changing appearance, enabling the generating of precise and intact facial details.
    
  * We propose a Face-Mouth Decomposition module to facilitate motion modeling via decomposing conflicted learning tasks for deformation, therefore providing accurate mouth reconstruction and lip synchronization.
    
  * Extensive experiments show that the proposed TalkingGaussian renders realistic lip-synchronized talking head videos with high visual quality, generalization ability, and efficiency, outperforming state-of-the-art methods on both objective evaluation and human judgment.






§ RELATED WORK



Talking Head Synthesis. 
Driving talking heads by arbitrary input audio is an active research topic, aiming to reenact the specific person to generate highly audio-visual consistent videos. 
Early methods based on 2D generative models synthesize audio-synchronized lip motions for a given facial image <cit.>. Later advancements <cit.> incorporate intermediate representations like facial landmarks and morphable models for better control, but suffer from errors and information loss during the intermediate estimation. Due to the lack of an explicit 3D structure, these 2D-based methods are short in keeping the naturalness and consistency when the head pose changes.

Recently, Neural Radiance Fields (NeRF) <cit.> has been introduced as a 3D representation of the talking head structure, providing photorealistic rendering and personalized talking style via person-specific training. Earlier NeRF-based works <cit.> suffer from the expensive cost of vanilla NeRF. Successfully driving efficient neural fields <cit.> with audio, RAD-NeRF <cit.> and ER-NeRF <cit.> have gained tremendous improvements in both visual quality and efficiency. To improve the generalizability of cross-domain audio inputs, GeneFace <cit.> and SyncTalk <cit.> pre-train the audio encoder with large audio-visual datasets. However, most of these methods represent facial motions by changing the appearance of each sampling point, which burdens the network with learning the jumping and unsmooth appearance changes, resulting in distorted facial features. Although some works <cit.> have introduced a pre-trained deformable fields module for few-shot settings, the lack of fine-grained point control and precise head structure brings drawbacks in static and dynamic quality. Instead, utilizing 3DGS to maintain an accurate head structure, our method simplifies the learning difficulty of facial motions with a pure deformation representation, therefore improving facial fidelity and lip-synchronization. 



Deformation in Radiance Fields.  Deformation has been widely applied in radiance fields to synthesize dynamic novel views. Some NeRF methods <cit.> use a static canonical radiance field to capture geometry and appearance and a time-dependent deformation field for dynamics. These methods predict an offset referring to the sampling position, which is opposite to the motion path and would bring extra difficulties in fitting. To solve this problem, <cit.> use a deformation that directly warps the canonical fields to represent dynamics. However, this method is costly since the spatial points cannot be accurately and stably controlled in its grid-based NeRF representation.

More recently, 3D Gaussian Splatting <cit.> introduces an explicit point-based representation for radiance fields, where deformation can be easily applied to a definite set of Gaussian primitives to directly warp the canonical fields. Based on this idea, considerable dynamic 3DGS works <cit.> get significant improvements in visual quality and efficiency for dynamic novel views synthesis. However, these methods only aim to remember the fixed motion at each time stamp, insufficient to represent various fine-grained motions driven by conditions, especially on the mouth. Despite some attempts conducted to reconstruct the human head <cit.> driven by parametrized facial models, the mapping from audio to these parameters is not easy to learn and would cause information loss, and thus they can not be easily transferred to our audio-driven task. In this paper, we introduce deformable Gaussian fields with an incremental sampling strategy to facilitate learning multiple complex facial motions from a monocular speech video via pure deformation, and decomposite inconsistent motions of the face and inside mouth areas to improve the quality of delicate talking motions.










§ METHOD




 §.§ Preliminaries and Problem Setting



3D Gaussian Splatting.  
3D Gaussian splatting (3DGS) <cit.> represents 3D information with a set of 3D Gaussians. It computes pixel-wise color 𝒞 with a set of 3D Gaussian primitives θ and the camera model information at the observing view. Specifically, a Gaussian primitive can be described with a center μ∈ℝ^3, a scaling factor s ∈ℝ^3, and a rotation quaternion q ∈ℝ^4. For rendering purposes, each Gaussian primitive also retains an opacity value α∈ℝ and a Z-dimensional color feature f ∈ℝ^Z. Thus, the i-th Gaussian primitive 𝒢_i keeps a set of parameters θ_i = {μ_i, s_i, q_i, α_i, f_i}. Its basis function is in the form of:

    𝒢_i(𝐱) = e^-1/2(𝐱-μ_𝐢)^TΣ_i^-1(𝐱-μ_𝐢),

where the covariance matrix Σ can be calculated from s and q.


During the point-based rendering, a rasterizer would gather N Gaussians following the camera model to compute the color 𝒞 of pixel 𝐱_p, with the decoded color c of feature f and the projected opacity α calculated by their projected 2D Gaussians 𝒢^proj on image plane:

    𝒞(𝐱_p) = ∑_i ∈ Nc_iα_i∏_j=1^i-1(1-α_j),   α_i = α_i 𝒢^proj_i(𝐱_p).

Similarly, the opacity 𝒜∈ [0,1] of pixel 𝐱_p can be given:

    𝒜(𝐱_p) = ∑_i ∈ Nα_i∏_j=1^i-1(1-α_j).


3DGS optimizes the parameters θ for all Gaussians through gradient descent under color supervision. During the optimization process, it applies a densification strategy to control the growth of the primitives, while also pruning unnecessary ones. This work inherits these optimization strategies for color supervision.


Problem Setting.  
In this paper, we aim to present an audio-driven framework based on 3DGS representation for high-fidelity talking head synthesis. Adopting a similar problem setting as NeRF-based works <cit.>, we take a few-minute speech video with a single person as the training data. A 3DMM model <cit.> is utilized to estimate the head pose and therefore to infer the camera pose. To keep aligned with previous works <cit.>, we use a pre-trained DeepSpeed <cit.> model as the basic audio encoder to get a generalizable audio feature from the raw input speech audio. 








 §.§ Deformable Gaussian Fields for Talking Head. 






Despite previous NeRF-based methods <cit.> have achieved great success in synthesizing high-quality talking heads via generating point-wise appearance, they can still not tackle the problem of generating distorted facial features on dynamic regions. One main reason is the appearance space, including color and density, is jumping and unsmooth, which makes it difficult for the continuous and smooth neural fields to fit. In comparison, deformation is another choice to represent motions with better smoothness and continuity, as shown in Fig. <ref>. In this work, we propose to purely use deformation in the Gaussian radiance fields to represent different motions of the talking head in 3D space. In particular, the whole representation is decomposed into Persistent Gaussian Fields and Grid-based Motion Fields, as shown in Fig. <ref>. These fields will be further refined for different regions in the next section.


Persistent Gaussian Fields. 
Persistent Gaussian Fields preserve the persistent Gaussian primitive with the canonical parameters θ_C = {μ, s, q, α, f}. Firstly, we initialize this module with the static 3DGS by the speech video frames to get a coarse mean field. Later, it attends a joint optimization with the Grid-based Motion Fields.  


Grid-based Motion Fields. 
Although the primitives in Persistent Gaussian Fields can effectively represent the correct 3D head, a regional position encoding is lacking due to their fully explicit space structure. Considering most facial motions are regionally smooth and continuous, we adopt an efficient and expressive tri-plane hash encoder ℋ <cit.> for position encoding with an MLP decoder to build Grid-based Motion Fields for a continuous deformation space. 

Specifically, the motion fields aim to represent the facial motion by predicting a point-wise deformation δ_i = {Δμ_i, Δ s_i, Δ q_i} for each primitive with the input of its center μ_i, which is irrelevant to the color and opacity changing. For the given condition feature set 𝐂, the deformation δ_i can be calculated by:

    δ_i = MLP(ℋ(μ_i) ⊕𝐂),

where ⊕ denotes concatenation.

Through a 3DGS rasterizer, these two fields are combined to generate deformed Gaussian primitives to render the output image, of which the deformed parameters θ_D are got from the canonical parameters θ_C and deformation δ:

    θ_D = {μ+Δμ, s+Δ s, q+Δ q, α, f}.



Optimization with Incremental Sampling. 
 
While learning the deformation, once the target primitive position is too far from the predicted results, the gradient would vanish and thus the motion fields may fail to be effectively updated. To tackle this problem, we introduce an incremental sampling strategy. Specifically, we first find a valid metric m (e.g. action units <cit.> or landmarks) to measure the deformation degree of each target facial motion. Then, at the k-th training iteration, we use a sliding window to sample a required training frame at position j, of which the motion metric m_j satisfies the condition:

    m_j ∈ [B_lower+k × T, B_upper+k × T],

where B_lower and B_upper denote the initial lower and upper bound of the sliding window, and T denotes the step length.
This selected training frame can offer sufficient new knowledge for the deformable fields to learn, but would not be too hard.

To avoid catastrophic forgetting, we apply the incremental sampling strategy once every K iterations.





 §.§ Face-Mouth Decomposition


Although the Grid-based Motion Fields can predict the point-wise deformation at arbitrary positions due to the continuous and dense 3D space representation, this representation still encounters a granularity problem caused by the motion inconsistency between the face and the inside mouth. Since the inside area of the mouth is spatially too close to the lips but does not always move together, their motions would interfere with each other in a single interpolation-based motion field. This can also further lead to a bad reconstruction quality in static structure as well, as shown in Fig. <ref>. 

To tackle this problem, we propose decomposing these two regions in 3D space and building two individual branches with separate optimization.
For each training video frame, we first use the off-the-shelf face parsing models to get a semantic mask of the inside mouth region in 2D space [Additional descriptions and details can be found in the supplementary material. ]. Then, we take the masked image of the inside mouth and the remaining surface region (containing the face, hair, and other head parts) to train two separate deformable Gaussian fields as two branches of our framework.





Face Branch. 
The face branch serves as the main part to fit the appearance and motion of the talking head, including all facial motions except the one of the inside mouth. 
In this branch, we adopt a region attention mechanism <cit.> in the Grid-based Motion Fields to facilitate the learning of the conditioned deformation driven by the features of audio a and upper-face expression e. To fully decouple these two conditions, the upper-face expression feature e is composed of 7 action units <cit.> that are explicitly irrelevant to the mouth. The deformation δ_i^F for the i-th primitive in the face branch can be predicted by: 

    δ_i^F = MLP(ℋ^F(μ_i) ⊕a_r, i⊕e_r, i),

where a_r, i = V_a, i⊙a and e_r, i = V_e, i⊙e denote the region-aware feature at position μ_i in the region attention mechanism, calculated by the attention vectors V_a, i and V_e, i with the Hadamard product ⊙.




During the optimization, we apply the Incremental Sampling strategy for the lips action and eye-blinking. Specifically, we measure the lips opening degree by the height of the mouth area according to the detected facial landmarks, and use AU45 <cit.> to describe the degree of eye close. Then, we gradually move the sliding window to guide the face branch to learn the deformations of the lips from close to open and eyes from open to close.




Inside Mouth Branch. 
The inside mouth branch represents the audio-driven dynamic inside mouth region in 3D space. Considering the inside mouth moves in a much simpler manner and is only driven by audio, we use a lightweight deformable Gaussian field to build this branch. In particular, we only predict the translation Δμ_i conditioned by the audio feature a for the i-th primitive:

    δ_i^M = {Δμ_i^M} = MLP(ℋ^M(μ_i) ⊕a).


To get a better reconstruction quality of the teeth part, we apply an incremental sampling strategy that smooths the learning of the overlapping between teeth and lips with the quantitative metric AU25 <cit.>.


Rendering. 
The final talking head image is fused with the two rendered face and inside mouth images. Based on the physical structure, we assume the rendering results from the Inside Mouth Branch are behind that from the Face Branch. Therefore, the talking head color 𝒞_head of pixel 𝐱_p can be rendered by:

    𝒞_head(𝐱_p) = 𝒞_face(𝐱_p) ×𝒜_face(𝐱_p) 
                        + 𝒞_mouth(𝐱_p) × (1 - 𝒜_face(𝐱_p)),

where  𝒞_face and 𝒜_face denote the predicted face color and opacity from the face branch, and 𝒞_mouth is the color predicted by the inside mouth branch.





 §.§ Training Details

We keep the basic 3DGS optimization strategies to train our framework. The full process can be divided into three stages, of which the first two stages are individually applied for the two branches and the last stage is for fusion.


Static Initialization. 
At the beginning of the training, we first conduct an initialization via the vanilla 3DGS for the Persistent Gaussian Fields to get a coarse head structure. Following 3DGS, we use a pixel-wise L1 loss and a D-SSIM term to measure the error between the image ℐ̂_C rendered by parameters θ_C and the masked ground-truth image ℐ_mask for each branch:

    ℒ_C = ℒ_1(ℐ̂_C, ℐ_mask) + λℒ_D-SSIM(ℐ̂_C, ℐ_mask) .



Motion Learning. 
After the initialization, we add the motion fields into training via its predicted deformation δ. In practice, we take the deformed parameters θ_D from Equation <ref> as the input for the 3DGS rasterizer to render the output image ℐ̂_D. The loss function is:

    ℒ_D = ℒ_1(ℐ̂_D, ℐ_mask) + λℒ_D-SSIM(ℐ̂_D, ℐ_mask) .



Fine-tuning. 
Finally, a color fine-tuning stage is conducted to better fuse the head and inside mouth branches. We calculate the reconstruction loss between the fused image ℐ̂_head rendered by Equation <ref> and the ground-truth video frame ℐ̂ with pixel-wise L1 loss, D-SSIM, and LPIPS terms:

    ℒ_F = ℒ_1(ℐ̂_head, ℐ) + λℒ_D-SSIM(ℐ̂_head, ℐ) + γℒ_LPIPS(ℐ̂_head, ℐ).

At this stage, we only update the color parameter f ∈θ_C and stop the densification strategy of 3DGS for stability.




§ EXPERIMENT






 §.§ Experimental Settings


Dataset.  
We collect four high-definition speaking video clips from previous publicly-released video sets <cit.> to build the video datasets for our experiments, including three male portraits "Macron", "Lieu", "Obama", and one female portrait "May". The video clips have an average length of about 6500 frames in 25 FPS with a center portrait, three  ("May", "Macron", and "Lieu") of which are cropped and resized to 512×512 and one  ("Obama") to 450×450. 


Comparison Baselines. 
In the experiments, we mainly compare our method with the most related NeRF-based methods AD-NeRF <cit.>, DFRF <cit.>, RAD-NeRF <cit.>, GeneFace <cit.> and ER-NeRF <cit.>, which render talking head via person-specific radiance fields trained with speech videos. Additionally, we also take the state-of-the-art 2D generative models (Wav2Lip <cit.>, IP-LAP <cit.> and DINet <cit.>), which do not need person-specific training, and person-specific methods (SynObama <cit.>, NVP <cit.>, and LSP <cit.>) as the baselines.







Implementation Details. 
Our method is implemented on PyTorch. For a specific portrait, we first train both the face and inside mouth branches for 50,000 iterations parallelly and then jointly fine-tune them for 10,000 iterations. Adam <cit.> and AdamW <cit.> optimizers are used in training. In the loss functions, λ and γ are set to 0.2 and 0.5. All experiments are performed on RTX 3080 Ti GPUs. The overall training process takes about 0.5 hours. A pre-trained DeepSpeech model <cit.> is used as a basic audio feature extractor.





 §.§ Quantitative Evaluation








Comparison Settings. 
To evaluate the reconstruction quality and lip-audio synchronization ability, our quantitative comparison contains two settings: 1) The self-reconstruction setting, where we split each of the four videos into training and test sets, and use the audio, expression, and pose sequences in the unseen test set to reconstruct the talking head in a self-driven way for quality evaluation.  2) The lip-synchronization setting, where we use the audio track from other videos to drive the models trained in the first setting and evaluate lip-synchronization, focusing on situations with cross-domain input audios. Specifically, we use the same audio samples as previous works <cit.> from NVP and SynObama as two test audios A and B to evaluate the "Obama" and "May" portraits. Since both audio A and B are from unseen videos with male voices, the evaluation results, especially on "May" with a different gender, can well illustrate the generalization ability. Tests for NVP, SynObama, and SSP-NeRF are conducted only on their released demos due to the lack of training codes. 




Metrics and Measurements.
In the aspect of static image quality, we employ PSNR for the overall quality, LPIPS <cit.> for high-frequency details, and SSIM <cit.> to evaluate face structure. For dynamic motions, we also utilize the landmark distance  (LMD) <cit.>  and the confidence score (Sync-C) and error distance (Sync-D) of SyncNet <cit.> for lip synchronization. Additionally, we estimate the action units <cit.> of the videos by OpenFace <cit.> and divide them into an upper-face action unit error (AUE-U) and lower-face action unit error (AUE-L) according to their definitions to separately evaluate the upper-face and mouth motions.

In the first setting, we measure the PSNR and LPIPS on the whole image, and SSIM on the face region. We also record the person-specific training time and inference FPS of all methods. Since all the 2D-based generative baselines do not modify the upper part of the face, we do not measure their AUE-U. Notably, we provide another video clip as the image reference for Wav2Lip to avoid information leakage, for which PSNR, LPIPS, and SSIM are not valid. In the second setting, we use the non-comparison-based Sync-C and Sync-E to quantitatively measure the lip-synchronization quality.



 












 

Evaluation Results. 
We report the results of the two settings in Table <ref> and Table <ref>, respectively. Considering the upper-face expression condition would also influence performance <cit.>, we add our upper-face feature e to ER-NeRF <cit.> as "ER-NeRF+e" for a fair comparison. 
1) In the self-reconstruction setting, our method achieves the best overall image quality, motion quality, and efficiency. For image quality, our method performs best in rendering accurate details (LPIPS) and structure (SSIM), due to our deformation-based motion representation and persistent head structure. 

In the aspect of motion quality, our method outperforms all NeRF methods in all metrics. Notably, TalkingGaussian gets a Sync-C score even higher than the generative method IP-LAP and DINet, demonstrating the powerful modeling ability of our method. Although Wav2Lip gets the best scores in Sync-C, its shortcomings in preserving personal talking styles lead to poor AUE-L and LMD. Moreover, due to the efficiency improvement brought by 3DGS, our method reaches the fastest training and inference speed in all baselines.
2) In the results in the lip-synchronization setting, our method shows the best generalization performance. Although ER-NeRF can also get good scores on "Obama", it performs much worse in the more challenging cross-gender situation of "May". This phenomenon can also be observed in many other NeRF-based methods, especially RAD-NeRF which keeps a complex audio encoding module. Surprisingly, AD-NeRF performs well in this situation, despite having a relatively blurry rendering. Besides the difference in model quality, we consider this decreasing generalizability also to be caused by the overfitting of the audio feature when previous NeRF-based methods try to fit the unsmooth changing appearance to reconstruct delicate talking heads. Instead, when just using the same unimproved audio feature extractor as most previous baselines <cit.>, our method can simultaneously keep high-level static rendering quality and best generalization ability for various training videos and input audios, thanks to our simpler and smoother deformation-based motion representation. 









 §.§ Qualitative Evaluation




Evaluation Results.  
To qualitatively evaluate the synthesis quality, we show the keyframes of a reconstructed sequence from the self-reconstruction setting and details of four portraits in Fig. <ref> and <ref>. 
1) Comparing the synthesized motion sequence in Fig. <ref>, our TalkingGaussian outperforms other methods by generating better visual-audio synchronized results. The generative methods (Wav2Lip, IP-LAP, and DINet) are short in generating high-quality images, as the trade-off for their one or few-shot ability. Lack of precise control signals, most NeRF-based baselines can not control audio-independent actions like eye blinking (orange arrow).  

While most NeRF-based baselines fail to synthesize some difficult mouth actions (blue box), our TalkingGaussian can precisely reappear these motions, without introducing any advanced audio encoders <cit.>. This demonstrates the effectiveness of the learning task simplifications brought by our two decomposition designs.
2) The comparison of synthesized details in Fig. <ref> shows our advantages in facial fidelity and fidelity. As illustrated in Sec. <ref>, both RAD-NeRF and ER-NeRF exhibit distorted (red arrow) and blurry (yellow arrow) facial features in the dynamic regions. Even using the more precise facial landmarks to condition the NeRF renderer, GeneFace still can't escape from this inherent trouble brought by the previous appearance-modification paradigm. By purely representing motions with deformation, our method tackles this problem and succeeds in synthesizing more accurate and intact facial features.






User Study.  
To better judge the visual quality in real scenarios judged by humans, we conducted a user study where a total of 32 talking head videos were generated by 8 methods. Then we invited 16 attendees to rate these methods according to their generated results from three aspects: (1) Lip-sync Accuracy; (2) Video Realness; and (3) Image Quality. The results are reported in Table <ref>, in which our TalkingGaussian performs the best in all three aspects, demonstrating the potential value of our method in real-world applications.










 §.§ Ablation Study












To prove the effectiveness of our contributions, we conduct the ablation study using the self-reconstruction setting. The results are reported in Table <ref>.


Motion Representation.  
First, we use the same module as our motion fields to predict the deformation and appearance modification for the Tri-Hash backbone from ER-NeRF <cit.> and 3DGS <cit.> to evaluate the motion representations. Due to the lack of accurate point-wise controls, deformation performs worse on the NeRF-based Tri-Hash. After introducing 3DGS, deformation shows its advantage in preserving better facial features and bringing higher image quality scores. We also visualize the results on 3DGS with the same conditions in Fig. <ref> for comparison. The results demonstrate the effectiveness of both our 3DGS-based persistent head structure and deformation-based motion representation. 


Face-Mouth Decomposition (FMD).  We apply our FMD to the 3DGS backbone to illustrate its effect. Although the results show that FMD can help in lip-synchronization in all situations, the improvement is much larger when combined with deformation, since it can solve the face-mouth motion conflict that seriously affects deformation learning. With this combination, our framework successfully reaches the best motion quality, especially in lip-synchronization. 



Incremental Sampling (IS). Raised by the unstable optimization process of deformation,  some jitters, incomplete motions, and errors in geometry structure can be observed in the generated videos, which lead to a lower SSIM. These problems are relieved after applying IS, demonstrating its contribution to generating more smooth and realistic talking heads. 





§ CONCLUSION
 



This paper represents a novel deformation-based framework TalkingGaussian for high-quality 3D talking head synthesis. Our study is the first to reveal a "facial distortion" problem caused by inaccurate predictions of the rapidly changing appearance. By maintaining a persistent head structure with 3DGS and decomposing inconsistent motions into different spaces, TalkingGaussian addresses this problem with a deformation paradigm and achieves superior performance in synthesizing realistic and accurate talking heads compared to existing methods.


Ethical Consideration.  We hope our method can promote the healthy development of digital industries. However, it must be noted that our method may be misused for malicious purposes and cause negative influence. We recommend the responsible use of this technique. As part of our responsibility, we will also assist in developing deepfake detection techniques by sharing our generated results.







 



J. Li et al.




School of Computer Science and Engineering, State Key Laboratory of 
 Complex & Critical Software Environment, Jiangxi Research Institute, 
 Beihang University Institute of Semiconductors, Chinese Academy of Sciences School of Information and Communication Technology, Griffith University RIKEN AIP The University of Tokyo

Supplementary Material for TalkingGaussian
    Jiahe Li1 Jiawei Zhang1 Xiao Bai1Corresponding author: Xiao Bai (baixiao@buaa.edu.cn). Jin Zheng1 Xin Ning2 Jun Zhou3 Lin Gu4,5

    Received xx; accepted xx
====================================================================================================================================









§ OVERVIEW

In the supplementary material, we first report additional experiments in Sec. <ref>. We further show an additional visualization in Sec. <ref>, and discuss the responsibility to human subjects and ethical consideration in Sec. <ref> and <ref>. Limitations and future work are summarized in Sec. <ref>. A supplementary video is also provided as an additional illustration.



§ ADDITIONAL EXPERIMENTS 



 §.§ Hybrid Motion Representation
 
We additionally conduct an experiment to explore whether a hybrid representation of mixing deformation and appearance modification could benefit the performance. The results are reported in Table <ref>. Upon the basic deformation δ, we first evaluate the setting of additionally predicting a factor to adjust the opacity α. Then we further try to predict the RGB color of each primitive directly rather than using the SH feature f. The results show that: 1) Adding alpha would not help more. This is reasonable since the opacity of each primitive should be a static value. 2) Adding RGB would result in a blurry rendering. In the first aspect, it would bring a larger burden for the network to store more information. On the other hand, a non-persistent color can still suffer from the distortion problem from inaccurate appearance prediction to some extent. 





 §.§ Extension
 
Besides the experiment settings in the main paper, our method is scalable and can extend to a wider range of applications. 


Audio Feature Extractor. 
Following previous baselines, we have adopted a pre-trained DeepSpeech <cit.> model to extract audio features in the main experiments, for a fair comparison. In fact, our method can also easily connect to more powerful feature extractors and become stronger. Table <ref> shows our performance with different audio feature extractors Wav2Vec 2.0 <cit.> and HuBERT <cit.> under the self-reconstruction setting. The results demonstrate a high-quality audio feature could boost the performance of TalkingGaussian, especially on lip-synchronization, showing the growth potential of our approach.









Cross-Lingual and Cross-Gender. 
Our method can also applied to more challenging cross-lingual and cross-gender cases. In this experiment, we collect 4 training videos, in which 2 males and 2 females with English and French audio are included, and use challenging 3 test audio clips to drive their corresponding models. The three test audios consist of two German audio clips separately with a female and a male voice, and a Chinese audio clip with a male voice. In this setting, we compare our method to two baselines ER-NeRF <cit.> and GeneFace <cit.> that utilize different extractors. 

In Table <ref>, the results show that our models all perform better than the baselines while using the same extractors. Notably, GeneFace has further used the HuBERT features to pre-train an intermediate representation on a large audio-visual corpus to enhance its generalizability for cross-domain audios. The generated results have been provided in the supplementary video.








Singing. 
While inputting a song, our method can even synthesize high-quality singing talking heads with no such training audio included. This demonstrates our surprising generalization ability and robustness for cross-domain inputs, and shows applicability for a wider range of situations. The generated videos can be found in the supplementary video.





§ ADDITIONAL VISUALIZATION 


Here we show some high-definition synthesized frames in Fig. <ref> for a convenient and intuitive visualization. Compared with current SOTA methods GeneFace <cit.> and ER-NeRF <cit.>, our method performs best in image quality while retaining a high lip-sync accuracy. We have also provided more additional results in our supplementary video. We strongly recommend watching it for better visualization.










§ DATASET DECLARATION 

In the experiments, all of the multimedia datasets we used were obtained from existing works <cit.>. To our knowledge, most of these data are collected from the internet. In our work, we have tried our best to use data containing only public figures to avoid invading personal privacy. All the data are manually checked to reduce the existence of offensive content.




§ ETHICAL CONSIDERATION 

As our target, we hope our TalkingGaussian can promote the healthy development of digital industries. However, it must be noted that our method may be misused for malicious purposes and cause negative influence. We recommend the responsible use of this technique:


    
  * Informed Consent. Whenever this technique is in use for spread purposes, ensure that all individuals in the training data have provided explicit, informed consent. 
    
  * Disclosure. Please disclose the use of our method, and any other deepfake techniques as well, in all synthesized products. This is critical to ensure all audiences are aware that the content is real, and may include misleading information.


For protection purposes, we will support the development of more powerful deepfake detectors to alert people to the presence of fake content.






§ LIMITATIONS AND FUTURE WORK 

In this paper, our proposed TalkingGaussian outperforms in rendering high-quality lip-synchronized talking head videos, with better facial fidelity and higher efficiency than previous methods. Despite that, our method still has some limitations. 

In the first aspect, some noisy primitives may randomly occur due to the densification operation of 3DGS. Although this can be relieved by a smoother optimization process provided by Incremental Sampling, it would still sometimes influence the quality. In future works, we will consider adding more constraints to better control the primitive's growth. 

On the other hand, the face and inside mouth branches are aligned via the audio feature in our method, enabling free and individual learning for their own motions. Nevertheless, this connection is not tight enough. Although it is sufficient for most in-domain audio inputs like speeches from the same person as that in the training data, the face and inside mouth area may be misaligned in some cross-domain situations. To solve this problem, we may build a better awareness of these two parts to enhance robustness in the future.











splncs04

