{'fraction_non_alphanumeric': 0.0229051998222283, 'fraction_numerical': 0.015144781374995727, 'mean_word_length': 4.8005155661312715, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction

In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and propose a novel algorithm that is able to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniformed. These fine 3D facial models can be represented as a 3D morphable model for rough shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different than the previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. The unprecedented dataset and code will be released to public for research purpose.


Introduction
Parsing and recovering 3D face models from images have been a hot research topic in both computer vision and computer graphics due to its many applications. As learning based methods have become the mainstream in face tracking, recognition, reconstruction and synthesis, 3D face datasets becomes increasingly important. While there are numerous 2D face datasets, the few 3D datasets lack in 3D details and scale. As such, learning-based methods that rely on the 3D information suffer.
Existing 3D face datasets capture the face geometry using sparse camera array [8,19,35] or active depth sensor such as Kinect [12] and coded light [34]. These setups limit the quality of the recovered faces. We captured the 3D face model using a dense 68-camera array under controlled illumination, which recovers the 3D face model with wrinkle and pore level detailed shapes, as shown in Figure 1. In addition to shape quality, our dataset provides considerable amount of scans for study. We invited 938 people between the ages of 16 and 70 as subjects, and each subject is guided to perform 20 specified expressions, generating 18,760 high quality 3D face models. The corresponding color images and subjects' basic information (such as age and gender) are also recorded.
Based on the high fidelity raw data, we build a powerful parametric model to represent the detailed face shape. All the raw scans are firstly transformed to a topologically uniformed base model representing the rough shape and a displacement map representing detailed shape. The transformed models are further used to build bilinear models in identity and expression dimension. Experiments show that our generated bilinear model exceeds previous methods in representative ability.
Using FaceScape dataset, we study how to predict a detailed riggable face model from a single image. Prior methods are able to estimate rough blendshapes where no wrinkle and subtle features are recovered. The main problem is how to predict the variation of small-scale geometry caused by expression changing, such as wrinkles. We propose the dynamic details which can be predicted from a single image by training a deep neural network on FaceScape dataset. Cooperated with bilinear model fitting method, a full system to predict detailed riggable model is presented. Our system consists of three stages: base model fitting, displacement map prediction and dynamic details synthesis. As shown in Figure 1, our method predicts detailed 3D face model which contains subtle geometry, and achieves high accuracy due to the powerful bilinear model generated from FaceScape dataset. The predicted model can be rigged to various expressions with plausible detailed geometry.
Our contributions are summarized as following: • We present a large-scale 3D face dataset, FaceScape, consisting of 18,760 extremely detailed 3D face models. All the models are processed to topologically uniformed base models for rough shape and displacement maps for detailed shape. The data are released free for non-commercial research.
• We model the variation of detailed geometry acrossing expressions as dynamic details, and propose to learn the dynamic detail from FaceScape using a deep neural network.
• A full pipeline is presented to predict detailed riggable face model from a single image. Our result model can be rigged to various expressions with plausible geometric details.

Related Work
3D Face Dataset. 3D face datasets are of great value in face-related research areas. Existing 3D face datasets could be categorized according to the acquisition of 3D face model. Model fitting datasets [34,63,24,5,7] fit the 3D morphable model to the collected images, which makes it convenient to build a large-scale dataset on the base of wild faces. The major problem of the fitted 3D model is the uncertainty of accuracy and the lack of detailed shape. To obtain the accurate 3D face shape, a number of works reconstructed the 3D face using active method including depth sensor or scanner [56,55,3,40,39,12,17], while the other works built sparse multi-view camera system [57,18]. Traditional depth sensors and 3D scanners suffer from the limited spatial resolution, so they can't recover detailed facial geometry. The sparse multi-view camera system suffers from the unstable and inaccurate reconstruction [41,59,58]. The drawbacks of these methods limit the quality of 3D face model in previous datasets. Different from the datasets above, FaceScape obtained the 3D face model from a dense multi-view system with 68 DSLR cameras, which provides extremely high quality face models. The parameters measuring 3D model quality are listed in Table 1. Our dataset outperforms previous works on both model quality and data amount. Note that Table 1 doesn't list the datasets which provide only parametric model but no source 3D models [8,6,34,31].
3D Morphable Model. 3DMM is a statistical model which transforms the shape and texture of the faces into a vector space representation [4]. As 3DMM inherently contains the explicit correspondences from model to model, it is widely used in model fitting, face synthesis, image manipulations, etc. The recent research on 3DMM can be generally divided into two directions. The first direction is to separate the parametric space to multiple dimensions like identity, expression and visemes, so that the model could be controlled by these attributes separately [52,12,30,27]. The models in expression dimension could be further transformed to a set of blendshapes [28], which can be rigged to generate individual-specific animation. Another direction is to enhance the representation power of 3DMM by using deep neural network to present 3DMM bases [2,45,48,50,49,16].
Single-view shape Prediction. Predicting 3D shape from a single image is a key problem to many applications like view synthesis [22,60,61] and stereoscopic video generation [13,25]. The emergence of 3DMM has simplified the single-view reconstruction of face to a model fitting problem, which could be well solved by fitting facial landmarks and other features [38,46] or regressing the parameter of 3DMM with a deep neural network [20,63]. However, fitting 3DMM is difficult in recovering small details from the input image due to the limited representation power. To solve this problem, several recent works adopt the multi-layer refinement structures. Richardson et al. [37] and Sela et al. [42] both proposed to firstly predict a rough facial shape and render it to the depth map, then refine the depth map to enhance the details from the registered source image. Sengupta et al. [43] proposed to train the SfSNet on combination of labeled synthetic data and unlabeled inthe-wild data to estimate plausible detailed shape in un-  [47] proposed to predict a bump map to represent the wrinkle-level geometry base on a rough base model. Huynh et al. [26] utilized imageto-image network and super-resolution network to recover the mesoscopic facial geometry in the form of displacement map. Chen et al. [15] also tried to predict the displacement map with a conditional GAN based on the 3DMM model, which enables to recover detailed shape from an in-the-wild image.
Our work advances the state of the art in multiple aspects. In dataset, our FaceScape is by far the largest with the highest quality. A detailed quantitative comparison with previous datasets are made in Table 1. In 3D face prediction, previous works focus on enhancing the static detailed facial shape, while we study the problem of recovering an animable model from a single image. We demonstrate for the first time that a detailed and rigged 3D face model can be recovered from a single image. The rigged model exhibits expression-depended geometric details such as wrinkles.

3D Face Capture
We use a multi-view 3D reconstruction system to capture the raw mesh model for the datasets. The multi-view system consists of 68 DSLR cameras, 30 of which capture 8K images focusing on front side, and the other cameras capture 4K level images for the side part. The camera shutters are synced to be triggered within 5ms. We spend six months to invite 938 people to be our capturing subjects. The subjects are between 16 and 70 years old, and are mostly from Asia. We follow FaceWarehouse [12] which asks each subject to perform 20 specific expressions including neutral expression for capturing. The total reconstructed number reach to roughly 18,760, which is the largest amount comparing to previous expression controlled 3D face datasets. The reconstructed model is triangle mesh with roughly 2 million vertices and 4 million triangle faces. The meta information for each subject is recorded, including age, gender, and job  In the lower side we show the pipeline from the captured multi-view images to topologically uniformed models (T.U. models).
(by voluntary). We show the statistical information about the subjects in our dataset in Figure 2, and a comparison with prior 3D face datasets in Table 1.

Topologically Uniformed Model
We down-sample the raw recovered mesh into rough mesh with less triangle faces, namely base shape, and then build 3DMM for these simplified meshes. Firstly, we roughly register all the meshes to the template face model by aligning 3D facial landmarks, then the NICP [1] is used to deform the templates to fit the scanned meshes. The deformed meshes can be used to represent the original scanned face with minor accuracy loss, and more importantly, all of the deformed models share the uniform topology. The detailed steps to register all the raw meshes are described in the supplementary material.
After obtaining the topology-uniformed base shape, we use displacement maps in UV space to represent middle and fine scale details that are not captured by the base model due to the small number of vertices and faces. We find the surface points of base mesh corresponding to the pixels in the displacement map, then inverse-project the points to the raw mesh along normal direction to find its corresponding points. The pixel values of the displacement map is set to the signed distance from the point on base mesh to its corresponding point.
We use base shapes to represent rough geometry and displacement maps to represent detailed geometry, which is a two-layer representation for our extremely detailed face shape. The new representation takes roughly 2% of the original mesh data size, while maintaining the mean absolute error to be less than 0.3mm.

Bilinear Model
Bilinear model is firstly proposed by Vlasic et al. [52], which is a special form of 3D morphable model to parameterize face models in both identity and expression dimensions. The bilinear model can be linked to a facefitting algorithm to extract identity, and the fitted individualspecific model can be further transformed to riggable blendshapes. Here we describe how to generate bilinear model from our topologically uniformed models. Given 20 registered meshes in different expressions, we use the example based facial rigging algorithm [28] to generate 52 blendshapes based on FACS [21] for each person. Then we follow the previous methods [52,12] to build the bilinear model from generated blendshapes in the space of 26317 vertices × 52 expressions × 938 identities. Specifically, we use Tucker decomposition to decompose the large rank-3 tensor to a small core tensor C r and two low dimensional components for identity and expression. New face shape can be generated given the the identity parameter w id and expression parameter w exp as: where V is the vertex position of the generated mesh.
The superiority in quality and quantity of FaceScape makes the generated bilinear model own higher representation power. We evaluate the representation power of our model by fitting it to scanned 3D meshes not part of the training data. We compare our model to FaceWarehouse(FW) [12] and FLAME [30]   to our self-captured test set, which consists of 1000 high quality meshes from 50 subjects performing 20 different expressions each. FW has 50 identity parameters and 47 expression parameters, so we use the same number of parameters for fair comparison. To compare with FLAME which has 300 identity parameters and 100 expression parameters, we use 300 identity parameters and all 52 expression paremeters. Figure 3 shows the cumulative reconstruction error. Our bilinear face model achieves much lower fitting error than FW using the same number of parameters and also outperform FLAME using even less expression parameters. The visually comparison in Figure 3 shows ours model could produce more mid-scale details than FW and FLAME, leading to more realistic fitting results.

Detailed Riggable Model Prediction
As reviewed in the related works in Section 2, existing methods have succeed in recovering extremely detailed 3D facial model from a single image. However, these recovered models are not riggable in expression space, since the recovered detail is static to the specific expression. Another group of works try to fit a parametric model to the source image, which will obtain an expression-riggable model, but the recovered geometry stays in the rough stage.
The emerge of FaceScape dataset makes it possible to estimate detailed and riggable 3D face model from a single image, as we can learn the dynamic details from the large amount of detailed facial models. We show our pipeline in Figure 5 to predict a detailed and riggable 3D face model from a single image. The pipeline consists of three stages: base model fitting, displacement map prediction and dynamic details synthesis. We will explain each stage in detail in the following sections.

Base Model Fitting
The bilinear model for base shape is inherently riggable as the parametric space is separated into identity dimension and expression dimension, so the rough riggable model can be generated by regressing the parameters of identity for the bilinear model. Following [46], we estimate parameters corresponding to a given image by optimizing an objective function consisting of three parts. The first part is landmark alignment term. Assuming the camera is weak perspective, the landmark alignment term E lan is defined as the distance between the detected 2D landmark and its corresponding vertex projected on the image space. The second part is pixel-level consistency term E pixel measur-ing how well the input image is explained by a synthesized image. The last part is regularization term which formulates identity, expression, and albedo parameters as multivariate Gaussians. The final objective function is given by: where E id , E exp and E alb are the regularization terms of expression, identity and albedo, respectively. λ 1 , λ 2 , λ 3 and λ 4 are the weights of different terms.
After obtaining the identity parameter w id , individualspecific blendshapes B i can be generated as: whereŵ exp (i) is the expression parameter corresponding to blendshape B i from Tucker decomposition.

Displacement Map Prediction
Detailed geometry is expressed by displacement maps for our predicted model. In contrast to the static detail which is only related to the specific expression in a certain moment, dynamic detail expresses the geometry details in varying expressions. Since the single displacement map cannot represent the dynamic details, we try to predict multiple displacement maps for 20 basic expressions in FaceScape using a deep neural network.
We observed that the displacement map in a certain expression could be decoupled into static part and dynamic part. The static part tends to keep static in different expressions, and is mostly related to the intrinsic feature like pores, nevus, and organs. The dynamic part varies in different expressions, and is related to the surface shrinking and stretching. We use a deforming map to model the surface motion, which is defined as the difference of vertices' 3D position from source expression to target expression in the UV space. As shown in Figure 4, we can see the variance between displacement maps is strongly related to the deforming map, and the static features in displacement maps are related to the texture. So we feed motion maps and textures to a CNN to predict the displacement map for multiple expressions.
We use pix2pixHD [53] as the backbone of our neural network to synthesize high resolution displacement maps. The input of the network is the stack of deforming map and texture in UV space, which can be computed from the recovered base model. Similar to [53], the combination of adversarial loss L adv and feature matching loss L F M is used to train our net with the loss function formulated as:  where G is the generator, D 1 , D 2 and D 3 are discriminators that have the same LSGAN [32] architecture but operate at different scales, λ is the weight of feature matching loss.

Dynamic Detail Synthesis
Inspired by [33], we synthesize displacement map F for an arbitrary expression corresponding to specific blendshape weight α, using a weighted linear combination of generated displacement mapsF 0 in neutral expression and F i in other 19 key expressions: where M is the weight mask with the pixel value between 0 and 1 , is element-wise multiplication operation. To calculate the weight mask, considering the blendshape expressions change locally, we first compute an activation mask A j in UV space for each blendshape mesh e j as: where A j (p) is the pixel value at position p of the jth activation mask, e j (p) and e 0 (p) is the corresponding vertices position on blendshape mesh e j and neutral blendshape mesh e 0 , respectively. The activation masks are further normalized between 0 and 1. Given the activation mask A j for each of the 51 blendshape meshes, the ith weight mask M i is formulated as a linear combination of the activation masks weighted by the current blendshape weight α and fixed blendshape weightα i corresponding to the ith key expression: where α j is the jth element of α. M 0 is given by There are many existing performance driven facial animation methods generating blendshape weights with depth camera [54,29,9] or single RGB camera [11,10,14]. As blendshape weights have semantic meaning, it's easy for artists to manually adjust the rigging parameters.

Implement Detail
We use 888 people in our dataset as training data with a total of 17760 displacement maps, leaving 50 people for testing. We use the Adam optimizer to train the network with learning rate as 2e −4 . The input textures and output displacement maps' resolution of our network is both 1024 × 1024. We use 50 identity parameters, 52 expression parameters and 100 albedo parameters for our parametric model in all experiments.

Evaluation of 3D Model Prediction
The predicted riggable 3D faces are shown in Figure 6. To show riggable feature of the recovered facial model, we rig the model to 5 specific expressions. We can see the results of rigged models contain the photo-realistic detailed wrinkles, which cannot be recovered by previous methods. The point-to-plane reconstruction error is computed between our model and the ground-truth shape. The mean error is reported in Table 2. More results and the generated animations are shown in the supplementary material.

Ablation Study
W/O dynamic detail. We try to use only one displacement map from source image for rigged expressions, and the other parts remain the same. As shown in Figure 9, we find that the rigged model with dynamic detail shows the wrinkles caused by various expressions, which are not found in W/O dynamic method.   We change the input of our displacement map prediction network by replacing the deforming map with one-hot encoding for each of 20 target expressions. As shown in Figure 9, we find the results without deforming map (W/O Def. Map) contain few details caused by expressions.

Comparisons to Prior Works
We show the predicted results of our result and other works in Figure 7. The comparison of detail prediction is  Figure 7: Comparison of static 3D face prediction with previous methods. The images in top two rows are from FaceScape, the image in third row is from volker sequence [51], and the image in bottom row is from Internet. The top three images are with ground truth shapes, so we evaluate the reconstruction error and show the heat map below each row. Our method predicts the lowest error comparing to three previous methods.
shown in Figure 8. As most of the detailed face predicted by other works cannot be directly rigged to other expressions, we only show the face shape in the source expression. Our results are visually better than previous methods, and also quantitatively better in the heat map of error. We consider the major reason for our method to perform the best in accuracy is the strong representation power of our bilinear model, and the predicted details contribute to the visually plausible detailed geometry.

Conclusion
We present a large-scale detailed 3D facial dataset, FaceScape. Comparing to previous public large-scale 3D face datasets, FaceScape provides the highest geometry quality and the largest model amount. We explore to pre-Source Image Ours DFDN Figure 8: Comparison of detail prediction. We adopt NICP [1] to register the base meshes of different methods to ground truth scans, and visualize the predicted details on common base meshes. dict a detailed riggable 3D face model from a single image, and achieve high fidelity in dynamic detail synthesis. We believe the release of FaceScape will spur the future researches including 3D facial modeling and parsing.
The generation of topologically uniformed model has been briefly introduced in Section 3.2 of the main paper.
Here we supplement a detailed description of model registration and displacement map generation.
Registration of base shape. We down-sample the raw recovered mesh into rough mesh with fewer triangle faces, namely base shape, and then build 3DMM for these simplified meshes. Firstly, the 2D landmarks are extracted from the frontal image, then the corresponding 3D landmarks are obtained by inverse-projection 2D landmarks. The Procrustes transformation [23] is used to register all landmarks to a standard 3D facial template with landmark annotations. In this way, the pose and scale for all the scanned meshes are roughly aligned to the standard facial template. Then we use Non-rigid ICP [1] to register the standard template mesh to scanned mesh in neutral expression. For scanned meshes in other 19 expressions, similar to [12], the deformation transfer algorithm [44] is firstly used to deform the registered mesh in neutral expression to other expressions mimicking the deformation of a set of template meshes in corresponding expressions. Then the Non-rigid ICP [1] is used to register these deformed individual-specific templates to scanned meshes to fit the scans in non-neutral expressions more accurately.
Displacement map generation. After obtaining the topology-uniformed base shape, we use displacement maps in UV space to represent middle and fine scale details that are not captured by the base model due to the small number of vertices and faces. The most straightforward way to compute the displacement map is to calculate the distance from the surface of the registered model to the raw mesh. However, we find that there will be artifacts in the displacement map caused by the defects in the registration procedure. Thus the raw scan is firstly smoothed with Laplacian mesh smoothing. Then we trace the surface points of base mesh corresponding to pixels in the displacement map, and inverse-project the points to the raw mesh along normal direction to find its corresponding points. The pixel value of the displacement map is set to the signed distance from the point on raw mesh to its corresponding point on the smoothed mesh.

C. Base Model Fitting
The base model fitting method has been briefly introduced in Section 4.1 of the main paper. Here we provide a detailed description of three parts in the objective function.
Landmark Alignment. Firstly the 2D landmarks L are extracted from the image using an off-the-shelf facial landmark detector. Assuming the camera is weak perspective, the landmark alignment term is defined as the distance between the detected 2D landmark L (k) and its corresponding vertex projected on the image space: where s is the scale factor of the weak perspective function, R is the rotation matrix and t is the translation. Pixel Level Consistency. The pixel-level reconstruction term is used to match the geometry more accurately in the regions where no feature points such as cheeks exists. Under the assumption of Lambertian surfaces, we use the first three bands of Spherical Harmonics(SH) [36] for illumination representation. The per-vertex albedo is represented as a PCA model based on our dataset with albedo parameter w alb . The objective function is formulated as: where V is the set of pixels corresponding to frontal vertices of the fitted mesh,Î is the synthetic face, I is the input image.
Regularization. We formulate the prior of identity, expression and albedo parameters as multivariate Gaussians around the average of our dataset for regularization. The final objective function is given by: where E id , E exp and E alb are the regularization terms of expression, identity and albedo, respectively. λ 1 , λ 2 , λ 3 and λ 4 are the weights of different terms. We optimize the parameters alternatively. Following [64], the vertex indices corresponding to contour landmarks of the face are updated after each iteration.

D. Facial Capture System
The capturing system has been briefly introduced in Section 3.1 of the main paper. Here we supplement the pictures System A System B Figure 10: Our multi-view system to reconstruct the high quality detailed 3D face. We captured the data in two different places, so there are two frameworks shown as system A and system B.

Source Image
Synthesized Exp. Zoom In Figure 11: We use our recovered model for synthesizing images in another expression with detailed shading. Figure 12: Failure cases. In the left, our prediction cannot recover the aquiline nose well, as this feature is not common in our dataset. In the right, the wrong displacement map is predicted due to occlusion.

Source Image Result Source Image Result
of our system in Figure 10. The system consists of the 68 DSLR camera array, controlled lighting and a centralized control sever.
Source Image Texture Prediction Ground Truth Figure 13: Predicted displacement maps using our method and ground truth.

E. More Results
More results are supplemented in Figure 14 as the extension of Figure 6 in our main paper. It shows that our results recover 3D faces with photo-realistic details. The faces can be further rigged to other expressions, and the details in the new expressions are synthesized to make the rigged model plausible.
We supplement the comparison of the predicted and ground-truth displacement maps in Figure 13 as the extension of Figure 7 in our main paper.

F. More Models
We show the 20 captured expressions for each subject in Figure 15, and show more subjects in neutral expression in Figure 16. The diversity of models in expression and identity dimensions ensures the quality of bilinear face model generated on FaceScape dataset.

G. Photo-realistic Image Synthesis
Similar to [52,12], given a facial image, our bilinear model can be used to synthesize images in other expressions. Specifically, we use the base model fitting method to estimate the face model. Then we change the expression parameter to generate the face model in the target expression and warp the image pixels guided by translations of vertices on the 3D face model. The details caused by the expression changing are further synthesized by adjusting the pixel shading. New pixel value is calculated based on the new normal from the predicted displacement map and estimated illumination in model fitting procedure. The synthesized images are shown in Figure 11.

H. Failure Case
We show some failure cases of our method in Figure 12.