FacePoseNet: Making a Case for Landmark-Free Face Alignment
Fengju Chang Anh Tuan Tran Tal Hassner Iacopo Masi Ram Nevatia Gerard Medioni







FacePoseNet: Making a Case for Landmark-Free Face Alignment
    Feng-Ju Chang1, Anh Tuan Tran1, Tal Hassner2,3, Iacopo Masi1, Ram Nevatia1, Gerard Medioni1

1 Institute for Robotics and Intelligent Systems, USC, CA, USA

2 Information Sciences Institute, USC, CA, USA

3 The Open University of Israel, Israel

{fengjuch,anhttran,iacopoma,nevatia,medioni}@usc.edu, hassner@isi.edu

    December 30, 2023
================================================================================================================================================================================================================================================================================================================================







We show how a simple convolutional neural network (CNN) can be trained to accurately and robustly regress 6 degrees of freedom (6DoF) 3D head pose, directly from image intensities. We further explain how this FacePoseNet (FPN) can be used to align faces in 2D and 3D as an alternative to explicit facial landmark detection for these tasks. We claim that in many cases the standard means of measuring landmark detector accuracy can be misleading when comparing different face alignments. Instead, we compare our FPN with existing methods by evaluating how they affect face recognition accuracy on the IJB-A and IJB-B benchmarks: using the same recognition pipeline, but varying the face alignment method. Our results show that (a) better landmark detection accuracy measured on the 300W benchmark does not necessarily imply better face recognition accuracy. (b) Our FPN provides superior 2D and 3D face alignment on both benchmarks. Finally, (c), FPN aligns faces at a small fraction of the computational cost of comparably accurate landmark detectors. For many purposes, FPN is thus a far faster and far more accurate face alignment method than using facial landmark detectors.





§ INTRODUCTION

Facial landmark detection is rarely, if ever, an application in its own right. Instead, it is typically a means to an end: It is one component out of many in pipelines designed for other face understanding and processing tasks, often providing effective means for aligning face photos and making them easier to process. Most facial landmark detectors, however, are developed without measuring their impact on these applications but rather using standard facial landmark detection benchmarks such as the popular AFW <cit.>, LFPW <cit.>, HELEN <cit.>, and IBUG <cit.>. These benchmarks contain face images with manually labeled ground truth landmarks. Better detection accuracy on these benchmarks equals better prediction of these manual positions. This raises an important question: Does better approximation of such human labeled landmarks imply better face alignment and consequently better face understanding?



[1]180Images one, three, and five are ground truth.

Why would higher accuracy on landmark detection benchmarks not imply better alignment? The many landmark detection benchmarks used by the community to measure detection accuracy typically offer 5, 49 or 68 landmarks painstakingly labeled on hundreds or thousands of unconstrained face images, reflecting wide viewpoint, resolution and noise variations. On low resolution images, however, even expert human operators can find it hard to accurately pinpoint landmark positions. More importantly, many landmark locations are not well defined even in high resolution (e.g., points along the jawline or behind occlusions). Thus, improved landmark detection accuracy may actually reflect better estimation of uncertain human labels rather than better face alignment (Fig. <ref>).




An additional concern relates to how landmarks are used for face alignment. Face alignment often implies using a global 2D or 3D transformation to warp faces to ideal, reference frames: Detected landmarks are matched with their corresponding landmarks in the reference coordinates and a 2D or 3D transformation is then computed by robust estimation methods. To our knowledge, the effects landmark detection noise, changing expressions or face shapes have on these estimated transformations were never fully explored. 

Responding to these concerns, we offer several contributions. (1) We propose comparing landmark detection methods by evaluating bottom line face recognition accuracy on faces aligned with these methods. (2) As an alternative to existing facial landmark detectors, we further present a robust and accurate, landmark-free method for face alignment: our deep FacePoseNet (FPN). We show it to excel at global, 3D face alignment even under the most challenging viewing conditions. Finally, (3), we test our FPN extensively and report that better landmark detection accuracy on the widely used 300W benchmark <cit.> does not imply better alignment and recognition on the highly challenging IJB-A <cit.> and IJB-B benchmarks <cit.>. In particular, recognition results on images aligned with our FPN surpass those on images aligned with state-of-the-art detectors.  

Some applications require landmark estimation. Our FPN provides a more accurate and far faster face alignment technique in the many cases where global alignment, rather than specific landmark positions, is needed. To support our claims, we make our code publicly available [<https://github.com/fengju514/Face-Pose-Net>].



§ RELATED WORK

Applications of facial landmark detectors.
Facial landmark detection is big business, as reflected by the numerous citation to relevant papers, the many facial landmark detection benchmarks <cit.>, and popular international events dedicated to this problem. With all this effort, a rigorous survey of the many applications of facial landmarks is outside the scope of this paper. In lieu of such a survey, and to get some idea of why this problem attracts so much attention, we offer the following cursory study.


We consider two of the most widely cited face landmark detector papers of the last decade, the tree based approach <cit.> and supervised descent method <cit.>. At the time of writing, based on Google Scholar, the latter accumulated nearly a thousand citations and the former well over a thousand. We found 23 application names appearing frequently (more than ten times) in the titles of the papers that cite these two and counted the number of times these applications were mentioned. The relative frequencies of these applications are reported in Fig. <ref>.

Of course, this simple survey is by no means accurate: the same term is counted twice if the paper using it in its title cites both <cit.> and <cit.> and many paper titles do not clearly state the application they describe (e.g., <cit.> describes a method for face alignment in 3D but does not mention “alignment” in the title). Nevertheless, with around two thousand papers included in this survey, the result is quite clear: Alignment, face recognition and pose estimation – also considered alignment – are overwhelmingly more popular than any other application. This, of course, excluding other landmark detection papers. 

 What does it mean to align a face?
The term alignment almost always appears in the titles of papers which present facial landmark detection methods <cit.> (and most others) implying that the two terms are used interchangeability. This reflects an interpretation of alignment as forming correspondences between particular spatial locations in one face image and another. A different interpretation of alignment, and the one used here, refers not only to establishing these correspondences but also to warping the two face images, thus making them easier to compare and match. Face warping with estimated 2D (in-plane) or 3D transformations is well known to have a profound impact on the performance of face recognition systems <cit.>. 

Although sometimes alignments involve non-parametric or part-based warps <cit.>, often, global 2D or 3D (parametric) transformation are all that is required for this purpose. Such aligned faces are then further processed in systems for face recognition <cit.>, emotion recognition <cit.>, age and gender estimation <cit.>, and more. In fact, it was recently claimed that a global alignment is both more robust and far faster to warp than non-parametric transformations <cit.>. This paper focuses on such global transformations, showing how they can be estimated quickly and accurately using a deep neural network.


Deep pose estimation. This work describes a deep network trained to estimate the 6DoF of 3D faces viewed in single images. Deep learning is increasingly used for similar purposes, though typically focusing on general object classes <cit.>. Some recently addressed faces in particular, though their methods are designed to estimate 2D landmarks along with 3D face shapes <cit.>. Unlike our proposed pose estimation, they regress poses by using iterative methods which involve computationally costly face rendering. We regress 6DoF directly from image intensities without such rendering steps.

In all these cases, absence of training data was cited as a major obstacle for training effective models. In response, some turned to larger 3D object data sets <cit.> or using synthetically generated examples <cit.>. We propose a far simpler alternative and show it to result in robust and accurate face alignment.







§ A CRITIQUE OF FACIAL LANDMARK DETECTION

Before using an existing state-of-the-art facial landmark detector in a face processing system, the following points should be considered. 


Landmark detection accuracy measures. Facial landmark detection accuracy is typically measured by considering the distances between estimated landmarks and ground truth (reference) landmarks, normalized by the reference inter-ocular distance of the face <cit.>:

    e(𝐋,𝐋̂)= 1/m𝐩̂_l-𝐩̂_r_2∑_i=1^m𝐩_𝐢 - 𝐩̂_̂𝐢̂_2,

Here, 𝐋 = {𝐩_𝐢} is the set of m 2D facial landmark coordinates, 𝐋̂ = {𝐩̂_̂𝐢̂} their ground truth locations, and 𝐩̂_l, 𝐩̂_r the reference left and right eye outer corner positions. These errors are then translated to a number of standard quantities, including the mean error rate (MER), the percentage of landmarks detected under certain error thresholds (e.g., below 5% or 10% error rates) or the area under the accumulative error curve (AUC). 

There are two key problems with this method of evaluating landmark errors. First, the ground truth compared against is manually specified, often by mechanical turk workers. These manual annotations can be noisy, they are ill-defined when images are low resolution, the landmarks are occluded (in case of large out-of-plane head rotations, facial hair and other obstructions), or located in featureless facial regions (e.g., along the jawline). Accurate facial landmark detection, as measured on these benchmarks, thus implies better matching human labels but not necessarily better detection. These problems are demonstrated in Fig. <ref>.



A second potential problem lies in the error measure itself: Normalizing detection errors by inter-ocular distances biases against images of faces appearing at non-frontal views. When faces are near profile, perspective projection of the 3D face onto the image plane shrinks the distances between the eyes thereby naturally inflating the errors computed for such images. 


Landmark detection speed. Some facial landmark detection methods emphasize impressive speeds <cit.>. Measured on standard landmark detection benchmarks, however, these methods do not necessarily claim state-of-the-art accuracy, falling behind more sophisticated, yet far slower detectors <cit.>. Moreover, aside from <cit.>, no existing landmark detector is designed to take advantage of GPU hardware, a standard feature in commodity computer systems and most, including <cit.>, apply iterative optimizations which may be hard to convert to parallel processing.



Effects of facial expression and shape on alignment. It was recently shown that 3D alignment and warping of faces to frontal viewpoints (i.e. frontalization) is effective regardless of the precise 3D face shape used for this purpose <cit.>. Facial expressions and 3D shapes in particular, appear to have little impact on the warped result as evident by the improved face recognition accuracy reported by that method. Moreover, it was recently demonstrated that by using such a generic 3D face shape, rendering faces from new viewpoints can be accelerated to the same speed as simple 2D image warping <cit.>.

Interestingly, they and many others used facial landmark detectors to compute parametric transformations – projection matrix <cit.> or 2D affine or similarity transforms <cit.> – by applying robust estimators to corresponding detected facial landmarks <cit.>. Variations in landmark locations due to expressions and face shapes essentially contribute noise to this estimation process. The effects these variations have on the quality of the alignment were, as far as we know, never truly studied.




§ DEEP, DIRECT HEAD POSE REGRESSION

Rather than align faces using landmark detection, we refer to alignment as a global, 6DoF 3D face pose, and propose to infer it directly from image intensities, using a simple deep network architecture. We next describe the network and the novel method used to train it. 




 §.§ Head pose representation

We define face alignment as the 3D head pose 𝐡, expressed using 6DoF: three for rotations, 𝐫=(r_x,r_y,r_z)^T, and three for translations, 𝐭=(t_x,t_y,t_z)^T:

    𝐡 = (r_x, r_y, r_z, t_x, t_y, t_z)^T

where (r_x, r_y, r_z) are represented as Euler angles (pitch, yaw, and roll). Given m 2D facial landmark coordinates on an input image, 𝐩_2× m, and their corresponding, reference 3D coordinates, 𝐏_3× m – selected on a fixed, generic 3D face model – we can obtain a 3D to 2D projection of the 3D landmarks onto the 2D image by solving the following equation for the standard pinhole model:

    [𝐩,1]^T = 𝐀[𝐑,𝐭][𝐏,1]^T,

where 𝐀 and 𝐑 are the camera matrix and rotation matrix respectively and 1 is a constant vector of 1. We then extract a rotation vector 𝐫=(r_x,r_y,r_z)^T from 𝐑 using the Rodrigues rotation formula:

    𝐑 = cosθ𝐈 + (1-cosθ)𝐫𝐫^T + sinθ[    0 -r_z  r_y;  r_z    0 -r_x; -r_y  r_x    0 ],

where we define θ = ||𝐫||_2.

Obtaining enough training examples.
Although our network architecture is not very deep compared to deep networks used today  for other tasks, training it still requires large quantities of labeled training data. We found the numbers of facial landmark annotated faces in standard data sets to be too small for this purpose. A key problem is therefore obtaining a large enough training set.


We produce our training set by synthesizing 6D, ground truth pose labels by running an existing facial landmark detector <cit.> on a large image set: the 2.6 million images in the VGG face dataset <cit.>. The detected landmarks were then used to compute the 6DoF labels for the images in this set. A potential danger in using an existing method to produce our training labels, is that our CNN will not improve beyond the accuracy of its training labels. As we show in Sec. <ref>, this is not necessarily the case.


To further improve the robustness of our CNN, we apply a number of face augmentation techniques to the images in the VGG face set, substantially enriching the appearance variations it provides. Fig. <ref> illustrates this augmentation process. Specifically, following face detection <cit.> and landmark detection <cit.>, we transform detected bounding boxes and their detected facial landmarks using a number of simple in-plane transformations. The parameters for these transformations are selected randomly from fixed distributions (Table. <ref>). The transformed faces are then used for training, along with their horizontally mirrored versions, to provide yaw rotation invariance. Ground truth labels are, of course, computed using the transformed landmarks. 

Some example augmented faces are provided in Fig. <ref>. Note that augmented images would often be too challenging for existing landmark detectors, due to extreme rotations or scaling. This, of course, does not affect the accuracy of the ground truth labels which were obtained from the original images. It does, however, force our CNN to learn to estimate poses even on such challenging images. 










FPN training.
For our FPN we use an AlexNet architecture <cit.> with its initialized weights provided by <cit.>. The only difference is that here the output regresses 6D floating point values rather than predicts one-hot encoded, multi class labels. Note that during training each dimension of the head pose labels is normalized by the corresponding mean and standard deviation of the training set, compensating for the large value differences among dimensions. The same normalization parameters are used at test time. 

2D and 3D face alignment with FPN.
Given a test image, it is processed by applying the same face detector <cit.>, cropping the face and scaling it to the dimension of the network's input layer. The 6D network output is then converted to a projection matrix. Specifically, the projection matrix is produced by the camera matrix 𝐀, rotation matrix 𝐑, and the translation vector 𝐭 in Eq. (<ref>). With this projection matrix we can render new views of the face, aligning it across 3D views as was recently proposed by others <cit.>.

For 2D alignment, we compute the 2D similarity transform to warp the 2D projected landmarks to pre-defined landmark locations. With frontal images (absolute yaw angle ≤ 30^∘), we use the eye centers, the nose tip, and the mouth corners for alignment. With profile images (absolute yaw angle > 30^∘), however, only the visible eye center and the nose tip are used.



§ RESULTS

We provide comparisons of our FPN with the following widely used, state-of-the-art, facial landmark detection methods: Dlib <cit.>, CLNF <cit.>, OpenFace <cit.>, DCLM <cit.>, RCPR <cit.>, and 3DDFA <cit.> evaluating them for their effects on face recognition vs. their landmark detection accuracy. 



 §.§ Effect of alignment on recognition

Sec. <ref> discusses the various potential problems of comparing face alignment methods by measuring their landmark detection accuracy. As an alternative, we propose comparing methods for face alignment and landmark detection by evaluating their effect on the bottom line accuracy of a face processing pipeline. Since face recognition is arguably one of the most popular applications for face alignment, we use recognition accuracy as a performance measure. To our knowledge, this is the first time alignment methods are compared based on their effect on recognition accuracy.

