Joint Voxel and Coordinate Regression for Accurate
3D Facial Landmark Localization
Hongwen Zhang†‡ , Qi Li† , Zhenan Sun†‡§
† Center

arXiv:1801.09242v1 [cs.CV] 28 Jan 2018

for Research on Intelligent Perception and Computing (CRIPAC)
† National Laboratory of Pattern Recognition (NLPR)
† Institute of Automation, Chinese Academy of Sciences (CASIA)
‡ University of Chinese Academy of Sciences (UCAS)
§ Center for Excellence in Brain Science and Intelligence Technology (CEBSIT), CAS
Email: hongwen.zhang@cripac.ia.ac.cn, {qli, znsun}@nlpr.ia.ac.cn

Abstract—3D face shape is more expressive and viewpointconsistent than its 2D counterpart. However, 3D facial landmark
localization in a single image is challenging due to the ambiguous
nature of landmarks under 3D perspective. Existing approaches
typically adopt a suboptimal two-step strategy, performing 2D
landmark localization followed by depth estimation. In this paper,
we propose the Joint Voxel and Coordinate Regression (JVCR)
method for 3D facial landmark localization, addressing it more
effectively in an end-to-end fashion. First, a compact volumetric
representation is proposed to encode the per-voxel likelihood of
positions being the 3D landmarks. The dimensionality of such
a representation is fixed regardless of the number of target
landmarks, so that the curse of dimensionality could be avoided.
Then, a stacked hourglass network is adopted to estimate the
volumetric representation from coarse to fine, followed by a 3D
convolution network that takes the estimated volume as input
and regresses 3D coordinates of the face shape. In this way, the
3D structural constraints between landmarks could be learned
by the neural network in a more efficient manner. Moreover,
the proposed pipeline enables end-to-end training and improves
the robustness and accuracy of 3D facial landmark localization.
The effectiveness of our approach is validated on the 3DFAW
and AFLW2000-3D datasets. Experimental results show that
the proposed method achieves state-of-the-art performance in
comparison with existing methods.

I. I NTRODUCTION
Facial landmark localization has been extensively studied
in the last decades and significant progress has been made
on solving this problem. Though impressive performance is
achieved in 2D face alignment recently [1], [2], 3D landmark
localization in a single image remains challenging due to the
ambiguous nature of landmarks under 3D perspective.
Currently, state-of-the-art approaches to facial landmark localization are dominated by regression based methods, demonstrating their effectiveness on addressing common issues such
as occlusions, large variations of appearance on face images in
the wild. Among them, cascaded regression methods [3], [4],
[5] attempt to learn the mapping from shape-index features
to the landmark coordinates. Though these methods could
achieve highly accurate results for nearly frontal face images,
their performances are barely satisfactory when it comes to
novel view images. On the other hand, heatmap regression
based methods [1], [2] estimate the heatmap for each individual landmark instead. Such heatmap representation encodes

the likelihood of positions being a specific landmark. The
heatmap regression strategy avoids the inefficient learning
of the non-linear mapping from feature space to landmark
positions, which has greatly facilitated landmark localization
problems including face alignment [1], [2] and human pose
estimation [6], [7]. Though these methods could work well
when the facial parts are visible, they might produce blurred
heatmaps when there are invisible landmarks due to occlusions, making it unstable and error-prone to estimate landmark
positions from those multi-mode heatmaps.
For 3D landmark localization, popular approaches employ
a two-step strategy lifting the 2D estimation to 3D shape.
These methods [8], [9], [10] typically perform 2D landmark
localization at first and then obtain the 3D face shape through
depth estimation or 3D face model fitting. Though such
strategy is effective, it is suboptimal and sensitive to the
result of 2D landmark localization. In [11], Pavlakos et al.
extend the 2D heatmap to 3D space and show that predicting
the body joints in a discretized 3D space could be more
effective for 3D human pose estimation. However, directly
extending the 2D heatmap to its 3D version for each landmark
is cumbersome and memory-demanding especially when the
number of landmarks increases.
To cope with these limitations, we propose the Joint Voxel
and Coordinate Regression (JVCR) method in this paper and
make the following contributions towords accurate 3D facial
landmark localization. Firstly, we propose the compact volumetric representation which encodes the voxel-wise likelihood
of positions being the target landmarks in 3D space. The
dimensionality of such a representation is fixed regardless
of the number of landmarks, so that the required memory
and computation could be reduced significantly. Secondly,
we adopt a coarse-to-fine strategy to regress the volumetric
representation so that the 3D structural constraints between
landmarks could be learned by the neural network more
efficiently. In addition, we employ 3D convolutions to regress
the 3D coordinates of landmarks from the volumetric representation. In this way, the 3D convolution network takes as
input the entire volumetric representation of all landmarks,
so that the prediction for those invisible landmarks could
be more robust. Finally, the proposed joint voxel and coor-

Image I

G1(I)

G2(I)

...

GM(I)

P(GM(I))

y

x
z

Coarse-to-fine Voxel Regression

Coordinate Regression

Figure 1: Pipeline of the Joint Voxel and Coordinate Regression (JVCR) method. The voxel regression subnetwork G consists of
M Hourglass modules, which estimate the the compact volumetric representation from coarse to fine. The coordinate regression
subnetwork P takes as input the estimated volume and regresses 3D coordinates of the face shape.

dinate regression framework enables end-to-end training and
shows promising results on 3D facial landmark localization.
Experimental results on 3DFAW [12] and AFLW2000-3D [13]
datasets demonstrate the superiority of our approach.
The remainder of this paper is organized as follows. Section II briefly reviews previous works related to ours. The
technical details of the proposed method are presented in
Section III. Experimental results are reported in Section IV.
Finally, we conclude the paper in Section V.
II. R ELATED WORK
A significant amount of work has been introduced for
landmark localization in the last decades. In this section,
we briefly review previous works related to ours, including
methods for 2D and 3D landmark localization.
2D landmark localization. Typical methods for 2D face
alignment include Constrained Local Models (CLMs) [14],
[15], [16], Active Appearance Models (AAMs) [17], [18]
and Cascaded Regression Methods (CRMs) [4], [3], [5]. For
CLMs and AAMs, they typically optimize the parametric
representation of face shape iteratively according to the pretrained appearance and shape models. For CRMs, however,
instead of representing the face shape parametricaly, they
regress the facial landmark coordinates directly from shapeindex features. To avoid the inefficient learning of the pixelto-coordinate mapping in CRMs, a majority of recent approaches [7], [1], [2] cast landmark localization as regressing
the heatmaps of landmarks instead of the coordinate vector.
Methods belonging to this type pursue regressing clear and
accurate 2D heatmaps for the target landmarks. For example,
Stacked Hourglass Networks [7] uses the symmetric topology
and intermediate supervision, which has been demonstrated to
be effective in both applications of human pose estimation [7]
and face alignment [2]. Several state-of-the-art works [19],
[2], [20] built upon this architecture achieve nearly saturate
performances on 2D landmark localization. Despite their effectiveness, it is usually unstable to estimate the positions
from multi-mode heatmaps for those invisible landmarks. Very
recently, instead of adopting the maximum operation, Sun et
al. [21] propose to infer the landmark coordinate from its
heatmap through the integral operation, which allows end-to-

end training and shows its effectiveness on 2D human pose
estimation.
3D landmark localization. One of the popular pipelines
for 3D landmark localization adopts a two-stage strategy which
performs the 2D landmark estimation at first and then predicts
the depth information for these 2D landmarks. In [8], Zhao et
al. propose a neural network to regress the 2D face shape
firstly and then estimate the depth of the landmarks. In [9],
Bulat et al. introduce a two-stage method which performs the
2D heatmap regression followed by depth prediction. Instead
of estimating the depth information directly, Gou et al. [10]
propose to recover the 3D face shape by fitting the 3D
morphable model to the 2D landmarks. Moreover, cascaded
regression methods are also extended to 3D landmark localization. In [22], Tulyakov et al. propose the 3D shape invariant
feature and estimate 3D face landmarks in a single step manner
using the cascaded regressors. On the other hand, Pavlakos et
al. [11] introduce the volumetric representation for 3D body
joints and show that predicting the joints in a discretized 3D
space could be more effective for 3D pose estimation. The
volumetric representation proposed in [11] could be viewed
as a natural extension of the 2D heatmap, which is highly
demanding for memory and computation. Though regressing
such a representation in a coarse-to-fine manner could alleviate
this problem [11], it still cannot avoid the curse of dimensionality when the number of target landmarks increases. Hence
it can not be easily generalized to other 3D object alignment
problems. Alternatively, we propose to encode the positions
of all landmarks in a single volume with the dimensionality
fixed regardless of the number of landmarks, providing a much
more efficient solution for general 3D landmark localization.
III. M ETHOD
In this section, we present the Joint Voxel and Coordinate
Regression (JVCR) method for 3D facial landmark localization
in detail. The pipeline of the proposed method is shown in
Fig. 1. Our network architecture consists of two main blocks,
a voxel regression subnetwork that estimates the compact
volumetric representation from coarse to fine, followed by a
3D convolution network that takes the estimated volume as
input and regresses the coordinate vector. In the following

refer to the compact volumetric representation as volumetric
representation or volume for simplicity.
B. Joint Voxel and Coordinate Regression
h

y

z

d

x
(a)

w

(b)

Figure 2: Visualization of the 3D face shape (a) and the
compact volumetric representation (b). The voxel values are
indicated by the density of the point cloud. Best viewed in
color.

subsection, we begin with the description of the compact volumetric representation. Then we introduce the voxel regression
subnetwork, the coordinate regression subnetwork as well as
the training scheme for the proposed method.
A. Compact Volumetric Representation for 3D Face Shape
Previous works [23], [11] have shown that encoding the
landmark positions into the heatmap-like or volumetric representation could provide much more discriminative information
than naively concatenating the coordinate vectors of 2D or 3D
landmarks. Such forms of supervision make it easier for fully
convolutional networks to learn the pixel to pixel mapping,
and have been used in the context of both facial landmark
localization [1] and human pose estimation [23].
For 3D landmark localization, the volumetric representation
proposed in [11] encodes the position of a specific landmark in
a volume with a 3D Gaussian centered around the groundtruth
position. Though this idea extends the typically used 2D
heatmap in a natural manner, it leads to a representation with
large dimensionality. Instead of representing each landmark
individually, we propose a volumetric representation encoding
the positions of all target landmarks in a more compact
manner. Specifically, coordinates of all the target landmarks
are converted into a discretized 3D volume V with the size of
w × h × d. Let vi,j,k denote the value of voxel (i, j, k). For
the n-th landmark located at xngt = (x, y, z), its contribution
to vi,j,k can be written as:
1 − (x−i)2 +(y−j)2 2 +(z−k)2
2σ
e
(1)
2πσ 2
where the kernel size σ could be set empirically. For 3D face
shape with N target landmarks, the overall contribution to
n
vi,j,k takes as the maximum value in {vi,j,k
}N
n=1 :
n
vi,j,k
=

n
vi,j,k = max vi,j,k
n

(2)

In this way, the dimensionality of the representation is fixed
regardless of the number of target landmarks. Fig. 2 visualizes
the 3D face shape and the corresponding compact volumetric
representation. It is worth noting that, in this paper, we also

Cascaded regression in a coarse-to-fine manner is wildly
employed in 2D landmark localization. Such a strategy could
make full use of the regressors and progressively refine the
output of the networks. Our method follows this technique and
decouples the 3D facial landmark localization problem into the
following two sub-tasks. The first one aims to regress the ideal
volumetric representation of 3D landmarks in a coarse-to-fine
manner. The second one aims to regress the coordinates of
landmarks from the volumetric representation.
1) Coarse-to-fine Voxel Regression: The voxel regression
subnetwork G learns the mapping from pixels of the face
image I to the volumetric representation V: G(I) → V.
Inspired by previous works [7], [11] on 2D and 3D human pose
estimation, we also adopt the stacked hourglass networks [7]
with intermediate supervision and skip connection. Specifically, the voxel regression subnetwork consists of M stacked
Hourglass modules [7] of which supervisions are volumes
denoted as {Vm }M
m=1 . Then, the voxel regression subnetwork
is trained using the voxel-wise mean squared error loss:
Lvox =

M X
X

m
Gm (I)i,j,k − Vi,j,k

2

(3)

m i,j,k

where Gm (·) denotes the volume outputted by the m-th
Hourglass module. Noted that GM (·) = G(·) is equivalent
to the final output of the voxel regression subnetwork.
As pointed out in [11], the prediction along the z dimension is much more challenging than another two dimensions.
Hence, coarse-to-fine regressing the volumes with the increasing resolution along z dimension could be more effective and
robust. In practice, the resolution d of Vm takes number from
preset values and progressively increases along with m.
2) Coordinate Regression: Typical heatmap regression
based methods [1], [24], [7] retrieve the coordinates of landmarks directly from the peak points of the corresponding
heatmaps. Considering that the positions of all landmarks
are encoded into a single volume, the conventional “tackingmaximum” operation is no longer applicable in our case
since the order of landmarks is not preserved in our compact volumetric representation. Hence, it needs to infer the
coordinates of landmarks from the corresponding volumetric
representation.
To this end, we propose a coordinate regression subnetwork P to learn the mapping from the compact volumetric
representation V to the corresponding coordinate vector x:
P (V) → x. Inspired by the work on 3D object recognition [25] and hand pose estimation [26], we adopt the 3D
convolution kernel instead of 2D convolution in our coordinate
regression subnetwork. The 3D convolution is typically used
to extract features from both spatial and temporal dimensions
for video analysis problems [27]. Hence, the 3D convolution
could be more naturally adopted to extract the 3D information

from the volumetric representation. The proposed coordinate
regression subnetwork consists of five 3D convolution layers,
with batch normalization and Leaky ReLU activation added in
between and a fully connected layer at the end. For training,
we employ the L2 regression loss on the predicted coordinate
vector:
2
Lcoord = kxgt − P (V)k2
(4)
where xgt denotes the concatenated vector of the ground-truth
3D landmark coordinates.
3) Training: Instead of training the whole network from
scratch, we adopt a two-stage training scheme which is more
stable and effective in our experiments. The two subnetworks
mentioned above are pre-trained separately for each subtask beforehand and fine-tuned as an integrated one finally.
Specifically, at the pre-training stage, the voxel regression subnetwork is trained with the face images and the ground-truth
volumes. Meanwhile, the coordinate regression subnetwork is
trained with the ground-truth volumes and the corresponding
coordinate vectors. At the fine-tuning stage, the coordinate
regression subnetwork is attached to the voxel regression
subnetwork, and the whole network is fine-tuned with the joint
supervision of both the ground-truth volumes and coordinate
vectors. Formally, the whole network is trained in an endto-end manner using the following loss function at the final
stage:
L = Lvox + λLcoord
=

M
X

2

kGm (I) − Vm k2 + λ xgt − P (GM (I))

2
2

(5)

m

where λ is used to balance the two terms.
IV. E XPERIMENTS
In this section, the implementation detail of the proposed
method are described firstly. Then, the datasets as well as
evaluation metrics used in our experiments are introduced.
Finally, we present the experimental results of the proposed
method.
A. Implementation Detail
The proposed network takes as input a 256 × 256 face
image and outputs predictions of the volumetric representation
and the coordinate vector. Inspired by the setting of [7], four
Hourglass modules are stacked together as the voxel regression
subnetwork (i.e. M = 4) and output volumes with the size of
64 × 64 × d, where the resolution d of z-dimension is chosen
from the set {1, 2, 4, 64} successively. The Gaussian kernel
size in Eq. 1 is set to σ = 1 in our experiments. During training, data augmentation, such as rotation, scaling and fliping,
was applied randomly to input images. The network was traind
for 25 epochs in total, including 15 epochs for the pre-training
stage and 10 epochs for the fine-tuning stage, respectively.
We adopted the RMSprop [28] optimization algorithm with
an initial learning rate of 2.5 × 10−4 , which was reduced by a
factor of 10 every 10 epochs. Our approach was implemented
using PyTorch. During testing, it takes about 50ms for our

model to process an image on a TITAN Xp GPU. Code is
made publicly available1 .
B. Datasets
3DFAW [12]. The 3DFAW dataset is provided by the
3D Face Alignment in the Wild (3DFAW) Challenge [12]
organizers, containing more than 23000 face images from BU4DFE [29], BP4D-Spontaneous [30] and MultiPIE [31]. 66
3D facial landmarks as well as the face bounding boxes are
annotated for each face image. The 3D points are annotated
consistently using a model-based structure-from-motion technique [32]. The 3DFAW dataset is divided into three subsets:
the training set, the validation set and the test set, containing
13969, 4725 and 4912 face images, respectively. Our method
is trained on the training set and tested on both the validation
and test set. It should be noted that the ground-truth 3D
landmarks of the test set are not publicly available. Hence
the evaluation results on the test set are provided by 3DFAW
Challenge organizers via the CodaLab platform2 .
300W-LP [13]. The 300W-LP dataset contains 61225 synthesized face images across large poses ranging from −90◦ to
90◦ . Those images are synthesized from 300W [33] using
the 3D morphable model based profiling algorithm proposed
in [13]. For each face, 68 3D landmarks are retrieved from
the parameters of the 3D morphable model, using the released
code of [13]. In our experiments, the depth values are normalized to have zero mean. We only use this dataset for training
and test our method on the AFLW2000-3D dataset mentioned
bellow.
AFLW2000-3D [13]. The AFLW2000-3D dataset contains
2000 face samples selected from the AFLW [34] dataset, introduced by Zhu et al. [13] along with the 300W-LP dataset. The
68 3D landmarks annotated in AFLW2000-3D are consistent
with those of 300W-LP. We use the AFLW2000-3D dataset
only for testing in our experiments, following the common
protocol in the literature [13], [35].
C. Evaluation Metrics
For fair comparison, the evaluation metrics are adopted in
consistency with previous works [12], [13]. For 3D landmark
localization, the Ground Truth Error (GTE) and Cross View
Ground Truth Consistency Error (CVGTCE) are used to
measure the performance as recommenced in the 3DFAW
Challenge [12]. The GTE is defined as the average pointto-point Euclidean error normalized by the distance between
the outer corners of the eyes. The CVGTCE is proposed in
the 3DFAW Challenge and aims at evaluating the cross-view
consistency of the predicted landmarks. For evaluating the 2D
projection of the 3D landmarks, the metric is the Normalized
Mean Error (NME), which is defined as the average 2D pointto-point Euclidean error normalized by the square root of the
bounding box size.
1 https://github.com/HongwenZhang/JVCR-3Dlandmark
2 https://competitions.codalab.org/competitions/10261

Table I: Comparison of GTE on the 3DFAW validation set.
100

GTE (%)

SDM+3DMM [10]
Gou et al. [10]
Bulat et al. [9]

6.34
5.90
4.94

JVCR

4.36

Fraction of Test Faces (%)

Method

Table II: Comparisons of CVGTCE and GTE on the 3DFAW
test set.
Method

CVGTCE (%)

GTE (%)

Zavan et al. [36]
Gou et al. [10]
Zhao et al. [8]
Bulat et al. [9]
Tulyakov et al. [22]

5.90
4.94
3.97
3.47
3.80

10.80
6.20
5.88
4.56
5.10

JVCR

3.46

4.35

90
80
70
60
50
40
30
20

FAN+Depth
JVCR

10
0
0

5

15

20

Figure 3: Comparison of cumulative errors distribution (CED)
curves on AFLW2000-3D.
Table IV: Comparison of NME (%) on the AFLW2000-3D
dataset. Only 2D coordinates are involved in the evaluation.

D. Experimental Results

AFLW2000-3D (68 pts)

In this subsection, we compare our approach with existing methods including top ranked methods on the 3DFAW
Challenge, state-of-the-art 2D face alignment method FAN [2]
and 3D face model based methods such as 3DDFA [13] and
3DSTN [35].
1) Evaluation on 3DFAW: The evaluation on 3DFAW consists of two parts. The first part is evaluated on the validation
set. The second part is evaluated on the test set, of which
performance is provided by the challenge organizers.
Table I shows the comparison results of GTE on the
validation set. It can be observed that the proposed method
outperforms others, all of which are based on the two-step
strategy. For comparison with top ranked methods on the
3DFAW Challenge, we further evaluate our method on the
test set. Comparisons of both the CVGTCE and GTE on
the 3DFAW test set are reported in Table II. Note that the
ground truth 3D landmarks of the test set are not available to
the participants, and the numbers for all methods are taken
from the CodaLab leaderboard and the literature [12], [22].
As shown in Table II, the proposed method achieves the
best result in comparison with other methods, including the
previously top-ranked method [9] and Tulyakov et al. [22]
which is built upon a 3D variant of cascaded regression
method. It demonstrates the effectiveness of the proposed
JVCR framework for accurate 3D facial landmark localization.
Fig. 4 shows example results of the proposed method on the
3DFAW test set.
2) Evaluation on AFLW2000-3D: We further evaluate our
method on AFLW2000-3D to demonstrate the effectiveness
Table III: Comparison of GTE on the AFLW2000-3D dataset.
Method

10
GTE (%)

GTE (%)

FAN+Depth [2]

7.45

JVCR

7.28

[0◦ ,30◦ ]

[30◦ ,60◦ ]

[60◦ ,90◦ ]

mean

std

RCPR [37]
ESR [4]
SDM [3]
3DDFA [13]
3DDFA+SDM [13]
3DSTN [35]
FAN [2]

4.26
4.60
3.67
3.78
3.43
3.15
2.77

5.96
6.70
4.94
4.54
4.24
4.33
3.48

13.18
12.67
9.76
7.93
7.17
5.98
4.60

7.80
7.99
6.12
5.42
4.94
4.49
3.62

4.74
4.19
3.21
2.21
1.97
1.42
0.92

JVCR

2.94

3.46

4.53

3.64

0.81

Method

of our method on face images with large pose and appearance
variations. Our method is trained on 300W-LP and tested on
AFLW2000-3D. Performance of both the 3D and 2D landmark
localization is evaluated for thorough comparison.
Since there are a limited number of previous works on 3D
facial landmark localization in the wild and the training code
for most of them are not available, we consider the most recent
state-of-the-art method FAN [2], with the code released by the
authors, as the baseline for comparison. Specifically, for 3D
facial landmark localization, the baseline method FAN+Depth
performs 2D landmark localization using FAN [2] at first and
then estimates the depth using ResNet [38]. Comparisons of
GTE and CED curves with the baseline method are shown
in Table III and Fig. 3 respectively. Benefiting from the endto-end pipeline, the proposed method outperforms the strong
baseline method considerably.
We also compare our method with other methods on 2D
facial landmark localization. In this case, only 2D coordinates
are involved in the evaluation and the metric is the Normalized Mean Error (NME), where the normalized distance is
square-root of the size of the bounding box enclosing all 2D
landmarks. Comparisons of NME across poses are reported
in Table IV. Note that the results of RCPR [37], ESR [4],
and SDM [3] are obtained from [13] and these methods have
been retrained on 300W-LP for adaptation to large poses. As
shown in Table IV, the proposed method achieve a superior

Figure 4: Example results of the proposed method on the 3DFAW dataset. The top row shows the face images as well as the
2D facial landmark localization results. The middle row shows the estimated volumetric representations. The last row shows
the predicted 3D facial landmarks.

Figure 5: Example results of the proposed method on the AFLW2000-3D dataset. Images are arranged as the same as Fig. 4

performance especially for large poses. Example results of
our method on AFLW2000-3D are depicted in Fig. 5. It can
be seen that our method is robust to occlusions and large
appearance variations occurred in face images in the wild.
V. C ONCLUSION
In this paper, we present the Joint Voxel and Coordinate
Regression (JVCR) method for 3D facial landmark localization. First, we introduce the compact volumetric representation
which encodes positions of all landmarks in a single volume.
In this way, the dimensionality of the representation could be
reduced greatly compared with the conventional volumetric
representation. For robust and accurate 3D facial landmark
localization, we perform the coarse-to-fine voxel regression
and coordinate regression via the stacked hourglass network
and 3D convolution network, respectively. Hence, the joint

voxel and coordinate regression could combine the merits of
both heatmap regression based methods and coordinate regression based methods. Moreover, the proposed method is able to
produce satisfying results on face images with occlusions and
large appearance variations. Experimental results on 3DFAW
and AFLW2000-3D datasets demonstrate the superiority of our
approaches in comparison with other state-of-the-art solutions.
In future work, we will investigate the proposed method further
in the context of general 3D object landmark localization such
as 3D human pose estimation in the wild.
R EFERENCES
[1] A. Bulat and G. Tzimiropoulos, “Convolutional aggregation of local
evidence for large pose face alignment,” in British Machine Vision
Conference, 2016, pp. 86.1–86.12.
[2] ——, “How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks),” in Proceedings of

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]
[19]

[20]

[21]
[22]

[23]

[24]

the IEEE International Conference on Computer Vision, Oct 2017, pp.
1021–1030.
X. Xiong and F. De la Torre, “Supervised descent method and its
applications to face alignment,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2013, pp. 532–539.
X. Cao, Y. Wei, F. Wen, and J. Sun, “Face alignment by explicit shape
regression,” International Journal of Computer Vision, vol. 107, no. 2,
pp. 177–190, 2014.
S. Ren, X. Cao, Y. Wei, and J. Sun, “Face alignment at 3000 fps via
regressing local binary features,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2014, pp. 1685–1692.
A. Bulat and G. Tzimiropoulos, “Human pose estimation via convolutional part heatmap regression,” in European Conference on Computer
Vision. Springer, 2016, pp. 717–732.
A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for
human pose estimation,” in European Conference on Computer Vision.
Springer, 2016, pp. 483–499.
R. Zhao, Y. Wang, C. F. Benitez-Quiroz, Y. Liu, and A. M. Martinez,
“Fast and precise face alignment and 3d shape reconstruction from
a single 2d image,” in European Conference on Computer Vision
Workshops. Springer, 2016, pp. 590–603.
A. Bulat and G. Tzimiropoulos, “Two-stage convolutional part heatmap
regression for the 1st 3d face alignment in the wild (3dfaw) challenge,”
in European Conference on Computer Vision Workshops. Springer,
2016, pp. 616–624.
C. Gou, Y. Wu, F.-Y. Wang, and Q. Ji, “Shape augmented regression
for 3d face alignment,” in European Conference on Computer Vision
Workshops. Springer, 2016, pp. 604–615.
G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis, “Coarse-to-fine
volumetric prediction for single-image 3d human pose,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition.
IEEE, 2017, pp. 1263–1272.
L. A. Jeni, S. Tulyakov, L. Yin, N. Sebe, and J. F. Cohn, “The first 3d
face alignment in the wild (3dfaw) challenge,” in European Conference
on Computer Vision. Springer, 2016, pp. 511–520.
X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, “Face alignment across
large poses: A 3d solution,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2016, pp. 146–155.
D. Cristinacce and T. F. Cootes, “Feature detection and tracking with
constrained local models.” in British Machine Vision Conference, vol. 1,
no. 2, 2006, p. 3.
J. M. Saragih, S. Lucey, and J. F. Cohn, “Deformable model fitting by
regularized landmark mean-shift,” International Journal of Computer
Vision, vol. 91, no. 2, pp. 200–215, 2011.
X. Yu, J. Huang, S. Zhang, W. Yan, and D. N. Metaxas, “Pose-free facial
landmark fitting via optimized part mixtures and cascaded deformable
shape model,” in Proceedings of the IEEE International Conference on
Computer Vision, 2013, pp. 1944–1951.
T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” IEEE Transactions on Pattern Analysis and Machine Intelligence,
no. 6, pp. 681–685, 2001.
I. Matthews and S. Baker, “Active appearance models revisited,” International Journal of Computer Vision, vol. 60, no. 2, pp. 135–164, 2004.
J. Yang, Q. Liu, and K. Zhang, “Stacked hourglass network for robust
facial landmark localisation,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition Workshops. IEEE, 2017,
pp. 2025–2033.
X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, and X. Wang,
“Multi-context attention for human pose estimation,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition,
2017, pp. 5669–5678.
X. Sun, B. Xiao, S. Liang, and Y. Wei, “Integral human pose regression,”
arXiv preprint arXiv:1711.08229, 2017.
S. Tulyakov, L. A. Jeni, J. F. Cohn, and N. Sebe, “Viewpoint-consistent
3d face alignment,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, 2017.
J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler, “Joint training
of a convolutional network and a graphical model for human pose
estimation,” in Advances in neural information processing systems, 2014,
pp. 1799–1807.
S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional
pose machines,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2016, pp. 4724–4732.

[25] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d
shapenets: A deep representation for volumetric shapes,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition,
2015, pp. 1912–1920.
[26] L. Ge, H. Liang, J. Yuan, and D. Thalmann, “3d convolutional neural
networks for efficient and robust hand pose estimation from single depth
images,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, vol. 1, 2017, p. 5.
[27] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks
for human action recognition,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 35, no. 1, pp. 221–231, 2013.
[28] T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient
by a running average of its recent magnitude,” COURSERA: Neural
Networks for Machine Learning, vol. 4, no. 2, pp. 26–31, 2012.
[29] L. Yin, X. Chen, Y. Sun, T. Worm, and M. Reale, “A high-resolution
3d dynamic facial expression database,” in Automatic Face & Gesture
Recognition, 2008. FG’08. 8th IEEE International Conference on.
IEEE, 2008, pp. 1–6.
[30] X. Zhang, L. Yin, J. F. Cohn, S. Canavan, M. Reale, A. Horowitz, P. Liu,
and J. M. Girard, “Bp4d-spontaneous: a high-resolution spontaneous
3d dynamic facial expression database,” Image and Vision Computing,
vol. 32, no. 10, pp. 692–706, 2014.
[31] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-pie,”
Image and Vision Computing, vol. 28, no. 5, pp. 807–813, 2010.
[32] L. A. Jeni, J. F. Cohn, and T. Kanade, “Dense 3d face alignment from
2d video for real-time use,” Image and Vision Computing, vol. 58, pp.
13–24, 2017.
[33] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 faces
in-the-wild challenge: The first facial landmark localization challenge,”
in Proceedings of the IEEE International Conference on Computer
Vision Workshops, 2013, pp. 397–403.
[34] M. Köstinger, P. Wohlhart, P. M. Roth, and H. Bischof, “Annotated
facial landmarks in the wild: A large-scale, real-world database for
facial landmark localization,” in Proceedings of the IEEE International
Conference on Computer Vision Workshops. IEEE, 2011, pp. 2144–
2151.
[35] C. Bhagavatula, C. Zhu, K. Luu, and M. Savvides, “Faster than real-time
facial alignment: A 3d spatial transformer network approach in unconstrained poses,” in Proceedings of the IEEE International Conference
on Computer Vision, Oct 2017, pp. 4000–4009.
[36] F. H. de Bittencourt Zavan, A. C. Nascimento, L. P. e Silva, O. R.
Bellon, and L. Silva, “3d face alignment in the wild: A landmarkfree, nose-based approach,” in European Conference on Computer Vision
Workshops. Springer, 2016, pp. 581–589.
[37] X. P. Burgos-Artizzu, P. Perona, and P. Dollár, “Robust face landmark
estimation under occlusion,” in Proceedings of the IEEE International
Conference on Computer Vision, 2013, pp. 1513–1520.
[38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2016, pp. 770–778.

