A Jointly Learned Deep Architecture for Facial Attribute Analysis and Face Detection in the Wild
Keke He Yanwei Fu Xiangyang Xue
cs.CV







A Jointly Learned Deep Architecture for Facial Attribute Analysis and Face Detection in the Wild
    Keke He, Yanwei Fu, Xiangyang Xue

Fudan University 

{kkhe15, yanweifu, xyxue}@fudan.edu.cn

    December 30, 2023
=================================================================================================







Facial attribute analysis in the real world scenario is very challenging mainly because of complex face variations. Existing works of analyzing face attributes are mostly based on the cropped and aligned face images. However, this result in the capability of attribute prediction heavily relies on the preprocessing of face detector. To address this problem, we present a novel jointly learned deep architecture for both facial attribute analysis and face detection. Our framework can process the natural images in the wild and our experiments on CelebA and LFWA datasets clearly show that the state-of-the-art performance is obtained. 






§ INTRODUCTION

The problem of automatically analyzing the facial attributes received increasing attention recently
<cit.>.
Face attributes may potentially benefit a number of real-world applications,
such as face alignment<cit.>,
head pose estimation <cit.> and face verification
<cit.>. Analyzing facial attributes
still remains challenging in real-world scenarios. For example, most
existing works <cit.> predict the facial attributes
on well-cropped face images. In an extension to real-world scenarios,
those works have to utilize the face detector to localize the bounding
box of human faces before facial attribute analysis. 

Such a pipeline of face detection followed by facial attribute prediction
is nevertheless undesirable in real-world facial attribute prediction
due to two reasons. Firstly, with the preprocessing of face detector,
the capability of facial attribute prediction has to heavily relies
on the results of face detection. Unfortunately in an uncontrolled
setting <cit.>, the face is likely to have large out-of-plane
tilting, occlusion and illumination variations, which may affect the
algorithms of face detection and facial attribute analysis simultaneously.
For example, the Viola-Jones face detector <cit.>
works well for near-frontal faces, but less effective for faces in
the pose far from frontal views. Second, face detection is heterogeneous
but subtly correlated with facial attribute prediction, and vice versa <cit.>.
Efficiently and effectively exploiting this correlation may help in
both detecting faces and predicting facial attributes.

We propose an end-to-end deep architecture to jointly learning
to detect faces and analyze facial attributes. Essentially, the two
tasks share the same bottom layers (feature map in Fig. <ref>)
in the architecture to alleviate the problems of face variations aforementioned.
For instance, one face region of one image is failed
to be detected as a face due to the partial occlusion, and yet the network
can detect the existence of some facial attributes such as “eyeglasses”,
and “smiling”. The face region can still be used to optimize our
network and in turn help supervise the learning of the face detection
part, to improve the performance of face detection part.

In this work, the deep learning architecture is proposed for both
face detection and facial attribute analysis. The network structure
is illustrated in Fig. <ref>. Given an entire image,
our architecture will firstly pass the image with convolutional layers
(conv1– conv5) and max pooling layers to
produce the conv feature map for face region proposals. 
The region of interest (RoI) pooling layer and face region proposal layer
are two layers introduced to facilitate the jointly learning
of our architecture. A region of interest (RoI) pooling layer pools
each face proposal on the conv5 feature map into a fixed-length
feature vector which is further processed by two fully connected layers
(fc6 and fc7). Built on the fc7 layer, each individual
task of facial attribute analysis and face detection are jointly optimized.
By virtue of the face detection subnet, our architecture can directly
predict the facial attributes from the whole images, rather than
using the well-cropped images as previous work <cit.>.
Extensive experiments on benchmark CelebA <cit.> and
LFWA <cit.> datasets demonstrate that our method outperforms state-of-the-art
alternatives.




Contribution. Our main contribution is to propose
a novel deep architecture of jointly learning of face detection
and facial attribute tasks, which is capable of utilizing both tasks
to better optimize the shared network and thus improve the performance
of both tasks. Two layers – region of interest (RoI)
pooling layer and face region proposal layer are introduced to enable learning these two tasks simultaneously. More importantly, we
can predict the facial attributes in the wild and the input images
do not need to be cropped and aligned as the standard practice in <cit.>.
Finally, our joint deep architecture achieves state-of-the-art result
on the biggest benchmark dataset for facial attribute analysis–
CelebA dataset <cit.> and LFWA <cit.>
datasets. 




§ RELATED WORK




Facial attribute analysis. It was first studied
by Kumar et al. <cit.>. In term of different visual
features and distinctive learning paradigm, the facial attribute analysis
has been developed into three categories: (1) the methods <cit.>
of using hand-crafted visual features, such as SIFT <cit.>
and LBP <cit.>; (2) the methods of utilizing the recent
deep features <cit.>; and
(3) multi-task methods of learning facial attribute <cit.>.
We here highlight the differences between our architecture and these
previous works. Liu et al. <cit.> cascaded three
deep networks pre-trained for facial attribute prediction. In contrast,
we show that the tasks of face detection and facial attribute prediction
are highly correlated and our jointly learning architecture can improve
both tasks. Rudd et al. <cit.> introduced a mixed
objective optimization network which utilizes distribution
of attribute labels to learn each task.
Abdulnabi et al. <cit.> proposed a multi-task
CNN model sharing of visual knowledge between tasks for facial attribute
analysis. Comparing with <cit.>,
we focus on jointly learning the face detection and facial attribute
analysis; and our model can predict facial attributes on the images
in the wild <cit.> i.e. without
cropped and aligned, as illustrated in Fig. <ref>
and Fig. <ref>. 




Face detection. The work of Viola and Jones <cit.>
made face detection usable in many real world applications. The cascaded
classifiers were built on Harr-like features to detect human faces.
The deformable part model (DPM) <cit.> on
top of HOG feature is a general object detector and can also be used
for face detection <cit.>. Recent advances of
deep learning architectures also inspire another category of methods
for face detection. Yang <cit.> used the fully convolutional
networks (FCN) to generate the heat map of facial parts for producing
face proposals. In contrast to these works, our face detection task
gets benefit from not only the well-designed architecture (in Fig.
<ref>), but also the jointly learning process with
facial attribute prediction. Further inspired by Fast-RCNN <cit.>,
we take the face detection as a special case of the general semi-rigid
object detection. More specifically, given an image, our face detection
will try to answer two questions: (1) whether this patch contains
faces or not? i.e. face score task in Fig. <ref>.
(2) can we detect the bounding box of faces if there is any face?
i.e. face bounding box task in Fig. <ref>.
The jointly optimizing these two sub-tasks will better solve face
detection.




Multi-task learning. Our framework can be categorized
as multi-task learning <cit.>,
which shares the information and explores the similarity of related
tasks on the same data. The multi-task learning can facilitate a wide
range of tasks and applications, including but not limited to action
recognition <cit.>, information retrieval <cit.>,
facial landmark detection <cit.>,
and facial attribute prediction  <cit.>.
The recent technical report <cit.> also combine
face detection with the tasks of locating face landmarks and recognizing
gender. However, unlike our facial attribute prediction on the images
in the wild, their work still has to firstly detect face regions for
the facial landmarks and predicting gender. 



§ OUR DEEP ARCHITECTURE 


In this section, we firstly overview our network in Sec. <ref>,
and then the tasks are defined in Sec. <ref>.
The Face region proposal and RoI pooling layers are explained accordingly
in Sec. <ref> and Sec. <ref>.
Finally, we utilize the network to solve the facial attribute prediction
and face detection in Sec. <ref>. 



 §.§ Overview


Figure <ref> shows our framework for jointly face
detection and attribute analysis. Our architecture takes an entire
image as input and a set of face bounding boxes as labels for training. The whole
network firstly processes the image with several convolutional layers
(conv1– conv5), and max pooling layers to
produce a conv feature map for face region proposal. For feature
map of each proposed face, we employ a region of interest (RoI) pooling
layer to pool it into a fixed-length feature vector. Each feature
vector is further processed by two fully connected layers (fc6
and fc7) and thus used for the tasks of facial attribute analysis
and face detection. On top layers, our architecture has face
detection branch and facial attribute branch. 

Our network follows the art and design of VGG-16 <cit.>.
Particularly, the kernel size, stride and the number of filters in
convolutional layers (conv1– conv5) and
the two fully connected layers (fc6 and fc7) are exactly
the same as the corresponding layers in VGG-16 architecture.



 §.§ Task formulation


Suppose we have the labelled source training dataset 𝒟_s={𝐈,𝐚,𝐋}
with N training instances and M attributes. 𝐈 denotes
the patches of training images and 𝐋 denotes the labels. We use the 𝐋 matrix to
both denote whether an image patch contains the face, and whether
a facial attribute exists in the image patch. Particularly, for the
i-th image patch 𝐈_i (i=1,⋯,N), we use 𝐋_i⋆=0
to indicate that this image patch does not contain a human face. If
𝐈_i is a face image patch, we use 𝐋_ij=+1
to denote the existence of j-th facial attribute 𝐚_j (j=1,⋯,M);
𝐋_ij=-1 otherwise. 

As illustrated in Fig. <ref>, our network extracts
the image patch to a 4096-dim feature vector, and it is denoted
as f(𝐈;Θ), where Θ is the parameter
set of the deep architecture. We have the prediction tasks of facial
attribute, face score and face bounding box. Each task has their own
parameters on the last layer.




Facial attribute prediction. To predict the attributes
of the image 𝐈^*, we need to learn a function 𝐋_i𝐚^*=Ψ(𝐈^*)
to predict the facial attribute 𝐚. We thus have the predicting
function Ψ=[ψ_i]_i=1,⋯,M, and ψ_i(𝐈):ℝ^4096→{ +1,-1};
specifically, we consider Ψ(x)=W_a^Tx, where
W_a⊆ℝ^4096×40 for all 40 attributes.




Face score. This task aims to predict the score
whether an image patch is face. Essentially, we need to learn Φ(x):ℝ^4096→{ +1,-1},
and Φ(x)=W_s^Tx, where W_a⊆ℝ^4096×2
for binary prediction task.




Face bounding box. We regress the bounding box
of faces by Ω(x):ℝ^4096→ℝ^4,
and we configure the form as Ω(x)=W_b^Tx, and
W_b⊆ℝ^4096×4 for the new bounding box.



 §.§ Face region proposal layer 


Inspired by the work of object detection <cit.>, our
network has the branch of face region proposal. Specifically, this
branch takes an image of any size as input and outputs a set of rectangular
face proposals. To generate the facial region proposal, we slide the
branch over the convolutional feature map after the five convolutional
layers (conv1– conv5) in Fig. <ref>.
The extracted conv5 features are further fed into two sibling
fully connected layers, i.e. face bounding box (bbox) and face
score layer. Face bounding box layer employs the smooth L_1
loss (defined in <cit.>) enables a regressor to predict
the facial bounding box. Face score layer utilizes the softmax loss
to indicate whether the bounding box is a face. For each image, we
selected top-300 face region proposals in term of the face scores
computed.



 §.§ The RoI pooling layer

The RoI pooling layer can convert the feature maps of face region
proposal (with the size of h× w) to a fixed spatial extent
of H× W, which facilitates the further process. Here, h,w,H,W
are heights and widths of each rectangle region of feature map in
Fig. <ref>. For the varying size of input feature patches,
we vary the size of filters of pooling layer with the sub-window of
approximate size h/H× w/W. The RoI pooling technique is 
 adopted from the work of object detection <cit.> which inspired by the SPPnets <cit.>.



 §.§ Facial attribute analysis and face detection 


The fixed-length feature vector extracted from RoI Pooling layer is
further fed into two fully connected layers (fc6 and fc7)
in order to enable the tasks of facial attribute analysis. Particularly,
built on the layers of fc6 and fc7, the multi-task network
structure is employed to analyze each attribute individually. This
can be modeled as the minimization of the expected loss over all the
training instances which is


    {Θ,W_a,W_s,W_b}    =argmin ℒ(𝐈;Θ,W_a,W_s,W_b)


where ℒ(𝐈;Θ,W_a,W_s,W_b)
is the loss function of jointly learning; and we have 

    ℒ(𝐈;Θ,W_a,W_s,W_b)     =λ_1ℒ_a(Φ(f(𝐈)))+λ_2ℒ_s(Ψ(f(𝐈)))
        +λ_3ℒ_b(Ω(f(𝐈)))

here the loss function ℒ_a(Φ(f(𝐈)))
on facial attribute prediction is the mean square error loss
of all attributes; we use softmax loss ℒ_s(Ψ(f(𝐈)))
for face score task, and finally the smooth L_1- loss
ℒ_b(Ω(f(𝐈)))
<cit.> is employed to regress the face bounding box.

We denote the shared parameters of deep architectures as
Θ which are jointly optimized by all these three tasks. Since
these three tasks are highly related, learning in such a way can not
only greatly reduce the prediction error of each individual task,
but also accelerate the convergence rate of learning the whole network.

For the testing image 𝐈_k, the predicted result 𝐋̂_kj
of the j-th facial attribute 𝐚_j (j=1,⋯,M)
is thresholded by


    𝐋̂_kj=
    1    ψ(𝐈_k)>τ
    
    -1    ψ(𝐈_k)≤τ


where τ is the threshold parameter.








§ EXPERIMENTS






 §.§ Datasets and settings




We conduct the experiments on the CelebA dataset <cit.>
and LFWA dataset <cit.>.

CelebA <cit.> contains approximately
200k images of 10k identities. Each image is annotated with 5 landmarks
(two eyes, the nose tips, the mouth corners) and binary labels of
40 attributes. To make a fair comparison with the other facial
attribute methods, the standard split is used here: the first 160k
images are used for training, 20k images for validation and remaining
20k for testing. CelebA provides two types of training images,
i.e., aligned and cropped face images and raw images as shown
in the first and second row of Fig. <ref> respectively.
We use the raw images for training our joint deep architecture. Since
there is no ground-truth bounding box label for faces in raw images,
the face detector <cit.> is employed here
to help generate the face bounding box for training images. We use
the implementation of dlib toolbox <cit.>
for generating labels. Note that face bounding box generation is only required in the training stages 
for synthesizing labels.
At testing phrase, raw images are directly
input for both facial attribute analysis and face detection.




LFWA <cit.> is constructed based on
face datasets LFW <cit.>. It contains approximately
13143 images of 10k identities. Following <cit.>,
50% of the images for training, and the other 50% are used
for testing. LFWA has 40 binary facial attributes, the same as
CelebA. We also generate face bounding box for LFWA to train our joint
learning network.




Evaluation metrics. We take the attribute prediction
as classification tasks and thus mean accuracy can be computed. Particularly,
we evaluate the performance as comparable to <cit.>
by the mean error which is defined as mean error=1-mean accuracy.




Implementation and Parameter settings. The τ
is set as 0 in Eq (<ref>); and square error loss
is used in Eq (<ref>). We empirically set λ_1=λ_2=1,λ_3=2
in Eq (<ref>). The convolutional layers and fully
connected layers are initialized by the 16 layer VGG network <cit.>
individually. We use the open source deep learning framework Caffe
<cit.> to implement our structure. A single end-to-end model
is used for all the testing. We employ the stochastic gradient descent
to train our network. Dropout is used for fully connected layers and
the ratio is set to 0.5. For training CelebA dataset, with initial
learning rate 0.001, and gradually decreased by 1/10 at 100k,
150k iterations, the total training iterations are 180k. For
training LFWA dataset, as there are only limited training images -
6263, We fine-tune from pre-trained CelebA model, with initial
learning rate 0.0001, and decreased by 1/10 at 40k iterations,
the total training iterations is 60k. 



Once trained, our framework
can predict the facial attributes and detect faces on the images in
the wild for any testing images. 


 

Running cost. Our facial model get converged with
180k iterations and it takes 29 hours on CelebA
with one NVIDIA TITANX GPU. 
On LFWA dataset, Our facial model get converged with 60k
iterations and takes 9 hours. For training
all the model, and it takes around 4
GB GPU memory.



 §.§ Competitors. 


Our model is compared against state-of-the-art methods and several
baselines. 
Particularly, (1) FaceTracer <cit.>
is one of the best methods with the hand-crafted features. The features
used including HOG <cit.> and color histograms of facial
regions of interested to train SVM classifier for predicting facial
attributes. (2) LNets+ANet <cit.> migrates two
deep CNN face localization networks to one deep CNN network for facial
attribute classification. (3) Walk and Learn <cit.>
learns good representations for facial attributes by exploiting videos
and contextual data (geo-location and weather) as the person walks.
(4) Moon <cit.> is a mixed objective optimization
multi-task network to learn all facial attributes, achieves the best result
on CelebA dataset. (5) Cropped model is a variant of our
model without using the face detection branch. The raw image is also
used as the input and we crop the faces from the images by Dlib toolkit
to train facial attribute models. The processed face images are utilized
to train the model. (6) Aligned model uses the aligned and cropped images provided
by CelebA and LFWA to train facial attribute model, which is used
by most of the state-of-the-art methods. To make a fair comparison, these
two baseline models are initialized by 16 layer VGG network; same
as the proposed structure.








 §.§ Comparison with the State of the Art

We compared our model with state-of-the-art methods: FaceTracer <cit.>,
LNets+ANet <cit.>, Walk and Learn <cit.>,
Moon <cit.> and two baselines. The results on the testing
split of CelebA and LFWA are reported in Tab. <ref>, Fig. <ref>
and Fig. <ref>, respectively. 
Note that please refer to the supplementary material for the full comparison results on each attribute. Comparing
with all the competitors, we draw the following conclusions.




Our model beats all the other methods by the mean error
on CelebA and LFWA dataset. As we can see from Fig. <ref>,
our approach obtains the mean error 8.41% which is the
lowest among all the competitors, and it outperforms all the other
methods on CelebA dataset. And on LFWA dataset as shown in Fig. <ref>,
our mean error is only 13.13% which is the lowest among
all the competitors. This validates the efficacy of our joint learning
architecture. Particularly, We compare the classification error
in each individual attribute in Fig. <ref>
and Fig. <ref>. We note that on more
than half among the total 40 attributes, our framework is significantly
better than the other competitors, since our jointly learning architecture
can efficiently leverage the information between face detection and
facial attribute prediction. Particularly, comparing with the other
works, all our attribute prediction tasks share the same deep learning
architecture, i.e. conv1-conv5, feature map and fc6,
fc7 layers as illustrated in Fig. <ref>).
These shared structures implicitly model the correlations between
each attribute task. Furthermore, on gender attribute
prediction, our framework can achieve 1.7% error on CelebA; in contrast,
the error of HyperFace <cit.> is 3.0%.
That indicates that our results can get 1.3% improvement over
that of HyperFace <cit.>.




Our model can process the images in the wild. Our face region proposal and RoI pooling layers are flexible
enough to directly process the images in the wild. This thus better
demonstrates the effectiveness of our models. Specifically, our model
not only achieves the best performance on the benchmark dataset –
CelebA and LFWA, but also our tasks of facial attribute prediction
do not need to align and crop the facial images as have done in many
previous work <cit.>.
Additionally, unlike the work of Walk and Learn <cit.>,
our model is not trained by the external data; and yet still obtains
better results than Walk and Learn <cit.>.




Our model is very efficient in term of jointly
learning to detect faces and predict facial attributes. Our joint
learning results have greatly improved over those of two baseline
models – Cropped, Aligned models. These two methods
are yet another two naive baselines of directly learning the facial
attribute tasks. Particularly, on CelebA dataset, our model beats
the two methods on the classification error of 27 attributes
(totally 40 facial attributes) as compared in Fig. <ref>.
Figure <ref> shows that our joint
model hit better results than the two baseline models on all the 40
attributes. This reveals attribute prediction tasks get benefit from
the face detection task. Also note that unlike LNets+ANet <cit.>
using two networks to localize the face and another one network to
extract features, we can train our model by a single end-to-end network
here since the shared architectures (conv1 – conv5)
can explicitly model the important facial parts, again thanks to the
face detection subnet. 




Qualitative results. The important facial parts
extracted by the shared feature map are visualized in Fig. <ref>.
In particular, four groups of images are shown; and the feature map
has higher activation on the regions of human faces. The first 3 columns
are success cases, even thought the images have large pose (Column
2) or very high occlusion (Column 3). We also list some failure examples
in Column 4, which are some extreme cases. Some of them are caused
by ambiguous or too small view (e.g. the cat face is also similar
to human's). The feature map reveals our model pay attention on the
important facial parts which helps to analyze facial attributes.




 §.§ Results of Face Detection


Our face detection is compared against the face detector of dlib toolbox
<cit.>, which is an implementation of the generic object detector
<cit.> on face images. On CelebA dataset, there
are only 5 landmarks of faces, but no labeled ground-truth bounding
box of faces for raw images. We generate the ground truth bounding box by 5 landmark coordinates for the test set. If the IOU (Intersection over Union) of predicted bounding box and the ground truth bounding box is larger than 0.5, we assume the prediction is correct.
We compare the two detectors on the testing split of CelebA.

The results are shown in Fig. <ref>. We compare
the Precision-Recall curve for two methods. We find that both the
Dlib detector and our face detector can achieve very high face detection
on the CelebA test set as shown in the left subfigure of Fig. <ref>,
which shows the efficacy of both detectors of solving the task of
face detection. Nevertheless, our face detector still beats the Dlib
detector by a relatively large margin. The AUC values of Dlib detector
and ours are 0.938 and 0.982 respectively. To better show the
difference, we highlight the up-right corner of PR curve in the right
subfigure as shown in Fig. <ref>. This result
further proves that our jointly learned architecture can make the
face detection and face attribute tasks help each other via sharing
the same parameters of the deep network. This shows that attribute learning
does help for face detection also it provides more detailed information
about the face.






§ CONCLUSION


In this paper, we propose a novel joint deep architecture for facial
attribute prediction and face detection. Different from the previous pipeline
of face detection followed by facial attribute prediction, our architecture
takes an entire image as input, enables both face detection and facial
attribute analysis. The proposed architecture can not only exploit
the correlation of face detection and face attribute prediction, but
also boost both tasks. The experimental results on CelebA and LFWA datasets
show the efficacy of proposed methods over the other state-of-the-art
methods.



ieee



