A Utility-Preserving GAN for Face Obscuration
Hanxiang Hao David Güera Amy R. Reibman Edward J. Delp


[
A Utility-Preserving GAN for Face Obscuration














equal*


Hanxiang Haoviper
David Güeraviper
Amy R. Reibmanece
Edward J. Delpviper


viperVideo and Image Processing Laboratory (VIPER), School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, USA
eceSchool of Electrical and Computer Engineering, Purdue University, Indiana, USA

Edward J. Delpace@ecn.purdue.edu




Privacy Protection, Face Obscuration, Deep Learning, GAN

0.3in
]









  



From TV news to Google StreetView, face obscuration has been used for privacy protection. 
Due to recent advances in the field of deep learning, obscuration methods such as Gaussian blurring and pixelation are not guaranteed to conceal identity.
In this paper, we propose a utility-preserving generative model, UP-GAN, that is able to provide an effective face obscuration, while preserving facial utility. 

By utility-preserving we mean preserving facial features that do not reveal identity, such as age, gender, skin tone, pose, and expression. 
We show that the proposed method achieves the best performance in terms of obscuration and utility preservation. 




§ INTRODUCTION
 




Major developments in the machine learning field have uncovered severe flaws in current face obscuration approaches.
As shown by <cit.>, machine learning methods are able to defeat Gaussian blurring or pixelation based obscuration methods.
These obscuration techniques have been widely used by Internet news outlets, social media platforms, and government agencies.
An extreme resort to prevent information leaking is to simply gray out the entire facial region by setting all pixels in the facial area to a fixed value. 
However, this approach is rarely used because its visual effect is unpleasant, especially if there are many faces to be redacted.
Besides the identifiable information, facial images also contain information that does not reveal identity, such as age, gender, and skin tone. 
Often, we want to preserve these features in many applications involving visual understanding and data mining <cit.>.


New obscuration methods are needed to remove identifiable facial information, while preserving the features that do not convey identity.
The proposed method, utility-preserving GAN (UP-GAN), aims to provide an effective obscuration by generating faces that only depend on the non-identifiable facial features. 
In this work, we define utility as the facial properties such as age, gender, skin tone, pose, and expression. 
We choose these properties because in practice, when dealing with a large number of identities, knowing these properties from the obscured images cannot reveal identity. 
One can also choose other properties to retain for different applications. 

As shown in Figure <ref>, UP-GAN is able to obscure the original faces by replacing them with synthetic faces that have the same utility. 



The main contributions of this work are summarized as follows. 
First, we develop a model that can produce new faces for obscuration. 
Then, we provide an evaluation to show the effectiveness of the proposed method with different training objectives. 
Finally, compared to different methods, we show that UP-GAN can provide the best performance in terms of face obscuration and utility preservation.



§ RELATED WORK
 


Standard approaches, such as pixelation and Gaussian blurring, achieve good obscuration performance in terms of human perception. 
However, <cit.> proposed a deep learning method with a simple structure that is able to defeat these obscuration techniques.
To provide better obscuration performance, a variety of approaches have been proposed to balance the need to remove identifiable information while preserving utility information. 



 §.§ Face De-identification

k-same Methods. 
This family of approaches first groups faces into clusters based on non-identifiable information such as expression, and then generates a surrogate face for each cluster. 
These methods can guarantee that any face recognition system cannot do better than 1/k in recognizing who a particular image corresponds to <cit.>, where k is the minimum number of faces among all clusters.
This property is also known as k-anonymity <cit.>.  
In <cit.> and <cit.>, they simply compute the average face for each cluster. 
Therefore, their obscured faces are blurry and cannot handle various facial poses. 
In <cit.>, the use of an active appearance model <cit.> to generate more realistic surrogate faces is presented. 
A generative neural network, k-same-net, that directly generates faces based on the cluster attributes is described in <cit.>. 
These two methods are able to produce more realistic obscured faces with the property of k-anonymity, but cannot handle different poses.  

GAN Methods. 
Generative adversarial network (GAN) <cit.> methods can provide more realistic faces. 
Their discriminator is designed to guide the generator by distinguishing real faces from generated faces. 
In <cit.>, a model that produces obscured faces directly from original faces based on conditional-GAN <cit.> is proposed. 
They use a contrastive loss to enforce the obscured face to be different than the input face. 
However, since they need to directly input the original faces, the obscuration performance is not guaranteed. 
<cit.> present a two-stage model that is able to generate an obscured face without the original identifiable facial information, which prevents the leakage of identifiable information directly from faces.  
GANs have also been used for face manipulation in videos. 
These techniques aim to create believable face swaps without tampering traces, by altering age <cit.> or skin color <cit.>. 
To prevent scenarios where these videos are used to create political distress or fake terrorism events, <cit.> design a deep learning model that is able to detect the altered frames using both the spatial and temporal information. 

Our proposed method tries to leverage the advantages of both types of methods. 
To achieve k-anonymity, it is designed to generate faces that depend only on the utility information without directly accessing original faces. 
Since it is also a GAN based method, with the discriminator guidance, it is able to produce more realistic faces than the k-same methods. 




§ PROPOSED METHOD
 
Recall that, in this implementation, we choose age, gender, skin tone, pose, and expression as the utility to be preserved. 
To better formulate our problem, we further divide the utility into two parts: attributes and landmarks. 
Attributes define the static part of the utility information that does not change with facial movement. 
Landmarks define a set of points of interest that describe the facial pose and expression. 
In order to obtain obscured faces, we first use an auxiliary system to detect the utility information: attribute vector v_a and landmark vector v_l from the original face I_real.

Since, in this work, we are not focusing on this auxiliary system, we use the UTKFace dataset <cit.> which provides the needed attributes (age, gender, and skin tone) and landmarks (7 points) to train and test our model. 

The fake face I_fake is then generated by the UP-GAN model using the attribute and landmark vectors.  
Given that the generated face has the same pose and expression, we can swap it with the original face to perform de-identification using face swapping algorithms <cit.>. 
Figure <ref> shows the swapping results using <cit.>. 











Figure <ref> shows the generator architecture of the UP-GAN model, which is based on the architecture proposed by <cit.>. 
Similar to the previous work, UP-GAN jointly learns the fake face and its binary mask. 
However, we modify the structure of the fully-connected layers to input the attribute and landmark vectors. 
As suggested by <cit.>, we also add a max pooling layer with stride 1 for dimension reduction before generating the output image and mask. 
More specifically, we first use two fully-connected layers to encode the input vectors and then apply de-convolution, followed by another convolution layer to upsample the feature maps.
The de-convolution layer contains an upsampling layer with stride 2 and a convolution layer with a kernel size of 5. 
For the following convolution layer after the de-convolution layer, we choose the kernel size to be 3. 
Note that the final output size of the generated face is 128×128×3 and the size of the binary mask is 128×128×2. 

The loss functions for the generator G and discriminator D are defined as:

    ℒ_G = 𝔼_v_a, v_l[logD(G(v_a, v_l)))]+ 
                                 λ_1ℒ_2 + λ_2ℒ_M + λ_3ℒ_P, 
       ℒ_D = 𝔼_I_real[logD(I_real)] + 
                                 𝔼_v_a, v_l[log(1-D(G(v_a, v_l)))],



where 

    ℒ_2 = ‖ I_real - I_fake‖_2^2, 
       ℒ_M = -1/N∑_i=1^N y_ilog(p_i) + (1 - y_i)log(1 - p_i). 
       ℒ_P = ∑_l ∈Ω‖ϕ_l(I_fake) - ϕ_l(I_real) ‖_2^2,

-0.3in
ℒ_2 is the reconstruction loss for learning the image content.  
ℒ_M is the binary cross entropy loss for learning the facial mask where p_i is the predicted probability of the i-th pixel in the binary mask, y_i is the ground truth label, and N is the total number of pixels. 
ℒ_P is the perceptual loss for learning the facial details, where Ω is a collection of convolution layers from the perceptual network and ϕ_l is the activation from the l-th layer.  
The perceptual loss was originally proposed by <cit.> for learning high level features extracted from a network pretrained on the ImageNet dataset <cit.>. 
In our work, the perceptual network is pretrained on a face identification dataset to enforce that the generated face contains similar facial features to the original face. 
More specifically, we choose the pretrained VGG-19 network <cit.> and finetune it with the FaceScrub dataset <cit.>. 
Lastly, λ_1, λ_2, and λ_3 are the scalar weights for their corresponding losses. 
Note that in our implementation, we have chosen λ_1=5, λ_2=1, and λ_3=1 to ensure that the terms in ℒ_G are within the same numerical order of magnitude.





§ EXPERIMENTS
 
In this section, we will evaluate different loss functions and analyze the obscuration performance of the generated faces compared to Gaussian blurring, pixelation, k-same method and k-same-net method. 



 §.§ Datasets

The UTKFace dataset <cit.> contains 23,708 images with annotations of 68-point facial landmarks and attributes of age, gender, and skin tone. 

To obscure the identifiable information present in the facial landmarks, we reduce the input landmark points from 68 points to 7 points.
These include the centers of the eyes, the center of the nose, and four points around the mouth.
Therefore, the dimensionality of the attribute vector is 3 and of the landmark vector is 14. 
From the perspective of k-anonymity, reducing landmark points is similar to increasing k.
When we increase k, the size of each cluster also increases, since they are grouped based on attribute and landmark vectors. 
Therefore, the upper bound of identification rate (1/k) decreases, meaning that the obscuration performance improves. 

To verify the obscuration performance, we use the FaceScrub dataset for face identification.
Note that this dataset contains 106,806 images from 530 identities. 
As this dataset does not provide attributes and landmarks, we use fixed attribute values and detect facial landmarks using the Dlib toolkit <cit.>. 
We can produce fake faces using the fixed attributes and detected landmarks. 
We then use a face identification model (VGG-19) to determine if we are able to identify these generated faces. 



 §.§ Data Augmentation

To prevent UP-GAN from simply memorizing the original face and replicating the output face using the input vectors, we use data augmentation on the original image I_real to increase its variation. 
First, we use elastic distortion <cit.> to add variety to the facial landmarks. 
As shown in Figure <ref>, the wave-like structure distorts the landmark points (e.g. the shape of the mouth). 
We also add random rotations, ranging from -30 to 30, to increase the variation of facial poses. 





 §.§ Results and Discussion

In Figure <ref>, we compare the results using different loss functions to show the effectiveness of training with the perceptual network and binary mask. 
We can also see that, compared to the original face, the generated face with adversarial loss and L_2 reconstruction loss can preserve the facial utility.
However, the facial details such as the outlines are partially missing.
By adding the mask loss, we can enhance the facial boundary, like the cheek and chin. 
If we add the perceptual loss, the generated face visually looks more realistic with fewer ripple-like artifacts. 



































We also evaluate the obscuration performance to see how well UP-GAN can conceal the original faces. 
We consider two threat models:
I) the attacker (identifier) has no information about the obscuration methods and
II) the attacker knows the obscuration methods. 
In threat model I, we train the identifier on the pristine images and test it on the obscured faces. 
In threat model II, we train and test the identifier on both clear and obscured images. 
To provide a fair comparison with the other obscuration methods, we use the generated faces I_fake as the obscured images, but we do not swap them into the original images. 
This is because the unobscured area (non-facial region) may contain identifiable information. 
Figure <ref> shows the visual quality of the obscured images with different methods including Gaussian blurring, pixelation, k-same method <cit.>, k-same-net method <cit.>s, and UP-GAN.

We modify the input layers of the k-same net method to input the same attribute and landmark vectors as UP-GAN. 
The obscured face from k-same method is really blurry like the areas of eyes, although the skin tone is preserved. 
The result from k-same-net method contains more facial structures, but compared to UP-GAN, the facial boundary is not clear.
To further quantify the visual performance, we compute the Fréchet inception distances (FID) <cit.> of the obscured faces. 
With the assumption that the real and obscured faces are two sets of realizations coming from two distributions, FID measures the distance of these two distributions. 
Therefore, we can use FID to estimate how realistic the obscured faces are.  
As shown in Table <ref>, UP-GAN achieves the minimum FID value, which confirms that the obscured face has the best visual quality. 

Table <ref> also compares the obscuration performance of UP-GAN against other methods. 
For the threat model I, Gaussian blurring with kernel size 5 and 15 fail to provide an effective obscuration, while all other methods achieve good performance. 
For the threat model II, the obscuration performance degrades for all methods, while pixelation with pixel size 25, k-same, k-same-net, and UP-GAN still achieve relatively good results. 

However, as shown in Figure <ref>, for pixelation-25 there are only 5×5 blocks representing the facial region. 
As with k-same and k-same-net, the visual quality of pixelation-25 is worse than UP-GAN. 







§ CONCLUSION
 
Gaussian blurring or pixelation cannot guarantee obscuration and preserve utility such as age, gender, skin tone, pose, and expression.
Our proposed approach, UP-GAN, is able to generate faces that preserve utility while also removing identifiable information from the original faces. 
By swapping the generated face back on the original image, we can produce an effective obscuration that not only removes personal identifiable information, but also retains the information that does not reveal identity. 
Based on our results, we show that UP-GAN is able to achieve a better performance than Gaussian blurring, pixelation, and k-same method in terms of face obscuration and utility preservation.  



§ ACKNOWLEDGMENTS
 
This material is based on research sponsored by the Department of Homeland Security (DHS) under agreement number 70RSAT18FR0000161. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DHS or the U.S. Government. 


icml2019

