Aurora Guard: Real-Time Face Anti-Spoofing via Light Reflection
Yao Liu Ying Tai Jilin Li Shouhong Ding Chengjie Wang Feiyue Huang Dongyang Li Wenshuai Qi Rongrong Ji





















Aurora Guard: Real-Time Face Anti-Spoofing via Light Reflection
    
First Author^1
Second Author^2Third Author^2,3Fourth Author^4
^1First Affiliation

^2Second Affiliation

^3Third Affiliation

^4Fourth Affiliation
{first, second}@example.com,
third@other.example.com,
fourth@example.com

    
=================================================================================================================================================================================================================================




   
   
   In this paper, we propose a light reflection based face anti-spoofing method named Aurora Guard (AG), which is fast, simple yet effective that has already been deployed in real-world systems serving for millions of users.
   Specifically, our method first extracts the normal cues via light reflection analysis, and then uses an end-to-end trainable multi-task Convolutional Neural Network (CNN) to not only recover subjects' depth maps to assist liveness classification, but also provide the light CAPTCHA checking mechanism in the regression branch to further improve the system reliability.
   Moreover, we further collect a large-scale dataset containing 12,000 live and spoofing samples, which covers abundant imaging qualities and Presentation Attack Instruments (PAI).
   Extensive experiments on both public and our datasets demonstrate the superiority of our proposed method over the state of the arts.
   





§ INTRODUCTION
 
Face anti-spoofing is currently a promising topic in computer vision research community, and also a very challenging problem in remote scenarios without specific hardware equipped in industry.
The existing methods <cit.> on face anti-spoofing are paying more attention on multi-modality information (e.g., depth or infrared light).
With the development of depth sensors, recent methods and commercial systems exploit hardwares that can be embedded with structured light (e.g., FaceID on iphone X), light field <cit.> or LIDAR to reconstruct accurate 3D shape, which can well address the limitation of 2D methods towards high-level security <cit.>. Although these methods can achieve good classification performance, they highly rely on the customized hardware design, which unavoidably increases the system cost.

As a replacement, recent advances on Presentation Attack Detection (PAD) tend to estimate depth directly from a single RGB image.
In particular, since 3D reconstruction from a single image is a highly under-constrained task due to the lack of strong prior of object shapes, such methods introduce certain prior by recovering sparse <cit.> or dense <cit.> depth features.
However, these methods still suffer from missing the solid depth clue.

As a result, the corresponding liveness classifiers are hard to generalize to real presentation attacks in the wild.

Towards high accuracy and security without using depth sensors, we propose a simple, fast yet effective face anti-spoofing method termed Aurora Guard (AG).
Its principle is to use light reflection to impose two auxiliary information, i.e., the depth map and light parameter sequence, to improve the accuracy and security of PAD respectively (as shown in Fig. <ref>).
In this paper, we propose and define the light parameters sequence as light CAPTCHA.


By only incorporating a single extra light source to generate the reflection frames, our method holds the efficiency and portability of cost-free software methods, which has already been deployed on smart phones and embedded terminals that serves for millions of users.





More specifically, our method mainly consists of two parts:
(1) 

Based on Lambertian model, we cast dynamic changing light specified by the random light CAPTCHA, and then extract the normal cues from facial reflection frames.

(2) We use an end-to-end trainable multi-task CNN to conduct liveness classification and light CAPTCHA regression simultaneously.
The classification branch estimates the depth maps from the normal cues, and classify liveness from the recovered depth map via a compact encoder-decoder structure.
 

The regression branch estimates the light parameter sequence, which forms a light CAPTCHA checking mechanism to handle one special type of attack named modality spoofing, which is a very common attack in real scenarios.

Moreover,

since the imaging quality (resolution, device) and the types of Presentation Attack Instruments (PAI) are essential to evaluate performance in practical remote face authentication, we further build a dataset containing videos of facial reflection frames collected by our system, which is the most comprehensive and largest one in both aspects compared with other public datasets.





To sum up, the main contributions of this work include:

∙ A simple, fast yet effective face anti-spoofing method is proposed, which is practical in real scenarios without the requirement on specific hardware design. 

∙ A cost-free depth recover net is proposed to estimate the facial depth maps via the normal cues extracted from the reflection frames for liveness classification.



∙ A novel light CAPTCHA checking mechanism is proposed to significantly improve the security against the attacks, especially the modality spoofing.


∙ A dataset containing comprehensive spoof attacks on various imaging qualities and mobile ends is built.





§ RELATED WORK




Local Texture based Methods
The majority of common presentation attacks are the recaptured images shown on printed photo and screens, in which the textures are different from the original images and can be leveraged to counter face spoofing.
For example, 


<cit.> adopted image distortion information as countermeasure against spoofing.
<cit.> proposed Deep Local Binary Pattern (LBP) to extract LBP descriptors on convolutional feature map extracted by CNN.
<cit.> converted the face image from RGB color space to HSV-YCbCr space and extracted channel-wise SURF features <cit.> to classify liveness result.
However, since the above methods operate on 2D images, they still suffer from poor generalization ability to unseen attacks and complex lighting conditions, especially when RGB sensors have low resolution or quality.
In contrast, our method exploits 3D information (e.g., depth) via the reflection increments from RGB images, which makes our method more robust and accurate to various attacks.



Depth Sensor based Methods
It is well known that the 3D facial cues can be used to defeat 2D presentation attacks.

For example, <cit.> directly exploit depth sensors such as Kinect to recover depth map and evaluate the anti-spoofing effectiveness combined with texture features.
<cit.> introduced a light field camera to extract depth information from multiple refocused images took in one snapshot.
Moreover, iPhone X incorporates a structured-light sensor to recover accurate facial depth map, which obtains impressive anti-spoofing performance.
However, although iPhone X achieves high accuracy, there are two practical problems.
First, it uses an expensive 3D camera for accurate depth.
Second, its implementation details are missing.
In contrast, our method is not only hardware-free that has competitive results against 3D hardware via a cost-free depth recover net, but also easy to follow for re-implementation.


























Depth Estimated from Single Image
<cit.> firstly attempted to recover a sparse 3D facial structure from RGB image for face anti-spoofing.
<cit.> proposed a two-steam depth-based CNN to estimate both texture and depth.
Recently, <cit.> fused multiple sequential depth predictions to regress to a temporal rPPG signal for liveness classification.
However, 3D reconstruction from a single image is still highly under-constrained, since these methods suffer from missing solid 3D information clue.
As a result, their anti-spoofing classifiers are hard to generalize to unseen spoof attacks, and is also sensitive against the quality of RGB camera.
To address the inaccurate depth issue, our method first obtains normal cues based on the light reflection, which better removes the effects of albedo and illuminance.
Then we train a compact encoder-decoder network to accurately recover the depth map. 

Lambertian Reflection based Methods

<cit.> firstly pointed out the importance of Lambertian modeling for face anti-spoofing, while only obtains rough approximations of illuminance and reflectance parts.

<cit.> also adopted Lambertian reflection model to extract simple statistics (i.e., standard deviation and mean) as features, but achieves limited performance.
Our method differs from the above methods in three key aspects:
(1) We actively perform light reflection via an extra light source specified by random light parameter sequence, while the above methods do NOT.
(2) We construct a regression branch to achieve the novel light CAPTCHA checking mechanism to make the system more robust, while the above methods again lack such scheme.
(3) We incorporate deep networks to learn powerful features, while the above methods use simple handcrafted features.














§ THE PROPOSED METHOD






Fig. <ref> illustrates the entire process of our method.
Specifically, we first set a smart phone (or any other devices) with front camera and light source (e.g., the screen)
in front of the subject.

Then, a random parameter sequence (i.e., light CAPTCHA) of light hues and intensities is generated, i.e., r={(α_i, β_i)}_i=1^n given n frames.
We manipulate the screen to cast dynamic light specified by the light CAPTCHA r.




After the reflection frames F_r are captured, we sequentially estimate the normal cues N, 

which are the input of a multi-task CNN to predict liveness label and regress the estimated light CAPTCHA r̂. The final judgement is been made from both of the predicted label and the matching result between r̂ and r.














 §.§ Normal Cues from Light Reflection



Given the facial reflection frames {F_r_i}_i=1^n,

we extract normal cues by estimating the reflection increments on subject's face.
Since objects with rough surfaces are diffuse reflectors (e.g. human face), light casted onto surface point is scattered and reflected, and then perceived as the final imaging in the camera.
Given images containing reflection on the object surface, we measure the magnitude variations among different images, under the assumption of Lambertian reflection model and weak perspective camera projection.

Lambert's Law regards the reflected part to be equal on all directions on the diffuse surface.
In other words, for any pixel point 𝐩 of the camera image under specific casting light L_r, its intensity F_r(𝐩) is formulated as:

    F_r(𝐩) = ρ_p(k_a+k_r𝐥·𝐧_p),

where k_a is the ambient weight, k_r is the diffuse weight, 𝐥 is the light source direction, ρ_p is the albedo and 𝐧_p is the point normal.
When light changes suddenly, k_a and 𝐥 (position of the screen) are not supposed to change temporally and can be regarded as constants. We adopt affine transformation to align 𝐩' and 𝐩 between image pairs, with transformation matrix estimated from the facial landmarks detected by PRNet <cit.>. Then in another image under casting light L_r', the intensity of the registered pixel 𝐩' is:


    F_r'(𝐩) = F_r'(𝐩') = ρ_p'(k_a+k_r'𝐥·𝐧_p').

We then attain the scalar product N_Δ r(𝐩) on each point, 

    N_Δ r(𝐩) = 𝐥·𝐧_p = F_r(𝐩) - F_r'(𝐩)/k_r - k_r',

where the scalar map arranged by N_Δ r(𝐩) is the normal cue. 





 §.§ Multi-task CNN


After obtaining the normal cues, we adopt a multi-task CNN that has two branches to achieve liveness classification and light CAPTCHA regression, respectively.
It should be noted that our multi-task structure is task-driven, which enables double checking mechanism to improve the robustness on modality spoofing in practical scenarios.


Liveness Classification.
Depending on the lighting environment, the normal cues extracted from facial reflection frames may be rough and noisy.
To efficiently obtain accurate depth information from the normal cues, we adopt an encoder-decoder network, which balances the performance and speed.
The network architecture is inspired by <cit.>, in which we use the inverted residual block <cit.>.



The recovered depth map is then sent to a simple classification structure to distinguish the real 3D face from those 2D presentation attacks.
The detailed structure is shown in Fig. <ref>.


After obtaining m frames of normal cues N_1, N_2,...,N_m of one video, the classifier has the following loss function:




    ℒ_cls =    1/m∑_i=1^m{ -(1-λ_depth)(c_ilog(𝒞(𝒰(𝒮(N_i)))) 
        + (1-c_i)log(1-𝒞(𝒰(𝒮(N_i))))) 
        + λ_depth∑_𝐩∈ℤ^2 - k log
              (
    
              e^d_k(𝐩) /
                (
                ∑_k'=1^256 e^d_k'(𝐩)
                )
              ) },

where 𝒮 denotes stem operation that contains two convolutional layers, 𝒞 denotes the depth liveness prediction net, 𝒰 denotes the depth recover net, c_i is the liveness label of the i-th normal cue, λ_depth is the weight of the depth estimation loss. In depth recovering part, we adopt 2D pixel-wise soft-max over the predicted depth map combined with the cross-entropy loss function, where k: Ω→1, ..., 256 is the ground truth depth label, d_k(𝐩) is the feature map activation on channel k at the pixel position 𝐩, while the feature map activation 𝒰(N_i) is the output of the depth recover net.









  
Light Parameter Regression.

 
 We reinforce the security of our method against modality spoofing, which is further discussed in Sec. <ref>, by customizing the casted light CAPTCHA and exploit a regression branch to decode it back for double checking automatically.

By feeding the same normal cues as the classification branch, the regression net has the loss function L_reg as:

    ℒ_reg =    1/m∑_i=1^m{ℛ(𝒮(N_i)) - Δ r_i^2},

where ℛ denotes the regression net, Δ r_i is the ground truth light parameter residual of reflection frames F_r_i and F_r_i-1.

Supposing there are V videos in the training set, the entire loss function of our multi-task CNN is formulated as:

    ℒ(Θ) = min_Θ1/2V∑_v=1^V{ℒ_cls^v + λ_regℒ_reg^v },




where Θ denotes the parameter set, λ_reg is the weight of CAPTCHA regression loss.
In practice, we set the light CAPTCHA sequence to be composed by 4 types of light in random order, which balances the robustness of CAPTCHA checking and time complexity. 


We set the rate of light changing identical to the frame rate, thus the frames hold different light reflection.
The length of F, r equals to m+1.
The Signal-to-Noise Ratio (SNR) is adopted to check if the estimated light parameter sequence matches the ground truth sequence.























































 §.§ Dataset Collection


As claimed in Sec. <ref>, various imaging qualities and the types of PAIs are very important for practical remote face authentication.
To address this need, we collect a new dataset, in which each data sample is obtained by casting dynamic light sequence onto the subject, and then record the 30-fps videos. 
Some statistics of the subjects are shown in Fig. <ref>.
Note that we mainly collect 2D attacks, the main target in most prior anti-spoofing methods <cit.>, rather than 3D ones because the cost to produce and conduct 3D attacks in real scenarios is much higher than 2D attacks.

Compared to the previous public datasets <cit.>, our dataset has three advantages:
1) Our dataset is the largest one that includes 12,000 live and spoof videos, with average duration to be 3s, collected from 200 subjects, compared to 4,620 videos from 165 subjects in <cit.>.
2) Our dataset uses the most extensive devices (i.e., 50 in ours vs. 4 in <cit.>) to obtain good simulation of real-world mobile verification scenarios.
3) Our dataset contains the most comprehensive attacks that include various print, replay, modality and another spoof face by light projector (see Fig. <ref>). 


We divide samples into 3 parts through the spoof types: paper attack, screen attack and other complex attacks consisting of cropped paper photos, projection attacks, etc.
In each part, the data is further divided into train set, develop set and test set, as shown in Tab. <ref>.
Moreover, the amounts of live data and spoof data stay equal in our dataset.
The live data is collected under multiple variations including interference illumination on face, noisy imaging quality and different poses. 
The spoof data are collected through abundant PAIs. 









§ EXPERIMENTS






 §.§ Implementation Details





















  
Model Training

We use Pytorch to implement our method and initialize all convolutional and fully-connected layers with normal weight distribution <cit.>.
For optimization solver, we adopt RMSprop <cit.> during training process.
Training our network roughly takes 5 hours using a single NVIDIA Tesla P100 GPU and iterates for 300 epochs. 



  
Evaluation Criteria


We use common criteria to evaluate the anti-spoofing performance, including False Rejection Rate (FRR), False Acceptance Rate (FAR) and Half Total Error Rate (HTER), which depends on the threshold value τ_cls.
To be specific, FRR and FAR are monotonic increasing and decreasing functions of τ_cls, respectively.
A more strict classification criterion corresponds to a larger threshold of τ_cls, which means spoof faces are less likely to be misclassified.
For certain data set 𝕋 and τ_cls, HTER is defined by



    HTER(τ_cls,𝕋)=FRR(τ_cls,𝕋)+FAR(τ_cls,𝕋)/2∈(0,1).

Lower HTER means better average performance on liveness classification, and HTER reaches its minimum when FAR=FRR, which is defined as Equal Error Rate (EER).
































 §.§ Ablation Study



  
Effectiveness of Depth Supervision
First, we conduct experiments to demonstrate the effects of the depth supervision.
To be specific, we monotonically increase the weight of depth loss λ_depth in Eq. <ref> and train multiple models, respectively.
Under each λ_depth, we train 10 different models, and then evaluate the mean and standard variance of EER and HTER, as shown in Tab. <ref>.
When λ_depth=0, the normal cues are directly used for liveness classification, which achieves the worst results.
As we increase the λ_depth to give more importance on the auxiliary depth supervision, the performance improves gradually, which verifies its effectiveness to helps denoise the normal cues and enhance the 3D information.



  
Light CAPTCHA Regression Branch
Although our system can well handle most of the normal 2D presentation attacks via depth information, it may still suffer from one special spoofing attack named modality spoofing, which directly forges the desired reflection patterns.


Specifically, modality spoofing will fail our classification net when meeting 2 requirement:
1) The formerly captured raw video consists of facial reflection frames, that contains the true reflection patterns, is leaked and replayed by Hi-res screen.
2) Within capture process of attack trial, the casted light doesn't interfere with the original facial reflection in video frames. 
Fig. <ref> illustrates the principle of our light CAPTCHA against the modality spoofing.

We further conduct experiments to prove the effectiveness of our light CAPTCHA checking mechanism in Fig. <ref>.
The |SNR| results of our regression branch are all below 0.35 and close with the ground truth CAPTHCHA, which demonstrates its ability to distinguish 4 types of casting light. 

Next, we quantitatively verify the effectiveness of our method against the modality spoofing attacks.
We set the light CAPTCHA formed as the compositions of 4 types of casting light in a random order for every checking trial.
To perform modality spoofing, we record videos consisting the same 4 types of true facial reflection (i.e., F_r_T1 in Fig. <ref>), and repeatedly replay the video for 3,000 times. 
In other words, the fixed video loop must match the randomly generated CAPTHCHA to bypass our system. 
The experiment results in Tab. <ref> show that the light CAPTCHA checking mechanism highly improves the security on modality spoofing. 















 §.§ Comparison to State-of-the-Art Methods





  
Depth Map Estimation

Next, we conduct comparisons on depth recovery against the recent state-of-the-art method <cit.>, as shown in Fig. <ref>.
We see that our method can recover more accurate depth map on various aspects, such as pose, facial contour and organ details, which demonstrate the effects to recover depth from solid depth clue instead of RGB texture.

It should also be noted that our method achieves comparable results to the Intel 3D sensor that can absolutely detect 2D presentation attacks without failure cases.




  
Face Anti-Spoofing
Here, we conduct comparisons on anti-spoofing, in which our method and several state-of-the-art methods are trained on our dataset (i.e., all the 3 training sets in each part), and then tested on public and our datasets, respectively.
After training, we determine the threshold τ_cls via the EER on the develop set and evaluate the HTER on the test set.
First, we conduct test on our dataset. 
Tab. <ref> shows that our method significantly outperforms the prior methods, where Aux Depth <cit.> ranks 2nd, while the conventional texture based methods <cit.> achieve relatively lower performance.

Next, we conduct tests on two public datasets: Replay-Attack <cit.> and CASIA <cit.>.
To better show the effectiveness and generalization of our method, NO additional fine-tuning is performed.
Since our method requires casting extra light onto the subjects, the only way to test the live subjects is to let the real person involved in the public dataset to be presented, which is impossible and unable us to measure FRR on public dataset.

For the spoof samples in these two public datasets, we print or broadcast the videos to act as the negative subjects and evaluate the FAR of various methods in Tab. <ref>.
The results again demonstrate the effectiveness and generalization of our method compared to the state-of-the-art methods.





  
Model Generalization 


Robust generalization ability is a key characteristic for face anti-spoofing in real scenarios. 
Here, we conduct generalization comparisons with a state-of-the-art local texture based method <cit.>, including live subject identity, device sensor interoperability and types of presentation attack.



Specifically, we train both models from training set of each part in our dataset, and evaluate the performance in all three test sets respectively.
Each model only learns one type of spoof attack with partial subject identities as well as device sensors, and the results are shown in Fig. <ref>. 
We can conclude that:
1) Curves (a),(e),(i) show that both models can achieve ideal FRR and FAR when training and evaluating are performed on the same attack.
2) Curves (b),(d) show that when testing on unseen attacks, the EER of <cit.> degrades to 0.10 while our method is still below 0.04 with degradation of only 0.01.
3) Harder negatives from complex dataset lead to worse performance, where the EER of <cit.> in (i) rises from 0.03 to 0.22 in (h), while our method only goes up to 0.13.
4) In the other cases, our method retains half of the degradation on FAR and FRR, which demonstrates its robust generalization ability.











  
Running Time Comparison
We compare cross-platform inference time with several SOTA methods.
We deploy and compare on 3 common platform architectures: GPU for cloud server, CPU (x86) for some embedded chips and CPU (arm) for smart phones, as shown in Fig. <ref>.

As we can see, our efficiency on mobile platform still meets the application requirement and even outperforms some methods on CPU (x86).
The results indicate that our method achieves real-time efficiency and is portable for cross-platform computation requirements along with state-of-the-art anti-spoofing performance.




 §.§ Comparison to Hardware-based Method



Finally, we compare our method with Structured-Light 3D (SL3D), which relies on the hardware that is embedded with structured-light 3D reconstruction algorithm. 
To be specific, we use an Intel^ RealSense SR300 3D camera to obtain the facial depth map and adopt the same CNN classifier as in our multi-task network for SL3D.
In contrast, our method only utilizes the ordinary RGB camera without specific hardware design.







To comprehensively compare our method with SL3D, we only select and perform the hardest presentation attacks that could cause failure cases of 2D texture based methods.


The results in Tab. <ref> indicate that our method can achieve comparable anti-spoofing performance compared to SL3D.







§ CONCLUSION

In this paper, an effective facial anti-spoofing method named Aurora Guard is proposed, which holds real-time cross-platform applicability.
The key novelty of our method is to leverage two kinds of auxiliary information, the depth map and the light CAPTCHA based on light reflection, which significantly improve the accuracy and reliability of anti-spoofing system against unlimited 2D presentation attacks.
Extensive experiments on public benchmark and our dataset show that AG is superior to the state of the art methods.



named



