Enhancing Privacy in Face Analytics Using Fully Homomorphic Encryption
Bharat Yalavarthi Arjun Ramesh Kaushik Arun Ross Vishnu Boddeti Nalini Ratha


empty
empty

plain
Heavy Dark Matter in White Dwarfs: Multiple-Scattering Capture and Thermalization
    [
    April 24, 2024
=================================================================================

fancy 





Modern face recognition systems utilize deep neural networks to extract salient features from a face. These features denote embeddings in latent space and are often stored as templates in a face recognition system. These embeddings are susceptible to data leakage and, in some cases, can even be used to reconstruct the original face image. To prevent compromising identities, template protection schemes are commonly employed. However, these schemes may still not prevent the leakage of soft biometric information such as age, gender and race. To alleviate this issue, we propose a novel technique that combines Fully Homomorphic Encryption (FHE) with an existing template protection scheme known as PolyProtect. We show that the embeddings can be compressed and encrypted using FHE and transformed into a secure PolyProtect template using polynomial transformation, for additional protection. We demonstrate the efficacy of the proposed approach through extensive experiments on multiple datasets. Our proposed approach ensures irreversibility and unlinkability, effectively preventing the leakage of soft biometric attributes from face embeddings without compromising recognition accuracy. 








§ INTRODUCTION




Face recognition entails the extraction of features from face images and comparing them to either validate a claimed identity (“verification") or determine an identity (“identification") <cit.>. Recent advancements in deep neural networks and AI have resulted in the development of powerful face recognition systems <cit.> that can be deployed in a wide range of applications such as personalized services, law enforcement, border security, and smartphone access <cit.>. However, this development has also raised questions about privacy accorded to subjects and the security of the templates (such as embeddings) stored in a face recognition application <cit.>. Even as ethical concerns attendant to the technology are being rightfully discussed in public forums, it is necessary for the technology itself, on the one hand, and its users, on the other hand, to embrace measures that can enhance privacy and security while mitigating potential biases <cit.>. Otherwise, the technology runs the risk of being overwhelmed by restrictive legislation <cit.> that can stifle the benefits of this technology in solving egregious crimes <cit.>. 










To address some of the privacy challenges associated with face embeddings stored as templates, we propose an approach in this paper that employs a polynomial transformation on homomorphically encrypted face embeddings. Using Fully Homomorphic Encryption (FHE) in our proposed method ensures that the face recognition result is only disclosed to authorized parties with the secret key for homomorphic encryption, and the face embeddings themselves are secured through encryption during the recognition process. We show through our experiments that using FHE also prevents leakage of soft biometrics (e.g., age, gender/sex, race/ethnicity) from face embeddings.

(It is necessary to point out that there is a difference between race and ethnicity, as well as sex and gender. In this paper, however, we use these terms interchangeably).


PolyProtect <cit.>, a template protection scheme, transforms face embeddings into more secure templates using multivariate polynomials with user-specific parameters. Our research demonstrates a symbiotic relationship between FHE and PolyProtect in ensuring optimal security measures. The synergy between these two techniques is essential, as each contributes unique strengths that, when combined, establish a robust security framework. According to the inversion attack analysis conducted by PolyProtect <cit.>, the risk of reversibility is notably high (30 - 99%) when an attacker possesses multiple (more than 4) templates of the same face. Even with the compromise of a single template, there is a 95% likelihood of reversibility when an overlap of 4 or greater is employed. However, PolyProtect introduces a tradeoff between accuracy and security, wherein increasing the overlap parameter enhances accuracy in face recognition and analysis tasks but concurrently diminishes template security. To address this challenge, we implement a strategy of encrypting the face embeddings using FHE and then applying the PolyProtect template with high overlap. This proactive step ensures that even under the full disclosure threat model, irreversibility, unlinkability and prevention of soft biometrics leakage are ensured without compromising identification performance.

Conversely, the integration of PolyProtect into our approach serves as a countermeasure against threats targeting compromised FHE systems, such as secret key leaks and passive attacks outlined in <cit.>. The identified risks pertain to FHE systems engaged in machine learning computations, such as mean and variance calculations. By combining both FHE and PolyProtect techniques (Fig. <ref>), our approach offers a more comprehensive and resilient privacy solution compared to relying on either method in isolation.































§ THREAT MODEL

We use the term facial analytics to refer to the process of deducing semantic information from a face image. This could include sensitive information such as age, gender, ethnicity, and health <cit.> – sometimes known as soft biometrics. The possibility of deducing soft biometric cues from face images or their embeddings using automated techniques is a source of concern. These automated techniques can be machine learning models such as SVMs or deep neural networks (DNNs). For example, a face image or its embedding can be “stolen" by hackers and various soft biometrics can be derived from them thereby revealing sensitive information.

In our work, we presume that the face embedding is provided in an encrypted form. Our goal is to ensure that the encrypted embedding does not reveal any soft biometric information to unauthorized users. Note that the threat remains unchanged even if the parameters of the models used for extracting soft biometric information (e.g., weights of a DNN) are encrypted. 


We consider the most challenging threat model according to ISO/IEC 30316, which is the full disclosure model, where the attacker possesses complete knowledge of the PolyProtect method <cit.>, including its algorithm, number of embedding elements (m), user-specific parameters (e.g., C, overlap, and E), and one or more PolyProtected templates corresponding to a face embedding. In addition, we assume that the public key used in FHE is available but not the private key. If the embeddings are not encrypted, the hacker can infer soft biometric information from the PolyProtect template as we show in Table <ref>. 





§ FHE BASICS

Encryption is the process where plain-text data is encrypted into ciphertext using a secret key and a cryptographic algorithm. Only authorized entities with a private key can decrypt the ciphertext back to the plaintext. Encryption is essential for protecting sensitive data from unauthorized access or modification. 
Homomorphic Encryption (HE) is a cryptographic system that permits certain computations to be performed on encrypted data without requiring decryption <cit.>. In this system, we have public (pk) and secret (sk) keys, encryption (E) and decryption (D) mechanisms, and plaintext values x and y. When x and y are encrypted as x' = E(x, pk) and y' = E(y, pk), respectively, a cryptosystem is considered homomorphic with respect to a chosen operator (e.g., addition or multiplication), denoted as ∘, if we can find another operator ∙ such that x ∘ y = D(x' ∙ y', sk). This means that we can conduct operations on encrypted data and obtain the same result when decrypting using the private secret key.

Specifically, given c_i = E(x_i, pk), i = 1, 2, ⋯, K, an FHE scheme allows the computation of c = g(c_1, c_2, ⋯, c_K) such that D(c, sk) = f(x_1, x_2, ⋯, x_K) for any arbitrary function f. 

It is essential to note that there are three types of homomorphic encryption schemes <cit.>: (a) Partial Homomorphic Encryption (PHE) permits addition or multiplication operations. (b) Somewhat Homomorphic Encryption (SHE) allows limited computations on ciphertexts. (c) Fully Homomorphic Encryption (FHE) enables computations on ciphertexts of any depth and complexity.









Numerous FHE systems have been introduced, including the BFV, BGV, and CKKS schemes <cit.>. The BFV and BGV schemes enable vector operations involving integers, while the CKKS scheme facilitates floating-point operations. These schemes achieve Single Instruction Multiple Data (SIMD) operations by bundling plaintext values into an array and then encrypting them to get ciphertext. In this work we use the HEAAN <cit.> library based on the CKKS scheme for FHE computations.





§ RELATED WORK

A notable body of literature explores privacy enhancements to soft biometrics at both the image and embedding (template) levels in face recognition. PFRNet <cit.> uses an Autoencoder framework to disentangle identity from attribute information to suppress gender information in face embeddings. Similarly, SensitiveNets <cit.> uses a privacy-preserving neural network that suppresses soft biometrics attributes. The approach adopts an adversarial regularizer, which incorporates a sensitive information removal function into the learning objective. The Multi Incremental Variable Elimination (Multi-IVE) method <cit.> works by eliminating those feature variables in embeddings that predict soft biometric attributes. Increasing the number of eliminations was shown to decrease the soft biometrics leakage but significantly affected the identification performance. In <cit.>, the authors introduce an adversarial attack approach designed to protect gender information in facial images. The method involves perturbing the image to minimize the estimated mutual information between the feature distribution acquired from a face recognition network and the gender variable. This method reduces gender leakage (prediction accuracy) by an average of 91% to 80% across three datasets. Reversible Attribute Privacy Preservation (RAPP) <cit.> uses a stream cipher to determine the sensitive attributes that have to be concealed with a user-defined password; it supports recovering the original attributes. It also uses an attribute adversarial network to generate perturbed images that conceal various attributes while retaining the utility of face verification. However, the identification performance is negatively impacted. The authors work around this challenge by reducing the number of features being concealed and the intensity of concealment. PrivacyNet <cit.> is another technique to impart soft biometric privacy to face images while preserving recognition capabilities via image perturbation using a GAN-based Semi-Adversarial Network (SAN). PrivacyNet also allows a person to choose the specific facial attributes to be obfuscated while allowing the other attributes to be extracted. One of the drawbacks of this image perturbation technique is that it sometimes does not generate realistic images and cannot conceal soft biometric features from a human observer.

Although current approaches mitigate the leakage of soft biometric attributes, they do not suppress it to the level of a “random guess". In our work, we show that through the use of FHE, we can restrict this leakage in face embeddings to a level that is equal to or lower than that of a random guess. In <cit.>, the authors show the susceptibility of privacy enhancement techniques such as homomorphic encryption. We address this susceptibility by employing a template protection scheme in addition to homomorphic encryption. Further, FHE encryption offers a stricter theoretical guarantee than existing methods for the security of soft biometrics.

FHE has been used in prior work for securing face recognition. Boddeti <cit.> proposed encrypting the face embeddings and performing face matching in the FHE domain. Batching and dimensionality reduction techniques are also explored to balance face-matching accuracy and computational complexity. In <cit.>, the authors introduce an efficient approach for searching encrypted probe images against a large gallery, using fixed-length representations. In <cit.>, the authors proposed a time-efficient and space-efficient face matching in the FHE domain for securing face templates. Our approach stands out from previous work by integrating a template protection scheme, a compression technique, and FHE to enhance the security of face templates. Moreover, we conducted thorough experiments to assess their efficacy in mitigating the leakage of soft biometrics.  









§ TEMPLATE PROTECTION


Amongst the many existing protection templates, we have adopted PolyProtect <cit.> to showcase the benefits of our work. Let V = [v_1, v_2, ..., v_n] denote an n-dimensional real-number face embedding. PolyProtect maps V to another real-numer feature vector, P = [p_1, p_2, ..., p_k] (where k<n). P is the PolyProtected template of V. m (where m<<n) consecutive elements from V are mapped to single elements in P via a polynomial equation of coefficients, C = [c_1, c_2, ..., c_m] and exponents, E = [e_1, e_2, ..., e_m]. C and E are user-defined, non-zero, and distinct for each user of the face recognition system. The first m elements of V are mapped to p_1 as :

    p_1    = c_1v_1^e_1 + c_2v_2^e_2 + ... + c_mv_m^e_m


Another important user-defined parameter is overlap, which defines the number of common elements from V used in successive values in P. When overlap = 0, the elements of V in each set are unique. The minimum and maximum values for overlap are 0 and m-1, respectively. The mapping for p_2 for overlaps 0 and m-1, respectively, are as follows:

    p_2    = c_1v_m+1^e_1 + c_2v_m+2^e_2 + ... + c_mv_m+m^e_m
    
         p_2    = c_1v_2^e_1 + c_2v_3^e_2 + ... + c_mv_m+1^e_m


The authors of PolyProtect <cit.> have also performed an extensive survey on a number of existing Biometric Template Protection (BTP) methodologies and evaluated them based on recognition accuracy, irreversibility, and unlinkability <cit.>. According to the survey, none of the existing BTP methodologies before PolyProtect <cit.> satisfy all three criteria - recognition accuracy, irreversibility, and unlinkability.



§ ABLATION STUDY






 §.§ Variation of user-defined parameters (C, m, overlap)

We conducted experiments with various user parameters in PolyProtect to investigate their impact on the leakage of soft biometrics. As overlap increased from 0 to 3, there was a notable increase in age leakage, particularly for values overlap>0. However, gender and ethnicity showed consistent levels of leakage across different values of overlap (Fig. <ref>(a)).  As the overlap increases, the amount of embedding information retained after the PolyProtect transformation also increases, potentially resulting in a higher risk of soft biometric leakage. 
In PolyProtect, the parameter m dictates the number of terms in the polynomial. At m=6, we observed maximum leakage for age, whereas for gender and ethnicity, maximum leakage occurred at m=7. Conversely, selecting m=5 minimized leakage across all three soft biometric attributes (Fig. <ref>(b)). Additionally, the parameter C, which defines the range of values for the polynomial coefficients [-C, C], was varied from 10 to 60 in our study. The leakage of ethnicity remained relatively stable across all tested values of C, while age exhibited an increase from C = 10 to 20, and gender showed a slight decrease from C = 45 to 60 (Fig. <ref>(c)).





 §.§ Summation of Ciphertext elements

Implementing PolyProtect, cosine similarity, and fully connected layers in FHE requires summing up the elements within a ciphertext. This is not straightforward as we cannot access individual elements of a ciphertext. The three approaches to efficiently achieve the summation are described below: 




  §.§.§ Naive Rotation

This is a brute-force method for summation within a ciphertext where expensive ciphertext rotations are performed N-1 times and the running sum is computed until all the elements are covered (Algorithm <ref>).  


















  §.§.§ Discrete Fourier Transform

When the Discrete Fourier Transform (DFT) of a signal is computed, the first value of DFT or the DC component will give the sum of the input signal values. We use this property to calculate the DFT of the ciphertext and get the sum of its values.     



  §.§.§ Fold and Add

This is a more efficient version of the naive rotation method described earlier, which can be visualized as iteratively folding the array into half and adding the corresponding folded parts log_2 N - 1 times as described in algorithm <ref>. Fig. <ref> shows that the Fold and Add method was the fastest among the three with a significant speedup than the naive rotation method, especially for large ciphertext sizes.






















 §.§ Polynomial approximation of inverse square root






Since computing cosine similarity requires inverse square root function, we use a polynomial approximation of the inverse square root function in the encrypted domain owing to the limitations of FHE in implementing non-linear functions. We restrict the input to the range (0, 1] to achieve a closer approximation. The performance of this approximation is measured through the relative error of 2000 random points in the range (0, 1]. We consider both 6-degree and 8-degree polynomials as a tradeoff between computational depth and accuracy, as in Fig. <ref>. 





§ PROPOSED APPROACH

The proposed methodology begins with the reception of encrypted face templates from users. These templates are generated through a pre-trained face recognition model and encrypted locally at the user's end using Fully Homomorphic Encryption (FHE), employing a private key exclusively known to the user. Subsequently, a template protection algorithm such as PolyProtect is applied to augment the security of the encrypted template, forming an additional layer of protection. The resulting encrypted PolyProtect template can be securely stored at the user's end and utilized as needed for tasks such as face identification or the extraction of soft biometrics, including gender, age, and ethnicity. The outcomes of these tasks are transmitted to the user in encrypted form and can only be decrypted using the user's private key.


 §.§ Dataset

We have performed our experiments on CelebSet <cit.> and Balanced Faces in the Wild (BFW) <cit.>. The statistics of these datasets have been detailed in Tables <ref>, <ref>. As the BFW dataset lacks age annotations for its face images, we utilized a pre-trained model trained on the CelebSet dataset to predict the ages of the BFW images. These predicted ages were then employed in our experiments.







 §.§ Embedding Compression

It is commonly believed that compressing embeddings can improve privacy leakage. We explore embedding compression through Matryoshka Representation Learning (MRL) <cit.> as a technique to improve privacy, in addition to template protection and encryption. MRL is an innovative approach that enhances representation learning by encoding information at various granularities within a single embedding. MRL achieves this by capturing details at different levels of abstraction, enabling the model to adapt to downstream tasks efficiently. The method focuses on learning coarse-to-fine representations, ensuring that high-level, generalized features are initially captured and progressively refined with finer details. MRL stands out by delivering representations that are comparable in accuracy and rich in information when compared to independently trained low-dimensional representations, making it a promising solution for efficient and effective representation learning.









In our research, we leveraged MRL to compress face embeddings obtained from FaceNet/AdaFace, which are originally of 512-dimension, down to 64-dimension. As depicted in Fig. <ref>, we noted a decrease in classification accuracies when the compressed dimensions fell below 64. It is important to note that, since HEAAN supports SIMD operations, compression does not aid in reducing computational depth. 



 §.§ Face Identification

Given a vector of face embeddings of size n, and user-defined parameters C, m, and overlap, we pad the embeddings vector such that it can be split into m small vectors V_i (where m<<n). Each m-sized vector is encrypted in FHE. We apply the PolyProtect algorithm as described in <cit.> on each V_i, resulting in vector P_i. Unlike in the PolyProtect algorithm <cit.>, we store every PolyProtect mapping as a vector of m elements which are the same, resulting from the sum obtained through Fold and Add (Algorithm <ref>). We have employed the Cosine similarity as a metric to calculate similarity scores in the 1:N search for identification. Cosine similarity in FHE is computed using the algorithm described in algorithm <ref>. 


  


 §.§ Age, Gender and Ethnicity Prediction




We perform our experiments using various combinations of embeddings, template protection, compression, and encryption. The embeddings are extracted through FaceNet <cit.> and AdaFace <cit.> models. Our experiments are primarily aimed at determining the identification accuracy and soft biometrics leakage for the described combinations to prove the efficacy of our proposed solution. Our networks (Fig. <ref>) have been tuned from the lens of an attacker to achieve maximum leakage of soft biometric features. For cases that require training with encrypted data in the FHE domain, we train an SVM classifier on the ASCII dump of the ciphertexts. 

Apart from a myriad of leakage metrics defined in <cit.> and <cit.>, we have chosen to define leakage through Suppression Rate (SR) and Privacy Gain (PG) <cit.>. Privacy Gain is defined as -

    PG = (1 - R_p) - (1-R_o)

where, R_o and R_p represent the recognition performances on the original data and the privacy-enhanced data, respectively. A positive value of PG signifies enhanced data protection. The ideal value for PG is 1. 

Suppression Rate measures the difference attribute prediction accuracy with and without privacy enhancement, A_p and A_o, respectively


    SR = (A_o - A_p)/A_o
















 §.§ Adaptation to FHE

Since the computations are performed in the FHE domain for face verification, gender, age, and ethnicity prediction there are some serious restrictions involved like lack of individual element access in the encrypted vector, high computational complexity, finite computational depth, and limited set of supported operations. 

To deal with the computational complexity, HEAAN offers standard mathematical operations like addition and multiplication in SIMD form. 
We implemented fully connected layers as a matrix multiplication between encrypted inputs and plaintext weights. For each output node, we multiply the weights with the input which is a ciphertext, and apply the Discrete Fourier transform (DFT) <cit.> to obtain the sum of the elements within a ciphertext as shown in Fig. <ref>. As we can not access individual elements within a ciphertext, it is not as straightforward as in plaintext to obtain the sum of elements within a ciphertext. DFT is used to obtain the sum efficiently because it uses fewer expensive ciphertext rotations than other alternatives.

Homomorphic encryption schemes and the HEAAN library do not directly support the comparison function. It is estimated using polynomial approximation. An asymptotically optimal comparison method named CompG was proposed by <cit.> by approximating a sign function using composite polynomial approximation. The approximate ReLU operator for FHE is implemented by 
using the CompG. For ReLU implementation, We employ the composite approximation technique to compare the input value a against 0. This comparison function produces a result of 1 if a is greater than b, 0 if a is less than b, and 0.5 if a is equal to b. To obtain the ReLU result, we multiply the output of the comparison function by the input value. 
CompG works only for input in the range of -1 to 1. To ensure that input to the ReLU falls within the specified range we normalize the output values from convolutional layers using a scaling factor before applying ReLU. This scaling factor is determined using the formula max(abs(maxValue), abs(minValue)), where abs(maxValue), abs(minValue) are the absolute maximum and minimum input values observed for the corresponding ReLU block on the train set after training is completed. The reciprocal of the scaling factor is applied after the ReLU to restore the original values of the positive inputs. 

Regular neural network operators like softmax or argmax are not feasible to implement in FHE without significant approximation. In the gender classification head of our network to obtain the classification output we need argmax to find the index of the largest logit value among the two logits. To replace argmax we used the CompG operator to find the largest value among these two logits. We are sending the result of this CompG comparison to the user. When the user decrypts, the larger value among the two can be identified giving the gender classification output.  

As operations in FHE are computationally time-consuming, it is important to parallelize the computations of efficiency. For our implementation used the parallelization mechanism provided in Number Theory Library (NTL). NTL is already used extensively by HEAAN for its operations.  





















   


























§ RESULTS

As evident from Tables <ref>, <ref>, <ref> our proposed approach prevents the leakage of soft biometrics from face embeddings with minimal loss in identification accuracy  (<2.5%), high Privacy Gain and high Suppression Rate. We can observe that our approach reduced the classification accuracies of soft biometric attributes to the level of random chance across the two datasets. We could also achieve an almost ideal Privacy Gain and Suppression Rate in certain scenarios, but we believe this could be because of the imbalanced nature of our datasets. Our experiments showcase that a combination of MRL, FHE, and PolyProtect in this order yields maximum protection against soft biometric leakage (Fig. <ref> and Fig. <ref>).

The consistent efficacy of our method across two different embeddings (FaceNet and AdaFace) and datasets (CelebSet and BFW) shows the ability of our method to generalize in different conditions. We also prove the ability of FHE to work seamlessly with template protection schemes and embedding compression techniques for preserving the privacy of face templates. 

Additionally, we find that utilizing a soft biometrics classifier network trained on plaintext data allows for seamless inference within the FHE domain, yielding the same predictive performance as that in the unencrypted domain (as depicted in Table <ref> in "None"). This underscores the feasibility of conducting not only identification but also soft biometric analysis in the secure FHE domain.





§ CONCLUSION

In this paper, we propose to use FHE in combination with template protection and compression to secure the face template and prevent soft biometric leakage. We show that soft biometric attributes from face embeddings can be strictly protected while preserving identification accuracy. In our approach, we compress the face embeddings using MRL (Matryoksha Representation Learning), encrypt them, and then apply PolyProtect as the template protection scheme. The identification performance of the encrypted template compared with the unencrypted version is unchanged. Since FHE guarantees are based on strong theoretical principles, privacy and security are ensured, and only authorized individuals with the secret key will be able to access the results from the FHE computation. 

This material is based upon work supported by the Center for Identification Technology Research and the National Science Foundation under Grant Nos. 1841517 and 1822190.










 











unsrt



