title: Fingerprint Feature Extraction by Combining Texture, Minutiae, and Frequency Spectrum Using Multi-Task CNN
authors: Ai Takahashi Yoshinori Koda Koichi Ito Takafumi Aoki
text: 
  
  
  
  
  
  
  Fingerprint Feature Extraction by Combining Texture, Minutiae, and Frequency Spectrum Using Multi-Task CNN
      Ai Takahashi^1, Yoshinori Koda^1,2, Koichi Ito^1, and Takafumi Aoki^1
  
  1 Graduate School of Information Sciences, Tohoku University, Japan.
  
  2 Biometrics Research Laboratories, NEC Corporation, Japan.
  
  ai@aoki.ecei.tohoku.ac.jp
  
      December 30, 2023
  ========================================================================================================================================================================================================================================
  
  empty
  
  
  
  Although most fingerprint matching methods utilize minutia points and/or texture of fingerprint images as fingerprint features, the frequency spectrum is also a useful feature since a fingerprint is composed of ridge patterns with its inherent frequency band.
  We propose a novel CNN-based method for extracting fingerprint features from texture, minutiae, and frequency spectrum.
  In order to extract effective texture features from local regions around the minutiae, the minutia attention module is introduced to the proposed method.
  We also propose new data augmentation methods, which takes into account the characteristics of fingerprint images to increase the number of images during training since we use only a public dataset in training, which includes a few fingerprint classes.
  Through a set of experiments using FVC2004 DB1 and DB2, we demonstrated that the proposed method exhibits the efficient performance on fingerprint verification compared with a commercial fingerprint matching software and the conventional method.
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  A fingerprint has the property of high permanence and high discrimination in biometric features that have been used in many biometric authentication systems, such as user authentication for mobile devices, mobile payments, and national IDs <cit.>.
  In general, fingerprint matching is based on the positions and angles of the feature points, called minutiae, and the relative positions between the feature points <cit.>.
  
  Fingerprint recognition using minutiae has two problems: (i) the recognition accuracy depends on the extraction accuracy of the minutiae, and (ii) the length of features depends on the number of minutiae.
  For example, in a scratchy or dry fingerprint, the break in the ridges is extracted as some minutiae.
  In wet or crushed fingerprints, the number of extracted minutiae is small since the ridges are collapsed.
  In order to achieve highly accurate recognition of such a fingerprint image, it is necessary to combine multiple features that can be extracted from the fingerprint image.
  
  In addition, the fingerprint image may be captured in a different area at the time of registration and authentication since the fingerprint image is acquired by placing the finger on the scanner.
  It is difficult to apply template protection to prevent leakage of biometric features since the number of extracted minutiae is different at the time of acquisition.
  
  In order to address the above problems, fingerprint matching methods based on deep learning have been proposed <cit.>.
  Li et al. <cit.> used Fully Convolutional Network (FCN) to extract feature based on the texture of fingerprint images.
  Engelsma et al. <cit.> proposed a Convolutional Neural Network (CNN) to extract texture feature of the whole fingerprint and minutia feature of the fingerprint.
  It is possible to extract fixed-length fingerprint feature using CNN that are independent of the number of minutiae.
  On the other hand, fingerprint matching methods using deep learning used a large-scale private fingerprint image dataset in training, and its effectiveness and reproducibility are not always guaranteed.
  
  In this paper, we propose a new CNN-based method for extracting fingerprint feature to achieve high accuracy fingerprint recognition with training using only a public dataset.
  The proposed method employs frequency spectrum as a new feature in addition to the texture and minutiae used in other fingerprint matching methods.
  We introduce the minutia attention module in order to extract effective texture feature from local regions around the minutiae.
  We propose new data augmentation methods, which takes into account the characteristics of fingerprint images to increase the number of images during training.
  We train our CNN model using 9,000 fingerprint images (1,000 classes) of IIIT-D Multi-sensor Optical and Latent Fingerprint (MOLF) <cit.> and show that the proposed method is more accurate than commercial fingerprint authentication software, VeriFinger, and <cit.> through experiments using FVC2004 <cit.>.
  
  The contributions of this paper are summarized below.
  
  
    * We propose a novel CNN architecture to extract feature from fingerprint images by combining texture, minutiae and frequency spectrum.
  
    * We propose a minutia attention module to pay attention to the position of the minutiae to extract global and local texture features.
  
    * We propose a new data augmentation method specific to fingerprint images.
  
    * We demonstrate that the recognition accuracy is higher than that of commercial software, VeriFinger SDK, and the conventional method by using only a public dataset in training.
  
  
  
  
  § RELATED WORK
  
  
  Fingerprint recognition is classified into two methods: one is based on local features, i.e., minutiae, and the other is based on global features, i.e., texture.
  
  A commonly used local features of a fingerprint is the minutiae, which are the end points and bifurcation points derived from fingerprint ridge patterns.
  The position, angle and relative relation of minutiae are used to compute the matching score.
  The minutia-based method is widely used in fingerprint recognition, for example, VeriFinger SDK <cit.> is available as a commercial software and NIST Biometric Image Software (NBIS) <cit.> as a free software.
  Although the minutiae are highly discriminative features, there is a problem that the recognition accuracy is decreased if the minutiae are not extracted correctly.
  Most of the methods using global features such as textures are based on image-based template matching <cit.>.
  Among them, the fingerprint matching method using frequency features of images has been proposed for matching fingerprint images from which minutiae cannot be extracted, such as dry or allergic skin <cit.>.
  Ito et al. <cit.> proposed a method called Band-Limited Phase-Only Correlation (BLPOC), which focuses on the phase information of the image.
  BLPOC has improved the accuracy of matching by utilizing the fact that the texture pattern of a fingerprint has an inherent frequency band and the energy is concentrated at that frequency band.
  Ito et al. <cit.> also showed that fingerprint recognition can be made more accurate by combining it with a minutia-based method.
  On the other hand, there is a problem that the recognition accuracy is decreased due to the rotation and nonlinear deformation between fingerprint images since only the translation between images can be considered.
  
  Recently, fingerprint recognition methods based on CNN have been proposed to address the above problems.
  Tang et al. <cit.> have proposed FingerNet, which can extract minutiae from low-quality fingerprint images with high accuracy by incorporating Gabor filter-based fingerprint image enhancement into CNNs.
  Through experiments using the NIST SD27 <cit.>, which is a latent fingerprint image dataset, they demonstrated that FingerNet exhibited higher accuracy of minutiae extraction than the commercial software.
  Nguyen et al. <cit.> proposed MinutiaeNet, which is a combination of CoarseNet to detect minutiae in the whole fingerprint image and FineNet to estimate the existence probability of minutiae.
  Li et al. <cit.> proposed a fingerprint feature extractor that does not need to be aligned and is robust to rotation and translation by training FCN on a local region centered on the minutiae.
  Engelsma et al. <cit.> proposed a multi-task learning method for texture feature extraction and minutiae detection to extract fingerprint features from textures and minutiae.
  Since the fingerprint features extracted using CNN are of fixed length, encryption can be applied to protect biometric information.
  On the other hand, most of CNN-based methods used a large-scale private fingerprint image dataset in training, and their effectiveness and reproducibility are not always guaranteed.
  
  
  
  § METHOD
  
  
  The CNN-based fingerprint recognition methods proposed so far use minutiae, textures, or a combination of both <cit.>.
  To improve recognition accuracy, we consider using frequency spectrum as a feature in addition to minutiae and texture.
  Since the ridge patterns of fingerprints have their inherent frequency band <cit.>, it is possible to match fingerprint images that cannot extract minutiae by focusing on the frequency band.
  The proposed method consists of three CNNs to extract three features: texture, minutiae, and frequency spectrum, which are then concatenated to form the fingerprint feature.
  We define a loss function for each feature and train it with multi-task learning, and introduce metric learning to improve the recognition accuracy.
  In order to train efficiently with fewer images, we introduce new data augmentation methods, which are specialized for fingerprint images.
  
  The network architecture of the proposed fingerprint feature extraction method is shown in Fig. <ref>.
  First, we use Spatial Transformer Networks (STN) <cit.> to align the rotation of the fingerprint image.
  Next, we extract texture-based, minutia-based, and frequency-based features from the fingerprint image using CNNs based on the Residual Network (ResNet) <cit.>.
  The three features are then concatenated to form the fingerprint feature, which is used for matching.
  The details of the proposed method are described below.
  
  
  
  
  
   §.§ Rotation Alignment Using STN
  
  
  Before processing with the proposed method, the input fingerprint image is enhanced to increase the recognition accuracy.
  In this paper, we employ the fingerprint enhancement method based on the intensity gradient, where we use the implementation of FastEnhanceTexture <cit.>.
  In the first step of the proposed method, we align the rotation of fingerprint images, which causes a decrease in accuracy in feature extraction.
  In this paper, we use STN <cit.> for end-to-end learning of CNNs.
  The STN estimates the transformation parameters in the localization network and transforms the image based on the estimated parameters.
  Fig. <ref> shows the architecture of the localization network used in the proposed method.
  Although it is possible to correct the translation and rotation using the STN, we have confirmed experimentally that the correction of the rotation angle alone is sufficient.
  As a result, the number of parameters to be estimated is reduced and the training is stabilized.
  
  
  
  
  
   §.§ Feature Extraction Using CNNs
  
  
  Frequency, texture and minutia features are extracted from the rotation-corrected fingerprint image using ResNet <cit.>.
  The frequency features are extracted using CNN as shown in Fig. <ref>, where ResBlock is the same as the one used in ResNet <cit.>.
  The input is two channels of the real and imaginary parts of the frequency spectrum obtained by Discrete Fourier Transform (DFT) of the fingerprint image.
  In addition, the following preprocessing is applied to extract the frequency features that represent the features of the fingerprint image.
  The DC component is much larger than the other frequency components and represents the gain inherent to the sensor.
  In order to reduce the effect of DC component, we apply normalization so that the average of the pixel values is zero, and then perform DFT.
  Since the energy of the fingerprint image is concentrated in an ellipse of the low-frequency band, the high-frequency region contains only perturbations such as noise and aliasing.
  In order to consider only the inherent frequency band of fingerprints similar to the BLPOC <cit.>, only the region containing elliptical frequency band is extracted as an input.
  The CNN network architecture for extracting texture and minutiae features is shown in Fig. <ref>.
  Note that the CNNs that extract texture and minutia features share weights up to the middle.
  The gray-scale fingerprint image is input to the CNN as one channel.
  In order to extract minutia features using CNN, we introduce a minutia map <cit.> which can represent the positions and angles of minutiae.
  The minutia map is obtained with a minutiae map generator as shown in Fig. <ref>, which is branched from a minutia feature extractor.
  
  
  
  
  
  
  
   §.§ Minutia Attention Module
  
  
  Texture features are extracted from the entire image through training of the CNN to be effective for fingerprint recognition.
  On the other hand, Li et al. <cit.> demonstrated that using texture features in the local area around the minutiae is more accurate than using features extracted from the entire fingerprint image.
  Based on this idea, we extract highly discriminative texture feature by using the location information of minutiae for texture feature extraction.
  In the proposed method, we employ a Minutia Attention Module (MAM) inspired by Spatially Attentive Output Layer (SAOL) <cit.> for extracting texture feature that takes into account local regions around the minutiae.
  Instead of the Global Average Pooling (GAP) Layer in the texture feature extractor, MAM is used with the minutia map as an attention mask to extract features that aggregate texture information around the minutiae.
  Fig. <ref> shows the architecture of MAM used in the proposed method.
  
  Let X^l ∈ℝ^C^l × H^l × W^l be the feature map output from the l-th layer of the texture feature extractor, where H^l, W^l, and C^l are the height, width, and the number of channels, respectively, of the feature map output from the l-th layer (1 ≤ l ≤ L).
  When using the GAP layer and Fully-Connected (FC) layer, the texture feature t_ tex is given by
  
      t_ tex =  GAP(X^L)^⊺W^ FC,
  
  where GAP(X^L) ∈ℝ^C^L × 1 indicates the spatially aggregated feature vector by GAP, W^ FC∈ℝ^C^L × K indicates the weight matrix of the output FC layer, (·)^⊺ indicates the transposition, C^L=1,024, and K=512.
  On the other hand, when using MAM instead of GAP, the texture feature with attention, t^ att_ tex, is given by
  
      t^ att_ tex
          =  MA(X^L)^⊺W^FC,
  
  where MA(X^L) ∈ℝ^C' × K is defined by
  
      MA_c'(X^L) = ∑_i,j A_ij (Y_c')_ij,
  
  for each c', where c' indicates the class index (1 ≤ c' ≤ C'), A∈ [0,1]^H^L × W^L indicates the attention mask, Y∈ [0,1]^C' × H^L × W^L indicates the spatial logits, which is obtained by applying softmax to X^L, and (Y_c')_ij indicates the (i,j)-th element of c'-th feature map of Y_c'.
  Note that the size of W^FC in Eq. (<ref>) is changed to ℝ^C' × K since the training is performed as 1,000-class classification problem, i.e., C'=1,000 in this paper.
  
  
  
  
  
  
  
   §.§ Loss Function and Deep Metric Learning
  
  
  This section describes the loss functions to be trained in the proposed method and deep metric learning to improve the accuracy of fingerprint recognition.
  
  We use the MSE loss for estimating the minutia map.
  Let the ground-truth and estimated minutia maps be H_g and H_e, respectively.
  The loss function for minutia map estimation is defined by
  
      L_ map = ∑_i,j,kρ{H_g (i,j,k) - H_e (i,j,k) }^2,
  
  where (i,j,k) indicates (x,y) coordinates and k channel, respectively, and ρ is a constant (ρ = 100 in this paper).
  The loss functions for texture, minutia, and frequency features are defined by the cross-entropy loss and are denoted by L_t, L_m, and L_f, respectively.
  The loss function of the proposed method L_ all is given by
  
      L_ all = L_t + L_m + L_f + λ_ map L_ map,
  
  where λ_ map is the weight for L_ map.
  Note that the weight parameters for STN are also optimized so as to minimize L_ all.
  
  The proposed method employs deep metric learning in training to improve the accuracy of fingerprint recognition.
  In this paper, we use AdaCos <cit.>, which has been demonstrated to be effective in face recognition.
  The use of AdaCos makes it possible to increase the cosine similarity of the genuine pairs and to decrease that of the impostor pairs during training.
  Let the class weight consisting of the center feature vector for each class be W.
  The probability P_i,y_i, which the i-th input image is classified into the class label y_i of W is given by
  
      P_i,y_i =
          exp(s̃_d^(t)·cosθ_i,y_i)/exp(s̃_d^(t)·cosθ_i,y_i)+∑_k ≠ y_iexp(s̃_d^(t)·cosθ_i,k),
  
  where cosθ_i,y_i indicates the cosine similarity between the i-th input feature and the corresponding class label y_i of W, s̃_d^(t) indicates the scaling parameter, k indicates the class label of W except for i, and t indicate the number of updates.
  In the case of using AdaCos, the final loss function L is given by
  
      L = - 1/N∑_i=1^Nlog P_i,y_i.
  
  
  Applying AdaCos to the proposed method, L_t, L_m, and L_f are calculated by Eq. (<ref>).
  
  
  
   §.§ Data Augmentation
  
  
  In this paper, data extension is important since the training is performed only on a public dataset that is not large.
  The general data augmentation methods are random contrast and random noise.
  In the proposed method, we introduce random deformation and random morphology as new data augmentation methods specialized for fingerprint images.
  During the acquisition process, nonlinear deformation may be added to the fingerprint image and the ridge pattern of the fingerprint may be collapsed.
  In order to represent deformation of fingers, the fingerprint image is deformed according to the fingerprint deformation model <cit.> by random deformation.
  In order to represent collapse of ridge patterns, the morphological filters of dilation and erosion are applied to a part of the fingerprint image.
  Note that the size of the filter for random morphology is 0.02 ∼ 0.2% of the image size according to random erasing <cit.>.
  Fig. <ref> shows examples of fingerprint images with data augmentation.
  
  
  
  
  
   §.§ Matching Score
  
  
  The matching score is calculated between fingerprint features extracted by the proposed method.
  Let the fingerprint features extracted two fingerprint images be t_1 and t_2, respectively.
  The matching score S is calculated by
  
      S = t_1^⊺·t_2.
  
  
  
  
  § EXPERIMENTS AND DISCUSSION
  
  
  This section describe the experiments of evaluating the proposed method and discuss with their results.
  
  
  
   §.§ Training
  
  
  The proposed CNN model is trained as follows.
  We use 12,000 fingerprint images (DB1, DB2, and DB3) out of 19,200 fingerprint images in MOLF in the experiments.
  Note that we do not use DB4 and DB5 consisting of latent fingerprint images since we only use the scanned fingerprint images in the experiments.
  The 12,000 fingerprint images are divided into 9,000 (1,000 fingers × 3 images × 3 DBs) for training and 3,000 (1,000 fingers × 1 image × 3 DBs) for validation.
  We perform multi-task learning for identifying 1,000-class labels and detecting minutiae.
  RMSProp is used as an optimizer and the weight decay for preventing overfitting is set to 10^-5.
  The learning rates for feature extractors and STN are set to 10^-3 and 5^-4, respectively.
  Fingerprint images are resized to 256 × 256 pixels as an input.
  The size of the minutia map is 128 × 128 pixels with 6 channels.
  The effective frequency band is set to 50% of the size of frequency spectrum.
  The weight for the minutia map is set to λ_ map=10.
  The probability of random noise, random contrast, random deformation, and random morphology is set to 80%, 80%, 50%, and 50%, respectively.
  We use the minutia information detected by VeriFinger SDK 10.0 <cit.> as the ground truth of the training data.
  
  
  
   §.§ Experimental Condition
  
  
  The performance of the proposed method is evaluated using the public fingerprint image dataset, FVC2004 <cit.> DB1 and DB2.
  DB1 and DB2 consist of 800 fingerprint images (100 fingers × 8 images).
  Fig. <ref> shows the example of fingerprint images from the genuine pairs in FVC2004 DB1 and DB2.
  DB1 has many low-quality fingerprint images that contain little overlap in the fingerprint region, nonlinear deformation in the same subjects, collapsed ridges due to wetness, or interrupted ridges due to drying.
  The DB2 has a lot of collapsed ridges, which makes it difficult to detect minutiae in many fingerprint images.
  We evaluate the matching scores of 2,800 genuine pairs and 368,500 impostor pairs, which are all the combinations of the dataset.
  The verification accuracy is evaluated by Equal Error Rate (EER), where is defined as the error rate where False Acceptance Rate (FAR) and False Rejection Rate (FRR) are equal.
  
  
  
  
  
  
  
   §.§ Ablation Study
  
  
  The effectiveness of refinement techniques introduced in the proposed method, i.e., frequency feature, deep metric learning, data augmentation, and MAM, is evaluated by the following ablation study using FVC2004 DB1 and DB2.
  In this experiment, we compare the verification accuracy of the proposed method with Verifinger SDK 10.0 <cit.> and the deep learning-based method <cit.> to demonstrate the effectiveness of the proposed method.
  Note that we replaced Inception modules used in <cit.> to ResNet modules used in the proposed method since the number of classes in the training data used in this experiment is quite small compared with the original implementation of <cit.>, where 382,914 classes are included in the private training data.
  Table <ref> shows the summary of experimental results.
  In FVC2004 DB1, the EERs of the proposed method have been improved by adding refinement techniques.
  The EER of the proposed method (D), which employs all the refinement techniques, is lower than that of VeriFinger and Engelsma <cit.>.
  In DB2, the accuracy of the proposed method has been also improved by adding refinement techniques.
  The proposed method (C) is more accurate than VeriFinger and <cit.>.
  On the other hand, the proposed method (D) with MAM had more errors than the proposed method (C) without MAM.
  This is because the fingerprint image in FVC2004 DB2 has many areas with collapsed ridges and blurred minutiae as shown in Fig. <ref>, and therefore, it may not be possible to apply proper attention to them.
  
  
  
   §.§ Neonate Fingerprint Recognition
  
  
  We evaluate the accuracy of the proposed method in neonate fingerprint recognition, which is an example of fingerprint images that are extremely difficult to extract minutiae from.
  In this experiment, we use the neonate fingerprint image dataset used in <cit.>.
  This dataset contains neonate fingerprint images taken with a 1,920-ppi sensor at 2, 6, and 18 hours of age.
  The number of genuine and impostor pairs is 664 and 9,632, respectively.
  The EERs of VeriFinger, the proposed method (C), and (D) are 42.6%, 34.9%, and 38.4%, respectively.
  In VeriFinger, wrinkles and pores are extracted as minutiae as shown in Fig. <ref>.
  The EER of VeriFinger is high due to the use of many wrong minutiae for fingerprint matching.
  On the other hand, the proposed method is more accurate than VeriFinger SDK in the neonate fingerprint recognition since the proposed method uses texture and frequency features in addition to minutia feature for matching.
  Minutiae extracted from neonate fingerprint images using the proposed method are shown in Fig. <ref>.
  This figure shows that the proposed method was able to extract relatively correct minutiae, however, the accuracy of the proposed method (C) with MAM was reduced due to the small number of minutiae.
  
  
  
  
  
   §.§ Qualitative Analysis
  
  
  We present qualitative evaluation of the minutia extraction using the proposed method.
  Fig. <ref> shows the minutiae extracted using MINDTCT <cit.>, MinutiaeNet <cit.> , VeriFinger, and the proposed method (D) for fingerprint images with quality values of 96 and 45 in VeriFinger.
  For the fingerprint image with a quality value of 96, the same minutiae were extracted by all the methods.
  
  In the case of the fingerprint image with a quality value of 45, there are many false positives for MINDTCT, and there are many undetected for MinutiaeNet, while the proposed method (D) is able to extract the majority of minutiae extracted by VeriFinger.
  
  The effectiveness of the proposed method is verified using a saliency map that visualizes the effect of each pixel in the input image on the extracted features.
  We use Guided-back propagation <cit.> to make a saliency map from CNN weights.
  Fig. <ref> shows the saliency maps of texture and minutia features.
  While CNN for extracting minutia feature focuses on the entire fingerprint image, CNN for extracting texture feature focuses on the core region.
  CNN with MAM focus not only on the core region, but also on the region around minutiae.
  Fig. <ref> shows the saliency maps of frequency feature.
  CNN for extracting frequency feature focuses on the frequency bands inherent in the fingerprint pattern and we confirmed that the individual frequency features are extracted.
  
  
  
  
  
  
  
  
  
  
  
  
  § CONCLUSION
  
  
  We proposed a novel CNN architecture to extract features from fingerprint images by combining texture, minutiae and frequency spectrum.
  A minutia attention module to pay attention to the position of the minutiae was proposed.
  We also introduced novel data augmentation methods specific to fingerprint images for efficient training with a small number of classes of training data.
  Through a set of experiments using FVC2004 DB1 and DB2, we demonstrated that the matching accuracy is higher than that of VeriFinger and the conventional method.
  In future work, we will investigate the use of local frequency features and the use of relationships between local regions to recognize low quality fingerprint images from which only a few minutiae are extracted.
  
  ieee
  
  
  
  