Robust Minutiae Extractor: Integrating Deep Networks and Fingerprint Domain Knowledge
Dinh-Luan Nguyen Kai Cao Anil K. Jain
cs.CV







Robust Minutiae Extractor:
 Integrating Deep Networks and Fingerprint Domain Knowledge
    Dinh-Luan Nguyen, Kai Cao and Anil K. Jain

Michigan State University

East Lansing, Michigan, USA

nguye590@msu.edu, {kaicao,jain}@cse.msu.edu






    December 30, 2023
=========================================================================================================================================================

empty



	We propose a fully automatic minutiae extractor, called MinutiaeNet, based on deep neural networks with compact feature representation for fast comparison of minutiae sets. Specifically, first a network, called CoarseNet, estimates the minutiae score map and minutiae orientation based on convolutional neural network and fingerprint domain knowledge (enhanced image, orientation field, and segmentation map). Subsequently, another network, called FineNet, refines the candidate minutiae locations based on score map. We demonstrate the effectiveness of using the fingerprint domain knowledge together with the deep networks. Experimental results on both latent (NIST SD27) and plain (FVC 2004) public domain fingerprint datasets provide comprehensive empirical support for the merits of our method. Further, our method finds minutiae sets that are better in terms of precision and recall in comparison with state-of-the-art on these two datasets. Given the lack of annotated fingerprint datasets with minutiae ground truth, the proposed approach to robust minutiae detection will be useful to train network-based fingerprint matching algorithms as well as for evaluating fingerprint individuality at scale. MinutiaeNet is implemented in Tensorflow: https://github.com/luannd/MinutiaeNethttps://github.com/luannd/MinutiaeNet 




§ INTRODUCTION

Automatic fingerprint recognition is one of the most widely studied topic in biometrics over the past 50 years <cit.>. One of the main challenges in fingerprint recognition is to increase the recognition accuracy, especially for latent fingerprints. Fingerprint comparison is primarily based on minutiae set comparison <cit.>. A number of hand-crafted approaches <cit.> have been used to augment the minutiae with their attributes to improve the recognition accuracy. However, robust automatic fingerprint minutiae extraction, particularly for noisy fingerprint images, continues to be a bottleneck in fingerprint recognition systems.





With rapid developments and success of deep learning techniques in a variety of applications in computer vision and pattern recognition <cit.>, we are beginning to see network-based approaches being proposed for fingerprint recognition. Still, the prevailing methods of minutiae extraction primarily utilize fingerprint domain knowledge and handcrafted features. Typically, minutiae extraction and matching involves pre-processing stages such as ridge extraction and ridge thinning, followed by minutiae extraction <cit.> and finally heuristics to define minutiae attributes. While such an approach works well for good quality fingerprint images, it provides inaccurate minutiae location and orientation for poor quality rolled/plain prints and, particularly for latent fingerprints. To overcome the noise in fingerprint images, Yoon <cit.> used Gabor filtering to calculate the reliability of extracted minutiae. Although this approach can work better than <cit.>, it also resulted in poor results with highly noisy images. Because these prevailing approaches are based on handcrafted methods or heuristics, they are only able to extract basic (or low level) features[Features such as edges, corners, etc.] of images. We believe learning based approaches using deep networks will have better ability to extract high level features[Abstract/semantic features retrieved from deep layers.] from low quality fingerprint images.

In this paper, we present a novel framework that exploits useful domain knowledge coded in the deep neural networks to overcome limitations of existing approaches to minutiae extraction. Figure <ref> visualizes results of the proposed framework on two latent fingerprints from the NIST SD27 dataset. 

Specifically, our proposed approach comprises of two networks, called CoarseNet and FineNet:

- CoarseNet is a residual learning <cit.> based convolutional neural network that takes a fingerprint image as initial input, and the corresponding enhanced image, segmentation map, and orientation field (computed by the early stages of CoarseNet) as secondary input to generate the minutiae score map. The minutiae orientation is also estimated by comparing with the fingerprint orientation.

- FineNet is a robust inception-resnet <cit.> based minutiae classifier. It processes each candidate patch, a square region whose center is the candidate minutiae point, to refine the minutiae score map and approximate minutiae orientation by regression. Final minutiae are the classification results.

Deep learning approach has been used by other researchers for minutiae extraction (see Table <ref>). But, our approach differs from published methods in the way we encode fingerprint domain knowledge in deep learning. Sankaran  <cit.> classified the minutiae and non-minutiae patches by using sparse autoencoders. Jiang  <cit.> introduced a combination of two networks: JudgeNet for classifying minutiae patches, and LocateNet for locating precise minutiae location. While Jiang use neural networks, their approach is very time-consuming due to use of sliding window to extract minutiae candidates. Another limitation of this approach is that it does not provide minutiae orientation information.



Tang  <cit.> utilized the idea of object detection to detect candidate minutiae patches, but it suffers from two major weaknesses: (i) hard threshold to delete the candidate patches, and (ii) the same network is used for both candidate generation and classification. By using sliding windows, Darlow  <cit.> fed each pixel of the input fingerprint to a convolutional neural network, called MENet, to classify whether it corresponds to a minutia or not. It also suffers from time-consuming sliding windows as in <cit.>, and separate modules for minutiae location and orientation estimates. Tang  <cit.> proposed FingerNet that maps traditional minutiae extraction pipeline including orientation estimation, segmentation, enhancement, and extraction to a network with fixed weights. Although this approach is promising because it combines domain knowledge and deep network, it still uses plain [A series of stacked layers.] network architecture and hard threshold in non-maximum suppression [A post-processing algorithm that merges all detections belonging to the same object.]. Finally, the accuracy of FingerNet depends largely on the quality of the enhanced and segmentation stage while ignoring texture information in the ridge pattern.

In summary, the published approaches suffer from using sliding windows to process each pixel in input images, setting hard threshold in post-processing step, and using plain convolutional neural network to classify candidate regions. Furthermore, the evaluation process in these studies is not consistent in terms of defining “correct” minutiae.

The contributions of our approach are as follows:




  * A network-based automatic minutiae extractor utilizing domain knowledge is proposed to provide reliable minutiae location and orientation without a hard threshold or fine tuning.



  * A robust patch based minutiae classifier that significantly boosts the precision and recall of candidate patches. This can be used as a robust minutiae extractor with compact embedding of minutiae features.



  * A non-maximum suppression is proposed to get precise locations for candidate patches. Experimental evaluations on FVC 2004 <cit.> and NIST SD27 <cit.> show that the proposed approach is superior to published approaches in terms of precision, recall, and F1 score values.






§ PROPOSED FRAMEWORK


Our minutiae extraction framework has two modules: (i) residual learning based convolutional neural network, called CoarseNet that generates candidate patches containing minutiae from input fingerprint image; (ii) inception-resnet based network architecture, called FineNet which is a strong minutiae classifier that classifies the candidate patches output by CoarseNet. These two networks also provide minutiae location and orientation information as outputs. Figure <ref> describes the complete network architecture for automatic minutiae location and orientation for an input fingerprint image. Section <ref> presents the architecture of CoarseNet. In Section <ref>, we introduce FineNet with details on training to make it a strong classifier.





 §.§ CoarseNet for minutiae extraction


We adopt the idea of combining domain knowledge and deep representation of neural networks in <cit.> to boost the minutiae detection accuracy. In essence, we utilize the automatically extracted segmentation map, enhanced image, and orientation map as complementary information to the input fingerprint image. The goal of CoarseNet is not to produce the segmentation map or enhanced image or orientation map. They are just the byproducts of the network. However, these byproducts as fingerprint domain knowledge must be reliable to get robust minutiae score map. Because Tang  <cit.> proposed an end-to-end unified network that maps handcrafted features to network based architecture, we use this as a baseline for our CoarseNet.



  §.§.§ Segmentation and orientation feature sharing


Adding more layers in the deep network with the hope of increasing accuracy might lead to the exploding or vanishing gradients problem. From the success of residual learning <cit.>, we use residual instead of just plain stacked convolutional layers in our network to make it more powerful. Figure <ref> shows the detailed architecture of the network. 

Unlike existing works using plain convolutional neural network <cit.> or sliding window <cit.> to process each patch with fixed size and stride, we use a deeper residual learning based network with more pooling layers to scale down the region patch. Specifically, we get the output after the 2^nd, 3^rd, and 4^th pooling layer to feed to an ASPP network <cit.> with corresponding rates for multiscale segmentation. This ensures the output has the same size as input without a loss of  information when upsampling the score map.

By using four pooling layers, each pixel in the j^th feature map, called level j, corresponds to a region 2^j × 2^j in the original input. Result layers at level 4 and 3 will be tested as coarse estimates while the level 2 serves as fine estimation.

Segmentation map.
Image segmentation and fingerprint orientation estimation share the same convolutional layers. Thus, by applying multi-level approach mentioned above, we get probability maps of each level-corresponding region in input image. For instance, to get finer-detailed segmentation for each region level j_l, we continue to process probability map of region level j_l/2.

Orientation map.
To get complete minutiae information from context, we adopt the fusion idea of Cao  <cit.>. We fuse the results of Dictionary-based method <cit.> with our orientation results from CoarseNet. Because <cit.> uses a hand-crafted approach, we set the fusion weight ratio of its output with our network-based approach as 1:3.




  §.§.§ Candidate generation

The input fingerprint image might contain large amounts of noise. So, without using domain knowledge we may not be able to identify prominent fingerprint features. The domain knowledge comprises of four things: raw input image, enhanced image, orientation map, and segmentation map. In the Gabor image enhancement module, we take the average of filtered image and the orientation map for ridge flow estimation. To emphasize texture information in fingerprints, we stack the original input image with the output of enhancement module to obtain the final enhancement map. To remove spurious noises, we apply segmentation map on the enhancement map and use it as input to coarse minutiae extractor module.



To obtain the precise location of minutiae, each level of residual net is fused to get the final minutiae score map with size h/16 × w/16, where h and w are the height and width of the input image. Figure <ref> shows the details of processing score map. To reduce the processing time, we use score map at level 4 as a coarse location. To get precise location, lower level score maps are used.




  §.§.§ Non-maximum suppression


Using non-maximum suppression to reduce the number of candidates is common in object detection <cit.>. Some of the candidate regions are deleted to get a reliable minutiae score map by setting a hard threshold <cit.> or using heuristics <cit.>. However, a hard threshold can also suppress valid minutiae locations. A commonly used  heuristics is to sort the candidate scores in ascending order. The L^2 distance between pairwise candidates is calculated with hard thresholds for distance and orientation. By iteratively comparing each candidate with the rest in the candidate list, only the candidate with higher score and score above the thresholds is kept. However, this approach fails when two minutiae are near each other and the inter-minutiae distance is below the hard thresholds.

Since each score in the minutiae map corresponds to a specific region in the input image, we propose to use the intersection over union strategy. Specifically, after sorting the scores of the candidate list, we keep high score candidates while ignoring the lower scoring candidates with at least 50% overlap with the candidates already selected.




  §.§.§ Training data for CoarseNet


Given the lack of datasets with ground truth, we use the approach in Tang  <cit.> to generate weak labels for training the segmentation and orientation module. The coarse minutiae extractor module uses minutiae location and minutiae orientation ground truth provided in the two datasets. We also use data augmentation techniques as mentioned in Section <ref>.



 §.§ FineNet


Extracting minutiae based on candidate patches is not adequate. Although CoarseNet is reliable, it still fails to detect true minutiae or detects spurious minutiae. This can lead to poor performance in fingerprint matching. This motivates our use of FineNet - a minutiae classifier from generated candidate patches. FineNet takes candidates from the output of CoarseNet as input to decide whether the region 10 × 10 in the center of the corresponding patch has a valid minutia or not.




  §.§.§ FineNet architecture


Figure <ref> describes the architecture of FineNet. As mentioned in Section <ref>, we use the Inception-Resnet v1 architecture as a core network in FineNet. 





For FineNet training, we extract an equal number of t_1 × t_1 sized minutiae and non-minutiae patches with t_1 = 45. FineNet determines the whether the 10 × 10 pixel region in the center of each patch contains a valid minutia or not. The candidate patches are resized into t_2 × t_2 pixels that feed to FineNet. Based on the observation that the original input image size (without rescaling) is not large in comparison with images for object classification, too much up scaling the image can cause blurring the tiny details, and too small an input image size is not sufficient for network with complex architecture, we choose t_2 = 160 pixels.

Training data for FineNet is extracted from the input gray scale images where minutiae data are based on the ground truth minutiae location and non-minutiae ones are from random sampling with the center regions do not contain partial or fully minutiae location. To make the network more robust, we use some small techniques like Batch Normalization <cit.>, rotation, flipping, scaled augmentation <cit.>, and bounding blurring <cit.> as pre-processing stage. 




  §.§.§ Losses and implementation details

Intra-Inter class losses.
Because input captured fingerprint image is not always in the ideal condition, it might be easily affected by distortion, finger movement, or quality (wet/dry) of finger. Thus, the variation of minutiae shapes and surrounding ridges can affect the accuracy of the classifier. To handle this situation and make the FineNet more robust with intra-class variations, we use Center Loss <cit.> as a complementary of softmax loss and minutiae orientation loss. While softmax loss tends to push away features of different classes, center loss tends to pull features in the same class closer. Let ℒ, ℒ_𝒞, ℒ_𝒮, ℒ_𝒪 be the total loss, center loss, sofmax loss, and orientation loss, the total loss for training is calculated as follows:



    ℒ = αℒ_𝒞 + (1-α)ℒ_𝒮 + βℒ_𝒪

where we set α = 0.5 to balance between intra-class (center) and inter-class (sofmax) loss and β = 2 to emphasize the importance of precision of minutiae orientation.

Parameter settings.
As mentioned in Section <ref>, fingerprint patches are input to FineNet where the patch size is 160 × 160. To ensure our network can handle distortion in input image, we apply scaled augmentation <cit.>, random cropping, and brightness adjustment. Horizontal and vertical flip with pixel mean subtraction are also adopted. We randomly initialize all variables by using a Gaussian distribution N(0, 0.01). Batch size is set to 100. We use schedule learning rate after particular iterations. Specifically, we set it as 0.01 in the beginning and reduce 10 times after 50K iterations. To prevent vanishing gradient problem, we set the maximum epoch to 200K. We use 0.0004 as the value for momentum and weight decay is 0.9.




§ EXPERIMENTAL RESULTS


We evaluate our method on two different datasets with different characteristics under different settings of parameters 𝒟 and 𝒪 (see Eq. (<ref>)). We also visualize examples of score maps with correct and incorrect minutiae extractions in Figure <ref>. All experiments were implemented in Tensorflow and ran on Nvidia GTX GeForce. 




 §.§ Datasets

We use FVC 2002 dataset <cit.> with data augmentation consisting of 3,200 plain fingerprint images for training. To compensate for the lack of a large scale dataset for training, we distort the input images in x and y coordinates in the spirit of hard-training with non-ideal input fingerprint images. Furthermore, we also apply additive random noise to the input images. Thus, for training CoarseNet, we have an augmented dataset of 8,000 images. To obtain data for training FineNet, we extract 45 × 45 pixel patches from these 8K training images for CoarseNet whose center is a ground truth minutia point. For non-minutiae patches, we randomly extract patches with the criteria that the 10 × 10 center of each patch does not contain any minutia. Thus, we collect around 100K minutia and non-minutia patches for training FineNet.

As mentioned in Section <ref>, we use FingerNet <cit.> to generate labels for domain knowledge groundtruth. Besides, we manually correct segmentation grountruth results from FingerNet to ensure better learning for CoarseNet.




 §.§ Evaluation


To demonstrate the robustness of our framework, we compare our results with published approaches on FVC 2004 <cit.>[We obtained the groundtruth from <cit.>] and NIST SD27 <cit.> datasets under different criteria of distance and orientation thresholds. Let the tuples (l_p,o_p) and (l_gt,o_gt) be the location coordinates and orientation values of predicted and ground truth minutia. The predicted minutia is called true if it satisfies the following constrains:



    ‖𝐥_𝐩-𝐥_𝐠𝐭‖_2 ≤𝒟
    ‖𝐨_𝐩-𝐨_𝐠𝐭‖_1 ≤𝒪


where 𝒟 and 𝒪 are the thresholds in pixels and degrees, respectively.
Specifically, we set the range of distances between detected and ground truth minutiaes from 8 to 16 pixels (in location) and 10 to 30 degree (in orientation) with default threshold value (0.5). We choose these settings to demonstrate the robust and precise results from the proposed approach while published works degrade rather quickly.


	
Table <ref> shows the precision and recall comparisons of different approaches to minutiae extraction. MINDTCT <cit.> is the open source NIST Biometric Image Software. VeriFinger <cit.> is a commercial SDK for minutiae extraction and matching. Since Gao  <cit.> did not release their code in public domain, we report their results on NIST SD27 and FVC 2004 database from <cit.>. Darlow  <cit.> use only a subset of the FVC dataset for training and the rest for testing, we do not include in our evaluation.

Table <ref> shows that the proposed method outperforms state-of-the-art techniques under all settings of parameters (thresholds) 𝒟 and 𝒪 for both FVC 2004 and NIST SD27. Our results also reveal that by using only rolled/plain fingerprint images for training, our framework can work pretty well for detecting minutiae in latents.



Table <ref> shows a comparison between using and not using our proposed non-maximum suppression method on the NIST SD27 dataset with setting 3 in Table <ref>. Because non-maximum suppression is a post processing step, it helps improve precision, recall and F1 values.

To make a complete comparison (at all the operating points) with published methods, we present the precision-recall curves in Figure <ref>. The proposed approach surpasses all published works on both FVC 2004 and NIST SD27 datasets.

Figure <ref> shows the minutiae extraction results on both FVC 2004 and NIST SD27 datasets with different quality images. Our framework works well in difficult situations such as noisy background or dry fingerprints. However, there are some cases where the proposed framework either misses the true minutiae or extracts spurious minutiae. For the FVC 2004 dataset and rolled fingerprints from NIST SD27 dataset, we obtain results that are close to the ground truth minutiae. However, some minutiae points are wrongly detected (image a) because of the discontinuity of ridges or missed detections (image c) because the location of minutiae is near the fingerprint edge. For the latent fingerprints from NIST SD27 dataset, besides the correctly extracted minutiae, the proposed method is sensitive to severe background noise (image e) and poor latent fingerprint quality (image g). The run time per image is around 1.5 seconds for NIST SD27 and 1.2 seconds for FVC 2004 on Nvidia GTX GeForce.





§ CONCLUSIONS




We have presented two network architectures for automatic and robust fingerprint minutiae extraction that fuse fingerprint domain knowledge and deep network representation:

- CoarseNet: an automatic robust minutiae extractor that provides candidate minutiae location and orientation without a hard threshold or fine tuning.

- FineNet: a strong patch based classifier that accelerates the reliability of candidates from CoarseNet to get final results.

A non-maximum suppression is proposed as a post processing step to boost the performance of the whole framework. We also reveal the impact of residual learning on minutiae extraction in the latent fingerprint dataset despite using only plain fingerprint images for training. Our experimental results show that the proposed framework is robust and achieves superior performance in terms of precision, recall and F1 values over published state-of-the-art on both benchmark datasets, namely FVC 2004 and NIST SD27.

The proposed framework can be further improved by (i) using larger training set for network training that includes latent images, (ii) constructing context descriptor to exploit the region surrounding minutiae, (iii) improving processing time, and (iv) unifying minutiae extractor into an end-to-end fingerprint matching framework.

ieee



