Finger-GAN: Generating Realistic Fingerprint Images Using Connectivity Imposed GAN
Shervin Minaee Amirali Abdolrashidi









Finger-GAN:
Generating
Realistic
Fingerprint Images
Using
Connectivity
Imposed
GAN
    Shervin Minaee^*, AmirAli Abdolrashidi^†  

^*New York University

 ^†University of California, Riverside
 


    

=================================================================================================================





Generating realistic biometric images has been an interesting and, at the same time, challenging problem. 
Classical statistical models fail to generate realistic-looking fingerprint images, as they are not powerful enough to capture the complicated texture representation in fingerprint images.
In this work, we present a machine learning framework based on generative adversarial networks (GAN), which is able to generate fingerprint images sampled from a prior distribution (learned from a set of training images).
We also add a suitable regularization term to the loss function, to impose the connectivity of generated fingerprint images. 
This is highly desirable for fingerprints, as the lines in each finger are usually connected. 
We apply this framework to two popular fingerprint databases, and generate images which look very realistic, and similar to the samples in those databases.
Through experimental results, we show that the generated fingerprint images have a good diversity, and are able to capture different parts of the prior distribution.
We also evaluate the Frechet Inception distance (FID) of our proposed model, and show that our model is able to achieve good quantitative performance in terms of this score.








§ INTRODUCTION


Fingerprint is arguably the most popular biometric used for identification and security. Simple to collect and rich in features, it provides decent benefits for an efficient recognition system. 
Although it is not as protected as features such as iris pattern <cit.>, fingerprint recognition is still among the most researched topics in biometrics along with iris <cit.> and face <cit.>, and is popularly used in forensics, airports and smartphones (such as in Iphone). All biometrics have unique features and patterns. Therefore, generating a synthetic fingerprint is a challenging effort. Our work seeks to accomplish this and measure the level of authenticity against real fingerprints.


Many approaches have been proposed for fingerprint recognition, which are based on matching minutiae features, such as ridge endings and bifurcation <cit.>.
There are also several approaches based on hand-crafted features extracted from fingerprint images <cit.>.
Also, with deep neural networks increasing in popularity in recent years <cit.>, end-to-end models have been also used for biometrics recognition  <cit.>. 






In this work, we focus on a different problem from fingerprint recognition, which is learning to generate fingerprint images.
Our work is mainly based on generative adversarial networks (GAN) <cit.>, which provides a powerful framework for learning to generate  samples from a given distribution. GAN is comprised of a generative network, which learns a training dataset's distribution and can generate new data (sampled from the same distribution), and a discriminative network, which tries to distinguish real samples from the ones generated with the generator model. 
On a high level, the generator and discriminator models are trained simultaneously, using a minimax type optimization. 
Since GAN's invention, it has been used for various computer vision and pattern generation tasks, such as image super-resolution, image to image translation, image generation based on text description, prediction of pedestrian walking patterns for autonomous vehicles, and iris generation <cit.>.

In this work, we propose a synthetic fingerprint generation framework based on deep convolutional generative adversarial networks (DC-GAN), which is able to generate realistic fingerprint images, which are hard to be distinguished from real ones. 
By far, to the best of our knowledge, no other work in literature has tackled fingerprint generation using GANs.
We add  a suitable term to the GAN loss function, imposing the generated fingerprint images to have connected lines. This is achieved by adding the total variation <cit.> of the generated images to the loss function. 
Figure <ref> provide 8 sample images generated by our proposed model.



The structure of the rest of this paper is as follows.
Section <ref> provides the details of our proposed framework, and the model architectures for both the generator, and discriminator networks.
In Section <ref>, we provide the experimental studies, and present the generated fingerprint images with our proposed framework. We show the generated fingerprint images over different epochs (for the sample input noise) to see how the generated images evolve over time, and finally the paper is concluded in Section <ref>.







§ THE PROPOSED FRAMEWORK


In this work we propose a generative model for fingerprint images based on deep convolutional generative adversarial networks, with a suitable regularization which suits fingerprint generation.
Let us first give more technical details on GAN, and then we dive into our proposed model.



 §.§ Generative Adversarial Network

To provide more details about how GANs work, the generator network in GANs learns a mapping from noise z (with a prior distribution) to a target distribution y, G= z → y.
The generator model, G, tries to generate samples which look similar to the real samples (provided during training), while the discriminator network, D, tries to distinguish the samples generated by the generator models from the real ones.
The general architecture of a GAN model is shown in Figure <ref>.


The generator and discriminator can be trained jointly, by minimizing a loss function.
The loss function of GAN can be written as:

    ℒ_GAN=  𝔼_x ∼ p_data(x)[log D(x)]+ 𝔼_z ∼ p_z(z)[log(1-D(G(z)))]

In this sense we can think of GAN, as a minimax game between D and G, where D is trying to minimize its classification error in distinguishing the fake samples from the real ones (maximize ℒ_GAN), and G is trying to maximize the discriminator network's error (minimize ℒ_GAN).
After training this model, the trained generator model would be: 

    G^*= arg min_G max_D  ℒ_GAN

In practice, ℒ_GAN may not provide enough gradient for G to get trained well, specially at the early steps of the training where the discriminator can easily detect fake samples from the real ones. 
One solution is to let the generator to maximize 𝔼_z ∼ p_z(z)[log(D(G(z)))] (instead of minimizing 𝔼_z ∼ p_z(z)[log(1-D(G(z)))]).



As discussed briefly in the introduction, there have been several works improving on GAN on one or all directions.
Since invention of GAN, there have been several works trying to improve/modify GAN in different aspects.
To name a few promising works, in <cit.>, Radford and colleagues proposed a convolutional GAN model for generating images, which generates visually higher quality images than fully connected networks.
In <cit.>, Mirza et al. proposed a conditional GAN model, which can generate images conditioned on class labels. 
This enables one to generate samples with a specified label (such as specific digit when trained on MNIST dataset).
The original paper on GAN uses KL divergence to measure the distance between distributions, and a problem can happen if these distribution have non-overlapping support in which KL divergence is not a good representative of their distance. 
The work in <cit.> addresses this issue, by proposing a new loss function based on Wasserstein distance (also known as earth mover's distance).
In <cit.>, Zhu proposed an image to image translation model based on a cycle-consistent GAN model, which learns to map a given image distribution into a target domain (e.g. horse to zebra, day to night images, summer to winter images).
In <cit.>, Zhang et al proposed a GAN-based high-resolution image generation network using a text description of the image.
The idea of adversarial training has been also applied to auto-encoder framework, to provide an unsupervised feature learning approach, e.g. in <cit.>,  where the adversarial module is trying to distinguish samples for which the latent representation is coming from a prior distribution from other samples.




 §.§ TV-Regularized GAN

We use a deep convolutional GAN (DC-GAN) as the main core of our fingerprint generation framework.
The fingerprint images consist of many lines, each forming a connected component, and help the model to generate fingerprint images with their lines being connected (instead of having broken lines).
Although the GAN model should learn this line connectivity from the training data, we found out imposing a regularization term which promotes the connectivity of the generated fingerprint images can help the generator learn this more easily.

There are several ways to impose connectivity within images, such as total variation (TV), group sparsity <cit.>. 
Total variation penalizes the solutions (in this case, the generated images) with large variations among neighboring pixels, leading to more connected and smoother solutions <cit.>.
We chose to use total variation to impose the connectivity of generated images here, as its gradient update is less complex during back-propagation. 
Total variation of a differentiable function f, defined on an interval [a,b] ⊂ R, has the following expression if f' is Riemann-integrable:

    V_a^b= ∫_a^b f' (x)  dx


Total variation of 1D discrete signals y=[y_1, y_2,  ..., y_N] is straightforward, and can be defined as:

    TV(y)=  ∑_n=1^N-1 |y_n+1-y_n|


where D is a (N-1)× N matrix as below:

    D= 
    [  1 -1  0  …  0;  0  1 -1  …  0;  ⋮  ⋮  ⋮  ⋱  ⋮;  0  0  …  1 -1 ]

For 2D signals, Y= [y_i,j], we can use the isotropic or the anisotropic version of 2D total variation <cit.>. 
To make our optimization problem simpler, we have used the anisotropic version of total variation, which is defined as the sum of horizontal and vertical gradients at each pixel:

    TV(Y)=  ∑_i,j |y_i+1,j-y_i,j|+|y_i,j+1-y_i,j|



We can add the total variation of the generated image to the loss function to promote the connectivity of the generated fingerprint images.
Adding this term to our framework, the new loss function for our model would be defined as:

    ℒ_GAN-TV=  𝔼_x ∼ p_data(x)[log D(x)]+ 
       𝔼_z ∼ p_z(z)[log(1-D(G(z)))] + λ TV(G(z))


We can then train both the generator and discriminator by stochastic gradient descent on mini-batches of training images.



 §.§ Network Architecture

We will now discuss the architectures used for our discriminator and generator networks.
As discussed above, both the discriminator and generator networks in our model are based on convolutional neural networks.
The discriminator model consist of 4 convolutional layers followed by batch normalization and leaky ReLU as non-linearity, and a fully connected layer at the end.
The generator model contains 5 fractionally-strided convolutional layers (aka deconvolution in literature), followed by batch-normalization and non-linearity.
The architecture of our generator and discriminator networks are shown in Figure <ref>.






§ EXPERIMENTAL RESULTS


Before presenting the generated images by the proposed model, let us first discuss the hyper-parameters used in our training procedure.
We train the proposed model for 120 epochs on a Nvidia Tesla GPU.
The batch size is set to 40, and ADAM optimizer is used to optimize the loss function, with a learning rate of 0.0002.
The input noise to the generator network, is 100-dimensional Gaussian distribution with zero mean, unit variance.
We present the details of the datasets used for our work in the next section, and then present experimental results.




 §.§ Databases

In this section, we provide a quick overview of two popular fingerprint databases used in our work, FVC-2006 fingerprint dataset <cit.>, and PolyU fingerprint dataset <cit.>.

FVC 2006 Fingerprint Database:
FVC has provided four distinct fingerprint databases, DB1, DB2, DB3 and DB4. 
Each database is 150 fingers wide and 12 samples per finger in depth (i.e., it consists of 1800 fingerprint images). 
Each database is partitioned in two disjoint subsets A and B, which contain the first 140 fingers (1680 images) and  the last 10 fingers (120 images) 
respectively. 
The image size and resolution vary depending on the database. 
For further details about this dataset, we refer the readers to <cit.>.
In our work, we have used the DB2-A dataset, which is taken by an “Optical Sensor”, and has a resolution of 400x560.
Twelve sample images from this database are shown in Figure <ref>.
It is worth mentioning that we first extract the central crop (400x400) of these images, and then resize them to 64x64 before feeding them to the TV-regularized GAN model. We also perform some data augmentation to increase the number of images during training.


PolyU Fingerprint Databases:
PolyU (high-resolution) fingerprint database is provided by the Biometrics Research Centre of the Hong Kong Polytechnic University.
They provide two high-resolution fingerprint image databases (denoted as DB-I and DB-II) <cit.>.
DB-I consists of 1600 images of 320x240 resolution, and DB-II provides 1480 images of 640x480.
In this work, we used the DB-II database.
We first extract the central crop (480x480) of these images, and then resize them to 64x64 before feeding them to our TV-regularized GAN model.


Twelve sample images from this database are shown in Figure <ref>.





 §.§ Experimental Analysis and Generated Fingerprint Images

We will now present the experimental results of the proposed fingerprint generator model trained on the above databases.
We trained one model per dataset in our experiments, with the model parameters discussed above.
After training these models, we can generate different images by sampling latent representation from our prior Gaussian distribution and feeding them to the generator network of the trained models.

In Figure <ref>, we present 16 generated fingerprint images over different epochs (0th, 20th, 40th, 60th, 80th and 100th), to see the amount of diversity among the generated images (in terms of the position of the width, height, and color of the fingerprints).
As it can be seen, there is a good amount of diversity in all the aforementioned aspects across the 16 generated images.








In Figure <ref>, we show the generated fingerprint images for four different latent representations, over every tenth epochs, when the model is trained on FVC2006 fingerprint dataset.
This will provide the granular changes happen after training the model for 10 more epochs.
As we can see, the generative model keeps improving over time, thus generating more realistic fingerprint images images.


In Figure <ref>, we show a similar results generating four samples images, using the trained model on PolyU fingerprint dataset.
We can make the same observation as above, where the generated sample images gets more realistic over time.




We also present the discriminator and generator loss functions for the trained Finger-GAN models on FVC2006 and PolyU databases.
The loss functions for these models are presented in Figures <ref> and <ref>.









 §.§ Numerical Results

To  measure  the quality  and  diversity of the model numerically,  we computed the Frechet Inception Distance (FID) <cit.> on the generated iris images by our model.
FID is an extension of Inception Score (IS) <cit.>, which was previously used for assessing the quality of generated images by GANs and other generative models.
FID compares the statistics of generated samples to real samples, using the Frechet distance between two multivariate Gaussians, defined as below:

    FID=  μ_r - μ_g ^2 + Tr (Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2) )

where X_r ∼ N(μ_r, Σ_r) and X_g ∼ N(μ_g, Σ_g) are the 2048-dimensional activations of the Inception-v3 pool3 layer for real and generated samples respectively.

The FID scores by our proposed FingerGAN model is shown in Table <ref>.
As we can see, the model achieves relatively low FID score, comparable with some of state-of-the-arts models on public image datasets.







§ CONCLUSION


In this work we propose a framework for fingerprint image generation using deep convolutional generative adversarial network. 
We use simple architectures for the generator and discriminator networks of our model, each containing 5 layers.
We also add a suitable regularization term  to our framework (the total variation of the output), to impose the connectivity of the generated fingerprint images.
These models are trained on two popular fingerprint databases, FVC 2006 and PolyU.
The experimental results show the synthetic fingerprint images look very realistic and similar to real fingerprints.
In the future, we plan to develop a model for fingerprint generation based on conditional-GAN model, which is able to generate image samples for a given identity.






§ ACKNOWLEDGMENT

We would like to thank the Hong Kong Polytechnic University (PolyU) and FVC group for providing us the fingerprint databases used in this work.
We would also like to thank Ian Goodfellow for his comments and suggestions regarding the model performance analysis.



1
daugman_howirisworks
Daugman, John. “How iris recognition works.” The essential guide to image processing. 2009. 715-739.
iris_intro
Daugman, John. "New methods in iris recognition." IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 37.5: 1167-1175, 2007.
face_intro
Xie, Weidi, and Andrew Zisserman. “Multicolumn networks for face recognition.” arXiv preprint arXiv:1807.09192 (2018).








finger_old1
Cappelli, Raffaele, Matteo Ferrara, and Davide Maltoni. “Minutia cylinder-code: A new representation and matching technique for fingerprint recognition.” IEEE Transactions on Pattern Analysis and Machine Intelligence 32.12 (2010): 2128-2141.
finger_old2
Zhao, Qijun, et al. “Adaptive fingerprint pore modeling and extraction.” Pattern Recognition 43.8 (2010): 2833-2844.


handcraft1
Abhyankar, Aditya, and Stephanie Schuckers, "Integrating a wavelet based perspiration liveness check with fingerprint recognition." Pattern Recognition 42.3: 452-464, 2009.
handcraft2
S Minaee, Y Wang. "Fingerprint recognition using translation invariant scattering network." Signal Processing in Medicine and Biology Symposium (SPMB), IEEE, 2015.

cnn1
A Krizhevsky, I Sutskever, GE Hinton, “Imagenet classification with deep convolutional neural networks”, Advances in neural information processing systems. 2012.
cnn2
He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.

finger_new1
Jiang, Lu, et al. “A direct fingerprint minutiae extraction approach based on convolutional neural networks.” Neural Networks (IJCNN), 2016 International Joint Conference on, IEEE, 2016.
finger_new2
Tang, Yao, Fei Gao, and Jufu Feng. “Latent fingerprint minutia extraction using fully convolutional network.” Biometrics (IJCB), 2017 IEEE International Joint Conference on. IEEE, 2017.
irisdeep1
Minaee, Shervin, AmirAli Abdolrashidi, and Yao Wang. “An experimental study of deep convolutional features for iris recognition.” Signal Processing in Medicine and Biology Symposium (SPMB), 2016 IEEE. IEEE, 2016.
irisdeep2
Al-Waisy, Alaa S., et al. "A multi-biometric iris recognition system based on a deep learning approach." Pattern Analysis and Applications 21.3: 783-802, 2018.
palmdeep
S Minaee and Y Wang, “Palmprint Recognition Using Deep Scattering Convolutional Network”, International Symposium on Circuits and Systems, IEEE, 2017.










gan
I Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, A Courville, and Y Bengio, “Generative adversarial nets.” In Advances in neural information processing systems, pp. 2672-2680. 2014.
sr-gan
Ledig, Christian, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken et al. “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.” In CVPR, 2017.
pix2pix
Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks.", arXiv preprint (2017).
gan_text
Zhang, Han, et al. “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.” arXiv preprint (2017).
gan_walk
Gupta, Agrim, et al. “Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR). No. CONF. 2018.
irisgan
S Minaee, A Abdolrashidi, "Iris-GAN: Learning to Generate Realistic Iris Images Using Convolutional GAN." arXiv preprint arXiv:1812.04822, 2018.
TV
A Chambolle, "An algorithm for total variation minimization and applications." Journal of Mathematical imaging and vision 20.1-2: 89-97, 2004.
dc-gan
Radford, Alec, Luke Metz, and Soumith Chintala. “Unsupervised representation learning with deep convolutional generative adversarial networks.” arXiv preprint arXiv:1511.06434, 2015.
con-gan
Mirza, Mehdi, and Simon Osindero. “Conditional generative adversarial nets.” arXiv preprint arXiv:1411.1784, 2014.
was-gan
Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein gan." arXiv preprint arXiv:1701.07875, 2017.
cycle-gan
JY Zhu, T Park, P Isola, AA Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks”, arXiv preprint, 2017.
adv_ae1
Makhzani, Alireza, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. "Adversarial autoencoders." arXiv preprint arXiv:1511.05644 (2015).
adv_ae2
Shervin Minaee, Yao Wang, Alp Aygar, Sohae Chung, Xiuyuan Wang, Yvonne W. Lui, Els Fieremans, Steven Flanagan, Joseph Rath, “MTBI Identification From Diffusion MR Images Using Bag of Adversarial Visual Features”, arXiv preprint arXiv:1806.10419, 2018.

group-sparse
Zhang, Jian, Debin Zhao, and Wen Gao. "Group-based sparse representation for image restoration." IEEE Transactions on Image Processing 23.8: 3336-3351, 2014.
TV1
Beck, Amir, and Marc Teboulle. "Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems." IEEE Transactions on Image Processing 18.11: 2419-2434, 2009.
TV2
Shervin Minaee and Yao Wang, "Screen Content Image Segmentation Using Robust Regression and Sparse Decomposition", IEEE Journal on Emerging and Selected Topics in Circuits and Systems, no.99, pp.1-12, 2016.
TV3
Minaee, Shervin, and Yao Wang. "Masked Signal Decomposition Using Subspace Representation and Its Applications." arXiv preprint arXiv:1704.07711, 2017.



FVC2006
R. Cappelli, M. Ferrara, A. Franco and D. Maltoni, "Fingerprint verification competition 2006", Biometric Technology Today, vol.15, no.7-8, pp.7-9, August 2007.
FVC_site
http://bias.csr.unibo.it/fvc2006/databases.asp
polyU
Q Zhao, D Zhang, L Zhang, N Luo, "High resolution partial fingerprint alignment using pore–valley descriptors." Pattern Recognition 43, no. 3: 1050-1061, 2010.
Poly_site
https://www4.comp.polyu.edu.hk/ biometrics/

FID
Heusel, Martin, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. "Gans trained by a two time-scale update rule converge to a local nash equilibrium." In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
IS
T Salimans, I Goodfellow, W Zaremba, V Cheung, A Radford, and X Chen. "Improved techniques for training gan", In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.








