A writer-independent approach for offline signature verification using deep convolutional neural networks features
Victor L. F. Souza Adriano L. I. Oliveira Robert Sabourin






A writer-independent approach for offline signature verification using deep convolutional neural networks features



 This article has been accepted for publication in BRACIS 2018 but has not yet been fully edited. Some content may change prior to final publication.

    Victor L. F. Souza
Centro de Informática 

Universidade Federal de Pernambuco

Recife, PE, Brazil 

vlfs@cin.ufpe.br
Adriano L. I. Oliveira
Centro de Informática 

Universidade Federal de Pernambuco

Recife, PE, Brazil 

alio@cin.ufpe.br
Robert Sabourin
LIVIA, École de Technologie Supérieure 

Université du Québec

Montreal, Canada 

robert.sabourin@etsmtl.ca









    December 30, 2023
======================================================================================================================================================================================================================================================================================================================================================================================




The use of features extracted using a deep convolutional neural network (CNN) combined with a writer-dependent (WD) SVM classifier resulted in significant improvement in performance of handwritten signature verification (HSV) when compared to the previous state-of-the-art methods. In this work it is investigated whether the use of these CNN features provide good results in a writer-independent (WI) HSV context, based on the dichotomy transformation combined with the use of an SVM writer-independent classifier. The experiments performed in the Brazilian and GPDS datasets show that (i) the proposed approach outperformed other WI-HSV methods from the literature, (ii) in the global threshold scenario, the proposed approach was able to outperform the writer-dependent method with CNN features in  the Brazilian dataset, (iii) in an user threshold scenario, the results are similar to those obtained by the writer-dependent method with CNN features.



Offline signature verification, Writer-independent signature verification, Dichotomy transformation.




§ INTRODUCTION


Signature Verification (SV) systems are used to automatically recognize whether the signature provided by an user actually belongs to the individual who he/she claims to be. Therefore, these systems are useful in many real-world applications, such as credit card transactions or document authentication. Specifically, the problem of automatic Handwritten Signature Verification (HSV) can be defined as follows: given a learning set containing genuine signatures of a set of users, a model is trained to classify the signatures as genuine or forgeries. Genuine signatures are those that really belong to the indicated user; in turn, forgeries are those created by other people <cit.>.

The signatures to be verified by the HSV systems can be acquired in two ways: offline (static) and online (dynamic). In offline SV, the signature is acquired after the writing process is completed. In this case, the signature is treated as an image. On the other hand, in online verification, a device is used to collect data as it is produced. So, online models can obtain additional information from the users during writing to perform the verification, such as the position or slope of the pen, or the writing pressure <cit.>.


Another important point that deserves to be highlighted when dealing with the HSV problem is the model's user horizon. If a model is trained for each user, the system is called writer-dependent (WD). In this case, initially, a training set is constructed as follows: genuine signatures of the tested user are treated as positive instances and signatures from other users as negative. Next, a binary classifier is trained for each user. Although WD systems achieve good results for the HSV task, requiring a classifier for each user increases the complexity and the cost of the system operations as more users are added <cit.>.

On the other hand, HSV systems used to classify signatures of any available user in the dataset are known as writer-independent (WI) systems. In this context, a single model is trained for all users from a dissimilarity space. Thus, the classification inputs are vectors of dissimilarity, which represent the difference between the features of a queried signature and a reference signature of the user. When compared to the WD approach, WI systems are less complex, but in general obtain worse results <cit.>.

Recently, Hafemann et al. <cit.> proposed an approach to deal with the offline HSV problem that uses concepts from both WI and WD systems. The approach carries out feature learning from the signature images in a WI format, using a deep convolutional neural network (CNN) called SigNet. After being trained, the CNN is used to extract representative features from the signatures, which are used to train a writer-dependent Support Vector Machine (SVM) classifier for each writer. Their results showed a significant improvement in performance when compared to the previous state-of-the-art methods.

The main objective of this paper is to investigate whether the deep CNN features learned by the Hafemann et al. model <cit.> (available online[http://en.etsmtl.ca/Unites-de-recherche/LIVIA/Recherche-et-innovation/Projets/Signature-Verification]) can also lead to good results in a writer-independent HSV context. To this end, it is proposed the use of dichotomy transformation <cit.> combined with an SVM as a writer-independent classifier to perform the signature verification. 

The following points will be analyzed: (i) which partial decisions fusion rule is the best (functions max, mean, median and min are tested). (ii) The influence of the number of signatures used in the reference set. (iii) A comparison  with other studies from the literature. The experiments are carried out using the GPDS and the Brazilian PUC-PR datasets.

The remaining of this paper is organized as follows: Section <ref> presents the related work on signature verification, separating WD and WI approaches. Section <ref> details the proposed method, and Section <ref> describes the used experimental protocol and the discussion of the results. Lastly, Section <ref> concludes the paper and discusses future works.



§ RELATED WORKS



In general, two approaches are used for offline handwritten signature verification, writer-dependent (WD) and writer-independent (WI). In the WD scenario, a classifier is trained for each writer and is responsible for authenticating his/her signatures. In the WI context a single classifier is trained for all writers and is responsible for associating the questioned entry signatures to one or more reference signatures in a dissimilarity space <cit.>. Most HSV systems presented in literature follow the WD approach <cit.>.

Batista et al. <cit.> proposed a hybrid generative–discriminative ensemble of classifiers which dynamically selects the classifiers for building a writer-dependent HSV system. During the generative stage, the signatures are divided in a grid format and multiple discrete left-to-right Hidden Markov Models (HMMs) are trained with different number of states and codebook sizes, to be able of working at different levels of perception. Then the HMM likelihoods for each enrolled signature are computed and grouped into a feature vector that is used through a specialized Random Subspace Method to build a pool of two-class classifiers (discriminative stage). For the verification task, the authors propose a new dynamic selection strategy based on the K-nearest-oracles (KNORA) algorithm and on Output Profiles to select the most accurate ensemble to classify the given signature <cit.>.

Guerbai et al. <cit.> proposed a writer-dependent HSV system based on One-Class SVM (OC-SVM) that tries to reduce the difficulties of having a large numbers of users. As a one-class classification problem, the proposed approach models only one class (genuine signatures). Which is a good characteristic, as, in general, the system only has the genuine signatures for each writer to train the classifier. Nevertheless, the low number of genuine signatures is still an important challenge <cit.>.

In the WI scenario, Bertolini et al. <cit.> proposed a  writer-independent approach for handwritten signature verification. This approach applies the ideas of dissimilarity representation and SVMs as classifiers <cit.>. The two main contributions of the authors are the following: (i) introduce a new graphometric feature set based on the curvature of the most important segments, simulated by using Bezier curves. (ii) The use of an ensemble of classifiers structure to improve the resistance against forgeries. This ensemble is built using a standard genetic algorithm and a pool of base classifiers trained with four different graphometric feature sets <cit.>.

Also, Rivard et al. <cit.> have proposed a  writer-independent approach that combines multiple feature extraction, dichotomy transformation, and boosting feature selection. The authors report that the accuracy and reliability of the system can be improved by integrating features from different sources of information, so initially they employ some techniques to extract features at different scales. Then they use the Dichotomy Transformation, which reduces the pattern recognition problem to a 2-class problem. 
A good point that deserves to be highlighted is that with this transformation the system alleviates the challenges of deal with limited number of reference signatures from a large number of users. Finally, an ensemble is built using boosting feature selection that uses low-cost classifiers capable of automatically select relevant features during training <cit.>. 

Some authors use a combination of both WD and WI approaches. For
example, Eskander et al. <cit.> proposed a hybrid writer-independent-writer-dependent model. The aim of the authors was to maximize the positives of each approach. In the scenario where only a few genuine signatures are available, they use the writer-independent classifier to perform the verification. On the other hand, the writer-dependent classifier is trained for an user when the number of genuine samples is above a defined threshold <cit.>.

Yilmaz <cit.> also propose a hybrid approach that combines WI and WD results. Using the main ideas from the WD and WI approaches, aiming to learn the importance of different dissimilarities, the writer-independent classifier is trained with dissimilarity vectors of query and reference signatures of all users. In its turn, the writer-dependent classifiers are trained separately for each user, to learn to differentiate genuine signatures and forgeries. The results are then combined using a score-level fusion of these complementary classifiers with different local features <cit.>.



§ PROPOSED METHOD



In the context of offline Handwritten Signature Verification, Hafemann et al. <cit.> have achieved significant improvements in results by using a deep CNN to extract writer-independent representative features from signatures (specifically, the 2048 features were obtained from the FC7 layer of the deep CNN). In  Hafemann et al. proposal, the deep CNN features are  used to train a writer-dependent SVM classifier for each user of the system. In contrast, in this paper a writer-independent (WI) SVM classifier is proposed to perform the signature verification.

WI systems use the dissimilarity between each questioned signature and the reference signatures to perform the authentication. To this end, the proposed WI approach employs the dichotomy transformation, which allows to transform K-class pattern recognition problems, where K is a large or unspecified value, into a 2-class problem (in this case, classify a handwriting sample into genuine or forgery) <cit.>. In this context, the single SVM classifier is trained for all users from this dichotomy space.

Among the advantages of using a WI model are the following: (i) it allows to exploit a system with only one signature per user, (ii) in contrast to WD systems, the number of users in WI is of little consequence, as input feature vectors are transformed into a distance space between signatures, (iii) the writers populating used during the verification do not necessarily need to be enrolled to the system <cit.>.



 §.§ Dichotomy Transformation


The Dichotomy Transformation (DT), proposed by Cha and Srihari <cit.>, transforms a multi-class problem into a binary problem (two classes). 
To this end, the vector of distances of every feature by the same writer samples are computed and categorized as a within author distance (which is denoted by x_+). On the other hand, the distance vectors from samples of different writers are categorized as between author distance (which is denoted by x_-). Let d_ij denote the j'th sample of the i'th writer <cit.>.













    x_+ = u(d_ij - d_ik)  
     where   i = 1   to   n; j,k = 1   to   m   and   j ≠ k




    x_- = u(d_ij - d_kl)  
     where   i, k = 1   to   n    and   j ≠ k; j,l = 1   to   m


where n is the number of writers, m is the number of samples per writer, u is the absolute value in the distance domain resulting from the dichotomy transformation. 

The following properties are desired  with the dichotomy transformation: (i) all distances between samples from the same writer in the feature domain should belong to the within class in the transformed space (that is, stay close to the origin in the dissimilarity representation space), and (ii) all distances between two different classes in the feature domain should belong to the between class in the transformed space (that is, stay away from the origin in the dissimilarity representation space) <cit.>. 

This is not always the case. One disadvantage of the dichotomy transformation is that perfectly clustered writers in the feature domain may not be perfectly dichotomized in the distance domain <cit.>. In other words, the broader the spread of the feature distributions among the writers, the less the dichotomizer is able to detect real differences between similar signatures <cit.>.

Other properties that are worth highlighting are as follows: (i) the dichotomy transformation affects the geometry of data distributions. So, if multiple boundaries are needed to separate the classes in the feature space, only one is needed in the distance space; (ii) the vectors resulting from the DT are always nonnegative since they consist of distances transformed in absolute values; and (iii) since each sample in the dichotomy space is formed by the distance of each pair of signatures, the limited number of samples is no longer a problem <cit.>.




§ EXPERIMENTS





 §.§ Datasets


The experiments are carried out using GPDS and Brazilian PUC-PR datasets. Table <ref> summarizes these datasets.

C[1]>p#1



In the first set of experiments the Brazilian PUC-PR dataset is considered. The data segmentation was done as in the paper by Rivard et al. <cit.>. The development set D is formed by users 61 to 168 and the exploitation dataset ε by users 1 to 60.
So, to generate the within class of the learning set L, 30 randomly selected genuine signatures for each writer from the development dataset D are used, resulting in 108 ·30 · 29/2 = 46,980 distance vectors. Aiming to generate a balanced dataset, the between class counterexamples of the learning set L are obtained as follows: for each writer, the dichotomy transformation is applied to 29 genuine signatures (references signatures) against 15 random forgeries, each one selected from a genuine signature of 15 different writers, giving 108 · 29 · 15 = 46,980 distance vectors <cit.>. 
To perform the verification, the reference set R is composed of 30 randomly selected genuine signatures from each writer of the exploitation dataset ε (writers from this dataset are unknown to the verification system). The questioned set Q is composed of the 10 remaining genuine signatures, 10 simple and 10 skilled forgeries from each writer plus 10 random forgeries, each one selected from a genuine signature of 10 different writers <cit.>.

The metrics used to evaluate the performance are based on Table 10 from <cit.>. 
The following metrics are used: (i) False Rejection Rate (FRR), which represents the percentage of genuine signatures that are rejected by the system, (ii) False Acceptance Rate (FAR), represents the percentage of forgeries that are accepted (it can be computed for each random, simple and skilled forgeries), (iii) Average Error Rate (AER), is the average error considering FRR, FAR_random, FAR_simple, FAR_skilled.  (iv) Equal Error Rate (EER), is the error obtained when FRR = FAR <cit.>.

The comparative analysis for the GPDS dataset are based on values from table 7 of Hafemann et al. paper <cit.>. 
For the GPDS-160 development set D is formed by users 161 to 881 and the exploitation dataset ε by user 1 to 160.
In turns, for the GPS-300 development set D is formed by users 301 to 881 and the exploitation dataset ε by user 1 to 300.
In this dataset, the within class of the learning set L is generated from 12 randomly selected genuine signatures for each writer from the development dataset D, which results in 721 ·12 · 11/2 = 47,586 and 581 ·12 · 11/2 = 38,346 distance vectors, respectively, for the GPDS-160 and the GPDS-300 datasets.
The between class is formed by using 11 genuine signatures (references signatures) against 6 random forgeries, each one selected from a genuine signature of 6 different writers, resulting in 721 · 11 · 6 = 47,586 and 581 · 11 · 6 = 38,346 distance vectors, respectively, for the GPDS-160 and the GPDS-300 datasets. To perform the verification, the reference set R is composed of 12 randomly selected genuine signatures from each writer of the exploitation database ε. The questioned set Q is composed by 10 of the remaining genuine signatures and 10 skilled forgeries from each writer plus 10 random forgeries, each one selected from a genuine signature of 10 different writers (trying to follow the segmentation from Rivard et al. paper <cit.>). 

In both datasets, the writer-independent SVM classifier is trained using the data from the learning set L and the verification is performed using the questioned Q and the reference R sets.



 §.§ Experimental setup


Support Vector Machines (SVM) have been widely used for both WD and WI signature verification tasks. This indicates that it is one of the most effective classifiers for this problem <cit.>.

In this paper, the SVM is used as writer-independent classifier with the following settings: RBF kernel, γ = 2^-11 and C=1.0 (the training data is balanced, so there is no need to use different weights for the positive and negative class) <cit.>. 

All data were randomly selected and a different SVM was trained for each replication (five replications for each experimental configuration were performed).

The point of intersection between FRR and FAR_skilled curves is used as a global threshold. The EER metric is also computed based on an user threshold, as presented in Hafemann et al. <cit.>.







 §.§ Results and discussion


The results are organized as follows: (i) initially, the number of signatures in the reference set R is fixed and an analysis of which partial decisions fusion rule is the best (functions max, mean, median and min were tested). (ii) In the sequence, the analysis about the influence of the number of signatures used in the reference set R. (iii) Finally, the comparison with the state-of-the-art for the used datasets (as the present work uses SigNet features, Hafemann et al. results are also presented using only these features <cit.>).



  §.§.§ Brazilian PUC-PR dataset




  
Partial decisions fusion rule analysis
To measure the impact of the partial decisions fusion rule, in this section  the number of references per user is fixed in 30 and different functions are tested. The functions used in the combinations were: (i) Mean, (ii) Max, (iii) Median and (iv) Min.










Figure <ref> presents the boxplots for AER and AER_genuine+skilled for the tested functions (Max, Mean, Median, Min). As can be seen, the Max function obtained the best results for both metrics. In the opposite direction, the Min function had the worst results in both cases.

The Wilcoxon paired signed-rank test with 5% significance level for both AER and AER_genuine+skilled metrics shows that the max function outperforms the other functions with statistical relevance. 

For the EER metric, the Wilcoxon paired signed-rank test also showed that the Max function is statistically better when compared to the Median and the Min. However, there is no statistical difference to the Mean function.

In Bertolini et al. <cit.> the Max rule also achieved the best results for the considered dataset.



  
Analysis of the influence of the number of reference signatures
In the previous section it was shown that the Max function obtained better results when compared to other functions. 
In this section, to measure the impact of the reference set cardinality, the Max function was fixed and the number of references per user was varied. To this end, reference subsets containing [1, 5, 10, 15, 20, 25, 30] randomly selected signatures are used as references for the authentication.



Figure <ref> depicts the boxplots for AER and AER_genuine+skilled for different reference sizes. As can be observed, using more references per user produces better results (the worst cases are with number of references = 1 and = 5). However, the variation among results decreases as the number of references increases. For instance, the experiments using 15 or 20 references present similar results; the same can be observed with 25 and 30.
  
The Wilcoxon paired signed-rank test for both metrics, using n_reference = 30 as baseline, shows that results are statistically better only when compared with the cases where number of references = 1 and = 5. There is no statistical difference to n = 10, 15, 20 or 25.

So far, the best results are obtained using the Max function with the highest value for the number of references (in this case, n_reference = 30) for both global and user threshold scenarios (this settings will be referenced as “best proposed approach”).





  
Comparison with the state-of-the-art
Table <ref> presents results obtained by the best proposed approach together with results from table 10 of Hafemann et al. work <cit.>. 




As shown in Table <ref>, when compared to Hafemann et al. work <cit.>, the results of the best proposed approach are worse in FRR metric, but are better in  FAR_skilled metric, in a global scenario. In summary, the results of the present work are better than Hafemann et al. work <cit.> (this can be seen through AER and AER_genuine+skilled). 
When compared to the other works, the presented approach produces a considerable improvement for all metrics. Keep in mind that the proposed method uses a simpler architecture (using only a single SVM to perform the verification) when compared to others, such as the ones proposed by Bertolini et al. <cit.> or Rivard et al. <cit.>, that use ensembles of classifiers.





As can be observed in Table <ref>, the results of the best proposed approach of this paper obtained slightly superior results in comparison with Hafemann, Sabourin and Oliveira <cit.> and Hafemann et al. <cit.> for the EER metric. It worth notice that the proposed method performs writer-independent authentication and both Hafemann's models operate in a writer-dependent way and, even so, the WI approach was able to improve the results.





  §.§.§ GPDS dataset

For the GPDS datasets, the same functions used as partial decisions fusion rule were tested. As the highest value for the number of references is 12, reference subsets containing [1, 2, 3, 4, 5, 10, 12] randomly selected signatures were tested as references for the authentication.

As in Brazilian PUC-PR dataset, in the GPDS dataset the best results are obtained using the Max function with the highest value for the number of references (in this case, n_reference = 12) for both global and user threshold scenarios.  
Figures <ref> and <ref> depict, respectively, the boxplots for AER and AER_genuine+skilled metrics (i) when varying the fusion rule and (ii) when we varied the reference sizes, for the GPDS-160 dataset. Similar behavior can be observed for both GPDS-160 and GPDS-300.

For the GPDS-160 dataset, performing the Wilcoxon paired signed-rank test for both metrics: (i) Max function outperforms
the other functions with statistical relevance. (ii) Using n_reference = 12 is statistically better when compared to the other cases.





Tables <ref> and <ref> presents the results when the best proposed approach is used and compares the obtained results with those from Table 7 of Hafemann et al. paper <cit.>.








As can be observed in Table <ref>, for the GPDS-160, our best proposed approach was able to outperform Guerbai et al. <cit.> and Hafemann, Sabourin and Oliveira <cit.> using both global and user thresholds. Also, it was able to outperform Yilmaz <cit.> (with 5 and 12 samples) using user thresholds. 
As in Brazilian PUC-PR dataset, in GPDS-160 dataset, the proposed approach was able to achieve better results with a simpler architecture. While we are using only a single SVM to perform the verification, Yilmaz <cit.> uses an ensemble of WI and WD classifiers.

For the GPDS-300, as presented in Table <ref>, our best proposed approach was able to outperform Soleimani et al. <cit.> and Hafemann, Sabourin and Oliveira <cit.> using both global and user thresholds. 

However, for both datasets, the best proposed approach using an user threshold obtained slightly inferior results in comparison with Hafemann et al. WD model <cit.> for the EER metric. 





§ CONCLUSION


This paper has introduced an approach for writer-independent offline signature verification that uses the dissimilarity representation of the deep CNN features from <cit.> and a single SVM as a writer-independent classifier to authenticate handwritten signatures. 

The experiments showed that, in general, for the tested datasets, the best results are obtained using the Max function as the partial decisions fusion rule with the highest value for the number of references for both global and user threshold scenarios.

Moreover, in the global threshold scenario, the proposed approach was able to outperform Hafemann et al. <cit.> in the Brazilian dataset. For the user threshold scenario, the proposed approach was able to obtain performance comparable to Hafemann et al. <cit.>. This was so even with the proposed method performing writer-independent authentication and Hafemann et al. <cit.> operating in a writer-dependent way. In the Brazilian dataset the proposed work was slightly superior and in the GPDS dataset slightly inferior. However, for both datasets, the proposed approach was able to outperform other methods from the literature that use WD classification and the WI dissimilarity representation with different features and more complex classification architectures (for instance, ensembles of classifiers). 

Future research will include the study of feature and prototype selection in the dissimilarity space, adaptation of WI classifier over time and the writer-dependent decision threshold. 



 
IEEEtran


