Quantification of the weight of fingerprint evidence using a ROC-based Approximate Bayesian Computation algorithm for model selection
J. H. Hendricks C. Neumann C. P. Saunders






t1This project was supported in part by Award No. 2014-IJ-CX-K088 awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice, and in part by the National Science Foundation under Grant DMS-1127914 to the Statistical and Applied Mathematical Sciences Institute. The opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect those of the Department of Justice or the National Science Foundation. We would like to thank Dr. Danica Ommen for her help in the initial implementation of ABC for forensic evidence.
ROC-based ABC algorithm for fingerprint evidence










Department of Mathematics and Statistics
 South Dakota State University
 Brookings, SD, USA
e1,e2,e3




Hendricks et al.


For more than a century, fingerprints have been used with considerable success to identify criminals or verify the identity of individuals. The categorical conclusion scheme used by fingerprint examiners, and more generally the inference process followed by forensic scientists, have been heavily criticised in the scientific and legal literature. Instead, scholars have proposed to characterise the weight of forensic evidence using the Bayes factor as the key element of the inference process. In forensic science, quantifying the magnitude of support is equally as important as determining which model is supported. Unfortunately, the complexity of fingerprint patterns render likelihood-based inference impossible. In this paper, we use an Approximate Bayesian Computation model selection algorithm to quantify the weight of fingerprint evidence. We supplement the ABC algorithm using a Receiver Operating Characteristic curve to mitigate the effect of the curse of dimensionality. Our modified algorithm is computationally efficient and makes it easier to monitor convergence as the number of simulations increase. We use our method to quantify the weight of fingerprint evidence in forensic science, but we note that it can be applied to any other forensic pattern evidence.









	Approximate Bayesian Computation
	Bayes factor
	Forensic science
	Weight of evidence
	Receiver Operating Characteristic












§ INTRODUCTION



For more than a century, fingerprints have been used with considerable success to identify criminals or verify the identity of individuals. In this paper, we define a fingermark (left panel of Figure <ref>), or mark, as the impression resulting from the inadvertent contact between the finger of an unknown donor and a surface (e.g., at a crime scene). We define a control print (right panel of Figure <ref>), or print, as a finger impression collected under controlled conditions from a known donor (e.g., a suspect). The purpose of forensic fingerprint examination is to support the inference of the donor of a fingermark. Currently, this inference process relies on the visual comparison between the fingermark and control prints from one or more candidate donors who may have been selected through a police investigation or by searching the fingermark in a database of prints from known individuals. 

 


Following the examination of fingerprint evidence, it is customary for the examiner to report one of two possible conclusions: an opinion of “identification", implying that the source of the fingermark is the donor of a given control print; or an opinion of “exclusion", implying that the source of the fingermark is not a considered candidate. Alternatively, the examiner may find the examination “inconclusive", indicating that the characteristics of the impressions being compared are not sufficient to reach one of the two possible conclusions (e.g., when the impressions are too small or too degraded).

The categorical conclusion scheme used by fingerprint examiners, and more generally the inference process followed by forensic scientists, have been heavily criticised in the scientific and legal literature <cit.>. Instead, scholars have proposed to characterise the weight of forensic evidence using the Bayes factor as the key element of the inference process (see <cit.> for a comprehensive discussion).

It is worth stressing that forensic scientists do not have a complete picture of all the evidence available in a given criminal case (e.g., eyewitness evidence; means, motive and opportunity of an individual of interest; other material and non-material evidence), which prevents them from assigning probabilities to the different propositions that the parties may have regarding a particular criminal activity. Furthermore, forensic scientists are not tasked with the fact-finding mission in the criminal justice system and are not in charge of the decision-making with respect to the propositions of the parties. Therefore, the role of the scientist is necessarily limited to reporting the amount of support of the forensic evidence for these propositions in the form of a Bayes factor. It is of primary importance to note that, in this setting, we are not only concerned with supporting the correct model, but also in supporting it with the appropriate magnitude. An appropriate magnitude of support is critical to ensure the coherent combination of the respective weight of the multiple pieces of evidence that may be considered in a given case. This imperative requirement is the main motivation behind the present work and is the main force that drives us away from more deterministic pattern recognition algorithms.

Assuming a fingermark recovered in connection with a crime and a suspect, Mr. X., we consider the following two alternative propositions:


	
  H_1: the fingermark was left by Mr. X.
	
  H_2: the fingermark was not left by Mr. X., but by another person in a relevant population of potential donors.


To address the so-called prosecution and defence propositions, H_1 and H_2, we define two corresponding models: M_1, representing how Mr. X. generates fingermarks, and M_2, representing how fingermarks are generated by the donors in the population of alternative sources. Let e_u denote the set of observations made on the fingermark. We are interested in evaluating the following Bayes factor:


    BF = π(M_1|e_u)π(M_2|e_u)·π(M_2)π(M_1) = f(e_u | M_1)f(e_u| M_2),


where π(·) is a measure of belief about the model and its parameters (see <cit.>, chapter 7, for a formal discussion on Bayesian model selection), and f(·) represents a (marginal) probability density function.

Fingerprints are usually characterised through certain features of the ridge pattern, such as the general pattern formed by the friction ridge flow and events disturbing the continuity of the ridges. These events, traditionally called minutiae, occur when a ridge ends (ridge ending) or bifurcates (bifurcation). Other types of events exist but are mainly combinations of these two basic types. 

 

When comparing two fingerprints, an examiner first verifies the compatibility of their general patterns and then determines whether the spatial relationships, types, and orientations of the minutiae on both impressions correspond. Mathematically characterising fingerprint patterns results in heterogeneous, high-dimension random vectors: minutia locations and spatial relationships are continuous measurements; minutia types are discrete observations; and minutia orientations are circular measurements. In addition, the dimensionality of the problem increases with the number of minutiae observed on a given impression. Therefore, modelling the joint likelihood of the characteristics of multiple features observed on an impression seems to be an unreasonable challenge.

While many attempts have been made to assign Bayes factors for fingerprint evidence (see <cit.> for a recent review), no algorithm has gained wide acceptance. In particular, while the results obtained with the algorithm proposed by <cit.> were used to support the general admissibility of fingerprint evidence in U.S. courts (State v. Hull, ; State v. Dixon, ), commentators argued that the model had two main shortcomings. Some commentators discussed that the algorithm did not result in a formal Bayes factor as it does not formally incorporate user beliefs <cit.>. Others noted that the algorithm relied on an ad-hoc weighting function used to palliate the inability of the authors to assign Bayes factors at 0, and that this function had no other justification than convenience <cit.>. 

In this paper, we take advantage of the similarities between the algorithm proposed by <cit.> and the Approximate Bayesian Computation framework to provide a method to formally and rigorously assign Bayes factors to fingerprint evidence. Our method leverages the property of the well-known Receiver Operating Characteristic (ROC) curve to address shortcomings in existing ABC algorithms. Our application addresses the issues raised in relation to <cit.> and provide a much needed general framework for the quantification of the weight of any type of forensic pattern evidence, as long as a similarity measure can be defined to compare two pieces of evidence. 



§ APPROXIMATE BAYESIAN COMPUTATION


Approximate Bayesian Computation (ABC) originated as a class of algorithms designed to sample from the approximate posterior density of a vector of parameters, θ, given an observed data set, D, without direct evaluation of the likelihood function, f(D|θ). This class of algorithms is especially useful in complex and high-dimensional settings where the likelihood function is not available in a usable form (see <cit.> or <cit.> for an overview).

To sample from an approximate posterior density, vectors of parameter values are first sampled from a prior distribution over the parameter space, and then used to generate pseudo-observations (each denoted D^∗) from an assumed generative model. A vector of parameter values is retained if the distance measured by a kernel function, Δ{·,·}, between values of a summary statistic, η(·), of D and D^∗ is less than some tolerance, t>0. In other words, the i^th sampled parameter vector, θ^(i), is retained if the corresponding distance satisfies Δ{η(D),η(D^∗(i))}<t.

In some application problems, such as forensic science, statisticians are not concerned with assigning posterior probability distributions in the parameter space, but are interested in performing model selection. However, performing likelihood-based inference using patterns of forensic interest, such as fingerprints, shoe sole impressions, or striations on bullets, is not feasible in the original feature space of the data. Starting with <cit.>, ABC has evolved into an algorithm that can be used for model selection by considering a model index parameter, ℳ, and its prior distribution over model indices. The model index determines the prior distribution over the parameter space and the likelihood structure used to generate pseudo-observations. The i^th sampled model index parameter, ℳ^(i), is retained if the corresponding score satisfies Δ{η(D),η(D^∗(i))}<t. 

The original ABC method for assigning a posterior probability to a model, given the observed data, is a function of the ratio of the number of times that model index has been retained over the total number of times any model was retained <cit.>. Mathematically, the ABC posterior odds for the comparison between two models is given by:


    π_t(ℳ=1|D)π_t(ℳ=2|D)
    	 = lim_N →∞∑^N_i=1𝕀(ℳ^(i)=1)·𝕀(Δ{η(D),η(D^∗(i))}≤ t)∑^N_i=1𝕀(ℳ^(i)=2)·𝕀(Δ{η(D),η(D^∗(i))}≤ t),
 

where 𝕀(·) is the indicator function and the subscript t in π_t indicates that this measure is a function of the choice of the tolerance level. 

The ABC Bayes Factor, BF_abc, can then be assigned using the ABC approximation of the posterior odds, divided by the prior odds   


    BF_abc=π_t(ℳ=1|D)/π_t(ℳ=2|D)·π(ℳ=2)/π(ℳ=1).


The ABC algorithm for comparing two models is summarised in Algorithm <ref> <cit.>. 




The benefits of Approximate Bayesian Computation methods are not without cost. Model selection using ABC is subject to two primary sources of error: 

	
  * the quality of the approximation to the likelihood function due to the use of the tolerance, t, 
	
  * and the loss of information engendered by using a non-sufficient summary statistic.


The effect of the tolerance level, t, on the performances of ABC algorithms has been widely discussed since the inception of ABC methods. In short, if t is too large, too many samples from the prior distribution are accepted and the approximation becomes invalid; and, if t is too small the rate of acceptance is too small to produce a stable posterior distribution <cit.>. 

The motivation for using summary statistics, rather than the full data set, stems from the curse of dimensionality encountered with the high-dimensional data sets that are common to scenarios in which ABC is necessary. Ideally a summary statistic that is sufficient across models should be used to enable the convergence of BF_abc to the Bayes factor <cit.>. In general, it seems difficult, if not impossible, to find sufficient summary statistics for the type of data that are typically used in ABC method. The goal is then to avoid the curse of dimensionality while minimising the loss of information encountered when using non-sufficient statistics. While it may seem that a general solution addressing the issue of the potential lack of sufficiency of summary statistics could consist in using a large set of summary statistics with the hope that they will jointly tend towards sufficiency by decreasing the loss of information, this is not the case. As the dimensionality of the set of summary statistics increases, the algorithm will also suffer from the curse of dimensionality <cit.>. 

















Several modifications of the traditional ABC algorithm for model selection have been proposed with the goal of addressing some of the issues related to the determination of the tolerance and the selection of low dimension summary statistics that minimise the loss of information. Most of these modified algorithms are rooted in the one proposed by<cit.>, who suggests using a polychotomous weighted logistic regression model that has been trained on the summary statistics of the pseudo-observations and corresponding model indices to predict the posterior probability of a model. Modifications include variable selection <cit.>, replacement of the logisitic regression by artificial neural networks <cit.> or random forest algorithm <cit.>, or the use of posterior probabilities as summary statistics <cit.>. 











Finally, we highlight the work by <cit.>, who propose a set of necessary and sufficient conditions for summary statistics that result in a corresponding BF_abc that supports the true model asymptotically. <cit.> suggest that summary statistics should be chosen so their asymptotic mean is different under opposing models - so different that their ranges are non-overlapping. We note that showing that summary statistics satisfy these conditions in a real application is not trivial, and furthermore, that we are not only interested in supporting the correct model, but we are interested in supporting it with the correct magnitude. Nevertheless, we will show below that our method relies on a similar argument to that of <cit.>. 

The literature shows that the issues of sufficiency, variable selection and tolerance level are usually entangled. The solutions to these issues proposed to date, and summarised above, have their own sets of complications:

	
  * None of these solutions enables formal monitoring of the convergence of BF_abc to the Bayes factor as the value of the tolerance, t, is reduced, or as the number of samples increases. Some convergence results for the regression adjustments in the context of parameter inference can be found in <cit.> but are not directly applicable to the model selection context. 
	
  * Most of these solutions are focusing on selecting the correct model, but are not necessarily designed to support the selected model with the appropriate magnitude, which is a critical requirement in forensic science. 
	
  * Most of these methods require to generate summary statistics from pseudo-observations that are as close as possible to the summary statistics of the observed data. That is, these methods attempt to estimate the behaviour of the model selection algorithm as t → 0^+ without ever observing t=0 (or any value even reasonably close due to the curse of dimensionality). This involves the use of potentially complicated variable selection methods, and the definition of an appropriate kernel function to compare summary statistics <cit.>.  
	
  * Methods based on generalised linear models <cit.>, artificial neural networks <cit.>, random forests <cit.> or other classifiers can be very computationally intensive depending on the dimension of the summary statistics or the number of pseudo-observations generated from the considered models. They also heavily rely on appropriately estimating the (very large number of) parameters of these classifiers. 
	
  * Some of the model selection algorithms, such as the one proposed by <cit.>, are trained using a limited subset of pseudo-observations. These pseudo-observations are selected or weighted based on the proximity of their summary statistics with those of the observed data. Unfortunately, this results in replacing user-defined prior probabilities on model indices with probabilities assigned in unpredictable ways by the algorithm based on the proportions of training data selected from each model. 


In this paper, we propose a modification to the ABC algorithm that utilises a relationship between the ABC Bayes factor and the Receiver Operating Characteristic (ROC) curve. Our novel ROC-based approach addresses the instability created by the use of a heuristic tolerance level. Our algorithm can accommodate sets of summary statistics of any size without being subject to the curse of dimensionality or worrying about the proximity of the summary statistics of the observed and pseudo-data, as long as conditions similar to the ones suggested by <cit.> are satisfied.  Critically, our solution enables us to rigorously control its convergence as the number of simulation increases. Furthermore, for the two proposed versions of our algorithm, one requires estimation of only four parameters, and the other does not require estimation of any parameters.







§ ROC-ABC ALGORITHM FOR MODEL SELECTION



ROC curves are traditionally used to measure the performance of a binary classifier as the decision threshold, t, is varied across the domain of (dis)similarity scores, Δ{·, ·}, that can be returned by the classifier <cit.>. ROC curves are obtained by plotting the rate of correct decisions in favour of the first model against the rate of incorrect decisions (false positives) in favour of the first model (i.e. when the second model is true) for all possible values of the decision threshold. 

Defining F(·) and G(·) as the cumulative distribution functions of scores under models 1 and 2, respectively, the general form of the ROC is given by <cit.> as ROC(p) = F(G^-1(p)),






where p denotes the rate of false positives in favour of the first model and G^-1(·) denotes the quantile function <cit.> for scores under the second model. Assuming equal priors for the model indices, we can show that the Bayes factor, BF_η, in Equation (<ref>), is a function of the ROC curve constructed on the set of Δ{η(D), ·} from model 1 and the set of Δ{η(D), ·} from model 2:


    BF_η    = lim_ t → 0^+BF_abc
        = lim_ t → 0^+ 
     N →∞∑^N_i=1𝕀[ℳ^(i)=1]·𝕀[Δ{η(D),η(D^∗(i))}≤ t]∑^N_i=1𝕀[ℳ^(i)=2]·𝕀[Δ{η(D),η(D^∗(i))}≤ t]·π(ℳ=2)π(ℳ=1)
        = lim_ t → 0^+ 
     N=K+L →∞∑^K_k=1𝕀[Δ{η(D),η(D^∗(k))}≤ t | ℳ=1]∑^L_l=1𝕀[Δ{η(D),η(D^∗(l))}≤ t|ℳ=2]·π(ℳ=2)π(ℳ=1)
        = lim_ t → 0^+ 
     K →∞
     L →∞K ·F̂_K(t)L ·Ĝ_L(t)·π(ℳ=2)π(ℳ=1)
       ≈lim_t → 0^+F(t)G(t)
     	
    	= lim_t → 0^+F(G^-1(G(t)))G(t)
        = lim_p → 0^+F(G^-1(p))p
    	= lim_p → 0^+ROC(p)p.



Equalities (<ref>) and (<ref>) come from <cit.>; the equality between (<ref>) and (<ref>) is a result of the definition of the empirical distribution function <cit.>; and, the approximate equality between (<ref>) and (<ref>) results from the convergence of the empirical distribution function to the true distribution function as K and L get arbitrarily large, and from the assumption that K and L grow at the same rate given that π(ℳ=2) = π(ℳ=1). 

The relationship expressed between Equations (<ref>) and (<ref>) shows that the ABC Bayes factor for two alternative models of interest can be assigned using the ratio between ROC(p) and p as the rate of false positives in favour of model 1 approaches 0. This notable result allows us to express the convergence of the ABC Bayes factor as a function of the rate of false positives in favour of model 1. Our result has significant practical implications when it comes to using ABC for model selection:


  * Our solution allows to monitor the convergence of the output of the algorithm to BF_η as function of a single, well-defined, measure, p, that only depends on the data generated under one of the considered models, as opposed to t which depends on both models and is usually set arbitrarily. 

  * Our solution is less sensitive to the curse of dimensionality as it does not require any of the Δ{η(D),·} to be close to 0. Indeed, our solution only considers the relative ranks of the Δ{η(D),·} calculated for the data generated under models 1 and 2.

  * Since our solution is less sensitive to the curse of dimensionality, it can accommodate vectors of summary statistics of any length. It only requires the design of a kernel function that ensures that the distributions of Δ{η(D),·} are well-separated under the competing models. This point is similar to the one made by <cit.>, except that the separation, in our solution, can be studied on the real line as opposed to a high-dimensional space as suggested in <cit.>.

  * Our solution uses the entire amount of pseudo-data generated and does so in a computationally efficient manner. 

  * Finally, our solution allows to formally preserve the user's priors on the model indices. 


Our solution requires estimation of the ROC curve, which is the topic of the next two sections.
 



 §.§ Empirical ROC



We can leverage the relationship between Equations (<ref>) and (<ref>) to assign an ABC Bayes factor in several ways. Our first method is purely data driven and uses the following approximation of the ratio in Equation (<ref>):


    ROC(p)p  = ∑^N_i=1𝕀[ℳ^(i)=1]·𝕀[Δ{η(D),η(D^∗(i))}≤ t]N∑^N_i=1𝕀[ℳ^(i)=2]·𝕀[Δ{η(D),η(D^∗(i))}≤ t]N.


We see that by defining t such that ∑^N_i=1𝕀[ℳ^(i)=2]·𝕀[Δ{η(D),η(D^∗(i))}≤ t] is constant for any N, and by increasing the number of simulations, the ratio in the denominator of Equation (<ref>) will be driven to 0; hence, approximating the limit as p → 0^+ in Equation (<ref>). This approach has several major advantages as compared to current practice: 


	
  * t is chosen only as a function of the distance scores between the value of the summary statistic of the observed data and the value of the summary statistic of the data generated under model 2 (versus all distance scores in other implementations of the ABC algorithm).
	
  * t is chosen such that the number of accepted distance scores under model 2 is fixed (versus a fixed value of t arbitrarily close to 0, or a varying value of t based on a fixed quantile of the empirical distribution of all the scores from models 1 and 2 combined).


For a given set of observed data, all current implementations of the ABC algorithm result in unpredictable variations of both the numerator and the denominator of the ratio in Equation (<ref>) as N increases, which makes its convergence difficult to monitor. By fixing the rate of convergence of the denominator in Equation (<ref>), our approach has the potential to better plan computing resources and monitor convergence.





 §.§ The non-central dual beta ROC model



Our second approach extends further the relationship between the ABC Bayes factor and the ROC curve by noting that <cit.>:  


    lim_p → 0^+ROC(p)p   = lim_p → 0^+d/dpROC(p) 
       = lim_p → 0^+d/dp F(G^-1(p)) 
       = lim_p → 0^+f(G^-1(p))g(G^-1(p))


where f(·) and g(·) denote the probability density functions of distance scores under models 1 and 2, respectively. 

Assigning an ABC Bayes factor using Equation (<ref>) requires evaluation of the first derivative of ROC(p), as p → 0^+, or the ratio of densities, f(·)/g(·), as p → 0^+. 

Following <cit.>, <cit.>, and <cit.>, we choose to fit a semi-parametric model to the empirical ROC curve obtained from the finite number of Δ{η(D), ·} generated by the algorithm. As noted by these authors, fitting a model to each of the score distributions makes the assumption that the scores follow this particular model. Instead, fitting a model directly to the ROC curve relies on the weaker assumption that there exists a monotonic transformation of the observed scores that results in scores whose distributions can be described by a simple model. Since the ROC curve is invariant to monotonic increasing transformations of the scores <cit.>, the simple model can be used to represent the ROC curve without knowledge of the underlying score distributions.

The common binormal representation of the ROC was considered <cit.>, however it can easily be shown that the limit of the binormal model as p → 0^+ is either taking a value of 0, 1 or ∞. Instead we use a model based on two non-central beta distributions <cit.>. Placing a restriction on the first shape parameter of each distribution (α_F=α_G=1) results in the following semi-parametric model for the ROC curve


    ROC(p)=F(G^-1(p  |  α_G=1, β_G, λ_G)  |  α_F=1, β_F, λ_F)


where F(·) and G(·) are non-central beta distribution functions with parameters α_F, β_F, λ_F, and α_G, β_G, λ_G corresponding to F and G respectively. Note that contrary to current ABC methods for model selection based on pattern recognition algorithm, our method only requires the estimation of four parameters. The restriction on the first shape parameter of the densities guarantees that their ratio, as p → 0^+, produces a stable result for all parameter values within the support: 


    lim_p → 0^+d/dpROC(p)
    	= lim_t → 0^+f(t)/g(t)
    	= exp{-1/2λ_F}B(1,β_F)B(1,β_G)exp{-1/2λ_G}.


Fitting the semi-parametric model requires the use of numerical optimisation techniques to estimate the parameter values. We use a two step fitting procedure:


	
  * Obtain an initial set of parameter estimates for β_F, λ_F,β_G, λ_G using a Maximum Likelihood Estimation approach for fitting a dual beta ROC model to continuous data <cit.>.
	
  * Refine these estimates using an objective function to minimise the L^2 norm between the semi-parametric model and the empirical ROC.	


The first step of our procedure enables us to include information on which model produced each distance score Δ{η(D), η(D^∗(i))}. However, during our implementation we found that the fit of the semi-parametric model to the empirical ROC curve could be improved from this method. Meanwhile, the L^2 norm objective function alone did not allow us to fit an adequate model when the score distributions overlapped heavily. Combining both processes gave the best results. 

Once estimates for β_F, λ_F, β_G, and λ_G are obtained, it is trivial to use Equation (<ref>) to assign the ROC-ABC Bayes factor. In practice, when a limited number of distance scores near 0 are observed, the quality of the fit of the ROC model near 0 can produce Bayes factors of meaningless magnitude (e.g., larger than 10^100 or smaller than 10^-100); these Bayes factors would vary wildly from one computation to another using the same observed data. We found that bounding the rate of false positives, p, to some low value, such as 10^-5, before evaluating Equation (<ref>) produced more robust ABC Bayes factors.





§ WEIGHT OF FINGERPRINT EVIDENCE USING ROC-ABC
 

ABC for model selection possesses some obvious similarities with the algorithm proposed by <cit.>. 
To assign a Bayes factor to an observed fingermark, Neumann et al.'s  algorithm considers two sets of scores. The first set contains scores measuring the level of dissimilarity between the observed fingermark and pseudo-fingermarks generated by Mr. X. The second set contains scores measuring the dissimilarity between the observed fingermark and pseudo-fingermarks originating from a sample of individuals from a population of potential alternative sources. The general idea behind <cit.>'s model is that, if H_1 is true, Mr. X. would generate many more pseudo-fingermarks that are similar to the observed fingermark than the individuals in the population of alternative sources. If we define:

	
  * D using the observed fingermark, e_u; 
	
  * models 1 and 2 in Equations (<ref>) and (<ref>) as the prosecution model and the defence model, respectively;
	
  * generative models corresponding to M_1 and M_2;
	
  * a kernel function, Δ{·,·}, to compare pairs of finger impressions;


we obtain an ROC-ABC algorithm to approximate a Bayes factor for fingerprint evidence. The ROC-ABC algorithm for fingerprint evidence is summarised in Algorithm <ref>. It will converge to the Bayes factor under the same conditions as discussed above (i.e. sufficient statistic across all models, optimal distance metric, infinite/large number of simulations, etc.). 






 §.§ Generation of pseudo-fingermark data



Implementation of the ROC-ABC algorithm for fingerprint evidence requires a model from which pseudo-fingermarks can be generated. We utilise the same fingerprint distortion model as <cit.> to generate pseudo-fingermarks from any given control print. This model mimics the way fingerprint features are displaced as the skin on the tip of a finger is distorted when pressed against a flat surface. The parameter space of the model represents a wide variety of distortion directions and pressures. The model allows many different distortions to be produced from a single finger. Our distortion model assumes that fingers distort in the same way, independently of factors related to the donor (e.g., age, weight, profession) and to the finger number (e.g., right vs. left hand, thumb vs. index finger). The model obviously does not cover all possible distortion and pressure conditions, and donor related factors; however, it is currently the only option to obtain a large number of pseudo-fingermarks from any given individual.  




  §.§.§ Generation of pseudo-fingermarks under the prosecution model



When comparing fingerprints, an examiner first detects k features of interest on the fingermark. Second, the examiner compares it to the 10 control prints from the donor considered by the prosecution proposition and selects the finger appearing to be the most likely source of the mark. Finally, the examiner attempts to identify the most similar k corresponding features out of the n features present on the control print from the selected finger. Note that, once selected, the sets of features on the fingermark and the control print remain fixed for the duration of the experiment, and that the selection process results in a unique bijective pairing between the two sets of k features. Our algorithm assumes that this selection and pairing process has been done before generating pseudo-fingermarks under M_1. When M_1 is selected by the algorithm, a pseudo-fingermark is generated from the k features selected on the control print using the distortion model. By construction, the features on this pseudo-fingermark have the same pairing with the features of the fingermark as the ones on the control print. The uncertainty on the type of the feature was modelled as described in <cit.>. By repeating this process each time M_1 is selected, we can obtain a set of pseudo-fingermarks from the k features selected on the control print of Mr. X. 





  §.§.§ Generation of pseudo-fingermarks under the defence model



The defence proposition considers that, if the fingermark was not left by Mr. X, it must have been left by another person in a relevant population of alternative sources. Assuming that fingerprint patterns result from a completely random process during the development of the foetus, we can generate data under M_2, first, by randomly selecting an individual from any representative sample of donors from the human population, and secondly, by generating a pseudo-fingermark from this individual's k minutiae configuration that is most similar to the k minutiae observed on the fingermark. As in the previous section, the features of this  pseudo-fingermark have a unique bijective pairing with the features on the fingermark by construction. This process is repeated each time M_2 is selected to obtain a random set of pseudo-fingermarks from the population of potential donors considered by M_2. Since it would be unrealistic to repeat manually the selection of the most similar k minutiae for each individual in a large sample, we use a commercially available automated fingerprint matching system to perform this task.  





 §.§ Summary statistic

Configurations of minutiae can be described numerically in the form of heterogeneous multi-dimensional random vectors containing the measurements summarised in Table <ref>. 





Minutiae locations and orientations are taken with respect to a coordinate system based on the frame of the impression's picture (Figure <ref> (a)). Different framings of the same impression result in different measurements for the same set of features. For this reason, the original measurements need to be summarised in a way that is rotation and translation invariant. 

 

Several invariant measurements capturing the spatial relationships between the minutiae in a configuration can be calculated, such as the distances between every pair of minutiae in the configuration (Figure <ref> (b)) and the distance of each minutiae from the centroid of the configuration (average of Cartesian coordinates of the minutiae in the configuration) (Figure <ref> (c)). A similar approach can be used to define invariant summaries of the direction of each feature by using fixed-length segments to represent minutiae directions and by taking the distances between the ends of these segments for every pair of minutiae in the configuration (Figure <ref> (d)). We also choose to represent minutiae directions as a function of the axes going from the centroid through the location of each minutia (angles are measured counterclockwise) (Figure <ref> (e)). Feature types can be directly compared between configurations without the need to summarise since types are rotation and translation invariant.

All of these measurements can be brought together to create a vector of summary statistic of the original representation. For the example in Figure <ref> with 7 features, a vector of summary statistic would include 21 cross-distances between pairs of minutiae, 7 distances between minutiae and the configuration's centroid, 21 cross-distances to capture the spatial representation of the minutiae directions, 7 angles and 7 types, for a total length of 63. For 10 features, the length of the vector of summary statistics would be 120; and for 15 features, it would be 255.

Given the heterogeneity and dimension of the measurements, it is unlikely that a sufficient summary statistic exists for fingerprint data. Here we adopt the approach which consists in pooling together as many individual summary statistics as possible in order to minimise the loss of information with respect to the original data. The curse of dimensionality is not a problem for our algorithm as discussed in Section <ref>. Furthermore, we note that, given that the model used to generate pseudo-fingermarks under M_2 is an extension of the model used under M_1, as our pooled summary statistic tends to sufficiency we can assume that it will be sufficient to compare between M_1 and M_2 <cit.>. Nevertheless, the vectors of summary statistics described above were designed to illustrate the concept of the ROC-ABC in the context of fingerprints and can certainly be improved upon through further investigation.





 §.§ Kernel function


Our ABC algorithm depends on a kernel function, Δ{·,·}, which compares pairs of summarised configurations of minutiae. As mentioned in Section <ref> and according to <cit.>, we wish to use a kernel function that best separates the distributions of Δ{η(e_u),·} obtained under the competing models considered in Section <ref>. 

We developed a metric to compare pairs of summarised configurations of minutiae, and optimised the weights of several components to best separate the two distributions of Δ{η(e_u),·}. For completeness, we have included this development and optimisation process in Appendix <ref>; however, we stress that other summary statistics, kernel functions and optimisation procedures could be considered without loss of generality of our proposed ROC-ABC method.





 §.§ Number of simulations


For the purpose of this application, we limited the total number of pseudo-fingermarks generated for each test configuration to 500,000 (approximately 250,000 under each model). Assuming that we are interested in assigning the ROC-ABC Bayes factor for a specific fingermark in forensic casework, a much larger number of pseudo-fingermarks can be generated.





§ DATASETS


The algorithm has been developed, optimised, and tested using several datasets. Each dataset comprises fingermarks or control prints captured digitally at 1:1 magnification and a resolution of 500 pixels per inch. All features were either labelled manually by an experienced fingerprint examiner or determined automatically by the encoding algorithm of the automated fingerprint matching system used in this study. The following features were extracted from every fingerprint used in the study:



  * the finger of origin of the impression, from 1 - right thumb - to 10 - left little finger (for control prints only);

  * the Cartesian coordinates of each minutia in pixels, using the bottom left of the image as the origin;

  * the direction of each minutia in radians, using the bottom left of the image as the origin and measuring counterclockwise; 

  * the type of each minutia: ridge ending, bifurcation or unknown;

  * the Cartesian coordinates of the centre of the impression (for control prints only). 	


The four datasets (A, B, C, D) used in this study are described in Appendix <ref>.




§ RESULTS


Three experiments were performed using 4067 trace configurations ranging from 3 to 25 minutiae and sampled from 207 fingermarks. For each trace configuration, we consider in turn that:


  [1.1cm]TS: Mr. X (the donor of the control configuration) is the true source of the trace configuration (dataset B);

  [1.1cm]CNM: Mr. X is selected based on the similarity of his fingerprints to the trace configuration (dataset C);

  [1.1cm]RS: Mr. X is randomly selected in a population of donors (dataset D).

In these experiments, the control configurations in datasets B to D were used to resample pseudo-fingermarks using M_1, and dataset A was used to resample pseudo-fingermarks using M_2. Results for all three experiments can be found in Figures <ref> and <ref>. ROC-ABC Bayes factors presented in Figure <ref> were assigned using the empirical ROC method discussed in Section <ref>. Figure <ref> presents ROC-ABC Bayes factors that were assigned using the non-central dual beta ROC model described in Section <ref>. Note that the vertical axis in <ref> has been truncated to focus on the mass of the distributions and that some extreme outliers may not be represented.





In experiment TS, the control prints originate from the true sources of the marks, and so H_1 is true.  Both figures show a similar behaviour where the magnitude of the ROC-ABC Bayes factor increases as the number of minutiae increases. In both cases, the ROC-ABC Bayes factors appear bounded. The bound for the empirical ROC-ABC Bayes factors stems from Equation (<ref>), which requires us to set a value for p. In this case, we define p as the false positive rate evaluated at the 10^th smallest Δ{η(e_u),·} generated under H_2. This corresponds to approximately 1 in 25,000. The bound for the non-central dual beta ROC model stems from using the same p as in the empirical model. Bayes factors erroneously supporting H_2 are noted in both series of results for smaller configurations of minutiae (3 to 7 minutiae), which is not surprising as these configurations contain less discriminative information and are more likely to be observed in fingers from different individuals. Nevertheless, we note that only a handful of ROC-ABC Bayes factors support the wrong proposition for larger configurations. These cases deserve further investigation; at this time, we believe that they are related to configurations displaying unusual distortion that cannot be handled by the current generation of the distortion algorithm. Improvement of the summary statistic, kernel function, and distortion algorithm may be able to minimise further the number of cases where the Bayes factor erroneously supports H_2. Finally, we observe that the range of values taken by the ROC-ABC Bayes factor for different configuration sizes overlap which supports the observations made by <cit.> that there is no scientific justification for the use of a fixed number of minutiae as a decision point to distinguish between H_1 and H_2, and that the evidential value of each configuration needs to be quantified based on its own characteristics. 

In experiment CNM, the control prints originate from non-mated sources that are chosen due to their similarity to the traces, and so H_2 is true. In both Figures <ref> and <ref>, we observe that a large proportion of the ROC-ABC Bayes factors erroneously support the hypothesis of common source, H_1, although the algorithm using the empirical ROC appears to perform significantly better. The high rate of misleading evidence is not a surprise for low numbers of minutiae since it is not difficult to find multiple similar configurations on different fingers in large a dataset. The high rate of misleading evidence is more surprising for larger configurations. Larger values of the ROC-ABC Bayes factor when H_2 is true occur when the kernel function used by the algorithm considers the pseudo-fingermarks generated using M_1 more similar to the observed fingermark than they really are. As mentioned before, improvement of the summary statistic and the kernel function should significantly reduce the rate of misleading evidence in favour of H_1. In practice, we believe that examiners comparing close non-matching finger impressions would be able to exclude that they originate from a common source by visual inspection using friction ridge characteristics that are not taken into account by our metric. 

In experiment RS, the control prints are obtained from randomly selected sources, and so H_2 is also true. In both cases the majority of observations correctly support the hypothesis of different sources, H_2. As in the second experiment, the algorithm using the empirical ROC curve significantly outperforms the one based on the non-central dual beta model of the ROC curve. 

Overall, we note that the semi-parametric modelling of the ROC curve needs to be improved. In many situations, it appears that the dual-beta model of the ROC curve is not optimal and that mixtures of beta distributions would better represent the data. 

These experiments were repeated using the logistic regression method by <cit.> (see Appendix <ref>). Results can be found in Figure <ref> of Appendix <ref>. It is important to note that the logistic method does not present an obvious upper bound for the results in experiment TS and assigns Bayes factors with notably large magnitudes. This property is not desirable since it may lead to unrealistic magnitude of support. 

A comparison of the computational times for the empirical and parametric ROC methods and the logistic method is presented in Figure <ref> of Appendix <ref>. The empirical ROC-ABC method was without rival in terms of computation time. The logistic method performed at a much slower rate, and the difference between the two methods increases with the dimensionality of the data. When compared to a widely used ABC algorithm, we find that our method provides the high computational efficiency that is necessary to provide real-time calculation of the weight of forensic evidence in casework.





§ DISCUSSION


The contribution of this paper is twofold. Firstly, we have proposed a method to rigorously quantify the weight of fingerprint evidence using the formal statistical framework provided by ABC. Secondly, our ROC-ABC algorithm helps with some of the issues commonly associated with ABC algorithms. 

Overall, our results are consistent with the results presented by <cit.> while capturing the user's belief about the parameters of the two competing models and providing an alternative to the weighting function that they proposed:

	
  * The probability of misleading evidence in favour of the defence decreases dramatically as the number of minutiae increases.
	
  * The probability of misleading evidence in favour of the prosecutor is very low for configurations greater than 7 minutiae, when the donor has been randomly selected. As expected, this probability is higher when the donor has been selected based on the similarity of its fingerprints with the fingermark. Improvements in the summary statistic used to describe fingerprint patterns and in the kernel function used to compare them, together with the ability of fingerprint examiners to account for more discriminative information than our model, will certainly reduce the rate of misleading evidence in favour of the prosecutor in an operational implementation of the model.
	
  * The overlap between the ranges of values of the ROC-ABC Bayes factor across different numbers of minutiae confirms that the use of the number of corresponding minutiae is only one of the criteria for inferring the identity of the source, and that the contribution of additional information regarding fingerprint pattern needs to be taken into account when determining the donor of a fingermark. 


In addition to the straightforward operational implementation of our ROC-ABC approach, Figures <ref> and <ref> illustrate the powerful visual representation of the probative value of fingerprint comparisons that is offered by the ROC-ABC algorithm. In Figure <ref>, the latent print originates from same source as the control print and BF_abc=2.51 × 10^7. The scores distributions and the ROC curve in the middle and right panels intuitively show that the data supports the hypothesis that the trace and control prints originate from the same source. In contrast, Figure <ref> shows a situation where the same latent print as in Figure <ref> is compared to a randomly selected control print. In this scenario, BF_abc=3.56 × 10^-153. The clear support of the algorithm for the hypothesis that the latent and control impressions do not originate from the same source can be seen using the scores distributions and ROC curve in the middle and right panels. We believe that this intuition can easily be conveyed to jurors and other factfinders. 





We do not claim that the kernel function that was used in this paper is optimal. It is worth exploring adaptive kernels that maximise the separation between the pseudo-data generated by both models in any specific case. Nevertheless, while the summary statistic and the metric used to generate the results presented in this paper can be improved upon, they are adequate to show the potential of the concept of the ROC-ABC algorithm. Operational implementation of the method would require further studies of the repeatability of the values obtained by the algorithm as a function of different samples (and different sample sizes) of the population of potential donors considered by H_2. 

Our proposed modification to the standard ABC model selection algorithm results in several improvements. Our approach, based on properties of the ROC curve, transforms the convergence of the algorithm into a function of the rate of false positives in favour of the model considered in the numerator of the Bayes factor. Most current implementations of ABC for model selection result in unpredictable variations of both the numerator and the denominator of the ABC Bayes factor as the number of simulations, N, increases, which makes the convergence of the algorithm more difficult to monitor. In our approach, the tolerance level, t, for a given data set is chosen such that the number of accepted samples under the model considered in the denominator of the Bayes factor is fixed for all N. Hence, as N increases, the approximation of the limit as the rate of false positives goes to 0 improves. Our approach has the potential to better plan computing resources. Critically, our method allows for rigorously monitoring convergence.

The shift from tolerance level on Δ{η(D),·} to rate of false positives in favour of model 1 does not require any of the Δ{η(D),·} to be close to 0. Instead, only the relative ranks of the Δ{η(D),·} calculated for the data generated under models 1 and 2 are considered. This implies that the kernel function used to assess level of similarity can accommodate vectors of summary statistics of any length, without encountering the curse of dimensionality because there is no need for any of the scores to be close to 0. Our algorithm only requires that the distributions of Δ{η(D),·} are well-separated under the competing models. This point is similar to the one made by <cit.>, except that the separation, in our solution, can be studied on the real line as opposed to a high-dimensional space as suggested in <cit.>. In addition to avoiding the curse of dimensionality, our method is also able to process the entire amount of pseudo-data generated in a computationally efficient manner and does not require filtering the pseudo-data: as the dimension of the summary statistic vector increases, the time required to assign Bayes factors using other methods (such as the logistic regression approach) increases exponentially (Figure <ref>). Finally, our solution allows to formally preserve the user's priors on the model indices. 

We have not formally addressed the issue of the sufficiency of the summary statistic that is required for the convergence of the ABC Bayes factor to the Bayes factor. This convergence is extremely important in some contexts, such as forensic science, where fact-finders are as equally interested in the proposition supported by the Bayes factor as in the magnitude of this support. Since the method that we propose is not affected by the curse of dimensionality, it permits including as much information in the summary statistic vectors as needed as in <cit.>. 

To implement our approach in practice, we have proposed two methods to assign ROC-ABC Bayes factors. Our results show significant differences in performance between the empirical ROC and the dual beta ROC method. The empirical model appears to produce more stable and meaningful results (i.e., ABC Bayes factors with reasonable magnitude). As we increase the number of simulations, the empirical model naturally approximates the limit p → 0^+ in Equation (<ref>). The non-central dual beta ROC model has the potential to explore the limit when p → 0^+ with a smaller sample size. In practice, we observed that the values obtained using Equation (<ref>) for multiple runs of the ROC-ABC algorithm for a given set of observed data differ greatly from one another (many orders of magnitude on the log_10 scale). It appears that Equation (<ref>) is very sensitive to small changes in the values of the estimates of the model's parameters. Instead, we tried to fix p=1/25,000 in Equation (<ref>) to obtain more robust values and generate the data in Figure <ref>. Our results show that in several cases our algorithm does not support the correct model; this may be due to our modelling of the ROC curve in the neighbourhood of 0 not being an accurate representation of the data. Once again, this shows that the non-central dual beta ROC model is very sensitive to small changes in the estimates of its parameters. Improvements can be made to the fitting procedure for the non-central dual beta ROC model, such as an explicit monotonic increasing transformation of the distance scores to initiate the numerical optimisation procedure with distributions that are closer to the assumed model; alternatively, other models whose limits at 0 exist can be investigated.




§ CONCLUSION


In this paper, we propose an algorithm to formally and rigorously assign Bayes factors to forensic fingerprint evidence. Our modified ABC model selection algorithm was used to address several criticisms of the model proposed by <cit.> by framing the problem into a formal Bayesian framework. The results presented here show that our method is promising, with low rates of misleading evidence, and has the potential to be applied to many other complex, high-dimension evidence forms such as shoe prints, questioned documents, firearms, and traces characterised by analytical chemistry. Ultimately, the widespread use of statistical approaches to quantify the weight of forensic evidence to replace the existing inference paradigm can only be enabled by technology providers offering commercial products to the forensic community. Our method leverages currently available technology that was designed to search forensic traces into large databases and retrieve the most likely candidates. For mainstream evidence types such as fingerprints, firearms, and shoe impressions, our algorithm can readily be implemented, validated, and integrated in current commercial offerings. Furthermore, we note that the use of ROC curves in the algorithm will be naturally familiar to engineers and scientists designing these systems, which may facilitate the implementation of our method in commercial systems. 





As an added benefit, our algorithm addresses several shortcomings of current ABC model selection methods. We use the properties of the Receiver Operating Characteristic curve to address the issue of choosing a suitable tolerance level when assigning ABC Bayes factors. Our modification allows for a natural convergence of the algorithm as the number of simulations increases, and for monitoring this convergence as a function of the sole rate of false positives in favour of the model considered in the numerator of the Bayes factor. Focusing on the rate of false positives (rather than  the tolerance level) allows our method to rely on the ordering of kernel scores, rather than the magnitudes of scores, and thus, is not subject to the curse of dimensionality. In addition, our method considers the entire amount of pseudo-data generated under the considered models in a computationally efficient manner. 




86




[Abraham et al.2013]Abraham2013
[author]
Abraham, J.J.,
  Champod, C.C.,
  Lennard, C.C. Roux, C.C.
(2013).
Modern statistical models for forensic fingerprint examinations: a
  critical review.
Forensic Science International
232
131-150.



[Aitken, Roberts and
  Jackson2010]Aitken2010
[author]
Aitken, C.C.,
  Roberts, P.P. Jackson, G.G.
(2010).
1. Fundamentals of Probability and Statistical Evidence in Criminal
  Proceedings.
In Communicating and Interpreting Statistical Evidence in the
  Administration of Criminal Justice, Guidance for Judges, Lawyers, Forensic
  Scientists and Expert Witnesses
Royal Statistical Society.



[Aitken and Taroni2004]AitkenTaroni:2004
[author]
Aitken, C. G. G.C. G. G. Taroni, F.F.
(2004).
Evaluation of Evidence for Forensic Scientists,
2nd ed.
Wiley and Sons Ltd, Chichester.



[Ashbaugh1999]Ashbaugh:1999
[author]
Ashbaugh, D. R.D. R.
(1999).
Quantitative-Qualitative Friction Ridge Analysis: an Introduction to
  Basic and Advanced Ridgeology.
CRC Press.



[Balding2012]Balding2012
[author]
Balding, David J.D. J.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Beaumont2008]Beaumont2008
[author]
Beaumont, M. A.M. A.
(2008).
Joint determination of topology, divergence time, and immigration in
  population trees.
In Simulation, Genetics and Human Prehistory,
(S.S. Matsumura,
  P.P. Forster C.C. Renfrew, eds.).
McDonald Institute Monographs
14,
135-154.
Cambridge McDonald Institute for Archeological Research,
  UK.



[Beaumont2010]Beaumont2010
[author]
Beaumont, M. A.M. A.
(2010).
Approximate Bayesian Computation in Evolution and Ecology.
Annual Review of Ecology, Evolution, and Systematics
41
379-406.



[Beaumont, Zhang and
  Balding2002]Beaumont2002
[author]
Beaumont, M. A.M. A.,
  Zhang, W.W. Balding, D. J.D. J.
(2002).
Approximate Bayesian Computation in Population Genetics.
Genetics
162
2025-2035.



[Bertillon1912]Bertillon:1912
[author]
Bertillon, M. A.M. A.
(1912).
Les empreintes digitales.
Archives d'Anthropologie Criminelle de Lacassagne
27
36-52.



[Blum2019]Blum2019
[author]
Blum, M.M.
(2019).
Regression approaches for ABC.
In Handbook of Approximate Bayesian Computation
(S. A.S. A. Sisson,
  Y.Y. Fan M. A.M. A. Beaumont, eds.)
3,
71-85.
CRC Press.



[Blum and Francois2010]Blum2010
[author]
Blum, M.M. Francois, O.O.
(2010).
Non-linear regression models for Approximate Bayesian Computation.
Statistics and Computing
20
63-73.



[Bookstein1989]Bookstein:1989
[author]
Bookstein, F. L.F. L.
(1989).
Principal Warps: Thin-Plate Splines and the Decomposition of
  Deformations.
IEEE Transactions on Pattern Analysis and Machine Intelligence
11
567-585.



[Champod1995]Champod:1995
[author]
Champod, C.C.
(1995).
Reconnaissance Automatique et Analyse Statistique des Minuties sur les
  Empreintes Digitales,
PhD thesis,
University of Lausanne.



[Champod, Evett and
  Jackson2004]ChampodEvettJackson:2004
[author]
Champod, C.C.,
  Evett, I. W.I. W. Jackson, G. A.G. A.
(2004).
Establishing the most appropriate databases for addressing source level
  propositions.
Science and Justice
44
153-164.



[Chen and Hu2016]ChenHu2016
[author]
Chen, W.W. Hu, N.N.
(2016).
Proper bibeta ROC model: algorithm, software, and performance
  evaluation.
In SPIE Medical Imaging
97870E–97870E.
International Society for Optics and Photonics.



[Cole2004]Cole:2004
[author]
Cole, S. A.S. A.
(2004).
Grandfathering evidence: fingerprint admissibility rulings from
  Jennings to Llera Plaza and back again.
American Criminal Law Review
41
1189-1276.



[Cole2005]Cole:2005
[author]
Cole, S. A.S. A.
(2005).
More than zero, accounting for error in latent fingerprint
  identification.
Journal of Criminal Law and Criminology
95
985-1078.



[Cole2009]Cole:2009
[author]
Cole, S. A.S. A.
(2009).
Forensics without uniqueness, conclusions without individualization:
  the new epistemology of forensic identification.
Law, Probability and Risk
In press.



[Cowell2012]Cowell2012
[author]
Cowell, RobertR.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Didelot et al.2011]Didelot2011
[author]
Didelot, X.X.,
  Everitt, R. G.R. G.,
  Johansen, A. M.A. M. Lawson, D. J.D. J.
(2011).
Likelihood-free estimation of model evidence.
Bayesian Analysis
6
49-76.



[Egli, Champod and Margot2006]Egli:2006
[author]
Egli, N.N.,
  Champod, C.C. Margot, P.P.
(2006).
Evidence evaluation in fingerprint comparison and automated fingerprint
  identification systems - modelling within finger variability.
Forensic Science International
176
189-195.



[Estoup et al.2012]Estoup2012
[author]
Estoup, A.A.,
  Lombaert, E.E.,
  Marin, J. M.J. M.,
  Guillemaud, T.T.,
  Pudlo, P.P.,
  Robert, C. P.C. P. Cornuet, J. M.J. M.
(2012).
Estimation of demo-genetic model probabilities with Approximate
  Bayesian Computation using linear discriminant analysis on summary
  statistics.
Molecular Ecology Resources
12
846-855.



[Evett and Williams1996]EvettEtAl:1996
[author]
Evett, I. W.I. W. Williams, R. L.R. L.
(1996).
A review of the sixteen point fingerprint standard in England and
  Wales.
Journal of Forensic Identification
46
49-73.



[Evett et al.2000]EvettEtAl:2000
[author]
Evett, I. W.I. W.,
  Foreman, L. A.L. A.,
  Jackson, G.G. Lambert, J. A.J. A.
(2000).
DNA profiling: a discussion of issues relating to the reporting of
  very small match probabilities.
Criminal Law Review
341-355.



[Faulds1880]Faulds:1880
[author]
Faulds, H.H.
(1880).
On the skin-furrows of the hand.
Nature
22
605-605.



[Fotopoulos2012]Fotopoulos2012
[author]
Fotopoulos, Stergios B.S. B.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Frazier et al.2017]Frazier2017
[author]
Frazier, D. T.D. T.,
  Martin, G. M.G. M.,
  Robert, C. P.C. P. Rousseau, J.J.
(2017).
Asymptotic Properties of Approximate Bayesian Computation
Technical Report No. arXiv:1607.06903v3.



[Galton1892]Galton:1892
[author]
Galton, F.F.
(1892).
Finger Prints.
Wiley.



[Gonzalez-Rodriguez
  et al.2006]Gonzalez:2006
[author]
Gonzalez-Rodriguez, J.J.,
  Drygajlo, A.A.,
  Ramos-Castro, D.D.,
  Garcia-Gomar, M.M. Ortega-Garcia, J.J.
(2006).
Robust estimation, interpretation and assessment of likelihood ratios
  in forensic speaker recognition.
Computer Speech and Language
20
331-355.



[Good1950a]Good1950
[author]
Good, I. J.I. J.
(1950a).
Probability and Weighing of the Evidence.
Charles Griffin & Co., Ltd., London.



[Good1950b]Good:1950
[author]
Good, I. J.I. J.
(1950b).
Probability and the Weighting of Evidence.
Charles Griffin, London.



[Good1983]good:1983
[author]
Good, I. J.I. J.
(1983).
Weight of Evidence: A Brief Survey.
In Bayesian Statistics 2
(J. M.J. M. Bernardo,
  M. H.M. H. Degroot,
  D. V.D. V. Lindley A. F. M.A. F. M. Smith, eds.)
249-269.
Second Valencia International Meeting on Bayesian Statistics,
  Valencia, Spain, 6-10 September 1983.
Elsevier Science, Amsterdam.



[Graversen2012]Graversen2012
[author]
Graversen, ThereseT.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Grelaud et al.2009]Grelaud2009
[author]
Grelaud, A.A.,
  Robert, C. P.C. P.,
  Marin, J. M.J. M.,
  Rodolphe, F.F. Taly, J. F.J. F.
(2009).
ABC likelihood-free methods for model choice in Gibbs random
  fields.
Bayesian Analysis
4
317-335.



[Henry1905]Henry:1905
[author]
Henry, E. R.E. R.
(1905).
Classification and Uses of Finger Prints.
London.



[Herschel1916]Herschel:1916
[author]
Herschel, W. J.W. J.
(1916).
The Origin of Finger-Printing.
Oxford University Press.



[Indovina et al.2009]NIST:2009
[author]
Indovina, M.M.,
  Dvornychenko, V.V.,
  Tabassi, E.E.,
  Quinn, G.G.,
  Grother, P.P.,
  Garris, M.M. Meagher, S.S.
(2009).
ELFT phase II - An evaluation of automated latent fingerprint
  identification technologies
Technical Report No. NISTIR 7577,
National Institute of Standards and Technology.



[Jandhyala2012]Jandhyala2012
[author]
Jandhyala, Venkata K.V. K.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Johnson, Kotz and
  Balakrishnan1995]Johnson1995
[author]
Johnson, N. L.N. L.,
  Kotz, S.S. Balakrishnan, N.N.
(1995).
Continuous univariate distributions
2.
Wiley, New York.



[Kadane2012]Kadane2012
[author]
Kadane, Joseph B.J. B.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Kaye2003]Kaye2003
[author]
Kaye, D. H.D. H.
(2003).
Questioning a courtroom proof of the uniqueness of fingerprints.
International Statistical Review
71
521-533.



[Kirk1963]Kirk:1963
[author]
Kirk, P. L.P. L.
(1963).
The Ontogeny of Criminalistics.
Journal of Criminal Law, Criminology and Police Science
54
235-238.



[Langenburg2009]Langenburg:2009
[author]
Langenburg, G.G.
(2009).
A performance study of the A.C.E.–V. process: a pilot study to
  measure the accuracy, precision, reproducibility, repeatability and
  biasability of conclusions resulting from the A.C.E.–V. process.
Journal of Forensic Identification
59
219-257.



[Lauritzen2012]Lauritzen2012
[author]
Lauritzen, SteffenS.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Lindley1977]Lindley:1977
[author]
Lindley, D. V.D. V.
(1977).
A problem in forensic science.
Biometrika
44
207-213.



[Locard1914]Locard:1914
[author]
Locard, E.E.
(1914).
La prevue judiciaire par les empreintes digitales.
Archives d'Anthropologie Criminelle, de Medecine Legale et de
  Psychologie Normale et Pathologique
28
312-348.



[Mansfield and
  Wayman2002]MansfieldWayman:2002
[author]
Mansfield, A. J.A. J. Wayman, J. L.J. L.
(2002).
Best Practices in Testing and Reporting Performance of Biometric
  Devices v2.0.1
Technical Report No. NPL report CMSC 14/02,
Center for Mathematics and Scientific Computing, UK National
  Physical Laboratory.
http://www.cesg.gov.uk/policy_technologies/biometrics/media/bestpractice.pdf.



[Margot and Quinche2010]Margot:2010
[author]
Margot, P.P. Quinche, N.N.
(2010).
Coulier, Paul-Jean (1824-1890): A Precursor in the History of
  Fingermark Detection and Their Potential Use for Identifying Their Source
  (1863).
Journal of Forensic Identification
60(2)
129-134.



[Marin and Robert2014]MarinRobert2014
[author]
Marin, J. M.J. M. Robert, C. P.C. P.
(2014).
Bayesian Essentials with R,
Second ed.
Springer Science + Business Media, New York.



[Marin et al.2012]Marin2012
[author]
Marin, J. M.J. M.,
  Pudlo, P.P.,
  Robert, C. P.C. P. Ryder, R. J.R. J.
(2012).
Approximate Bayesian Computational Methods.
Statistics and Computing
1-14.



[Marin et al.2014]Marin2014
[author]
Marin, J. M.J. M.,
  Pillai, N. S.N. S.,
  Robert, C. P.C. P. Rousseau, J.J.
(2014).
Relevant statistics for Bayesian model choice.
Royal Statistical Society, Series B
76
833-859.



[Marin et al.2019]Marin2019
[author]
Marin, J. M.J. M.,
  Pudlo, P.P.,
  Estoup, A.A. Robert, C.C.
(2019).
Likelihood-Free Model Choice.
In Handbook of Approximate BAyesian Computation
(S. A.S. A. Sisson,
  Y.Y. Fan M. A.M. A. Beaumont, eds.)
6,
153-178.
CRC Press.



[Metz, Herman and Shen1998]Metz1998
[author]
Metz, C. E.C. E.,
  Herman, B. A.B. A. Shen, J. H.J. H.
(1998).
Maximum Likelihood estimation of receiver operating characteristic
  (ROC) curves from continuously-distributed data.
Statistics in Medicine
17
1033-1053.



[Mitchell2004]Mitchell:2004
[author]
Mitchell
(2004).
U.S. v. Mitchell 365 F.3d 215 (3d Cir.).



[Mossman and Peng2016]MossmanPeng2016
[author]
Mossman, D.D. Peng, H.H.
(2016).
Using Dual Beta Distributions to Create “Proper” ROC Curves
  Based on Rating Category Data.
Medical Decision Making
36
349-365.



[Neumann, Evett and
  Skerrett2012]Neumann2012
[author]
Neumann, C.C.,
  Evett, I. W.I. W. Skerrett, J.J.
(2012).
Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Neumann et al.2006]Neumann:2006
[author]
Neumann, C.C.,
  Champod, C.C.,
  Puch-Solis, R.R.,
  Meuwly, D.D.,
  Egli, N.N. Anthonioz, A.A.
(2006).
Computation of likelihood ratios in finterprint identification for
  configurations of three minutiae.
Journal of Forensic Sciences
51
1255-1266.



[Neumann et al.2007]Neumann:2007
[author]
Neumann, C.C.,
  Champod, C.C.,
  Puch-Solis, R.R.,
  Egli, N.N.,
  Anthonioz, A.A. Bromage-Griffiths, A.A.
(2007).
Computation of likelihood ratios in fingerprint identification for
  configurations of any number of minutiae.
Journal of Forensic Sciences
52
54-64.



[Neumann et al.2015]Neumann2015
[author]
Neumann, C.C.,
  Champod, C.C.,
  Yoo, M.M.,
  Genessay, T.T. Langenburg, G.G.
(2015).
Quantifying the weight of fingerprint evidence through the spatial
  relationship, directions and types of minutiae observed on fingermarks.
Forensic Science International
248
154-171.



[National Research Council of the National
  Academies2009]NAS:2009
[author]
National Research Council of the National Academies
(2009).
Strengthening Forensic Science in the United States: A Path
  Forward.
The National Academies Press, Washington, D.C.



[Office of the US Inspector
  General2006]OIG:2006
[author]
Office of the US Inspector General
(2006).
A review of the FBI's handling of the Brandon Mayfield Case - Special
  report.
US Department of Justice.



[UK Home Office2005]HOffice:2005
[author]
UK Home Office
(2005).
DNA Expansion Programme 2000–2005: Reporting achievement.
Forensic Science and Pathology Unit, UK Home Office.



[Pepe2003]Pepe2003
[author]
Pepe, M. S.M. S.
(2003).
The Statistical Evaluation of Medical Tests for Classification and
  Prediction.
Oxford University Press Inc., New York.



[Peterson and Hickman2005]DOJ:2005
[author]
Peterson, J. L.J. L. Hickman, M. J.M. J.
(2005).
Census of Publicly Funded Forensic Crime Laboratories 2002
Technical Report,
Bureau of Justice Statistics.



[Plaza2002]LleraPlaza:2002
[author]
Plaza, LleraL.
(2002).
U.S. v. Llera-Plaza Nos C.R. 98–362–10, 11,12.
E.D. PA.



[Prangle2017]Prangle2017
[author]
Prangle, D.D.
(2017).
Adapting the ABC distance function.
Bayesian Analysis
12
289-309.



[Prangle et al.2014a]Prangle2014a
[author]
Prangle, D.D.,
  Fearnhead, P.P.,
  Cox, M. P.M. P.,
  Biggs, P. J.P. J. French, N. P.N. P.
(2014a).
Semi-automatic selection of summary statistics for ABC model choice.
Statistical Applications in Genetics and Molecular Biology
13
67-82.



[Prangle et al.2014b]Prangle2014
[author]
Prangle, D.D.,
  Blum, M.M.,
  Popovic, G.G. Sisson, S.S.
(2014b).
Diagnostic tools for approximate bayesian computation using the
  coverage property.
Australian and New Zealand Journal of Statistics
56
309-329.



[Pritchard et al.1999]Pritchard1999
[author]
Pritchard, J. K.J. K.,
  Seielstad, M. T.M. T.,
  Perez-Lezaun, A.A. Feldman, M. W.M. W.
(1999).
Population Growth of Human Y Chromosomes: A Study of Y Chromosome
  Microsatellites.
Molecular Biology and Evolution
16
1791-1798.



[Pudlo et al.2016]Pudlo2015
[author]
Pudlo, P.P.,
  Marin, J. M.J. M.,
  Estoup, A.A.,
  Cornuet, J. M.J. M.,
  Gautier, M.M. Robert, C. P.C. P.
(2016).
Reliable ABC model choice via random forests.
Bioinformatics
32
859–866.



[Robert2007]Robert2007
[author]
Robert, C. P.C. P.
(2007).
The Bayesian Choice.
Springer.



[Robert et al.2011]Robert2011
[author]
Robert, C. P.C. P.,
  Cornuet, J. M.J. M.,
  M., Marin J.M. J. Pillai, N. S.N. S.
(2011).
Lack of confidence in ABC model choice.
Proceedings of the National Academy of Sciences of the United States
  of America
108
5112-5117.



[Saks and Koehler2005a]Saks2005
[author]
Saks, M.M. Koehler, J. J.J. J.
(2005a).
The coming paradigm shift in forensic identification science.
Science
309.



[Saks and Koehler2005b]SaksKoehler:2005
[author]
Saks, M. J.M. J. Koehler, J. J.J. J.
(2005b).
The coming paradigm shift in forensic identification science.
Science
309
893-894.



[Saks and Koehler2008]SaksKoehler:2008
[author]
Saks, M. J.M. J. Koehler, J. J.J. J.
(2008).
The Individualization Fallacy in Forensic Science Evidence.
Vanderbilt Law Review
61
199-219.



[Sisson, Fan and Beaumont2019]Sisson2019
[author]
Sisson, S. A.S. A.,
  Fan, Y.Y. Beaumont, M. A.M. A.
(2019).
Overview of ABC.
In Handbook of Approximate Bayesian Computation
(S. A.S. A. Sisson,
  Y.Y. Fan M. A.M. A. Beaumont, eds.)
1,
3-54.
CRC Press.



[Stern2012]Stern2012
[author]
Stern, HalH.
(2012).
Comments on Neumann, C., Evett, I. W., Skerrett, J.
  (2012). Quantifying the weight of evidence from a forensic fingerprint
  comparison: a new paradigm.
J. R. Statist. Soc. A
175
371-415.



[Stoney2001]Stoney:2001
[author]
Stoney, D. A.D. A.
(2001).
Measurement of Fingerprint Individuality.
In Advances in Fingerprint Technology
(H. C.H. C. Lee R. E.R. E. Gaensslen, eds.)
327-388.
C.R.C. Press, Boca Raton.



[Taroni, Champod and
  Margot1999]TaroniChampod:1999
[author]
Taroni, F.F.,
  Champod, C.C. Margot, P.P.
(1999).
Forerunners of Bayesianism in Early Forensic Science.
Journal of Forensic Identification
49
285-305.



[Toni and Stumpf2010]ToniStumpf2010
[author]
Toni, T.T. Stumpf, M. P. H.M. P. H.
(2010).
Simulation-based model selection for dynamical systems in systems and
  population biology.
Bioinformatics
26
104-110.



[State v. Dixon2011]State.v.Dixon
[author]
State v. Dixon
(2011).
No. 27-CR-10-3378.
(D. Ct. Cty. Hennepin, Minn.).



[State v. Hull2008]State.v.Hull
[author]
State v. Hull
(2008).
No. 48-CR-07-2336.
(Minn. D. Ct. Cty. of Mille Lacs).



[van der Vaart1998]vanderVaart1998
[author]
van der Vaart, A. W.A. W.
(1998).
Cambridge Series in Statistical and Probabilistic Mathematics:
  Asymptotic Statistics.
Cambridge University Press, Cambridge, UK.



[Wasserman2013]Wasserman2013
[author]
Wasserman, L.L.
(2013).
All of statistics: a concise course in statistical inference.
Springer Science + Business Media.



[Wertheim, Langenburg and
  Moenssens2006]Wertheim:2006
[author]
Wertheim, K.K.,
  Langenburg, G.G. Moenssens, A.A.
(2006).
A report of latent print examiner accuracy during comparison training
  exercises.
Journal of Forensic Identification
56
55-93.



[Zabell2005]Zabell:2005
[author]
Zabell, S. L.S. L.
(2005).
Fingerprint evidence.
Journal of Law and Policy
13
143-170.











§ KERNEL FUNCTION DEVELOPMENT



As mentioned in Section <ref>, we wish to use a kernel function that best separates the distributions of Δ{η(e_u),·} obtained under the competing models considered in Section <ref>. Our kernel function, Δ{·,·}, is a linear combination of several metrics, denoted by Δ_1{·,·}, Δ_2{·,·}, Δ_3{·,·}, Δ_4{·,·}, and Δ_5{·,·}, corresponding to the different summary statistics described above and aimed at capturing differences in spatial relationships, directions, and types of the features: 


    Δ{·,·} = c_1Δ_1{·,·} + c_2Δ_2{·,·} + c_3Δ_3{·,·} + c_4Δ_4{·,·} + c_5Δ_5{·,·},


where c_i, for i ∈{1,2,3,4,5}, are real-valued constants. The components of the metric are described below. We remind the reader that by construction (see Section <ref>) the i^th measurement in e_u is uniquely paired with the i^th measurement in e_u^∗.




 §.§ Components of the distance metric


The first component, Δ_1{·,·}, captures the differences in cross-distances between the locations of the minutiae in a pair of configurations. Denoting the i^th cross-distance from e_u by d_i, and the i^th cross-distance from e_u^∗ by d_i^∗, the first component of the distance metric is given by Δ_1{e_u,e_u^∗} = ( ∑_i=1^k2(d_i-d^∗_i)^2/d_i)^1/2.
The second component, Δ_2{·,·}, captures the difference in the spatial spread of the features. Denoting the distance of the i^th minutiae from the centroid of the configuration from e_u by d_i, and the same from e_u^∗ by d_i^∗, this component is given by Δ_2{e_u,e_u^∗} = ( ∑_i=1^k(d_i-d^∗_i)^2/d_i)^1/2.
The third component, Δ_3{·,·}, takes the same form as Δ_1{·,·}, but instead uses d_i and d_i^∗ as the i^th cross-distance between location markers for feature directions (as illustrated in Figure <ref> (d)) on e_u and e_u^* respectively. This component captures differences in directions of the features.
The fourth component, Δ_4{·,·}, captures the difference in direction between the paired features for two configurations. Denoting the angle (measured in degrees) depicted in Figure <ref> (e) for the i^th minutiae of e_u by θ_i, and the same from e_u^∗ by θ_i^∗, the fourth component is given by 

    Δ_4{e_u,e_u^∗}=
       ∑_i=1^k {[               |θ_i - θ_i^∗|θ_i        if | θ_i - θ_i^∗| ≤ 180; (180 - | θ_i - θ_i^∗|)  180θ_i        if | θ_i - θ_i^∗| > 180;                                ]. .
 

Finally, denoting the type of the i^th minutiae from e_u by τ_i, and the same from e_u^∗ by τ_i^∗, the fifth component is given by Δ_5{e_u,e_u^∗} = (∑_i=1^k 𝕀[τ_i = τ_i^∗] )^1/2.





 §.§ Optimisation of the distance metric

Since our algorithm can accommodate vectors of summary statistics of any length, we are not interested in performing any form of variable selection. However, we are interested in obtaining the “best” separation between the score distributions under the competing models in order to recover the correct model. This argument extend the one made by <cit.>.

Values for all c_i in Equation (<ref>) can be obtained by maximising the separation between the distribution of Δ{·,·}'s from minutiae configurations generated by the same donor, and the distribution of Δ{·, ·}'s from minutiae configurations generated by different donors. To obtain the results presented later in this paper, we used numerical optimisation to maximise the average area under 450 ROC curves obtained from configurations with 5, 8, 12, 17, and 23 minutiae. Each ROC curve was based on 50,000 distance scores as calculated in Equation <ref> and obtained by comparing k minutiae on a fingermark to pseudo-fingermarks resulting from the distortion of the true source of the fingermark (Section <ref>) and from the distortion of other fingers (Section <ref>). Our results indicated that the second and fifth components of our metric did not help discriminate between same-source and different-sources distance scores, and were assigned c_2=c_5=0. Our results also indicated that component 3 was the most useful to maximise the average area under the ROC curve (c_3 = 6.5), followed by component 1 (c_1 = 1) and by component 4 (c_4 = 0.1). 

We stress that other summary statistics, kernel functions and optimisation procedures could be considered without loss of generality of our proposed ROC-ABC method.




§ DATASETS



The data used in this study includes four datasets, described below.




 §.§ Dataset A: relevant population of potential donors


A dataset of control prints taken under controlled conditions from more than 400,000 individuals (only identified through randomly assigned ID numbers) was used as a sample of a population of potential sources. The size of the dataset was not driven by scientific considerations, but corresponds to the number of control prints that the authors managed to gather for research purposes. Locations, directions, and types of fingerprint features were extracted automatically. This dataset was used to generate pseudo-fingermarks under the defence model (see Section <ref>) and to generate special test cases under the prosecution model (see below).




 §.§ Dataset B: True source dataset


A dataset of 207 pairs of fingermark and control print were obtained from casework archives. While the sources of the fingermarks are unknown, a trained fingerprint examiner deemed that, within each pair, the fingermark and control print originate from the same finger. Since the fingermarks originate from casework, they were developed on multiple surfaces using different physicochemical methods and represent a range of fingermarks that can be observed in casework. The fingerprint features on each pair of impressions were manually annotated by the fingerprint examiner. Corresponding features between paired impressions were manually designated by the fingerprint examiner. 
 
Test configurations, ranging from 3 to 25 minutiae, were sampled from the fingermarks (one configuration of each size per fingermark). For each test configuration, the corresponding minutiae on the control print were also selected, thus providing pairs of “matching” configurations truly originating from the same source (or assumed to be). 

The fingermarks described here were used as e_u when generating the results presented below. The corresponding control prints were used as control prints from Mr. X to test the ROC-ABC algorithm when the prosecution proposition, H_1, is true.







 §.§ Dataset C: “Close non-matching" source dataset


The 4067 test configurations generated from the 207 fingermarks from dataset B were automatically compared against the control prints from all individuals in the sample from the population in dataset A. The matching algorithm was set up to select the control configuration that appeared most similar to each test configuration.

These pairs of configurations were used to test the ROC-ABC algorithm when the defence proposition, H_2, is true and Mr. X is not the true source of the fingermark, but possesses a very similar configuration of minutiae. This is equivalent to testing the algorithm under the “worst possible case" scenario, which may correspond to the situation where Mr. X has been brought to the attention of the prosecution as a result of a database search. 




 §.§ Dataset D: “Random" source dataset


The experiment performed to construct dataset C was repeated. However, in this case, the matching algorithm was set up to return random configurations from any of the control prints, regardless of their levels of similarity to the test configurations. 
 
These pairs of configurations were used to test the ROC-ABC algorithm when the defence proposition, H_2, is true and Mr. X is not the true source of the fingermark, and he has not been selected based on the characteristics of his fingerprints. This is equivalent to testing the algorithm in situations where Mr. X has been brought to the attention of the prosecution as the result of evidence unrelated to fingerprint. 




§ RESULTS USING <CIT.>'S METHOD



Results for all three experiments were also generated using the logistic regression method proposed by <cit.> since this method is the basis for most ABC model selection algorithms and has been found to be working well. We use the logistic regression method with the same metric as the one used in our ROC-based ABC algorithm in order to have directly comparable results. We want to re-emphasise that it is critical for the forensic application to have a fair idea of the magnitude of support for a given model, and not only to be able to select the correct one. Thus, we did not compare our method to machine-learning-based ABC methods since they seem to be only focusing on the posterior distributions for the models <cit.>, while Equation <ref> shows that our method can converge to the Bayes factor of interest. 

These results are presented in Figure <ref>. A comparison of the computation times for each of the three methods (empirical ROC, parametric ROC, and logistic regression) is presented in Figure <ref>.


 


The general trend of the results of the logistic regression method are similar to those of the ROC-based methods. For the experiment TS, the magnitude of the ABC Bayes factor increases as the number of minutiae increases, while for experiments CNM and RS, the ABC Bayes factors tend to generally support H_2.

However, the logistic method does not present an obvious upper bound for the results in experiment TS and assigns Bayes factors with notably large magnitudes. Based on the convergence results shown in Equations (<ref>) to (<ref>), we do not believe that the larger magnitude of the Bayes factors in Figure <ref> can be justified by the number of simulations performed in this experiment. We can only conclude that these Bayes factors severely overstate the weight of the evidence observed and generated in these cases. In addition, we note the very large variance of the Bayes factor assigned using the logistic regression method during experiment TS. This large variance results in a high rate of misleading evidence in favour of H_2. This rate is noticeably greater than that from the empirical method. 

Interestingly, results from experiments CNM and RS show that the logistic regression method produces less misleading evidence in favour of H_1 when H_2 is true, even when very similar prints are used. The logistic regression method maximises the separation between the two models by leveraging all of the content of the vectors of summary statistics, while the kernel function described in Section <ref> has been optimised for the average case. The ROC-ABC method may be improved in this aspect by using an adaptable kernel function that would also maximise the separation in each case. 

Overall, while using the logistic regression method as proposed by <cit.>, we noted that the weighting of the pseudo-data prior to fitting the logistic regression model resulted in the removal of large portions, if not all, of the data generated under either M_1 or M_2. As discussed previously, this results in altering the user-defined priors on the model index and replacing them by unpredictable data-driven priors. Furthermore, this led to instability when fitting the logistic regression model.

A comparison of the computation time among the three methods (empirical ROC, parametric ROC, and logistic regression) is presented in Figure <ref>. This computation time represents the time require to assign ABC Bayes factors using the three different methods once the pseudo-data has been generated. The empirical ROC-ABC method was without rival in terms of computation time. Even as data complexity/dimensionality increased, computation time was relatively constant. The logistic method outperformed the parametric ROC-ABC method up until 9 minutiae. At this point, the total computation time for the logistic method continued to increase at an exponential rate while the computation time for the parametric ROC-ABC method remained fairly uniform. This is unsurprising since the computational complexity of the parametric ROC-ABC is driven by the number of univariate scores in the ROC curve, and not by the dimension of the vectors of summary statistics. In addition to an increase in computing time, an increase in computing resources was also required by the logistic regression method such that fingerprint data with more than 22 features could not be processed on a standard desktop computer, while the ROC-based methods have a very small computing footprint (once the initial pseudo-data has been generated).



