The power metric: a new statistically robust enrichment-type metric for virtual screening applications with early recovery capability
A new metric for the evaluation of model performance in the field of virtual screening and quantitative structure–activity relationship applications is described. This metric has been termed the power metric and is defined as the fraction of the true positive rate divided by the sum of the true positive and false positive rates, for a given cutoff threshold. The performance of this metric is compared with alternative metrics such as the enrichment factor, the relative enrichment factor, the receiver operating curve enrichment factor, the correct classification rate, Matthews correlation coefficient and Cohen’s kappa coefficient. The performance of this new metric is found to be quite robust with respect to variations in the applied cutoff threshold and ratio of the number of active compounds to the total number of compounds, and at the same time being sensitive to variations in model quality. It possesses the correct characteristics for its application in early-recognition virtual screening problems.
<!>Background<!>Definitions<!><!>Definitions<!>Definition and calculation of established metrics<!>Derivation of a new metric: the power metric<!><!>Derivation of a new metric: the power metric<!>Probability distribution function to evaluate the metrics<!><!>Probability distribution function to evaluate the metrics<!><!>Dependency on both model quality λ and cutoff threshold χ<!>Conclusions<!>
<p>independence to extensive variables,</p><p>statistical robustness,</p><p>straightforward assessment of error bounds,</p><p>no free parameters,</p><p>easily understandable and interpretable.</p><!><p>In this paper a new metric is proposed that adheres to the six desired characteristics of an ideal metric. The metric is based on the principles behind the power of hypothesis test, which is the probability of making the correct decision if the alternative hypothesis is true. Comparison of the new power metric with more established metrics, including the enrichment factor (EF) [6, 7], the relative enrichment factor (REF) [8], the receiver operating characteristic (ROC) enrichment ROCE [9–11], the correct classification rate (CCR) [12, 13], Matthews correlation coefficient (MCC) [14], Cohen's kappa coefficient (CKC) [15, 16] together with the standard precision (PRE), accuracy (ACC), sensitivity (SEN) and specificity (SPE) metrics, is presented in this paper.</p><!><p>In the field of virtual screening, the quality of a model can be quantified by a number of metrics. The area under the curve (AUC) represents the overall accuracy of a model, with a value approaching 1.0 indicating a high sensitivity and high specificity [17]. A model with an AUC of 0.5 represents a test with zero discrimination. AUC metrics are calculated from typical ROC curves; these are plots of the (1 − SPE) values on the x-axis against the SEN values plotted on the y-axis for all possible cutoff points. Sensitivity and specificity, and thus the AUC, are good indicators of the validity of a method but are not measuring the predictive value of a method [18].</p><!><p>Illustration of the relation between cutoff χ and number of predicted actives/non-actives. Assuming a list of compounds ranked according to their predicted activity values, all compounds that are located on the left side of χ on this ranked list are predicted to be active, while all compounds that are located on the right-hand side of χ on this list are predicted to be non-active. All compounds that fall along the left-hand side of χ define the 'selection set'; in this example this includes five compounds. The total number of compounds in the selection set is N
s (here: 5), while the total number of compounds in the entire collection is N (here: 15). The number of true actives in the selection set is n
s (here: 3) and the number of true actives in the entire data collection is n (here: 4). Using these abbreviations, one can define the number of true positives TP as being equal to n
s, the number of true negatives TN equal to (N − N
s − n + n
s), the number of false positives FP equal to (N
s − n
s), and the number of false negatives FN being equal to (n − n
s)</p><!><p>Apart from the N
s and N variables, two other definitions are used in the following sections: the number of true active compounds in the selection set that is defined as n
s, and the number of true active compounds in the entire dataset defined as n. Finally, the prevalence of actives R
a in the entire dataset can be defined as:2\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$R_{a} = n/N$$\end{document}Ra=n/N</p><!><p>The sensitivity of a model is defined as the ability of the model to correctly identify active compounds from all the actives in the screening set (also termed the true positive rate or TPR), while specificity refers to the ability of the model to correctly identify inactives from all inactives in the dataset at a given cutoff threshold χ:3\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$${SEN}\left( \chi 
ight) = TPR\left( \chi 
ight) = rac{TP}{TP + FN} = rac{{n_{s} }}{n}$$\end{document}SENχ=TPRχ=TPTP+FN=nsn
4\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$SPE\left( \chi 
ight) = rac{TN}{FP + TN} = rac{{N - N_{s} - n + n_{s} }}{N - n}$$\end{document}SPEχ=TNFP+TN=N-Ns-n+nsN-n</p><p>In line with the true positive rate, one can also define a false positive rate FPR as the number of true inactives in the selection set in relation to the total number of inactives in the entire dataset:5\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$FPR\left( \chi 
ight) = rac{FP}{FP + TN} = rac{{N_{s} - n_{s} }}{N - n}$$\end{document}FPRχ=FPFP+TN=Ns-nsN-n</p><p>Other well-established metrics include the precision and accuracy:6\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$PRE\left( \chi 
ight) = rac{TP}{TP + FP} = rac{{n_{s} }}{{N_{s} }}$$\end{document}PREχ=TPTP+FP=nsNs
7\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$ACC\left( \chi 
ight) = rac{TP + TN}{TP + TN + FP + FN} = rac{{2n_{s} + N - N_{s} - n}}{N}$$\end{document}ACCχ=TP+TNTP+TN+FP+FN=2ns+N-Ns-nN</p><p>The enrichment factor is probably the most used metric in virtual screening and other fields as well. The EF at a given cutoff χ is calculated from the proportion of true active compounds in the selection set in relation to the proportion of true active compounds in the entire dataset:8\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$EF\left( \chi 
ight) = rac{TP/TP + FP}{TP + FN/TP + TN + FP + FN} = rac{{N 	imes n_{s} }}{{n 	imes N_{s} }}$$\end{document}EFχ=TP/TP+FPTP+FN/TP+TN+FP+FN=N×nsn×Ns</p><p>The enrichment factor is very intuitive and easy to understand, but it lacks a strong statistic background and has some drawbacks, including the lack of a well-defined upper boundary [the EF(χ) can vary from 0 in the case that there are no active compounds in the selection set (n
s = 0), and up to 1/χ when all active compounds are located in the selection set (n
s = n); see Ref. [19] for the derivation], the dependency of the value on the ratio of active to inactive compounds in the dataset, and a pronounced 'saturation effect' when the actives saturate the early positions of the ranking list and the performance metric cannot get any higher, thereby preventing to distinguish between good and excellent models [6].</p><p>To avoid the problems associated to EF, a number of other metrics have been proposed. The first of these is the relative enrichment factor [8], a metric in which the problem associated to the saturation effect is fixed by considering the maximum EF achievable at the cutoff point:9\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$REF\left( \chi 
ight) = rac{{100 	imes n_{s} }}{{\hbox{min} \left( {N 	imes \chi ,n} 
ight)}}$$\end{document}REFχ=100×nsminN×χ,n</p><p>The REF, has well defined boundaries—ranging from 0 to 100—and is less subject to the saturation effect.</p><p>The ROC enrichment metric is defined as the fraction of actives found when a given fraction of inactives has been found [9]:10\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$\it {{ROCE}}\left(pchi 
ight) = rac{{{{n}}_{{s}} /{{n}}}}{{\left( {{{N}}_{{s}} - {{n}}_{{s}} } 
ight)/\left( {{{N}} - {{n}}} 
ight)}} = rac{{{{n}}_{{s}} 	imes \left( {{{N}} - {{n}}} 
ight)}}{{{{n}} 	imes \left( {{{N}}_{{s}} - {{n}}_{{s}} } 
ight)}}$$\end{document}ROCEχ=ns/nNs-ns/N-n=ns×N-nn×Ns-ns</p><p>The ROCE metric has been advocated by some researches as a better approach to address early recovery [5, 9]. However, some issues still remain, such as the lack of a well-defined upper boundary [which is equal to 1/χ when TPR(χ) equals 1], a smaller but still noticeable saturation effect, and a statistic robustness which is not as desirable as we will demonstrate later.</p><p>Another metric often considered to measure classification performances is the correct classification rate [12], defined as the percentage of instances correctly classified:11\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$\it {{CCR}}\left(pchi 
ight) = rac{1}{2}\left[ {rac{{TP}}{{{{TP}} + {{FN}}}} + rac{{TN}}{{{{TN}} + {{FP}}}}} 
ight] = rac{1}{2}\left[ {rac{{{{n}}_{{s}} }}{{n}} + rac{{{{N}} - {{N}}_{{s}} - {{n}} + {{n}}_{{s}} }}{{{{N}} - {{n}}}}} 
ight]$$\end{document}CCRχ=12TPTP+FN+TNTN+FP=12nsn+N-Ns-n+nsN-n</p><p>The CCR is sometimes also called the balanced accuracy [20].</p><p>Matthews correlation coefficient has been advocated as a balanced measure that can be used on classes of different sizes [14]. The MCC is in essence a correlation coefficient between the measured and predicted classifications; it returns a coefficient of +1 in the case of a perfect prediction, 0 when no better than random prediction and −1 in cases of total disagreement between prediction and observation:12\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$\it {{MCC}}\left(pchi 
ight) = rac{TP 	imes TN - FP 	imes FN}{{\sqrt {\left( {TP + FP} 
ight)\left( {TP + FN} 
ight)\left( {TN + FP} 
ight)\left( {TN + FN} 
ight)} }} = rac{{N 	imes n_{s} - N_{s} 	imes n}}{{\sqrt {N_{s} 	imes n 	imes \left( {N - n} 
ight) 	imes \left( {N - N_{s} } 
ight)} }}$$\end{document}MCCχ=TP×TN-FP×FNTP+FPTP+FNTN+FPTN+FN=N×ns-Ns×nNs×n×N-n×N-Ns</p><p>The last metric that is evaluated with respect to its performance as compared to the here developed power metric is Cohen's kappa coefficient [21–24]:13\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$\it {{CKC}}\left(pchi 
ight) = 1 - rac{{1 - rac{TP + TN}{TP + TN + FP + FN}}}{{1 - rac{{\left( {TP + FN} 
ight)\left( {TP + FP} 
ight) + \left( {FP + TN} 
ight)\left( {FN + TN} 
ight)}}{{\left( {TP + TN + FP + {FN}^{2}} 
ight) }}}} = 1 - rac{{N 	imes n + N 	imes N_{s} - 2 	imes n_{s} 	imes N}}{{N 	imes n + N 	imes N_{s} - 2 	imes n 	imes N_{s} }}$$\end{document}CKCχ=1-1-TP+TNTP+TN+FP+FN1-TP+FNTP+FP+FP+TNFN+TNTP+TN+FP+FN2=1-N×n+N×Ns-2×ns×NN×n+N×Ns-2×n×Ns</p><!><p>In virtual screening studies, we can assume all compounds being inactive as the null hypothesis, and the assumption that some compounds are active as the alternative hypothesis. The statistical power, also known as sensitivity or recall, is equal to the true positive rate.</p><p>However, the statistical power alone does not include information about the distribution of negative instances or the size effect. Therefore, a metric based on statistical power and suited for applications in the field of virtual screening should incorporate information about the negative instances as well. Ideally, a good virtual screening method must be able to perform a good prediction of true positive instances combined with a small false positive prediction rate. This translates in a metric that combines the TPR with the false positive rate:14\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$'net\;power '\;(\chi ) = TPR(\chi ) - FPR(\chi )$$\end{document}′netpower′(χ)=TPR(χ)-FPR(χ)</p><!><p>Distribution curves of null and alternative hypothesis. The red area is the area defined by the alternative hypothesis minus the area defined by the null hypothesis, or, put differently, as the true positive rate (TPR, or 1 − β) minus the false positive rate (FPR, or α or type I error), hence 'net power' = 1 – β − α. The cutoff point is defined as the crossing point of the two distributions</p><!><p>The metric is not new; it has been developed independently several times in the past. Its origin can be traced back to the seminal paper of Peirce [25] with his 'science of the method' [26]. More than 70 years later, it was proposed again by Youden as Youden's index (Y'J) [27]. Youden's index is often used in conjunction with the ROC curve as a criterion for selecting the optimum cutoff point [28]. The index has been used to calculate the best cutoff point in the ROC curve. Once more, almost 50 years later in 2003, it was proposed again by Powers who called it 'informedness' [10].</p><p>Despite the success of this metric to evaluate the prediction power of a method, it is not entirely appropriate for virtual screening studies due to the lack of early recovery capabilities that are very desirable in any virtual screening application. Consider, for instance, a database of 10,000 compounds of which 1% are active compounds. In this hypothetical thought experiment, we can think of different methods that yield identical Youden's indices calculated from different TPR and FPR values. Thinking of two methods, each produce a Youden's index of 0.5, with the first one characterized by a TPR = 0.9 and a FPR = 0.4, and the second method characterized by a TPR = 0.51 and a FPR = 0.01. In the case of the first method, 4050 compounds will be marked as 'hits' of which only 90 compounds being true active (or 5.7% of the selected compounds). However, for the second method only 150 compounds are flagged as 'hits', of which 51 compounds are true actives (or 34% of the selected compounds). Obviously, for virtual screening applications, the second method provides a more optimal early recovery rate since only 1.5% of the original dataset needs to be tested in order to recover 51% of all active compounds.</p><p>Normalization of the 'net power' metric by dividing by the sum of the true positive and false negative rates introduces early recovery capabilities bias into the 'net power' metric. This difference-over-the-sum normalized 'net power' expresses the dominance of the true positive rate over the false positive rate among those instances predict as positive, expressed by its rates:15\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$normalized\;'net power^{\prime} = rac{TPR\left( \chi 
ight) - FPR\left( \chi 
ight)}{TPR\left( \chi 
ight) + FPR\left( \chi 
ight)}$$\end{document}normalized′netpower′=TPRχ-FPRχTPRχ+FPRχ</p><p>The metric ranges from −1 to +1 and can easily be modified to range from 0 to +1 by adding 1 to the metric and dividing by 2. We call this new metric the power metric (PM) and is defined as follows:16\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$${PM}\left( \chi 
ight) = rac{{\left( {rac{TPR\left( \chi 
ight) - FPR\left( \chi 
ight)}{TPR\left( \chi 
ight) + FPR\left( \chi 
ight)} + 1} 
ight)}}{2} = rac{TPR\left( \chi 
ight)}{TPR\left( \chi 
ight) + FPR\left( \chi 
ight)} = rac{{n_{s} 	imes N - n 	imes n_{s} }}{{n_{s} 	imes N - 2 	imes n 	imes n_{s} + n 	imes N_{s} }}$$\end{document}PMχ=TPRχ-FPRχTPRχ+FPRχ+12=TPRχTPRχ+FPRχ=ns×N-n×nsns×N-2×n×ns+n×Ns</p><!><p>In order to evaluate the performance of several metrics used in the field of virtual screening, we used the probability distribution function approach as suggested by Truchon and Bayly to build hypothetical models of different qualities [6]. For a typical virtual screening study with N compounds of which n being active compounds, we generated the ranks of these active compounds according to the exponential distribution as proposed by Truchon and Bayly [6]:17\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$X_{i} = rac{ - 1}{\lambda }ln\left( {1 - U_{i} \left( {1 - e^{ - \lambda } } 
ight)} 
ight)$$\end{document}Xi=-1Î»ln1-Ui1-e-Î»</p><p>The generated real number X
i corresponds to the relative position of active compound i and U
i is a pseudo random number with values between 0 and 1. In this exponential distribution, the Î» parameter represents the model quality (lower Î» values correspond to poor models and larger Î» values correspond to better models). The number X
i is transformed into a rank integer r
i that falls within 1 and N:18\documentclass[12pt]{minimal}
				sepackage{amsmath}
				sepackage{wasysym} 
				sepackage{amsfonts} 
				sepackage{amssymb} 
				sepackage{amsbsy}
				sepackage{mathrsfs}
				sepackage{upgreek}
				\setlength{\oddsidemargin}{-69pt}
				egin{document}$$r_{i} = int\left( {N 	imes X_{i} + 0.5} 
ight)$$\end{document}ri=intN×Xi+0.5</p><!><p>Visualization of the quality of the test models. In each case, 100 active compounds were distributed in a total set of 1000 compounds according to the distribution as defined by Eq. 17. The term 'quality' corresponds to the λ value of Eq. 17. The 'ideal' case was generated by positioning all 100 actives at the top-100 positions of the dataset. For each model, the AUC was calculated by integration using the composite trapezoidal rule</p><!><p>To illustrate the model generation process by example, consider a model with quality λ = 20 and consisting of n = 100 active compounds on a total of N = 10,000 compounds. To generate the relative rankings of these 100 active compounds, Eq. 17 was called 100 times, each time with a different random number U
i. Using Eq. 18, the 100 generated X
i numbers are then converted into 100 rankings r
i with N set to 10,000. These 100 rankings are the absolute positions of the active compounds; the remaining 9900 ranks (10,000 − 100 = 9900) are those of the inactive compounds.</p><p>In order to evaluate the quality of the PM metric and to compare its behavior to the other metrics, a large number of datasets were generated and analyzed. The total number of compounds N, number of actives n, model quality λ and cutoff parameter χ were varied. Each simulation was repeated 10,000 times and the results were analyzed by inspecting the variations of mean and standard deviation (STD) of the metrics as a function of the number of actives and total compounds. The eleven enrichment-type metrics that were analyzed were the PM, EF, ROCE, CCR, REF, MCC, CKC, together with the standard PRE, ACC, SEN and SPE metrics.</p><p>All calculations were performed under Python 2.7 using Numpy and Scipy [29]. The IPython notebook [30] was used as programming environment and figures were generated with Matplotlib [31]. MarvinSketch was used for drawing chemical structures [32].</p><!><p>Dependency on the model quality parameter λ using models generated from datasets with 100 actives (n) on 10,000 compounds in total (N)</p><p>Metric abbreviations are given in the Methods section. All metrics are dependent on the model quality, but in case of the ROCE, EF, REF, MCC, CKC, SEN and PRE metrics there is at least a tenfold increase when moving from a bad model (λ = 2) to a good model (λ = 40), while for the PM metric there is a doubling of the value. The accuracy ACC and specificity SPE metrics are not dependent on the quality of model, while the correct classification rate metric (CCR) shifts from 0.5 in the case of a bad model to a maximum of 0.75 for the best model. Good models have a PM of >0.9; for good models this value is largely independent on the applied cutoff value χ (see Table 3 as well)</p><p>Dependency on the R
a value</p><p>In the case of bad model quality (λ = 1), the metrics most sensitive to variations in the R
a value include the REF, PRE and ACC metrics, and also the CKC metric in the case of a large cutoff value of χ = 10%. This dependency is not so outspoken for the PM metric, except in the case when a very bad model is combined with a low cutoff value (χ = 1%). In cases with better model quality (λ = 20), significant dependencies are observed for the ROCE, EF, REF, MCC, CKC, SEN, PRE and ACC metrics, while the PM, CCR and SPE metrics are more stable. The metric that is least sensitive to variations in the R
a value, irrespective of the underlying model quality or cutoff threshold, is the CCR metric</p><p>aIn this case the ROCE metric could not be calculated from Eq. 10 since (N
s − n
s) is equal to 0</p><p>Dependency on the χ cutoff value using models generated from datasets with 250 actives (n) on 10,000 compounds in total (N)</p><p>The PM is not so much dependent on the applied cutoff value. For good models the EF and ROCE metrics decrease when the cutoff is increased, while the REF, CCR, MCC and CKC values always increase when the cutoff is increased from 2.5% up to 10%</p><p>Comparison of the power metric with the five main other metrics (CCR, ROCE, MCC, REF and CKC) using a model dataset of 250 active compounds on a total number of 10,000. The logarithm of the quality parameter λ is varied along the abscissa [a log(λ) of 2 corresponds to a quality λ of 100] while the applied cutoff threshold χ is varied along the ordinates. The black dotted line at a cutoff value χ of 2.5% indicates the boundary of 250 compounds on a total of 10,000. In a perfect model, all 250 active compounds would be located along the topside of this boundary</p><!><p>Based on these observations, it can be concluded that the CCR, MCC and CKC metrics are all less suitable for early-recognition problems; for these problems the PM and ROCE metrics are better suited. The REF metric might also be an option to some extend but some cautions are warranted when used in combination with cutoff thresholds χ that are equal or larger than the fraction of true actives in the entire dataset. In these cases an increase in the REF metric is observed, which makes it less suitable for early-recognition problems. As already mentioned, the main disadvantage of the ROCE metric is the lack of a well-defined upper boundary, and for this reason the PM metric seems to posses powerful early-recognition properties and might be one of the preferred metrics for evaluating virtual screening models.</p><!><p>The power metric PM as described in this paper is a statistically solid metric with little sensitivity to the ratio of actives to the total number of compounds (the R
a value; see Table 2) and little sensitivity to the cutoff threshold parameter χ (Table 3). The metric is dependent on the underlying model quality, in such sense PM values around 0.5 are calculated for poor to random models, and values between 0.9 and 1.0 for high quality models. It is statistically robust in the sense that the calculated standard deviations are small and largely insensitive to the applied threshold cutoff value χ.</p><!><p>accuracy</p><p>area under the curve</p><p>correct classification rate</p><p>Cohen's Kappa coefficient</p><p>enrichment factor</p><p>Matthews correlation coefficient</p><p>power metric</p><p>precision</p><p>quantitative structure–activity relationship</p><p>relative enrichment factor</p><p>receiver operating characteristic</p><p>ROC enrichment</p><p>sensitivity</p><p>specificity</p><p>standard deviation</p><p>true negative rate</p><p>true positive rate</p><p>A comment to this article is available at https://doi.org/10.1186/s13321-018-0267-x.</p>