the accurate prediction 
 of protein  ligand affinity remains 
 one of the great challenges in computational chemistry . a fast and generally applicable method would 
 greatly benefit the pharmaceutical industry by speeding up the discovery 
 of new drugs and reducing reliance on expensive wet lab experiments . in principle , one can calculate the binding affinity of any protein  ligand 
 complex using molecular simulations and rigorous statistical mechanics 
 techniques . these methods have been successfully applied 
 in lead - optimization , especially when calculating the relative affinities 
 of congeneric ligands . despite their general applicability , these techniques remain too time - consuming to assay the vast chemical 
 libraries used in pharmaceutical research . as a result , very rapid 
 models , called scoring functions , are typically used in virtual screens . scoring functions predict affinity using only a single snapshot of 
 a molecular complex , so that the objective of scoring function research 
 is to elucidate any emergent laws that operate above the physical 
 laws of atomic motion . they are designed to be applicable to all proteins 
 and ligands , in contrast to quantitative structure , the ideal scoring 
 function would not need to be optimized for any particular protein 
 or a set of ligands and could accurately predict hits early in a drug 
 discovery project . despite decades of development , however , 
 the performance of scoring 
 functions varies greatly between different protein systems , and predictions 
 often correlate poorly with experimental data . the cause of scoring function 
 error remains as yet uncertain and has been attributed to a number 
 of different factors . physical arguments , for instance , have highlighted 
 the poor treatment of explicit solvent effects and protein on the other hand , recent efforts have sought to 
 improve scoring functions by empirical means , employing advanced machine 
 learning techniques , many protein however , it is still not clear whether such approaches 
 significantly improve accuracy . a technique 
 known as consensus scoring has been been shown to 
 reliably lower error by combining the predictions of different scoring 
 functions , although accuracy still remains far below that of experiment . given the potential impact on drug discovery 
 and the substantial 
 effort in scoring function development , it is vital to understand 
 the fundamental uncertainties in these rapid affinity models . recently , 
 faver et al . investigated the systematic and random errors associated 
 with the interaction energies of protein  ligand complexes . using a fragment based approach , they found that the random error in electrostatic 
 interaction calculations rises with the system size . protein targeted 
 scoring functions have also been observed to be more accurate than 
 generalized models , challenging the utility 
 of a universal scoring function , yet questions still remain with regards 
 to the limits of scoring function accuracy . for instance , is it possible 
 for a particular set of descriptors and functional form of a model 
 to achieve a negligible error ? also , can a universally applicable 
 scoring function ever be better than a targeted one ? without a formal 
 analysis of the structure - based modeling process , questions such as 
 these can not be answered fully . in this work , we investigate 
 the inherent uncertainties in empirical 
 structure - based models with a rigorous mathematical analysis utilizing 
 statistical learning theory and information theory . as statistical 
 learning theory is implicitly applied whenever one trains a model 
 using regression or classification , we establish the statistical relationship 
 between a structural snapshot of a protein  ligand complex and 
 its affinity . by doing so , we find that a critical assumption of statistical 
 learning theory is often violated when training and applying empirical 
 structure - based models . an implication of this is that a scoring function 
 that performs optimally on one set of protein  ligand complexes 
 necessarily performs poorly on another set . also , we prove that even 
 the very best universal structure - based model is significantly limited 
 in its accuracy , and protein - specific models are always likely to 
 be better . throughout , we use an information theoretic measure to 
 quantify scoring function error to ensure that our analysis is independent 
 of how the protein  ligand interactions are modeled , so that 
 our results apply to any set of descriptors and regression method . to understand empirical scoring function error we must first appeal 
 to the theory that underpins regression analysis : statistical learning 
 theory . we denote the structural features 
 of a protein  ligand complex as x and the binding 
 affinity as y. statistical learning theory formalizes 
 the process of elucidating the functional relationship between x and y by assuming there is a probabilistic 
 process that generates the data used to train and test a model . we 
 denote the probability distribution function ( pdf ) over x and y as p(x , y ) . the functional relationship between structure and affinity 
 is encoded in the conditional pdf p(y|x ) , as with this pdf , one can find the most likely 
 binding affinity for a given structural description of a protein a further , critical assumption in statistical learning theory 
 is that the data used to train a model are generated by the same probabilistic 
 process as the data found in the predictive setting . a popular measure , for instance , is the mean squared error 
 between the affinity predictions of a model and their experimental 
 values . for our loss function , we utilize the cross ( shannon ) entropy 
 between the true and modeled probability distributions . the cross 
 entropy , denoted c(y|x ) , is given by1and is a quantification 
 of the uncertainty 
 in the affinity given a set of structural descriptors sampled from p(x , y ) , and a model , denoted q(y|x ) . with this loss 
 function , the task in scoring function development is to find the 
 model that lowers the uncertainty as much as possible . cross entropy 
 is a very general loss function , and while 
 commonly used in classification , judicious 
 selection of specific forms of q(y|x ) allows for more familiar measures of regression 
 error . indeed , if q(y|x ) encodes for a scoring function that minimizes c(y|x ) , then the same scoring function 
 also minimizes the mean squared error ( see supporting 
 information section 2.1 ) . in a well - known information theoretic 
 result , cross entropy expands to2where h(y|x ) is 
 the conditional entropy of the true distribution , 
 given by3and d(p(y|x)||q(y|x ) ) is the kullback  leibler 
 divergence or 
 relative entropy between the true and modeled distributions . it is 
 given by4 relative entropy is a convex 
 function for the two conditional distributions ; 
 it is zero only when the distributions are the same , and positive 
 otherwise . equation 2 can be understood as representing 
 the minimum uncertainty , or error , of the binding affinity given the 
 structure , plus the uncertainty due to our model ; choosing the wrong 
 model only increases our uncertainty of the system . in our information theoretic perspective , h(y|x ) is the minimum achievable error , which is irreducible 
 for a given set of descriptors , and d(p(y|x)||q(y|x ) ) is the bias for assuming the structure - affinity 
 relationship is encoded in q(y|x ) when in reality it is encoded in p(y|x ) . this bias is a type of random error , 
 as opposed to a systematic error . any bias ensures that the minimum 
 error is not achieved for the wrong model , and borrowing a term from 
 financial decision theory and signal 
 processing , we shall refer to the positive 
 deviation from the minimum error as regret . many factors can cloud the reasons for an empirical model 's 
 inaccuracy . 
 in structure - based affinity models one has to choose the representation 
 of a complex - a possible description might be the surface complementarity 
 between a drug bound to a protein - and the functional form of the 
 model . both choices may introduce errors that are difficult to disentangle 
 from the fundamental uncertainties arising from rapid affinity prediction . the benefit of using c(y|x ) to quantify a error is that all the confounding factors 
 that hinder a scoring function 's accuracy unravel , and we need 
 only 
 consider the relationship between a protein ligand 's affinity 
 and a complete structure of the complex obtained from solution at 
 equilibrium . first , the minimum error , h(y|x ) , is a property of the data and its 
 underlying pdf only , and thus is independent of the functional form 
 of a scoring model . second , the mere process of selecting a subset 
 of structural features of a protein  ligand complex to use in 
 a model means some data about the complex is discarded . discarding 
 data can either maintain or increase h(y|x ) , so to analyze 
 the minimum achievable error of a typical scoring function , we use x to denote snap - shots of entire complexes of structures 
 in solvent , thereby avoiding any reduction in information . third , 
 we use conditional pdfs and their corresponding optimal models so 
 as not to consider the variance that occurs in when fitting a scoring 
 function in practice . thus , we focus 
 on an idealization of a structure - based model , the error of which 
 is a lower bound of what can be achieved in practice . our analysis begins by placing the training 
 and testing of an empirical structure - based scoring function in a 
 statistical framework . using this framework , we derive an upper bound 
 for the regret of a model that is trained and applied to protein ligand 
 complexes sampled from different distributions , a common occurrence 
 in virtual screening . we also analyze and compare the bias and minimum 
 error of a universal structure - based model to those of protein - targeted 
 models . we verified our analytic 
 predictions by fitting and testing a scoring function that is archetypal 
 of popular empirical models . ligand 
 interaction terms including shape complementarity and electrostatic 
 energy , and the number of rotatable bonds in the ligand . as is typical , 
 the functional form of the scoring function is linear . this method improves model robustness by controlling for the l1 and l2 norms of 
 the regression coefficients at the cost 
 of having two free parameters that can be optimized on a validation 
 data set or by using cross - validation . we tested l1 fractional penalty between 0 and 1 in 0.1 increments , 
 and following zou and hastie , we tested l2 penalties of 0 , 0.01 , 0.1 , 1 , 10 , and 100 . as our analysis is an idealization of the structure - based 
 modeling process , we trained and tested our canonical scoring function 
 using data sets of protein  ligand complexes with known binding 
 affinities and whose structures had been resolved by x - ray crystallography . we used 10 data sets in total which were constructed using the 24th 
 september 2010 version of the csar high quality data set of protein the csar data set was 
 used to construct one training set ( 207 complexes ) and one test set 
 ( 83 complexes ) that both consist of a diverse range of protein the csar data set was provided with a list that contained 
 all of the complexes assembled into groups of 90% protein sequence 
 identity . one complex from each group was selected at random to form 
 the training set . the complexes that did not share 90% sequence identity 
 with any other protein were also selected for the training set . from 
 the complexes that remained , another random selection was made to 
 form the diverse test set , so it too was composed of proteins with 
 less than 90% sequence identity . this test set is labeled as data 
 set a in table 1 and in the results section . the pdbbind data set was used to assemble 
 8 single - protein test sets which are labeled as b additional complexes were added to the hiv1 protease ( labeled a ) , 
 trypsin ( c ) , carbonic anhydrase ii ( e ) , and thrombin ( e ) data sets , 
 in order to match the single - protein data sets in the 2009 study by 
 cheng et al . in that study , the predictive 
 performance of thirty - three variants of popular scoring functions 
 ( including gold , glidescore and x - score ) was evaluated . this allowed 
 us to test how representative our scoring function is of these popular 
 models . we created a diverse scoring function by fitting 
 our canonical model on the training set of diverse protein ligand 
 complexes optimizing the free parameters to data set a. our model 
 was also fitted to data sets b  i and optimized using leave - one - out 
 cross - validation to create 8 single - protein scoring 
 functions . all of these scoring functions were then applied across 
 the data sets listed in table 1 , and their 
 accuracy was measured using the mean absolute error and the spearman 
 rank coefficients of the affinity predictions against the experimental 
 values . ligand s 
 structure x , and affinity y , is 
 in general different for each complexes . this means that the fundamental 
 condition in regression analysis for the training and test set to 
 be sampled from the same probability distribution is often violated 
 for structure - based scoring functions . in rigorous free energy 
 calculations , a control parameter , denoted  , is often used to define a molecular system or set 
 of constraints on that system . in ligand binding free energy calculations , 
 the control parameter associated with the bound state , denoted b , is switched over the 
 course of possibly many simulation windows to the value associated 
 with the unbound state , denoted u . as free energy is a state function , the binding 
 affinity y depends only on b and u . experimental error gives rise to uncertainty in 
 the value of the  true free energy difference between 
 the two states . we represent this intrinsic uncertainty in the free 
 energy between the states via the pdf s(y|b,u ) . in contrast to rigorous methods , 
 scoring functions use a single 
 snapshot of the protein  ligand complex to predict affinity . as sampled from the bound state , any such snapshot is dependent only 
 on b . for instance , 
 if the snapshot is sampled from the equilibrium distribution of the 
 complex in solvent , then from statistical mechanics , the pdf of a 
 structure x given b , denoted r(x|b ) , is equal to the boltzmann 
 distribution5where f(b ) is the free 
 energy of the bound state , e(x,b ) is the potential energy 
 of the structural snapshot , 
  is the integral over all momenta , and  is equal to the reciprocal of the boltzmann constant multiplied 
 by the temperature . as discussed , our information theoretic approach 
 to scoring function error means that x represents 
 a particular snapshot of a protein ligand complex , without any loss 
 of information . to understand scoring function error , we consider 
 the probabilistic 
 relationship between structure x and affinity y. for a given protein ligand complex  , we denote the pdf of observing x and y as p(x , y| ) . the question  what is the binding affinity 
 of a particular protein - ligand complex  ? is implicitly asking  what is the free energy difference between 
 states defined by b and u ?  . in other words ,  is really a surrogate label 
 for b and u . from the above discussion , thus , x and y are conditionally independent for a given complex , so 
 that6 critically , by sampling the structure x from the 
 equilibrium distribution as in eq 5 , it is apparent 
 that each p(x , y| ) is unique for a particular . this follows by acknowledging that altering either 
 the protein or the ligand will change the boltzmann distribution . we have 
 shown that the pdf of structures and affinities for a particular protein ligand 
 complex , p(x , y| ) , is different for each molecular pair . yet implicitly assumes 
 that the complexes in a data set set have been sampled in a probabilistic 
 manner . we denote the probability for selecting a complex  for a particular data set as ( ) . a scoring function is not trained on 
 the complexes themselves , but on their structures and affinities sampled 
 from the weighted sum7where the sum is over all 
 protein ligand 
 complexes , and we have made the dependency of p(x , y ) on ( ) explicit 
 with a subscript . the optimal scoring function for complexes sampled 
 from ( ) is encoded 
 in the conditional pdf8 by inserting 
 the pdfs 
 defined in eqs 6 and 5 into the above , it is apparent that p(y|x ) contains 
 contributions from all the individual boltzmann distributions of each 
 complex , so that different protein using the 
 above definitions , we now investigate the error of a 
 scoring function that occurs when it is applied to a set of complexes 
 sampled from ( ) but 
 is trained on complexes sampled from a different distribution , 
 denoted as ( ) . following 
 the theory section , the bias incurred by applying 
 the scoring function that is optimal on data sampled from ( ) , which is encoded in p(y|x ) , to complexes sampled from ( ) is given by d(p(y|x)||p(y|x ) ) . our first main result ( proven in supporting information section 3.1 ) is that9 thus , the relative entropy 
 between the complex selection probabilities 
 is an upper bound to the bias of a misapplied scoring function . scoring 
 function bias is minimized when ( ) = ( ) , so that the 
 protein  ligand complexes in the training and test sets of a 
 scoring function have been sampled from the same probability mass 
 function . however , if ( ) and ( ) do not overlap , 
 then d(()||( ) ) is unbounded , 
 implying that scoring function error can be arbitrarily large . this can occur when the probability of finding any complex 
 from the training set in the test set is zero , such as when a protein - specific 
 qsar model is applied to another protein . a universally applicable 
 structure - based model should , by definition , 
 have the lowest possible error when applied to the widest conceivable 
 range of protein  ligand complexes . if a training set is composed 
 of a diverse range of proteins and ligands , we know from eq 9 that for a scoring function s bias to be 
 zero , the test set should be similarly diverse . in a typical virtual 
 screen , however , scoring functions that have been trained on a diverse 
 range of protein  ligand complexes are applied only to ligands 
 binding to a single protein , implying different complex 
 selection probabilities between the training and test sets and a potentially 
 large bias . to 
 investigate the error of a universal scoring function , we consider n different proteins that one can use to construct data 
 sets of protein  ligand structures x and affinities y. protein - specific complexes are sampled according to i( ) , i = 1,2,3, ... ,n , with corresponding 
 joint pdfs pi(x , y ) . many samples from pi(x , y ) results in a data set of bound structures and affinities of different 
 ligands bound to protein i. to model the creation 
 of a data set composed of a diverse range of protein ligand 
 complexes , we select which pi(x , y ) to sample from with 
 probability i . similar to eq 6 , the appropriate joint pdf for 
 this diverse scoring function is given by the weighted sum10 the corresponding 
 conditional pdf , p(y|x ) , encodes for the optimal 
 diverse scoring function . if n is sufficiently large 
 and  suitably broad , then p(y|x ) represents a however , it is important to note that this generalized optimality 
 is defined only for its particular sampling probabilities . from 
 the previous section , we know that a scoring function that 
 has been designed for a diverse range of protein  ligand complexes 
 will have a nonzero regret when applied to a protein - specific data 
 set . a relevant question is , therefore , how large is the average bias 
 incurred by applying a scoring function defined from p(y|x ) to complexes sampled from i( ) . our second main result 
 ( see supporting information section 3.2 
 for the proof ) is that for an arbitrary scoring function encoded by q(y|x)11meaning the generalized model has the lowest 
 average bias over all the protein  ligand complexes sampled 
 by . although the universal model p(y|x ) is optimal over the all n proteins , 
 we now show , perhaps counterintuitively , that it is not optimal for 
 each specific protein . in our information theoretic perspective ( see 
 eq 2 ) , the minimum error of the general model p(y|x ) over the n proteins is given by its 
 conditional shannon entropy , denoted h(y|x ) . also , 
 the minimum achievable error of an optimal protein - specific model pi(y|x ) on protein i is denoted hi(y|x ) . our third main result ( see supporting information section 3.3 for the proof ) is that12 hence , the minimum error of a generalized structure - based 
 model 
 is greater than the average minimum error for protein specific models . thus , on average , a scoring function targeted for a specific protein 
 will outperform a scoring that has been designed for a diverse range 
 of protein  ligand complexes . eq 12 also shows that a universal model has a broader error distribution 
 than the average single - protein model . for a given model , there 
 is an unavoidable trade - off between accuracy 
 over a broad spectrum of complexes and accuracy for certain individual 
 cases . it is important to highlight that the generalized pdf , p(y|x ) , and the pdf for a specific data set pi(y|x ) are , in general , different . as these conditional distributions encode their own optimal functional relationships between structure 
 and affinity , the model that best predicts the binding affinities 
 for many proteins may be very different from the best model for specific 
 proteins . the previous 
 section showed that regret is integral to a generalized empirical 
 scoring function . there is a similar trade - off in accuracy when optimizing 
 a scoring function for any set of complexes . as described , the 
 relative entropy quantifies the error cost of misapplying a scoring 
 function to a data set . this cost is a bias , and as it ensures that 
 the minimum error is not achieved , it contributes to the regret of 
 a model . the bias can be reduced for a particular sampling regime 
 of protein  ligand complexes by refitting a scoring function 
 for those complexes . however , when the regret for those complexes 
 decreases , the regret between the scoring function and another sampling 
 regime may increase . this occurs by virtue of the fact that different 
 complex sampling probabilities results in distinct structure - affinity 
 pdfs and that the relative entropy is a convex function of two distributions . this means that a structure - based model that achieves the lowest possible 
 error on a group of protein  ligand complexes will necessarily 
 perform poorly on another group . this represents a  no free 
 lunch scenario for structure - based 
 models : without any a priori information , no structure - based model 
 can be said to outperform another on any particular data set . in conceptualizing the practical implications of this , it is fruitful 
 to consider two protein - specific conditional pdfs , p(y|x ) and p(y|x ) , as normal distributions with variances 
 approximately equal to  and corresponding 
 optimal scoring functions f(x ) and f(x ) respectively . following from heskes , we show in the supporting 
 information section 2.2 that13 this equation implies 
 that we can represent the theoretically optimal 
 scoring functions f(x ) and f(x ) as being embedded in a space that preserves 
 the mean squared distances between them . optimizing a scoring function 
 to a particular protein can then be considered as moving through this 
 space toward the optimal model , increasing the distance from another 
 model which is optimal on a different protein . a schematic diagram 
 of this hypothetical scoring function space is shown in figure 1 . every location in the space corresponds to a scoring function , 
 and each colored protein represents the optimal model for a particular 
 set of protein  ligand complexes . beginning with a fitted model , 
 shown as the lightest gray cross , that performs close to optimally 
 on the green data set , its large distance from the purple model means 
 that the bias on this data set is also large . optimizing this model 
 for the latter , shown by darkening gray crosses , increases the distance 
 from the green and hence the error of the model on this data set . 
 having a low error on one set of complexes necessarily means a scoring 
 function performs poorly on others . four of our test 
 sets were chosen to match the single - protein tests 
 sets used by a study by cheng et al . , so 
 that we could compare our scoring function to the thirty - three popular 
 scoring functions they tested . our scoring function achieved a higher 
 spearman rank coefficient than 79% , 61% , 39% , and 100% of the thirty - three 
 models on the hiv1 protease ( b ) , trypsin ( c ) , carbonic anhydrase ii 
 ( e ) , and thrombin ( g ) data sets respectively . a full comparison between 
 our model and the best and worst models evaluated in the study by 
 cheng et al . is shown in table s2 of the supporting 
 information . these results show our model is a relatively good 
 scoring function , and we take the following results to be representative 
 of the state of the art . in 2010 , a study by kramer and gedeck 
 showed that while the scoring 
 function , rf - score , performed well on 
 a diverse range of protein  ligand complexes , its accuracy was 
 highly variable on single - protein data sets . our information theoretic analysis indicates that this behavior 
 is independent of the functional form of the model and the protein , we assessed the performance 
 of our own diverse scoring function on the data sets shown in table 1 . our own scoring function is linear and uses interaction 
 terms such as hydrogen bond strengths . as figure 2 demonstrates , our diverse scoring function correlates well 
 with experimental affinities on the diverse protein ligand 
 data set and worse on average on the single - protein data sets , in 
 agreement with what has been observed in previous scoring functions . the 
 predictions of the diverse scoring function on each of our 
 test sets ( see table 1 ) . the dashed red line 
 indicates the line of perfect prediction and the spearman rank coefficient , 
  , for each data set is shown . in table s2 of the supporting information , these rank coefficients 
 along with the pearson correlation coefficients and standard deviations 
 from a linear fit are shown for data sets b , c , e , and g and compared 
 against the performance of the scoring functions tested by cheng et 
 al . while the rank order of the ligand 
 affinities is well predicted for the diverse range of complexes ( a ) , 
 the accuracy varies dramatically for the single - protein data sets 
 ( b  i ) . this behavior is similar to what has been observed previously 
 with other scoring functions and is explained by our analysis . figure 2 shows that , also like many popular 
 scoring functions , the performance of our diverse 
 model is heavily dependent on the data set it is applied to . this 
 ultimately follows from the  no free lunch theorem 
 for supervised learning ; a scoring function 
 that performs well on one sampling regime of protein  ligand 
 complexes necessarily means it will perform poorly on others . also , 
 we know from eq 9 that the regret of a misapplied 
 scoring function can be arbitrarily large if the complexes in the 
 training and test sets have been sampled in a significantly different 
 manner . we further illustrate this difficulty in scoring function 
 optimization by calibrating the free parameters ( see methods ) of the diverse scoring function for each protein - specific 
 data set . changing these parameters can be thought of as moving through 
 a hypothetical scoring function space as depicted in figure 1 toward the location of an optimal protein - specific 
 model . the relative mean absolute error of the scoring function for 
 each free parameter pair on each data set is shown in figure 3 . no single choice of parameter pair yields the 
 minimum error on all of the sets , and figure 3 shows that optimizing the scoring function for a data set can increase 
 the error on others . grid of heatmaps showing the mean absolute error for our 
 own scoring 
 function when applied to data sets a ligand 
 complexes , while data sets b  i are single - protein data sets . each heatmap shows the relative error of the model on the data set 
 as the two free parameters of the scoring function - trained using 
 elastic net regularization - are varied ( x- and y - axis ) . the color gradation indicates parameter - pair choices 
 that give rise to the lowest ( white ) through to the highest ( red ) 
 error for that data set . coloring by relative error highlights that 
 no single parameter - pair choice achieves the lowest error on all nine 
 data sets , so that a model that has the lowest average error over 
 all data sets will not be the best on each individual data set . to 
 compare the absolute values of the errors , figure s3 of the supporting information utilizes an absolute coloring 
 scale , and table s3 shows the absolute 
 values of the highest and lowest errors on each data set . equation 11 shows that , for 
 a particular 
 sampling regime , the conditional pdf that encodes for the scoring 
 function with the lowest error over a diverse range of complexes also 
 has the lowest average bias than any other . yet eq 12 shows that this comparatively low bias is compensated by 
 an intrinsic error that is larger than the average minimum error for 
 specific protein models . this implies that protein - specific scoring 
 functions will have a lower error when applied to their respective 
 protein than the general model . similarly , misapplying a protein - specific 
 model to the wrong protein will have a higher average error than a 
 generalized scoring function . these analytical results explain the 
 results observed with our own single - protein and diverse scoring functions 
 ( see methods ) . the average error from cross - validation 
 of each single - protein model is 1.2 kcal / mol compared to an average 
 error of 1.8 kcal / mol of the diverse scoring function . by applying 
 each single - protein scoring functions to the other data sets , we found 
 the average misapplied error to be 2.7 kcal / mol . the relative sizes 
 of each of these errors are in complete agreement with our analytical 
 results . the mean absolute errors of these models on each data set 
 are shown in figure 4 . the mean absolute errors 
 of our scoring model on our test sets 
 ( see table 1 ) when fitted in three different 
 ways . yellow bars show the cross - validation error of the model when 
 it is fitted to each specific data set ; green bars show the error 
 of the model when it is fitted to a diverse range of protein ligand 
 complexes ; blue bars show average errors of the protein - specific scoring 
 functions when misapplied to that data set . in agreement with our 
 mathematical analysis , the error of a protein - specific scoring function 
 is on average less than the error of a general scoring function , which 
 itself is more accurate than the average error of a misapplied protein - specific 
 scoring function . by assuming that the 
 cross - validation error of the single - protein 
 models is the minimum error achievable for our scoring function descriptors , 
 we can approximate the regret of the diverse scoring function on a 
 single - protein data set by the difference between the error it achieves 
 and the cross - validation error . on the carbonic anhydrase ii data 
 set ( e ) , the diverse scoring function has an exceptional large regret 
 of around 4 kcal / mol . removing this data set from the analysis , the 
 diverse - scoring function has an apparently encouraging average regret 
 of 0.3 kcal / mol . in actuality , a truly protein - specific scoring function 
 would be have to be designed from the bottom up , and may include descriptors 
 designed especially for the protein , have a particular functional 
 form , and have the method of regression chosen after experimentation . our own protein - specific scoring functions are simply recalibrations 
 of our own diverse scoring function . thus , the regret of our diverse 
 model is likely to be larger when truly protein - specific models are 
 considered . we attribute the large error on carbonic anhydrase 
 ii to the relative 
 infrequency of complexes with metal  ligand interactions appearing 
 in the diverse training set . carbonic anhydrase ii has a catalytic 
 zinc ion in its binding site that interacts with its inhibitors . in 
 the diverse training set , only 37 out of the 207 ligands are within 
 a hydrogen bonding distance to a metal ion . thus , the data set contains 
 relatively little information about metallic interactions and a correspondingly 
 large regret for these type of complexes is to be expected . metalloproteins 
 are notoriously difficult to account for in scoring functions , and 
 further optimization or extra protocols are typically required for 
 specific systems . the substantially lower error our scoring function attains when 
 recalibrated for carbonic anhydrase ii is indicative of this general 
 trend . as our analytical results show , the accuracy of an empirical 
 structure - based scoring function depends on the degree of informational 
 overlap between the training and test sets , such that specific scoring 
 functions have a lower error on average than generalized models . by formally analyzing the structure - based modeling process , we 
 have conclusively proven that protein - specific scoring functions will 
 on average achieve a lower error than the very best universal model . we have shown how training and applying a scoring function on different 
 sets of protein  ligand complexes can result in an arbitrarily 
 large error and that a model which performs optimally on one set of 
 complexes necessarily performs poorly on another set by virtue of 
 the  no free lunch theorem for supervised learning . our results follow from the fact that data sets of protein  ligand 
 structures and affinities are , in general , governed by distinct probability 
 distributions , so that there may be a cost in transferring empirical 
 scoring functions between data sets . regret would typically occur in a virtual 
 screening context , where a scoring function that is trained on a diverse 
 range of complexes is used to predict affinities of ligands binding 
 to a single protein . we employed an information theoretic analysis 
 to demonstrate that this error is independent of the way protein while previous research into the sources of scoring function 
 error has focused on errors from energetic calculations , bias - derived error has remained unreported and 
 explains the variability of scoring function performance on different 
 protein data sets . in contrast , techniques 
 based on statistical mechanics that utilize well sampled molecular 
 simulations - in principal - represent universally applicable methodologies . however , even these methods are not free from error , as their accuracy 
 hinges on the type of forcefield used and its parametrization ; understanding 
 forcefield errors is an active area of research . as atomic resolution 
 simulations represent a  bottom - up approach to free 
 energy calculations and empirical scoring functions a  top - down 
 approach , some of the sources of error in both methods may be fundamentally 
 different . as such , the extent to which both approaches are complementary 
 merits investigation . although generalized empirical models are often used , prior 
 knowledge often exists that may help optimize the scoring function 
 to the protein . one may know some of the active ligands and may be 
 fortunate to have a three - dimensional structure of the target available . 
 a fruitful way forward may be to develop scoring functions that are 
 able to robustly incorporate any prior information one has of the 
 protein into the model . bayesian methods are ideal for such an approach , 
 and while they have previously been applied to predict ligand activity 
 and protein selectivity , the formal exploitation 
 of all prior information , to our knowledge , has not yet been explored . indeed , rigorous physics - based approaches and empirical models could 
 be combined within a bayesian framework . if one has extremely limited 
 data of the protein , the development of protein class or family specific 
 models may be a sufficient compromise between generality and accuracy . when fitting to small data sets , these generalized models can also 
 provide the prior distributions on the regression coefficients for 
 bayessian regression , as it is well established that bayesian regression 
 greatly stabilizes fitted models in such cases . given that model regret is intrinsic to protein ligand 
 scoring , 
 it remains of paramount importance to be able to estimate its magnitude 
 in a predictive setting and is the subject of our ongoing research . any further understanding regarding the limits of scoring function 
 accuracy is vital for a more informed and efficient synergy between 
 theoretical affinity predictions and experimentally driven drug development .
a major 
 goal in computational chemistry has been to discover the 
 set of rules that can accurately predict the binding affinity of any 
 protein - drug complex , using only a single snapshot of its three - dimensional 
 structure . despite the continual development of structure - based models , 
 predictive accuracy remains low , and the fundamental factors that 
 inhibit the inference of all - encompassing rules have yet to be fully 
 explored . 
 using statistical learning theory and information theory , 
 here we prove that even the very best generalized structure - based 
 model is inherently limited in its accuracy , and protein - specific 
 models are always likely to be better . 
 our results refute the prevailing 
 assumption that large data sets and advanced machine learning techniques 
 will yield accurate , universally applicable models . 
 we anticipate 
 that the results will aid the development of more robust virtual screening 
 strategies and scoring function error estimations .