Debiasing Algorithms for Protein Ligand Binding Data do not Improve Generalisation
The structured nature of chemical data means machine learning models trained to predict proteinligand binding risk overfitting the data, impairing their ability to generalise and make accurate predictions for novel candidate ligands. To address this limitation, data debiasing algorithms systematically partition the data to reduce bias. When models are trained using debiased data splits, the reward for simply memorising the training data is reduced, suggesting that the ability of the model to make accurate predictions for novel candidate ligands will improve. To test this hypothesis, we use distancebased data splits to measure how well a model can generalise. We first confirm that models perform better for randomly split held-out sets than for distant held-out sets. We then debias the data and find, surprisingly, that debiasing typically reduces the ability of models to make accurate predictions for distant held-out test sets. These results suggest that debiasing reduces the information available to a model, impairing its ability to generalise.
Introduction<!>Methods<!>Results<!>Discussion<!>Supporting Information Available:
<p>The accurate identification of ligands that bind tightly and specifically to a given protein target is a crucial step in drug discovery. Experimental high-throughput screening is expensive, time-consuming, and far from comprehensive due to the numerous potential ligands in chemical space. 1,2 Physics-based methods such as docking and Molecular Dynamics can be inaccurate and are computationally expensive. [3][4][5] Databases such as ChEMBL 6 are growing rapidly, increasing the popularity of machine learning (ML)-based approaches to molecular property prediction. 7,8 In this paper, we focus on data-driven approaches to virtual screening, i.e. using data from high-throughput screening experiments to train models that predict whether ligands will have activity against a particular protein target. 9 Many recent ML approaches have achieved outstanding success on benchmark datasets that are randomly partitioned into train and validation sets, with AUCs (area under the Receiver Operator Characteristic curve) routinely exceeding 0.9. [10][11][12][13][14][15][16][17] However, it is unclear whether this impressive performance indicates that a model that can truly generalize across chemical space, or instead sim-ply overfits the training data. [18][19][20][21][22][23] Since chemical space contains clusters of molecules around scaffolds, memorizing the properties of a few scaffolds can be sufficient to perform well, masking the fact that the model may not generalize beyond close analogues. 24,25 Further, molecules tested experimentally are generally designed by humans and therefore likely to be easy to synthesize and similar to previously known binders. 19,26 To counter this limitation, data bias definitions and corresponding debiasing algorithms have been introduced. 20,22,27,28 Two popular bias measures, Maximum Unbiased Validation (MUV) and Asymmetric Validation Embedding (AVE), are illustrated in Fig. 1. 20,22 In each case the bias measure is used by a genetic algorithm to rearrange the train/validation split such that the bias is reduced. Specifically, MUV ensures that active ligands are uniformly embedded among inactive ligands according to some distance metric, while AVE adds the requirement that inactive ligands are not tightly clustered. Across benchmark datasets for multiple protein targets, both bias metrics were shown to correlate to model performance, suggesting that heavily biased datasets provide a falsely optimistic picture of the predictive ability of the trained model. Furthermore, debiasing was shown A B to decrease classification accuracy for the debiased validation set, presumably because the resulting models could not perform well by simply memorising the training data. 20,22 The purported advantage of models trained using debiased data splits is that they overfit less, so generalise better to make accurate predictions for novel candidate ligands. In this paper, we develop a framework to measure generalisation ability, and explicitly test this hypothesis. We assemble data for 189 targets with > 500 reported active ligands, and split each dataset into a train set and a distant held-out test set used to define the far-AUC, a metric of model generalisation (see Fig. 2a). We then randomly split the train set to produce a random held-out validation set, which is used to measure the standard AUC. Despite achieving stateof-the-art AUCs on the held-out validation sets, our trained ML models struggle to generalise effectively when challenged with the distant held-out test sets. We then apply MUV and AVE debiasing to our 189 random train/validation splits, and use the resulting debiased train sets to build new models. We find that counter to the stated aim, the debiased models make less accurate predictions for novel candidate ligands, as illustrated by their performance on the distant held-out test sets.</p><!><p>We filter protein-ligand binding data from ChEMBL 24.1, 6,29 acquiring active ligands (IC 50 , K i , K d , or EC 50 of less than 1 µM) for each protein target. We acquired data for inactive ligands from PubChem indexed by UniProt Protein ID. 30,31 The handful of cases where ligands were marked both active and inactive by different assays were eliminated. Some targets have fewer inactives than actives, in which case we randomly drew inactives from ChEMBL to achieve an even split for every target. This procedure was repeated every time the algorithm was run, contributing to the error bars shown in the figures.</p><p>We use ECFP6 fingerprints with 2048 bits as the feature set for all models. 32 For consistency we use Tanimoto similarity as the distance metric throughout, computed as 1 − d where d is the Jaccard distance metric from Scipy 33 (note this differs from the specific metric used in MUV 20 ). We first randomly split both the actives and inactives for each protein target into a 70% set and a 30% set. Each 30% set is filtered to build distant heldout test-sets that contain approximately 10% and approximately 25% of the total active and inactive ligands respectively, wherein all ligands are at least 0.4 from every ligand in the 70% set. Our far-AUC metric measured the performance of models on this distant held-out test set, which only contains molecules distant from any molecules (active or inactive) that the model has previously seen.</p><p>We further randomly split the remaining 70% set for each target into train (80%) and validation (20%) sets and train Naive Bayes, Logistic Regression, and Random Forest models using these data splits. All models used were implemented with scikit-learn. 34 We use no prior for Naive Bayes, C = 1 for Logistic Regression, and 100 trees with a maximum depth of 10 for Random Forest. We do not tune model hyperparameters since the focus of our work is the debiasing algorithms. The 3-way train/validation/test split and the requirement for data points far from the training set restricted us We follow the original definitions of AVE and MUV bias. 20,22 Specifically, given a set V of validation molecules and T of training molecules with a similarity threshold d ∈ [0, 1], we define a nearestneighbor function</p><p>where I d (v, T ) = 1 if the distance from the validation molecule v to its nearest neighboring training molecule is smaller than d. We then define a distance function on sets</p><p>with D = {0, 0.01, . . . , 1}. For convenience, let V a , V i , T a , T i be the sets of validation actives, validation inactives, test actives, and test inactives, respectively. Then the AVE bias is defined as</p><p>and the MUV bias is defined as</p><p>To minimise MUV or AVE bias we use the implementation developed in, 22 which largely follows that in. 20 A number of random initial train/validation splits are generated; then the mutation phase of the genetic algorithm involves randomly merging two train/validation splits, moving compounds between the training and validation sets, and deleting compounds from either set. The algorithm ran for 300 iterations or until bias was < 0.01, whichever occurred first, and then produced the least biased split.</p><!><p>We first use our framework to test the extent to which ML models are able to generalise and make accurate predictions for novel candidate ligands. To assess this we compare the ability of each trained model to accurately classify ligands in (i) the random held-out validation set, reported by the standard AUC and (ii) the distant held-out test set, reported by the far-AUC. The distant held-out test sets mimic the real-world need to make accurate predictions for novel candidate ligands that are distinct from the training data. The results of our analysis for Naive Bayes, Logistic Regression and Random Forest models are shown in Fig. 2b. We find that the far-AUC is significantly lower than the MUV debiasing does not consistently help the models to generalise. These results suggest that debiasing does not improve the ability of the models to generalise to novel candidate ligands.</p><p>standard AUC for all models tested across all 189 target datasets, indicating that our models do not generalise well. This confirms the hypothesis that generalisation is a challenge for ML models trained on protein/ligand binding datasets. Although we have not tested more complex models, it is likely that as the complexity of the ML model increases, overfitting to the training data will increase.</p><p>We next assess the extent to which MUV and AVE debiasing alleviate this issue. The key question is whether these algorithms remove data biases that prevent the models from generalising or instead remove useful information that is necessary for the model to learn. This is a nontrivial question, as both MUV and AVE rely on the inherent assumption that the metrics being used to measure distances between ligands do not correlate strongly with the binding activity of the ligand. The ultimate goal of a ML model is to learn a function of the data features that distinguishes ligands that bind to a given protein from those that don't. In contrast, the goal of debiasing is to ensure that actives and inactives are well-mixed according to some distance metric, which is itself a function of the data features. If this distance metric happens to be correlated with the physico-chemical criteria required for binding, then debiasing will remove important information from the training data, potentially harming the performance of the model.</p><p>To address this question we use the distant heldout test sets to compare the existing models with versions trained using debiased train/validation splits. We evaluate the effect of debiasing by computing the change in the far-AUC score between the models trained on the debiased data, and the models trained on the original data for each target. As shown in Fig. 3, neither AVE nor MUV debiasing improve the generalisation ability of the trained models. On average, AVE decreases the far-AUC of Logistic Regression by 0.024 ± 0.030 (mean ± standard deviation) and that of Random Forests by 0.021 ± 0.029, while MUV debiasing decreases the far-AUC of Logistic Regression by 0.004±0.034 and of Random Forest by 0.002 ± 0.036.</p><p>To check whether the far-AUC depends on the extent to which the data were debiased by either MUV or AVE, Figs. 4a and 4b show the change in far-AUC as a function of the final dataset bias achieved by AVE and MUV respectively. We find no correlation in each case. Supplementary Fig. S1 confirms that this still holds if we instead consider the change in MUV bias, to account for the initial MUV bias measured.</p><p>We further probe if the number of active ligands for a target indicates whether debiasing will prove effective. Figs. 4c and 4d show no correlation between the number of active ligands and the change in far-AUC achieved by debiasing. However, the magnitude of the change in far-AUC decreases slightly as the number of active ligands increases, suggesting that debiasing has a smaller effect for large datasets. In addition, we checked These results suggest that neither approach has much effect for large datasets.</p><p>whether there was any relationship between the change in the far-AUC and the number of decoy molecules added for a given target to achieve equal numbers of active and inactive ligands. Supplementary Fig. S2 confirms that there is no correlation.</p><p>To better understand these findings, and ask whether there are situations in which debiasing does improve the ability to generalise, we examine the data generated by our experiment in more detail. We first examine far-AUC trajectories measured during debiasing. Supplementary Figs. S3 and S4 suggest that there is no positive correlation between the far-AUC achieved and reduction of either the MUV or AVE bias. The jagged behaviour seen for both algorithms suggests that certain moves that significantly decrease the far-AUC are sometimes preferred.</p><p>We hypothesised that moves in these trajecto-ries that decrease the far-AUC correspond to data deletion, a move allowed by the genetic algorithm. For example, Supplementary Fig. S5 shows how the size of the dataset for ChEMBL 5508 decreases during the debiasing process. To evaluate debiasing in the absence of data deletion, we modified the genetic algorithm to prevent data from being deleted. Supplementary Fig. S5 shows that the modified versions of both MUV and AVE debiasing still succeed at reducing the bias of the dataset. Could this provide an approach that better enables the models to generalise? To test this we repeated our earlier analysis with the modified debiasing algorithms.</p><p>As shown in Fig. 5, we find that forbidding deletion does slightly improve the generalisation ability of the resulting debiased models for both AVE and MUV compared to the versions with deletion. However, even without deletion, AVE decreases the  far-AUC of Logistic Regression by 0.016 ± 0.041 and of Random Forest by 0.017 ± 0.044 on average across the 189 datasets, while MUV debiasing decreases the far-AUC of Logistic Regression by 0.0006 ± 0.037 and of Random Forest by 0.001 ± 0.038. As in the case where deletion is allowed, Figs. 6a and 6b confirm that there is no relationship between the extent to which the data were debiased, and the change in far-AUC obtained.</p><p>Overall, our results indicate that it is difficult to predict when AVE or MUV debiasing will improved generalisation. Further work is needed to determine the conditions under which debiasing has the potential to improve the ability of a model to generalise and make accurate predictions for novel candidate ligands.</p><!><p>In this paper we develop a simple far-AUC metric that measures the ability of a protein-ligand binding model to generalise and make accurate predictions for novel candidate ligands. We use this metric to evaluate the AVE and MUV debiasing algorithms that were designed to reduce overfitting to the training data, and thus potentially improve the ability of models to generalise. Our analysis for both AVE and MUV debiasing in Fig. 3 finds that debiasing does not systematically improve the ability of the trained models to generalise, despite the fact that Fig. 2b shows there is significant room for improvement.</p><p>This suggests that debiasing algorithms are not able to accurately distinguish signal from bias, and in many cases remove relevant information from the training data. Our analysis of the debiasing trajectories suggests that the deletion operation used by the genetic algorithm in both MUV and AVE debiasing may exacerbate this loss of useful information or signal from the data. To address this we implemented versions that did not allow data points to be deleted. This did not result in models that were better able to generalise compared to those built without debiasing.</p><p>Dataset bias is clearly an important issue, particularly in chemistry; however, current debiasing approaches need to be applied carefully to ensure that they do not eliminate relevant information. Indeed, some clustering among actives is to expected in fingerprint space, since active ligands for a given protein can have structurally similar features. It is important to distinguish between this clustering and artificial clustering that may result from the fact that only portions of chemical space have been explored by synthetic chemists, or other potential sources of bias.</p><p>Another approach is to better understand the regions of chemical space in which protein-ligand binding models trained using a particular dataset are able to make accurate predictions. Generalisation is challenging for ML models across many contexts, even when trained with unbiased datasets, so given the highly biased nature of chemical data, expecting protein-ligand binding models to generalise may be ambitious. Methods that establish the domain of applicability for trained models need to be developed to provide confidence in those predictions that fall within this domain. This approach would have the advantage of avoiding the information/bias distinguishability problem described above while still allowing the resulting models to generalise to some degree. Acknowledgement V.S. acknowledges support from the Winston Churchill Foundation of the USA. Computations for this project were run on the ziggy cluster at Centre for Molecular Informatics, Chemistry Department, University of Cambridge. L.J.C. acknowledges support from the Simons Foundation.</p><!><p>The supplement includes figures showing the relationship between the change in far-AUC and the change in MUV bias, as well as the number of decoy molecules added. It also shows how the far-AUC changes during the process of running the debiasing algorithm and contains an analysis of how dataset bias depends on the size of the dataset and demonstrates the debiasing without deleting data points is possible. Code necessary for generating the dataset used in this paper, running the debiasing algorithm, and splitting the dataset to compute the far-AUC for an arbitrary model has also been provided. This material is available free of charge via the Internet at http://pubs.acs.org/.</p>