To evaluate the effectiveness of the machine learning models that we developed, we considered three different HTVS scenarios and various possible strategies available in each case. We consider different scenarios because the available resources and requirements for a catalyst influence the optimal HTVS strategy. In both scenarios, we use the model predictions on the data withheld as the test set in each part of 10-fold cross validation such that we can reuse the entire dataset as the search space. The metric for computational cost in each scenario is the number of DFT O2-binding calculations performed, as this is by far the most expensive step of HTVS for these catalysts. Although it would be desirable to eliminate O2binding calculations entirely, we assume for all strategies (even those that use machine learning) that they are necessary to verify that a particular site is indeed active. This is how such a search would likely be done in practice, as DFT will always be trusted over a model trained on DFT and verifying the ML model with DFT calculations, increasing the probability that the desired criteria are met when the catalyst is tested experimentally. This is important because experimental confirmation of a catalyst's activity is generally orders of magnitude more expensive than the DFT compute time for that catalyst. In the first scenario, the goal was to find 100 catalysts each with at least one confirmed active site. This scenario applies whenever the relative activity ordering of the catalysts is less important than the possibility of catalytic activity, for example when identifying structures that can be used as starting points for further modification, or which will be filtered using additional models or criteria. One strategy, which requires only DFT, is to try binding O2 to potential active sites at random until an active one is found. We compare this to a different strategy, which uses our classifier to predict which sites are most likely to be active and then runs O2-binding calculations at only those sites. In this scenario, it is not necessary to find all of the active sites for a particular catalyst, only to confirm that at least one is present, so in both methods all of a catalyst's sites were removed from the search space as soon as the first active site on that catalyst was found. The number of DFT O2 binding calculations necessary to find 100 active catalysts using each of these strategies is shown in Figure . The number of calculations over 10 trials using the classifier trained with all features averaged to 101.3 ± 0.4, the classifier trained using only structural features averaged to 115.2 ± 2.6 calculations, and the randomly testing sites averaged to 576.6 ± 58.1 calculations. The performance of the classifier trained on structural features overall is not unexpected, as its accuracy in this trial was similar to its accuracy of 88.3% during cross-validation. In contrast, the performance of the classifier trained on all features is much better than a classification accuracy of 95.3% suggested, indicating this classifier had a much greater accuracy for the sites it was most confident were active. This principle is illustrated in Figure , which depicts the RFC class probability for the active and inactive sites. Notably, there are only a handful of inactive sites that the model is incorrectly confident are active, entirely analogous to the RFC's performance in this HTVS scenario. As at minimum 100 O2-binding calculations are necessary in this scenario to confirm the active sites in this scenario, there is little room for further improvement on this classifier. In the second scenario we developed, the goal was to find active sites within the set of active sites that bind O2 most strongly. In contrast to the previous scenario, the goal here is focused on finding particular active sites rather than catalysts, so catalysts are not removed from the search space after one active site on them is found as a catalyst could have multiple active sites. This scenario could apply, for example, if the activity of a set of catalysts were found to be limited by the binding strength of oxygen intermediates and the goal were to increase the binding strength. This goal is similar to the scenario where there is a specific known target value for the binding strength of all oxygen intermediates, as is common for many classes of ORR catalysts with strongly correlated intermediate energies frequently depicted in what are referred to as volcano plots. As before, the baseline we will compare against is the strategy that chooses active sites randomly. Specifically, this randomized approach selects catalysts randomly and binds O2 to all possible active sites. We designated the 15 strongest binding sites (out of the total 4039) as the set of "best" sites that each method is trying to find.