To illustrate these challenges, let us consider as example the discovery of double perovskite oxides with high ab initio computed bulk modulus, where we use two popular statistical models, Gaussian process (GP) regression and random forest (RF), and two BBO data acquisition strategies, expected improvement (EI) of rewards and pure exploitation (XT). GPs are the traditional BBO model, because their Bayesian approach provides a principled quantification of "epistemic" uncertainty, i.e., uncertainty from a lack of training data related to a specific test point. However, they can struggle already with moderately high-dimensional representations such as the 24 features used in this example. In contrast, RFs are known to work robustly well with high-dimensional feature spaces , while their ensemble-based uncertainty quantification does not represent epistemic uncertainty. Interestingly, as shown in Figure , CV indicates that RF has the better in-distribution predictive performance not only in terms of squared error but also in terms of log loss, which takes uncertainty into account. Nevertheless, RF is outperformed by GP in terms of the produced discovery rewards, demonstrating that standard in-distribution performance estimation techniques can suggest sub-optimal methods. This demonstrates that already method selection is a real challenge for practical problems. However, the situation is much worse for methodological research that aims to not only determine, which of a small number of established methods works best, but to test dozens of combinations of models and acquisition functions. Absent innovation in performance estimation, comparing 𝐾 methods in terms of their expected discovery reward across 𝐿 repetitons of 𝑇 rounds requires the acquisition of 𝐾𝐿𝑇 labels in addition to any pre-generated initial data. This is because, even when starting from a common initial training distribution, each method produces its own sequence of proposal distributions. Since these distributions are unknown a priori, there is no way to pre-generate data from them, blocking the usual collaborative improvement process around an initially released dataset. Thus, the prohibitive cost of expected reward estimation currently blocks substantial progress in addressing other important challenges like unsound uncertainty quantification or acquisition function optimization with infinite candidate populations particularly when using non-invertible materials representations.