Recognizing these issues, the community increasingly focusses on active learning approaches like Bayesian optimization for model-driven blackbox optimization (BBO). These methods manage an iterative modelling and data acquisition process and aim to optimize the cumulative "reward" received for the acquired data points over time, such as the maximum property value discovered so far. This process, illustrated in Figure , is enabled by an acquisition function that leverages the predictions of a statistical model together with its uncertainty quantification to effectively manage the underlying tradeoff of exploration (learning more about the candidate space) and exploitation (aim to sample high value candidates). This shift to consider actions instead of just predictions constitutes an important step towards accelerated materials discovery, but it reveals shortcomings not only in existing modelling approaches but more fundamentally in the methodological framework used to improve those models. In particular, the inapplicability of established performance estimation frameworks based on pre-generated data renders it extremely costly to conclusively compare and to systematically improve methods. where 𝑃 is the modelled cumulative distribution function; and (iii) label for top-ranked material is acquired and added to data sample generating reward, e.g., defined as 𝑅 𝑡 = 𝑚𝑎𝑥{𝑦(𝑋 𝑖 ): -𝑁 < 𝑖 ≤ 𝑡} when maximizing a single property or figure of merit 𝑦, which incentivizes the discovery of materials with high 𝑦-value as early as possible in the process. While standard statistical analysis assumes the initial data points 𝑋 -𝑁+1 , … , 𝑋 0 to be drawn with respect to some sampling distribution 𝐷 0 , this distribution does not have to be balanced or representative of the whole population. However, any concentration away from a representative, i.e., uniform, sampling distribution, poses the risk of delayed reward generation, and a misspecified acquisition function or model, in particular one with over-confident predictions, even risks to never escape local maxima represented in the initial data collection. The sampling distribution of subsequent points 𝐷 1 , 𝐷 2 , … , 𝐷 𝑇 vary and depend on the combination of model 𝑝 and acquisition function 𝑎. Hence, they cannot be pre-generated for new methods rendering label generation a key bottleneck in method development.