In recent years, the materials science community has established a large-scale infrastructure for data sharing that promises to increase the efficiency of the "data-driven" discovery of novel useful materials . Growing data collections are envisioned to lead to increasingly accurate statistical models for property prediction that can significantly reduce the number of necessary experiments or first principles computations and, thus, substantially improve the cost and time for critical discoveries . Indeed, the combination of public datasets and robust statistical estimation techniques like cross validation (CV) enables a collaborative improvement process ("common task framework" ). As a result, there are now models that can predict certain materials properties well on average with respect to the same distribution as the training data. Unfortunately, the in-distribution expected performance, as estimated by CV, is not directly coupled with the performance for the discovery of novel materials: expected performance fails to capture the model behavior for the very few exceptional materials that one aims to discover, and, fundamentally, in-distribution performance is irrelevant for a discovery process that is designed to generate high-performing materials more frequently than they occur in the initial training data.