Many machine learning models are built to be useful for materials discovery. Discovery implies predicting something unknown. However, the common practice of using a random train/test split does not necessarily measure the model performance in predicting the unknown. First, a simple random train/test split cannot account for the fact that many databases contain very similar structures (see Case Study 1). As we have shown, (hypothetical) databases often contain multiple structures that are only different in the type or position of one functional group. Dividing such a group of structures with identical scaffolds across train and test set will not give a faithful measure of the generalization ability of the model as one of the main assumptions of testing is violated-train and test set are not entirely independent of each other (one might see the functionalized graphs as children of the same scaffold). Second, for practical applications, there will almost always be a data shift; that is, the data distribution the model will be used with will be different from the distribution the model has been trained on. For example, it is well known that structures in hypothetical databases do not have the same distribution (e.g., lower density, less metal diversity) as structures in experimental databases. This has already been recognized by others such as Meredig et al. who utilized leave-one-cluster-out cross-validation to estimate extrapolation performance (or Xiong et al. using k-fold forward cross-validation). A similar approach, in which one clusters the principal components of the data into k clusters and trains on k -1 clusters and tests on the cluster that was not used for training, is implemented in mofdscribe. As each cluster has specific properties, this method tests how well the model can extrapolate to new properties.