No clear standard has emerged for how to quantify the extrapolative limits of machine learning models. Standard cross-validation is designed for interpolation; alternate methods such as LOCO-CV are needed to ensure that the test set is significantly different than the training set for evaluating extrapolation. For example, several reports have indicated that the current set of crystal graph neural network algorithms, despite performing well on the matbench data set, may have issues generalizing out of distribution . To this end, extensions like matbench-discovery attempts to test on out of sample (i.e., not in Materials Project ) structures to better test extrapolation. Nevertheless, the issue is far from solved. As training data sets grow in size and scope, it becomes more difficult to find and evaluate samples that are significantly outside the training domain. More fundamentally, evaluating ML models for open-ended exploration (e.g., as in iterative machine learning or generative models) remains a fruitful area of research because standard performance metrics do not translate directly to these areas and validating each new prediction for every algorithm can be expensive. Thus, many opportunties remain to better investigate the extrapolation capability of ML models.