4 Design of the test set From the above examples, it should be apparent that the distribution of test set uncertainties is a key factor determining what we can expect from the UQ validation metrics. Especially ranking-based UQ validation metrics are sensitive to the uncertainty distribution, e.g. for the models presented here we observed ρ sim rank values ranging between 0.04 (practically no correlation expected) and 0.57. A high ρ rank is only possible if the test set uncertainties are well separated (wide distribution). In other words, there should be a big difference in the model performance across different parts of the test set. Thus, if one wants to be able to test the ability of the UQ estimates to distinguish between very high and very low uncertainty predictions, it is important to think about the design on the test set. Many ML models in chemical research are still trained and tested based on a random split of some data set. Assuming a relatively homogeneous data set we should not expect an especially big difference in model performance across the test set. How to design training and test set in order to test e.g. the ability of the model to generalize is subject to increasing attention in the chemical ML literature and we expect similar considerations on test set design for UQ validation to be important going forward. Here, we illustrate the effect of how the test set is designed on a ranking-based method such as ρ rank through a simple toy example introduced in the original deep evidential regression paper by Amini et al. . Amini et al. tested the performance of their epistemic evidential uncertainties by training a model on y = x 3 + ϵ, with ϵ drawn from a Gaussian error distribution with standard deviation 3 representing the aleatoric uncertainty. They train the evidential model in the range X train ∈ [-4; 4] but test in the range X test ∈ [-6; 6] resulting in a very convincing plot (Figure in ) showing how the uncertainty (and error) increases outside the training interval, where the model is forced to extrapolate. Similarly, we train a model with X train ∈ [-4; 4] but use the same model on two different test sets; one with samples drawn within the training set range [-4;4] and one with samples drawn in the range as in the original study where 1/3 of the test samples are outside of the training set interval. Figure illustrates the different situations and are accompanied by the corresponding Spearman's rank correlation coefficients (0.05 vs 0.65). Note that while ρ sim rank = 0.65±0.02 is in good agreement for the test set containing out-of-distribution (OOD) samples, that is not the case for the test set within the training distribution (ρ sim rank = 0.27±0.03) which indicates that the uncertainties within the training distribution but close to the border are overestimated. This clearly illustrates the effect of how the test set is designed on ranking-based metrics; if we want to test the UQ method's ability to distinguish between high and low uncertainty prediction we have to design the test set accordingly. Another point should also be noted here; a method for testing the ability of a model to generalize is to compare model error for an in-distribution test set with an OOD test set. Success is in this scenario as little difference in test error/model accuracy as possible. However, for uncertainty predictions we should expect a change in the ranking-based metrics as we add OOD data points to the test set.