Abbr. Possibilities Based on the user's choices of D, T , K, L, and C K , MatFold can typically create thousands of splits. The feasibility of training this many models may depend on the dataset size and modeling approach and may be less feasible, for example, in the training of recently developed universal ML potentials. However, dataset and model sizes are often small enough for more specialized ML-based materials discovery models to perform splits across at least some subset of the criteria summarized in Table . Subsequent exemplars based on our previous work (modeling vacancy defect formation energies and surface work functions ), we are able to train thousands of model splits generated by MatFold to obtain improved insights into our model's generalizability and limitations. An overview of the two datasets and the chosen Mat-Fold split protocols for each are listed in Table . Model hyperparameters are fixed at the optimal conditions as determined in the respective previous work. To evaluate the model performances, we denote the mean absolute error of an outer test set MAE = 1/N k i |p i -p i |, where N k is the number of samples in outer fold k, pi is the model prediction of sample i, and p i the truth value. The expected model performance is given as the ensemble average over the set of all K folds, ⟨{MAE} K ⟩. For non-nested CV, i.e., K-fold, pi in the k th test set is predicted by a single model trained on the k th train set. For nested CV, i.e., (K, L)-fold, the final prediction is the ensemble average over the set of inner model predictions on the outer test set, ⟨{ pi } L ⟩. The deviation of that ensemble average prediction from the true value is referred to as residuals, calculated as |p i -⟨{ pi } L ⟩|. Importantly, nested CV also yields an uncertainty metric We note that for datasets with strong imbalances in splitting labels (e.g., an element present in almost the entire dataset vs. another element being present only in a tiny fraction of the dataset) the MAE and its standard deviation may be affected by the random seed during split generation. This can be mitigated in MatFold by specifying a minimum and maximum threshold of split label prevalence that determines whether that label is considered during the CV procedure or is always enforced to be in the training set. For example, if oxygen is present in 90% of structures in the dataset and the user specifies a maximum threshold of 0.9, then oxygen-containing structures will be part of the training set by default during CV.