One approach to constructing OOD test sets is to utilize unsupervised clustering with a chosen materials featurization and then conduct leave-one-cluster-out CV (LOCO-CV). For example, on compositional models for superconducting transition temperatures, LOCO-CV revealed how generalizability and expected accuracy are drastically overestimated due to data leakage in random train/test splits. Omee et al. have investigated the performance of OOD prediction tasks on MatBench datasets (refractive index, shear modulus, and formation energy) utilizing structure-based graph neural network (GNN) models and LOCO-CV (k-means clustering and t-distributed stochastic neighbor embedding). Hu et al. similarly have utilized LOCO-CV to study the benefit of various domain adaptation algorithms for materials property predictions (experimental band gaps and bulk metallic glass formation ability). Further examples of studying generalization error based on featurization and clustering algorithms include kernel density estimate and uniform manifold approximation and projection. Another approach to assess OOD generalization is based on ensembling ML models to obtain a set of predictions. The averaged predictions can exhibit more robust OOD generalization behavior and the standard deviation yields an uncertainty metric. Ensembling can be effectively applied to any bagged regressor ML model and has also been implemented for GNNs and other deep neural networks. Other recent work includes fitting a single model to estimate ensemble error bars by leveraging synthetic data augmentation, 10 mitigating data bias arising from uneven coverage of materials families by entropytargeted active learning, and a study on OOD generalization of formation energy models with structural and chemical hold-outs. To further encourage standardized reporting of these types of detailed insights into generalization performance and limitations of ML-based models in the materials sciences, here we provide "MatFold" as a model-agnostic programmatic tool for automatically generating CV splits for arbitrary materials datasets and model architectures, such as structure-based or composition-based 14 models. Specifically, we propose a standardized series of CV splits based on increasingly difficult chemical/structural holdout criteria, dataset size reduction, nested vs. non-nested splits, and others. By assessing model performance across various combinations of MatFold splitting criteria, one could, for example, more fairly compare the performance of differing approaches with the same modeling objectives. This approach allows for a better understanding of how well models' predictions generalize with increasingly difficult chemical or structural hold-out criteria. Additionally, it can determine the expected model improvement with continued data acquisition and assess whether this improvement depends on the splitting criteria used for OOD generalization. Furthermore, the method evaluates whether nested CV ensembles enhance OOD predictions and quantifies the extent of this improvement. It also examines the reliability of uncertainty estimates derived from nested CV ensembles and whether this reliability varies based on the splitting criteria used for assessing generalization.