Understanding and quantifying the generalizability, improvability, and uncertainty of machine learning (ML)based materials discovery models is critical, especially in applications where downstream experimental validation (synthesis, characterization, and testing) is often time-and cost-intensive. Careful, and sometimes extensive, cross-validation (CV) is required to both avoid erroneous conclusions regarding a model's capabilities and to fully understand its limitations. Withholding randomly selected test data is often insufficient for quantifying a model's performance as this sub-set is drawn from the same distribution that potentially suffers from data leakage. This in-distribution (ID) generalization error is typ-Supplementary Information: the ∆H V dataset is provided in the supplementary_files_defects.zip. Additional CV analysis showing inference performance for additional hold-out strategies. MAE heatmaps and parity plots for leave-one-element-out splits.