For the model-agnostic approach, we calculated the distance between the OOD test set and training set using a k-nearest neighbors approach (k=5). For both the Tanimoto and TMD metrics, all pairwise comparisons between splitting strategies showed significant differences (Tukey HSD 45 test, p < 0.05), indicating that each strategy produced distinctly different distribution shifts (Figure ). for definitions) as the least challenging ones under investigation (median Tanimoto distances of 0.577, 0.610, and 0.614; median TMDs of 120, 133, and 138, respectively). Prior studies also reported that scaffold splitting can lead to overly optimistic performance estimates. We hypothesize that generating scaffolds with the Bemis-Murcko algorithm results in scaffolds that represent relatively few molecules each. As the ratio of unique scaffolds to the total number of molecules increases and approaches unity, scaffold splitting converges toward random splitting (in the extreme case, where each molecule corresponds to a unique scaffold, the two splitting meth-ods become mathematically equivalent). To validate this hypothesis, we quantified the ratio of unique scaffolds to the total number of molecules for each data set (Figure ). The two distance metrics disagree on identifying the most challenging splitters. According to the Tanimoto distance, the most challenging splitters are molecular weight reverse and max dissimilarity (medians of 0.682 and 0.679, respectively). In contrast, according to the TMD, the hardest splitters are molecular weight and max dissimilarity (medians of 237 and 176, respectively). These findings indicate that, although there is a moderate correlation between the rankings based on these two metrics, they capture different aspects of splitting strategies.