To investigate the impact of splitting strategies, we train models on experimental data (for which we can also perform a time-based split) and then evaluate how well the models generalize to hypothetical materials. We trained the gradient boosted decision tree models using the default feature set of CoRE dataset in mofdscribe (currently including histograms of persistence diagrams, AMD, geometric properties, APRDF) to predict the methane deliverable capacity. For all experiments, we remove duplicates, i.e., materials with identical structure graphs. We then optimize hyperparameters of gradient boosted decision trees using Bayesian optimization on the validation set and train the model using different splitting strategies, always keeping the train/validation/test ratios fixed. The figure uses the random split as the control group and computes bootstrapped mean effect sizes for the different splitting approaches. We see that all splitting strategies lead to models with different generalization performances (better in all cases except for the density-based split) than the random control group. Box Fig. Bootstrapped mean difference in mean absolute error for out-of-domain predictions as a function of the splitting strategy. We tuned gradient boosted decision tree models using the different feature sets implemented in mofdscribe (currently including histograms of persistence diagrams, AMD, geometric properties, APRDF) using different featurizers on the CoRE MOF dataset to predict the methane deliverable capacity and evaluate the performance on the ARABG dataset. We ran every experiment around 20 times and then computed bootstrapped effect sizes with respect to the random split performance.