Using increasingly general (i.e., more structures are considered as duplicates) definitions of duplicates (Figure ), we analyze how many matching structures we find in the all-solvents removed (ASR) subset of the CSD MOF subset (2019), and the CoRE and QMOF databases. The inverse cumulative histograms below plot how often we find n or more identical hashes (e.g. at n = 10, the count represents the number of structures with 10 or more identical hashes). The "graph" strategy considers all structures which share the same connectivity and atom types as identical. The "undecorated graph" (undec. graph) strategy does not consider the atom types. The "scaffold" strategy removes all functional groups, solvents, hydrogens, and terminal atoms from consideration (formally, all subgraphs connected via bridges). Again, we can also remove the atom types from consideration. As expected, we see an increase in the larger number of duplicate counts from left to right. This analysis shows that, for instance, the Co-CPO-74 structure appears 114 times in the ASR CSD MOF subset. This implies that this structure will likely appear both in the training and test set. Importantly, this is not the only structure with many duplicates; there are of the order of 100 structures in which structure graph appears more than ten times in the CoRE MOF database-and hence do not contribute to a meaningful measure of the generalization performance of the model. Analyzing a database of in-silico assembled MOF structures (BW) we find notably many scaffold duplicates: Among nearly 20 thousand structures we only find 1584 unique undecorated scaffolds. Many in-silico MOF assembly approaches enumerate all possible combinations of building blocks, nets, and functional groups. While this approach can give a lot of detailed insights into regions of chemical space, it will also give rise to many very similar structures that can lead to the violation of the assumption of independence between training and test set. To illustrate this, we can simulate a thousand random train/test splits (as commonly done) and measure how often a scaffold occurs both in the training and test set. For practical train/test ratios, the majority of scaffolds (e.g., 55 % for a train/test ratio of 0.8/0.2) will be found in both training and test set. To showcase the potential impact of such data leakage on model evaluation, we computed learning curves (for CO 2 Henry coefficients in the BW database as implemented in mofdscribe) for two deduplication levels: No deduplication, and removal of identical undecorated scaffolds. If there would not have been any data leakage, the learning curves should be similar. However, for the deduplicated datasets we observe that the initial learning is faster (presumably because of a higher information density in deduplicated datasets) but much lower than for the dataset containing undecorated scaffold duplicates. The increase in performance we observe for the data set with duplicates is most likely caused by data leakage, the test set contains many structures that are only marginally different from the training set, and hence if we remove these duplicates from our training and test, our error worsens significantly. Depending on the strictness of the duplicate definition (Figure ) one might see-with the same train and test set sizes-drastically larger errors. Moreover, it is important to realize that in our case study, the removal of duplicates increased the learning (steeper learning curves; in fact, this can sometimes lead to better models) and led to a more faithful measure of generalization performance. Box Fig. Learning curves with and without duplicated identical scaffolds. For this experiment, we trained gradient boosted decision trees using the default feature set (currently including histograms of persistence diagrams, AMD, geometric properties, APRDF) implemented in mofdscribe on the BW database subset used in Moosavi et al. to predict the CO 2 Henry coefficient (which we reuse from Moosavi et al. ). We used a train/test/valid split of 0.7/0.2/0.1 and performed the experiment 100 times. The shaded areas indicate 95% confidence intervals around the mean.