The data leakage described above is easy to spot, but sometimes test set contamination occurs via more complex mechanisms. One example is our own Ref. 78 where a multitask model was trained on computational activation energies and reaction enthalpies of chemical reactions. Both forward and reverse reactions were included and were treated as independent datapoints, so some of the reactions in the test set had their reverse counterpart in the training set. Out of several developed model architectures, one performed especially well, with accuracies close to chemical accuracy. However, this chosen architecture mainly excelled over the other potential architectures in exploiting the data leakage efficiently, leading to a seemingly good performance. When tested on independent reactions, however, the model produced errors about half an order of magnitude larger than reported. Only after removing the data leakage were we able to develop different architectures with better performance. In this case, test set contamination not only caused the reported test set error to be too low, but also hindered model development and optimization. Similar cases were reported in literature, where without a rigorous splitting strategy it was impossible to select the best architecture and parameters for reaction models. Splitting in systems involving multiple molecules that each may need rigorous splitting (e.g., solute-solvent pairs) creates an additional complication. In our previous work, we showed the importance of data splitting by excluding several solvents, solutes, and substructures from the training set. For chemical datasets, splitting according to molecular scaffolds or time-stamps 80 can be an appropriate measure to prevent data leakage. In fact, the performance of many models was shown to drop significantly if evaluated on a more rigorous basis than simple random splits. However, scaffold splitting also comes with shortcomings, some of which have no easy remedy, as chemical space is inherently non-uniform and unbalanced. A detailed determination of optimal splitting strategies extends beyond the scope of this study. In general, we recommend rigorously splitting datasets when developing new models, and paying increased attention to possible sources of data leakage.