Splitting the datasets. To estimate the generalization performance of a model, it needs to be evaluated on data it has not seen before (i.e. is independent of the training data) and, ideally, mimics the distribution of data the model will be used on. For this, one typically splits the dataset collected in step into multiple parts. However, as we show, the chosen strategy can have an important impact on model selection and interpretation of the results.