Why you shouldn't upsample before cross validation
To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler &quot;baseline&quot; case, where we simply upsample (i.e. create duplicate samples) without SMOTE.
The first reason why such a procedure is invalid is that, this way, some of the duplicates due to upsampling will end up both to the training and the validation splits (CV folds); the result being that the algorithm is validated with some samples that have already been seen during training, which invalidates the very fundamental requirement of a validation set (fold) and it is actually the very definition of data leakage. For more details, see own answer in the SO thread Process for oversampling data for imbalanced binary classification; quoting from there:

I once witnessed a case where the modeller was struggling to understand why he was getting a ~ 100% test accuracy, much higher than his training one; turned out his initial dataset was full of duplicates -no class imbalance here, but the idea is similar- and several of these duplicates naturally ended up in his test set after the split, without of course being new or unseen data...

But there is also a second reason: this procedure shows biased performance measures in our validation folds that are no longer representative of reality: remember, we want our validation folds to be representative of the real unseen data, which of course will be imbalanced. Performing CV after upsampling results also to artificially balancing our validation folds; doing so, and claiming that we get X% accuracy when a great part of this accuracy will be due to the artificially upsampled minority class makes no sense, and gives misleading impressions. For details, see own answer in the SO thread Balance classes in cross validation. Notice that the author of the post you have linked to says (rather cryptically, and only in a parenthesis):

(we are smart enough not to oversample the test data)

For more corroboration, here is Max Kuhn, creator of the caret R package and co-author of the (highly recommended) Applied Predictive Modelling textbook, in Chapter 11: Subsampling For Class Imbalances of the caret ebook:

You would never want to artificially balance the test set; its class frequencies should be in-line with what one would see “in the wild”.


Now, it's true that the above hold for the case of balancing through simple upsampling of the minority class; but SMOTE does not do that - it uses interpolation to create synthetic samples that are &quot;close&quot; enough to the real minority ones. How does this change the situation?
Not much.

The second reason stated above (biased performance measures in the validation folds) is still fully applicable - in fact, it holds independently of the exact nature of the upsampling (duplicate samples or synthetic ones).

Given that the synthetic samples generated by SMOTE are indeed highly correlated with the real ones, the problems due to the first reason mentioned above are still largely present, although somewhat ameliorated.


In contrast, the pipeline approach does not suffer from these issues, because it first splits into training and validation folds, and applies SMOTE subsequently only to the training ones.