Cross validation schema for imbalanced dataset
I'm not sure if there's a question here, but I'll add some comments.
Firstly, if you can get it in the wild, always work with balanced data. However, if you are going to manually create a &quot;balanced&quot; data set yourself, make sure that the selection criteria that you use to create that data is appropriate. As an example, choosing the 100k most recent positive and negative outcomes may not be appropriate because the time frame of the positive outcomes may extend well beyond that of the more common negative outcomes. So in this 200k data set, your negative 100k outcomes may relate to data from the last year while the data relating to your 100k positive outcomes may relate to the last ten years.
Secondly, if you are going to balance your data be aware of how the balancing technique works and try to understand its weaknesses / limitations. Be mindful that rebalancing a data set will result in a new data set, and remember that you will have to check that the new data set is still appropriate to use. As an example, you will need to check that the distribution of input variables is still roughly the same as before. Applying this thinking to your options above, can you be certain that the data in each fold will be roughly similar?
Lastly, if you are going to use a modelling framework which can handle imbalanced data then make sure you understand why it can handle the imbalanced data. In particular, if the framework applies some weighting / balancing technique in the background you should be aware of this and be able to explain it.