XGBoost: what to do when Kfold is not enough? I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform the detection over some features that I have created. The time-series are not sampled uniformly and the time-order looks not so important for this specific problem so far.

The problem is that when I perform the StratifiedKFold (as per Sklearn) the results looks promising and the standard deviation of the relevant metrics among the kfold is really small. Nevertheless, if I remove one time-series entirely from the training set and fitting the model over the other ones I am not able to replicate the same results.

This gap between Kfold performances and "real test" looks to me like the training is not really generalising the problem, despite the good results during the Kfold validation. 

Do you have any idea to fix this problem? or any advice?
Using normal KFold cross-validation for time-series data will yield a highly optimistic error estimate since you are using data from the future to predict the past. The model just has to learn to interpolate, not to predict.
Therefore you have to use time-wise CV:

Furthermore, if your goal is to predict the future for a time-series you do not know the past of, you have to use leave-one-group-out, time-wise-CV to get a realistic performance estimate: Train on all time-series except for a test-set of time-series, but only using data up to a certain point in time. Validate on the test-time-series but only using data after the timepoint. 

I doubt you will have success with this approach since this is often a very difficult problem.

For choosing the right cross-validation method you have to be clear about the goal of your method.


Do you want to interpolate missing data for a couple of time-series?
Use K-Fold CV
Do you want to predict the future of some time-series, for which you
know the past? Use time-wise CV
Do you want to predict the future of some time-series, but you only
know the past of some other time series? Use Leave-one-Group-out,
time-wise CV


I assume you are after the second point since this is the most common problem.