Significant difference in testing vs cross validation accuracy What could be the possible reasons for a significant difference in cross validation and testing f1_scores? I am performing 3 fold Stratified cross validation and the testing f1_score is almost 0.15 less than cross validation score. How can I come up with a more effective cross validation strategy so that the two scores are closer?
Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted. 

A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation. 

Possible strategies to get a better estimation of the model performance would be:


useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)
considering simpler models (e.g., less variables, more general parameters)
considering other machine learning algorithms