Gap leaderboard score and model scoring on a Competition I'm working on a Veolia challenge on Ens Data Challenge ens-data (equivalent to Kaggle) the goal is to classify very rare binary events (the failure of a pipeline ) for 2014 and 2015 (y={2014,2015}). In input we have 5 features 3 categorical features (which I turned into dummy variable) and two continuous. The score is average AUC, $0.6*AUC_1 + 0.4*AUC_2$.

My problem is the following, when I compute each AUC (for 2014 and 2015) with a stratified kfold cross validation and I compute the average AUC I get roughly 0.88 and when I submit on the website I end up with 0.67, I guess there is a problem in my code.

Here is my code for choosing the best model for 2014:

Rk: to predict on the test set (2014,2015 unknown), I first predict with all 5 features, 2014. Then I add the prediction of 2014 to my feature to predict 2015

# Spot Check Algorithms
models = []
models.append(('LG', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('RF', RandomForestClassifier()))
models.append(('GBC', GradientBoostingClassifier()))
models.append(('SVM', SVC()))
# evaluate each model in turn
results = []
names = []

# stratifiedkfold is defined by default when there is an integer
scoring = 'roc_auc'
num_folds = 10

for name, model in models:
    cv_results = cross_validation.cross_val_score(model, X, Y, cv=num_folds, scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)
As you have commented, you are concerning about over-fitting.

In fact, cross validation will help to weaken over-fitting, but it can't eliminate over-fitting. Model scoring on TRAIN dataset sometimes exceeds scoring on TEST dataset. Here are some examples I can find:


Deep Residual Learning for Image Recognition (the famous ResNet paper). Checkout figure.1
This kernel of Two Sigma Financial Modeling Challenge on Kaggle says:



  we are getting a public score of 0.0169 which is slightly better than the previous one.
  Submitting this model to the LB gave me a score of 0.006


In my practice, I am more willing to concern the scoring gap between TRAIN, VALIDATION(DEVELOP) and TEST dataset. Personally speaking, TRAIN>VALIDATION=TEST is better than TRAIN=VALIDATION>TEST.

Edit: For class imbalance problem, there are some resources:


8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset. This blog shows a common workflow dealing with imbalanced class issue.
Class Imbalance Problem in Data Mining: Review. This paper compares several algorithms created for solving the class imbalance problem.