I'm working on a Veolia challenge on Ens Data Challenge ens-data (equivalent to Kaggle) the goal is to classify very rare binary events (the failure of a pipeline ) for 2014 and 2015 (y={2014,2015}). In input we have 5 features 3 categorical features (which I turned into dummy variable) and two continuous. The score is average AUC, $0.6*AUC_1 + 0.4*AUC_2$.My problem is the following, when I compute each AUC (for 2014 and 2015) with a stratified kfold cross validation and I compute the average AUC I get roughly 0.88 and when I submit on the website I end up with 0.67, I guess there is a problem in my code.Here is my code for choosing the best model for 2014:Rk: to predict on the test set (2014,2015 unknown), I first predict with all 5 features, 2014. Then I add the prediction of 2014 to my feature to predict 2015# Spot Check Algorithmsmodels = []models.append(('LG', LogisticRegression()))models.append(('LDA', LinearDiscriminantAnalysis()))models.append(('RF', RandomForestClassifier()))models.append(('GBC', GradientBoostingClassifier()))models.append(('SVM', SVC()))# evaluate each model in turnresults = []names = []# stratifiedkfold is defined by default when there is an integerscoring = 'roc_auc'num_folds = 10for name, model in models:    cv_results = cross_validation.cross_val_score(model, X, Y, cv=num_folds, scoring=scoring)    results.append(cv_results)    names.append(name)    msg = %s: %f (%f) % (name, cv_results.mean(), cv_results.std())    print(msg)
Gap leaderboard score and model scoring on a Competition
machine learning;python