Why is there a big drop off in performance in my GBM? I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 observations that I use for a final validation. All 10,200 observations were obtained in similar fashion.

I carried out a grid search with 5-fold cross-validation in order to obtain a suitable set of hyper parameters. The results for my test set are good and very stable. However, there is a big drop off in performance when use my final validation data.

Results for the test set 

-&gt;  Precision: 0.836 / Recall: 0.629 / Accuracy 0.874


Results for the final validation set

-&gt;  Precision: 0.149 / Recall: 0.725 / Accuracy 0.484


At first I thought this could be the caused by data leakage, but even after removing "suspicious" features, there is still a big drop off when comparing the test results with the final validation results. 

Surely I'm doing something wrong, but I'm at a loss as to what exactly. Here are the relevant lines of code (nothing fancy):

&gt; X = pd.read_csv('train_test.csv')
&gt; y = X.pop('Target')
&gt; X_final = pd.read_csv('final_validation.csv')
&gt; y_final = X_final.pop('Target')

&gt; X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

&gt; gb = GradientBoostingClassifier(n_estimators=300, max_depth=5, learning_rate=0.2)
&gt; gb_model = gb.fit(X_train, y_train)

&gt; # test set
&gt; y_pred = gb_model.predict(X_test)
&gt; precision, recall, fscore, support = score(y_test, y_pred, average='binary')

&gt; # final validation set
&gt; y_hat = gb_model.predict(X_final)
&gt; precision, recall, fscore, support = score(y_final, y_hat, average='binary')


Any thoughts?
I would first suggest trying to plot the results during training. How do your metrics (or at least the loss) vary over the training process for training and cross-validation datasets?
The loss at each iteration is appended to your GBC object gb_model in the train_score_ attribute.

Normally, when there is such a big gap between training and test data, it indicates that you are overfitting to your training data, and that the model does not generalise well to unseen data. You could think about doing  shuffling your data, in order to balance the training/validation/test datasets - [if you are looking at a time-series problem, you should be careful as to how you do this].