Why is my model accuracy high in train-test split but actually worse than chance in validation set? I have trained a XGboost model to predict survival for the Kaggle Titanic ML competition.

As with all Kaggle competitions there is a train dataset with the target variable included and a test dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.

My problem:

I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the train data. The accuracy I get from this validation is ~80% which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).

The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.

However when I submit my predictions for the test set my public score is ~35% which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.

Why is my model so much worse on the test?

I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.

However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.

So can you help me understand how those two issues happen at the same time:


Drastic difference between local accuracy and public score
Reversing actually leads to the better public score.


Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):

https://www.kaggle.com/fnguyen/titanicrising-test
Looking at your code, one set of data transformations were applied to the train data and a different set of transformations were applied to the test data. Different data transformations could account for different evaluation metric performance. 

It is best practices to put all data transformations in a function so they can be applied to all data in a similar way. 

Since you are using scikit-learn, sklearn.compose.ColumnTransformer is designed for this purpose. Example code for the Titanic dataset is here.