Overfitting with sklearn pipeline - reasons why? So....

I've been playing around with this for FAR TOOO LONG now and I really need some advice. 

Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems to provide good results when scoring but i believe this to be data leakage and an incorrect process to perform in real world solutions.

I've kept apart 15% of data as a holdoutset and the rest for CV.

The issue I have is that if I put the scaler into a pipeline, the scale changes on each fold of the CV and gives me wildly different results to if I prescaled the whole dataset prior to hyper parameter tuning.

I'm even getting worse results once I fit to the whole training data (including the holdout set) which suggests its overfitting to the smaller data set.

Is there anything I can do to combat this?

Am I doing something wrong? (should I be putting a scaler into a pipeline?)

Or is this just what it is in the real world?

Any other help regarding overfitting etc would be amazing.
If your dataset is giving really different results for different folds and you just have a scaler and a model in your pipeline could be for different reasons:


The splits of your validation have different difficulties for some reason. Some are just easy to predicto of you haven't choosen the proper validation and crossfold. An example of this would be that there is a temporal dependency and you are not getting it. If this was the case my reccomendation will be to choose a proper validation and crossfold.
Other possible problem would be that the features are not equally distributed, meaning that there are some outliers. If there is outliers when you scale you will have different intervals. If this was the case I would say that before doing anything more complicated doing some pre processing for the whole dataset. 


Note: Doing what Kagglers do of concateneting train and test can be dangerous when you put your model into production.