80-20 better than full dataset for LightGBM Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables.

I have two modes, 1) Production and 2) Testing. The first one just trains a model with the whole dataset. The second does the same with an 80% of the dataset and tests over the remaining 20% (80-20 done with train_test_split, from sklearn.model_selection, no seed used).

In both cases I show the neg_mean_squared_error at the end of the execution. In the first case for the whole dataset, in the second I get two values (training and testing)

I am shocked to see that while in the first case my error is around -10, in the second the values are -5 (training) and -5.3 (testing). An average value of my objective variable can be 80. I would expect to lose accuracy since I train with less data and then I test over a different sample.

Question: There is any theoretical reason that explains that a 80-20 test gets a "better" neg_mean_squared_error than in the full data case? Or it has to be a (sneaky) bug in my code?
I cannot think about any theoretical reason. However:


It may be the case that the neg_mean_squared_error is not correctly implemented. Try another metric, like Median or Mean Percentage Error and see if that continues to happen.
It may be related to how the data is distributed, but still that's not common.


Lastly, I wouldn't put a model in production (if by that you mean deploying into a product, for example) without testing its performance in a test set, even if in another instance of the model, trained using cross validation, has good performance. One reason: it may be the case that a small part of the training data breaks your model, and does not generalise.