regularization error vs over fitting I have a collected data from 50 unique blocks, and then merged data from 49 blocks into one data set, and saved the data from 1 block for testing purpose.
I then split the merged data set from 49 blocks using train_test_split(sklearn). Then used training data to train a random forest regressor using cross validation and get a good model score(R^2 score from sklearn random forest regressor model) on train (0.99) and test set(0.94). But when I use the trained model on the reserved data from the 1 block, the performance is very bad (-1.0).
If I merge the data from all 50 blocks, then use train test split and keep 60% data as training set, 20% as test set, and 20% (reserved set), I get good scores from all three sets. Training set score(0.98), test set(0.93) and the reserved set(0.96).
Any intuition regarding what could be causing this? And any suggestions on how to improve the model score for the 1 block of unseen data?
Welcome to the community!
There are points coming to my mind:

Check the amount of data in each block and then their distribution. First experience might be due to the lack of enough data in reserved block (i.e. you literally just trained but did not validated resulting to a complete overfitting) or having a totally different distribution (i.e. the difference between blocks is NOT random).
Be sure about wht you called &quot;Cross Validation&quot;. If you splitted 49 blocks into a training and a testing set, then it is minimal version of Cross Validation. 49 blocks should be splitted to $k$ different set, each time one of them plays the test set role and then the final error is averaged over all $k$ runs. That is statistically more robust (first experience shows problem in robustness of statistical inference)
Second experiment uses 100 blocks instead of 50 (where were those other 50 in first experiment?) so first, you have more data, and second, you did not split according to &quot;blocks&quot; (whatever they are) but merged all and randomly chose train, validation and test. This procedure reduces the danger of distributional bias within train/valid/test and also might be an indicator of the fact that statistically those &quot;blocks&quot; are different (their distribution)

hope it helped. Please share your results so everyone can learn from it.
Good Luck!