How to train a Machine Learning model for blocked data
I think your approach is more or less valid, but you’re overfitting the model because you don’t have a lot of data. So that’s my main point.

There are many ways in principle to deal with this, but if you have little data, perhaps you simply shouldn’t be expecting great results. If you use the averages, you get better results, but “the problem isn’t solved completely”. A priori I’m not sure you have reason to think you can do better.

Anyway, step 1 should be to simplify the model (fewer and more shallow trees). You should find that your test error deteriorates but your validation error improves, because rather than simply memorizing records, the model is now forced to learn something about the blocks. This information can be reused on records the model hasn’t seen yet.

You can also look at regularization, but it is essentially an automatic way of finding a simpler, but better model. Also, data augmentation. Is it possible to generate more data by varying the existing data a bit? Like the way you can generate more images by changing the images that you have: By cropping, flipping, rotating, etc.

Finally, make sure that your split between train-test-validate sets is random. I guess it depends on your data but it seems you can’t expect the model to learn about a block that doesn’t exist at all in the training set. If that block appears in the validation set only, I guess the model would be random on that data.