object image segmentation I have 2 different datasets with similar objects, one where each object is 50 pixels wide and the other where they are 150 pixels. Each photo is 512x512 for both datasets. These two datasets have the same number of photos. Taken with the same camera, so same focal length, resolution etc...
So far, we agree that only the size of the objects differs.

I segment with U-net, for each dataset. It's all right, I have good predictions.
For fun, I train with 150 pixels dataset and test on 50 pixels dataset, and vice versa.
Again, everything is fine, the results are bad so we deduce that the results are better when the size of the objects is similar. Logical.

Now I train a model on a dataset composed of both the 50 and 150 pixels objects (half 50 and half 150), and in total the same number of images as before.
When I test my model on a set of the images composed exclusively of 50 pixels objects (respectively 150), I get better results than when I trained my network using only 50 pixels object to train (resp 150)
Is this due to scale-(in)variant features issues? Is there a similar case you dealt with ?
Thanks a lot
That's a tough one.  I don't know what could cause that.
It can be hard to know why we see the results we do, when working with neural nets.  Often the best we can do is form several hypotheses, and then devise experiments to try to test those hypotheses.
Some possible explanations that you could try to test:

Perhaps it is noise / random chance

Perhaps the 50-pixel images are fairly homogenous and the 150-pixel images are fairly homogenous, and taking a combination of these two yields more diversity, which helps the network learn (e.g., it doesn't overfit to either)

Perhaps the 50-pixel objects don't come from the same distribution as the 150-pixel objects (e.g., the 50-pixel images are of children and the 150-pixel images are of adults; these differ in more ways than just size; or the background is correlated to the size of the object), and you get more regularization from exposing the network to both, or something (this is vague and might not be very different from the second explanation)


Maybe you can think of other possibilities.