Are Intersection over Union (IoU) scores preferring larger objects?
There are a lot of reasons for this behaviour:


training data: In your training data, there could be more larger objects than smaller objects, so the model didn't have enough information about smaller objects to learn to segment them well. 
training procedure: Depending on your training procedure (e.g. loss function), your model will focus more on some type of objects (in your case, larger objects ...)
model architecture: Every model has some advantages, so some models are better in recognizing objects of various sizes. E.g.  convolutional and pooling layers can increase a receptive field of neurons so that they could recognize objects of various sizes more easily. Also, some model architectures are specialized in recognizing models of varying sizes.


Possible directions:


training data: Know your data! :) Inspect the dataset to see what's in. The things to look for: are all classes represented well, are labels (ground truth) good, sizes of objects, ...
training procedure: Try with various loss functions (cross entropy, dice loss, cross entropy + dice loss, focal loss, ...). Also, dataset augmentation techniques are important (e.g. with dataset augmentation you can create objects of various sizes ...).
model architecture: Resnet, Inception, InceptionResnet - those are the backbones that you might want to use for your backbone as they have proved to be good for semantic segmentation.