Semantic segmentation in high-resolution images with high variance - cannot avoid underfitting
Given your mask is very small, you should look at reducing your convolutions to 2x2 since that will help aggregate more information from these smaller masks. EfficientNet has 3x3 and 5x5 convolutions which may not be suited for your purpose.
It is a better idea for you to train from scratch using smaller convolutions (2x2). Also, since you will lose the edge of transfer learning, look at self-training, if it is possible for your network.