Semantic segmentation failing in small instance detection
You may find it useful to categorize these smaller defects as a separate class, then introduce a class weights matrix to penalize incorrect classification of the smaller defects more heavily. If these small defects represent a very small portion of the total number of pixels in your training data, then the model may be stuck at a local minimum, where it just predicts them as zero because the loss is not heavily penalized for this. So you need to add extra penalty for the incorrect classification.
Assuming you create a new class for the small defects, the class weights matrix is implemented as a keras callback function.
If you do not want to create a multiclass problem, there is also this paper : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977596/#!po=32.4561
The multilevel unet architecture will not solve class imbalance if that is truly the underlying issue, but if may make the network more aware of smaller scale changes in the images you're working with.