Sementic segmentation data and model compile in Keras
Two things that stand out is that you are using sigmoid as activation which is used for binary classification (it just squash values between 0 and 1). The other is that your learning rate is 5x lower than the default value. 

Also because you are classifying images, I think the title is misleading; semantics is associated with "meaning" i.e. with language.