Deep Learning Network decreasing in accuracy
Possible tricks which can improve the accuracy of semantic segmentation models:


try lowering the learning rate
try variating the batch size
train longer (more epochs)
train the model on larger images
increase the dataset size