How to deal with unbalanced data in pixelwise classification?
Looks like the problem lied just in poorly chosen parameters. In the example given above, simply increasing epoch count significantly improved the results. In my actual image processing problem, I had to decrease learning rate, since with the default one, ReLU's were saturating very quickly, hindering further progress.