UNet Loss function for non-categorical Mask?
If I understand your question correctly - the "ground truth" mask is just a gray-scale image with values in range [0,255], meaning , there is a strong relation between it's values (for example - 25 is closer to 26 then to 70. this is not the case with regular segmentation, where you assign a different class to each pixel and the classes values may represent arbitrary objects such as "bicycle" or "person"). In other words, this is a regression problem, and to be more specific an image-to-image regression. You are trying to reconstruct a gray-scale image which should be identical to the ground truth mask, pixel-wise. 
If I understood you correctly - you should look for regression losses. Common examples that can be used are Mean Squared Error (aka MSE, L2 norm) or Mean Absolute Error (aka MAE, L1 norm). Those are the "usual suspects" and I suggest you start with them, although many other losses exists.
I have a UNet Segmentation network implemented in Keras that simply maps all pixels in an RGB image to 4 categories which is trained on a heat map mask (Low, Low-Med, High-Med, High). Using CCE or categorical Dice loss I am able to get decent results.
However, The mask in it's original form is a heat map image with 255 bits of resolution. It seems like a totally arbitrary introduction of error to shoehorn it into the Unet by reducing the 255 bits of resolution into 4 categories.
I would like the network to output an image with each pixel having a value between (0,1), and train the network with masks that are produced by multiplying the heat map image by 1./255.
Where, in this case, the loss function would incorporate the mathematical difference between the mask and the prediction from the network. Can anyone point me in the direction of someone who has done something similar? I think I am just awful at describing what I'm looking for with relevant terminology because it seems like this would be a fairly common goal in computer vision..?