How to Precondition A Fully Convolutional Neural Network
I've been thinking about this problem before and while I have not researched this or even implemented this, I have a few ideas, from easy/obvious to more elaborate:


Don't make it fully convolutional but add some dense layers where you concat this information
If it's just one category, one-hot encode it and broadcast it over your picture as extra channels at the input or maybe halfway in the network
Similar to the previous suggestion but learn an embedding for your categories that you broadcast either at the beginning or the middle
This time, use one-hot encoding as input and learn an embedding that is for example 8x8 and upsample this to your image resolution
Learn an embedding again, but this time use some upconvolutions to scale it to the input (or middle) size


Let me know how much it helps!