How does one input images and labels for Semantic Instance Segmentation with neural networks?
You'll want to train your NN in such as way that you'll be able to use it for prediction.  

If you want to just predict the classes from the image, then all you want to send to your NN is 

the original image (probably color balanced) and
predict the classes from the XML (convert that into a 1 hot class encoding)

if you want to predict the mask also, then send

the original image (probably color balanced) and
predict the mask and classes from the XML (convert that into a 1 hot class encoding)


The above objectives (just classes, or classes+mask prediction) drives the decision to store the classes or the classes + mask.
So I know for a standard convolutional neural network you can provide the neural net (NN) a file with a list of labels or simply separate your classes by folders but for instance segmentation I imagine it's different right? 
For example using a site like labelme2 you can annotate and segment images and then download them along with mask files and XML files for labels. Does one need to subsequently input the original image, mask image and XML file to the instance segmentation NN?  
Thanks in advance.