title: How to load Image Masks (Labels) for Image Segmentation in Keras
body: <p>I am using Tensorflow as a backend to Keras and I am trying to understand how to bring in my labels for image segmentation training.</p>
  
  <p>I am using the <a href="http://vis-www.cs.umass.edu/lfw/part_labels/" rel="noreferrer">LFW Parts Dataset</a> which has both the ground truth image and the ground truth mask which looks like this * 1500 training images:</p>
  
  <p><a href="https://i.stack.imgur.com/2CGHc.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/2CGHc.jpg" alt="Aaron_Peirsol_0002_Image"></a>
  <a href="https://i.stack.imgur.com/7o3sC.png" rel="noreferrer"><img src="https://i.stack.imgur.com/7o3sC.png" alt="Aaron_Peirsol_0002_Mask"></a></p>
  
  <p>As I understand the process, during training, I load both the </p>
  
  <ul>
  <li>(X) Image</li>
  <li>(Y) Mask Image</li>
  </ul>
  
  <p>Doing this in batches to meet my needs. Now my question is, is it sufficient to just load them both (Image and Mask Image) as NumPy arrays (N, N, 3) or do I need to process/reshape the Mask image in some way. Effectively, the mask/labels are represented as [R, G, B] pixels where:</p>
  
  <ul>
  <li>[255, 0, 0] Hair </li>
  <li>[0, 255, 0] Face </li>
  <li>[0, 0, 255] Background</li>
  </ul>
  
  <p>I could do something like this to normalize it to 0-1, I don't know if I should though:</p>
  
  <pre><code>im = Image.open(path)
  label = np.array(im, dtype=np.uint8)
  label = np.multiply(label, 1.0/255)
  </code></pre>
  
  <p>so I end up with:</p>
  
  <ul>
  <li>[1, 0, 0] Hair </li>
  <li>[0, 1, 0] Face </li>
  <li>[0, 0, 1] Background </li>
  </ul>
  
  <p>Everything I found online uses existing datasets in tensorflow or keras. Nothing is really all that clear on how to pull things off if you have what could be a considered a custom dataset.</p>
  
  <p>I found this related to Caffe: <a href="https://groups.google.com/forum/#!topic/caffe-users/9qNggEa8EaQ" rel="noreferrer">https://groups.google.com/forum/#!topic/caffe-users/9qNggEa8EaQ</a></p>
  
  <p>And they advocate for converting the mask images to a <code>(H, W, 1)</code> (HWC) ?where my classes would be <code>0, 1 ,2</code> for Background, Hair, and Face respectively.</p>
  
  <p>It may be that this is a duplicate here (combination of similar quesiton/answers):</p>
  
  <p><a href="https://stackoverflow.com/questions/43900125/how-to-implement-multi-class-semantic-segmentation">How to implement multi-class semantic segmentation?</a></p>
  
  <p><a href="https://stackoverflow.com/questions/42959364/tensorflow-how-to-create-a-pascal-voc-style-image">Tensorflow: How to create a Pascal VOC style image</a></p>
  
  <p>I found one example that processes PascalVOC into (N, N, 1) that I adapted:</p>
  
  <pre class="lang-py prettyprint-override"><code>LFW_PARTS_PALETTE = {
      (0, 0, 255) : 0 , # background (blue)
      (255, 0, 0) : 1 , # hair (red)
      (0, 0, 255) : 2 , # face (green)
  }
  
  def convert_from_color_segmentation(arr_3d):
      arr_2d = np.zeros((arr_3d.shape[0], arr_3d.shape[1]), dtype=np.uint8)
      palette = LFW_PARTS_PALETTE
  
      for i in range(0, arr_3d.shape[0]):
          for j in range(0, arr_3d.shape[1]):
              key = (arr_3d[i, j, 0], arr_3d[i, j, 1], arr_3d[i, j, 2])
              arr_2d[i, j] = palette.get(key, 0) # default value if key was not found is 0
  
      return arr_2d
  </code></pre>
  
  <p>I think this might be close to what I want but not spot on. I think I need it to be (N, N, 3) since I have 3 classes? The above version and there is another one originated from these 2 locations:</p>
  
  <p><a href="https://github.com/martinkersner/train-CRF-RNN/blob/master/utils.py#L50" rel="noreferrer">https://github.com/martinkersner/train-CRF-RNN/blob/master/utils.py#L50</a></p>
  
  <p><a href="https://github.com/DrSleep/tensorflow-deeplab-resnet/blob/ce75c97fc1337a676e32214ba74865e55adc362c/deeplab_resnet/utils.py#L41" rel="noreferrer">https://github.com/DrSleep/tensorflow-deeplab-resnet/blob/ce75c97fc1337a676e32214ba74865e55adc362c/deeplab_resnet/utils.py#L41</a> (this link one-hot's the values)</p>