title: Python image types, shapes, and channels for segmentation
body: <p>I am using this <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=cFHKCvCTxiff" rel="nofollow noreferrer">tutorial</a> for instance segmentation in PyTorch. The test data the tutorial uses includes images and accompanying image masks from a dataset available <a href="https://www.cis.upenn.edu/%7Ejshi/ped_html/PennFudanPed.zip" rel="nofollow noreferrer">here</a>. I have an example of one of the image masks from that data set <a href="https://drive.google.com/file/d/1NQGL4r45QL5Z9pzAH9-If8ARTJcQioqC/view?usp=sharing" rel="nofollow noreferrer">here</a> (example data for this question). That mask looks like this by default in the dataset:</p>
  <p><a href="https://i.stack.imgur.com/O3Bqq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/O3Bqq.png" alt="enter image description here" /></a></p>
  <p>The tutorial <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=cFHKCvCTxiff&amp;line=11&amp;uniqifier=1" rel="nofollow noreferrer">uses this code</a>:</p>
  <pre><code>mask.putpalette([
      0, 0, 0, # black background
      255, 0, 0, # index 1 is red
      255, 255, 0, # index 2 is yellow
      255, 153, 0, # index 3 is orange
  ])
  </code></pre>
  <p>as explanatory step for the mask to make it look like this:</p>
  <p><a href="https://i.stack.imgur.com/L7sHW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/L7sHW.png" alt="enter image description here" /></a></p>
  <p>but that code is not requisite in the segmentation process itself. It's just used to show what the mask contains.</p>
  <p>I am trying to use my own image data. I created masks for the images in <a href="https://www.gimp.org/" rel="nofollow noreferrer">G.I.M.P.</a> This is <a href="https://drive.google.com/file/d/1uu9Uok6yLf1nGiBL-TjxlEOwTaMxwOYE/view?usp=sharing" rel="nofollow noreferrer">one of the masks</a> I made. It looks like this by default.</p>
  <p><a href="https://i.stack.imgur.com/p7c42.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/p7c42.png" alt="enter image description here" /></a></p>
  <p>As I try to run the tutorial code, I have problems with the masks. This code chunk creates a class that creates PyTorch datasets.</p>
  <pre><code>import os
  import numpy as np
  import torch
  import torch.utils.data
  from PIL import Image
  
  
  class PennFudanDataset(torch.utils.data.Dataset):
      def __init__(self, root, transforms=None):
          self.root = root
          self.transforms = transforms
          # load all image files, sorting them to
          # ensure that they are aligned
          self.imgs = list(sorted(os.listdir(os.path.join(root, &quot;PNGImages&quot;))))
          self.masks = list(sorted(os.listdir(os.path.join(root, &quot;PedMasks&quot;))))
  
      def __getitem__(self, idx):
          # load images ad masks
          img_path = os.path.join(self.root, &quot;PNGImages&quot;, self.imgs[idx])
          mask_path = os.path.join(self.root, &quot;PedMasks&quot;, self.masks[idx])
          img = Image.open(img_path).convert(&quot;RGB&quot;)
          # note that we haven't converted the mask to RGB,
          # because each color corresponds to a different instance
          # with 0 being background
          mask = Image.open(mask_path)
  
          mask = np.array(mask)
          # instances are encoded as different colors
          obj_ids = np.unique(mask)
          # first id is the background, so remove it
          obj_ids = obj_ids[1:]
  
          # split the color-encoded mask into a set
          # of binary masks
          masks = mask == obj_ids[:, None, None]
  
          # get bounding box coordinates for each mask
          num_objs = len(obj_ids)
          boxes = []
          for i in range(num_objs):
              pos = np.where(masks[i])
              xmin = np.min(pos[1])
              xmax = np.max(pos[1])
              ymin = np.min(pos[0])
              ymax = np.max(pos[0])
              boxes.append([xmin, ymin, xmax, ymax])
  
          boxes = torch.as_tensor(boxes, dtype=torch.float32)
          # there is only one class
          labels = torch.ones((num_objs,), dtype=torch.int64)
          masks = torch.as_tensor(masks, dtype=torch.uint8)
  
          image_id = torch.tensor([idx])
          area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
          # suppose all instances are not crowd
          iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
  
          target = {}
          target[&quot;boxes&quot;] = boxes
          target[&quot;labels&quot;] = labels
          target[&quot;masks&quot;] = masks
          target[&quot;image_id&quot;] = image_id
          target[&quot;area&quot;] = area
          target[&quot;iscrowd&quot;] = iscrowd
  
          if self.transforms is not None:
              img, target = self.transforms(img, target)
  
          return img, target
  
      def __len__(self):
          return len(self.imgs)
  
  dataset = PennFudanDataset('PennFudanPed/')
  dataset[0]
  </code></pre>
  <p>The last line returns:</p>
  <pre><code>(&lt;PIL.Image.Image image mode=RGB size=559x536 at 0x7FCB4267C390&gt;,
   {'area': tensor([35358., 36225.]), 'boxes': tensor([[159., 181., 301., 430.],
            [419., 170., 534., 485.]]), 'image_id': tensor([0]), 'iscrowd': tensor([0, 0]), 'labels': tensor([1, 1]), 'masks': tensor([[[0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0],
             ...,
             [0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0]],
    
            [[0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0],
             ...,
             [0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0],
             [0, 0, 0,  ..., 0, 0, 0]]], dtype=torch.uint8)})
  </code></pre>
  <p>When I run this code with my data,</p>
  <pre><code>...
  dataset = four_chs('drive/MyDrive/chambers/')
  dataset[0]
  </code></pre>
  <p>I get this error:</p>
  <pre><code>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:38: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
  
  ---------------------------------------------------------------------------
  
  TypeError                                 Traceback (most recent call last)
  
  &lt;ipython-input-17-12074ae9ab35&gt; in &lt;module&gt;()
        1 len(dataset)
  ----&gt; 2 dataset[0]
  
  &lt;ipython-input-1-99ab92a46ebe&gt; in __getitem__(self, idx)
       42         boxes = []
       43         for i in range(num_objs):
  ---&gt; 44             pos = np.where(masks[i])
       45             xmin = np.min(pos[1])
       46             xmax = np.max(pos[1])
  
  TypeError: 'bool' object is not subscriptable
  </code></pre>
  <p>I'm not sure exactly what is going on, but there is a difference between my mask and the mask that is in the test data. They are both PNG file types, but it seems that mine has the red, blue, green channels split out with another channel or something, but I don't know what it is, based on the shape of the object in Python. This comes from one of the masks I made:</p>
  <pre><code>mask2 = np.array(mask1)
  mask2.shape
  (5312, 2988, 4)
  
  </code></pre>
  <p>For one of the test data masks:</p>
  <pre><code>mask2 = np.array(mask)
  mask2.shape
  (536, 559)
  </code></pre>
  <p>There appears to be only one channel? So with their different shapes, I guess that why I get the error from (this is an excerpt from the code I pasted earlier)</p>
  <pre><code>...
          mask_path = os.path.join(self.root, &quot;masks&quot;, self.masks[idx])
          # note that we haven't converted the mask to RGB,
          # because each color corresponds to a different instance
          # with 0 being background
          mask = Image.open(mask_path)
  
          mask = np.array(mask)
          # instances are encoded as different colors
          obj_ids = np.unique(mask)
          # first id is the background, so remove it
          obj_ids = obj_ids[1:]
  
          # split the color-encoded mask into a set
          # of binary masks
          masks = mask == obj_ids[:, None, None]
  
          # get bounding box coordinates for each mask
          num_objs = len(obj_ids)
  for i in range(num_objs):
               pos = np.where(masks[i])
  ...
  </code></pre>
  <p>How do I get my masks' shapes to match those of the masks in the test data so that I can use the rest of segmentation code create my PyTorch compatible dataset that will work with the segmentation algorithm? I'm not trying to get the same height and width, but change the number of channels/layers, but I don't think I want it to be grayscale.</p>
  <p><strong>EDIT after HansHirse's comment:</strong></p>
  <p>I went to back to G.I.M.P and used the Image Mode menu to change the image to grayscale. I exported with that setting. I tried to run the code with that file and it did not work.</p>
  <p>I also found a way to convert the R.B.G. image to grayscale with upon import with <code>Image.open().convert(&quot;L&quot;)</code>. This does not work either.</p>
  <p>In both cases, the problem has to do with speckles of colors that I thought were separate being mixed together. For example, I used HansHirse's advice to fill in the areas of interest with grey &quot;colors&quot; of 1,2,3,and 4, while the background stayed 0. Upon importing the file that created, the values of those files are 3,5,8, and 10. And while one of the shapes may be mostly of value 3, there are vagrant pixels with that value in the other shapes, so there is no value that is contained entirely in one shape. With that situation, the code draws bounding boxes that surround all 4 shapes, instead of around one shape.</p>
  <p>I'm aware of using the hue, saturation, value (H.S.V.) color space and tried converting to that color space. That still doesn't solve the problem for me.</p>
  <p>I'm trying to figure out how to use something like</p>
  <pre><code>np.where( mask[&lt;buffered shape1 xmin&gt;,&lt;buffered shape1 xmax&gt;, &lt;buffered shape1 ymin&gt;, &lt;buffered shape1 ymax&gt;,0] == &lt;majority color value for shape&gt;)
  </code></pre>
  <p>to sort of quarter up the mask, filter based on the main color's value for that shape in that quarter, and get the actual x and y values for the shape in that quarter. With those values, I figure I can use the min and max from the actual values to create my bounding boxes.</p>
  <p>Another note is that when exporting from G.I.M.P., there is a drop-down menu in the export window to set the file to 8 bit R.G.B. or Gray. Select 8bpc Gray for the format needed.</p>