title: How to improve depth segmentation of Lego blocks
body: <h4>Input</h4>
  <p>I have the following <strong>depth images</strong>  of type <code>uint16</code> obtained from <strong>Intel realsense <a href="https://www.intelrealsense.com/download/7691" rel="nofollow noreferrer">L515</a></strong> camera which is supposed to have an <code>Avg Depth Accuracy&lt; 5mm @ 1m</code>.</p>
  <p><img src="https://cloud.univ-grenoble-alpes.fr/index.php/s/skGWWtAK5ygJNpx/download/d1.png" alt="d1" /></p>
  <p><img src="https://cloud.univ-grenoble-alpes.fr/index.php/s/EZFxSXgJgfn7Ljm/download/d2.png" alt="d2" /></p>
  <h4>Goal</h4>
  <p>I want to <strong>quantify the depth of the blocks</strong> inside this image to get a discrete representation of the blocks inside my region of interest of <code>23 x 11 block positions</code> such as</p>
  <p><code>P_x1_y1 : z = 1</code>(one block), <code>P_x2_y2: z = 2</code> (two blocks), up to 5 blocks (as in the image center).</p>
  <p>The <strong>ROI RGB</strong> Image can clarify my aim <em>(but it is not used as an input)</em>:</p>
  <p><a href="https://i.stack.imgur.com/14rr6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/14rr6.png" alt="roi" /></a></p>
  <h4>What I have tried so far:</h4>
  <ol>
  <li>Obtaining the <strong>ROI</strong>:</li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>#!/usr/bin/python3
  # -*- coding: utf-8 -*-
  
  import numpy as np
  import cv2
  import matplotlib.pyplot as plt
  
  def get_roi(d1, output_size=(736, 384), ratio=(0.77, 0.54), shift=(0, 80), verbose=False):
      &quot;&quot;&quot;
      Function: get_roi, to find and resize the ROI.
      ---
      Parameters:
      @param: d1, nd array, depth image.
      @param: output_size, tuple, the output ROI size.
      @param: ratio, tuple, the ratio of the ROI to the detected zone.
      @param: shift, tuple, the shift in pixles to align the ROI.
      @param: verbose, bool, to vizualize the result.
      ---
      @return: roi, nd array, ROI resized.
      &quot;&quot;&quot;
      d = d1.copy()
  
      th =  cv2.threshold(d, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]
      th = th.astype(np.uint8)
  
      contours = cv2.findContours(th, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE )[0]
      cnt_thresh = 10000
      fx, fy = ratio
      x_shift, y_shift = shift
      for i, cnt in enumerate(contours):
          if(cv2.contourArea(cnt)&gt;cnt_thresh):
              x,y,w,h = cv2.boundingRect(cnt)
              cx = x + w//2 + x_shift
              cy = y + h//2 + y_shift
              nw = int(fx * w)
              nh = int(fy * h)
              # cv2.rectangle(d1,(cx-nw//2,cy-nh//2),(cx+nw//2,cy+nh//2),color=0)
      d_roi = d1[cy-nh//2:cy+nh//2,cx-nw//2:cx+nw//2]
      roi = cv2.resize(d_roi, output_size)
  
      # Visualize results
      if(verbose):
          plt.imshow(roi)
          plt.show()
  
      return roi
  </code></pre>
  <ol start="2">
  <li>Finding the <strong>mode(most frequent) non-zero</strong> value of each cell in the grid:</li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>def mode(arr):
      &quot;&quot;&quot;
      Function: mode, to find the mode of an array.
      ---
      Parameters:
      @param: arr, nd array, any.
      ---
      @return: the mode value (whatever int/float/etc) of this array.
      &quot;&quot;&quot;
      vals,counts = np.unique(arr, return_counts=True)
      if 0 in vals:
          z_idx = np.where(vals == 0)
          vals   = np.delete(vals,   z_idx)
          counts = np.delete(counts, z_idx)
      index = np.argmax(counts)
      return vals[index]
  </code></pre>
  <ol start="3">
  <li>Quantifying the values of each cell:</li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>def mapVal(val):
      &quot;&quot;&quot;
      Function: mapVal, to map depth values.
      ---
      Parameters:
      @param: val, int, any.
      ---
      @return: int val, specific value 0, 50, 150, 200, 250, val.
      &quot;&quot;&quot;
      if val&lt;=183:
          return 0
      if val&gt;183 and val &lt;=230:
          return 50
      if val&gt;230 and val &lt;=295:
          return 100
      if val&gt;295 and val &lt;=390:
          return 150
      if val&gt;390 and val &lt;=470:
          return 200
      if val&gt;470:
          return 250
      else:
          return val
  </code></pre>
  <ol start="4">
  <li><strong>grid the ROI</strong> into cells, and applying <strong>Linear correction</strong> for the depth static error:</li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>def gridWorkspace(roi, gridSize=(23, 11), shift=[0, 5], verbose=False):
      &quot;&quot;&quot;
      Function: gridWorkspace, to find the contours of the red markers.
      ---
      Parameters:
      @param: roi, nd array, cropped region of interest.
      @param: gridSize, tuple, lenght/width or the Workspace.
      @param: shift, to make static error compensation for alignment.
      @param: verbose, boolean, to show the output of the function. 
      ---
      @return: None.
      &quot;&quot;&quot;
      # Store a deep copy for results:
      roi_copy = roi.copy()
  
      # Divide the image into a grid:
      verticalCells   = gridSize[1]
      horizontalCells = gridSize[0]
  
      # Cell dimensions
      bigRectWidth  =  roi_copy.shape[1] 
      bigRectHeight =  roi_copy.shape[0]
  
      cellWidth = bigRectWidth // horizontalCells
      cellHeight = bigRectHeight // verticalCells
  
      x_shift, y_shift = shift
  
      # # Correction values
      origin = mode(roi[y_shift:cellHeight+ y_shift, x_shift:cellWidth+x_shift])
      x_max  = mode(roi[y_shift:y_shift+cellHeight, x_shift+(horizontalCells-1)*cellWidth:x_shift+horizontalCells*cellWidth])
      y_max  = mode(roi[y_shift++(verticalCells-1)*cellHeight:y_shift+verticalCells*cellHeight, x_shift:x_shift+cellWidth])
      print(&quot;origin= {}, x_max= {}, y_max= {}&quot;.format(origin, x_max, y_max))
  
      x_corr = ( int(x_max) - int(origin) ) // horizontalCells
      y_corr = ( int(y_max) - int(origin) ) // verticalCells
      print(&quot;x_corr = {}, y_corr = {}&quot;.format(x_corr, y_corr))
  
      # Loop thru vertical dimension:
      for j in range(verticalCells):
  
          # Cell starting y position:
          yo = j * cellHeight + y_shift
  
          # Loop thru horizontal dimension:
          for i in range(horizontalCells):
  
              # Cell starting x position:
              xo = i * cellWidth + x_shift
  
              # Cell Dimensions:
              cX = int(xo)
              cY = int(yo)
  
              # Quantify current cell:
              # print(i, j, mode(roi[cY:cY + cellHeight, cX:cX + cellWidth]))
              roi_copy[cY:cY + cellHeight, cX:cX + cellWidth] =  mapVal(mode(roi[cY:cY + cellHeight, cX:cX + cellWidth]) - j*y_corr - i*x_corr)# mapVal(371 - mode(roi[cY:cY + cellHeight, cX:cX + cellWidth]))
              # Draw Cell
              cv2.rectangle(roi_copy, (cX, cY), (cX + cellWidth, cY + cellHeight), (100, 100, 255), 1)
  
      # Visualize results
      if(verbose):
          plt.imshow(roi_copy)
          plt.show()
  </code></pre>
  <hr />
  <p>So when I try:</p>
  <pre class="lang-py prettyprint-override"><code>path = &quot;&quot;
  imName = &quot;d1.png&quot;
  
  d1 = cv2.imread(path+imName, -1)
  
  roi = get_roi(d1, verbose=False)
  
  roi = np.max(roi) - roi
  
  roi[roi&lt;0] = 0
  roi[roi&gt;500] = 0
  
  gridWorkspace(roi, verbose=True)
  </code></pre>
  <p>I get this result:</p>
  <p><a href="https://i.stack.imgur.com/1QEAq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1QEAq.png" alt="result" /></a></p>
  <p>Can you please tell me what can I do to improve my segmentation? thanks in advance.</p>