title: OpenCV Extract shapes with extra contrast from backgrounds
body: <p>For a project I'm working on, I have to extract shapes which have extra contrast (made to stand out) from backgrounds of arbitrary content.</p>
  <p>The extractor is meant to be fully automated.</p>
  <p>The major recurring facts in my favor are:</p>
  <ol>
  <li>The shapes I need to extract are already highlighted with a border (for human consumption)</li>
  <li>The shapes always have a medium/high contrast color for borders</li>
  <li>Shapes only have 0 or 1 holes</li>
  <li>All shapes (including if split into 2 or 3 parts) have the same size when those parts are put together.
  5. If you want to think in pictogorics. Think like a bathroom male sign <a href="https://i.stack.imgur.com/ysp9R.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ysp9R.png" alt="enter image description here" /></a> which has 2 parts.</li>
  </ol>
  <p>The major drawbacks include:</p>
  <ol>
  <li>The shapes can have the same color as some elements in background color</li>
  <li>The highlight color isn't always the same.</li>
  <li>The shapes' colors aren't always the same.</li>
  <li>Shapes may be split into two parts. With text, it's about the same as with letter <code>i</code> or <code>!</code></li>
  <li>I don't know the size of the shapes.</li>
  <li>Images have much noise outside the shapes. However, the shapes are very clearly drawn.</li>
  </ol>
  <p>Here's a google maps screenshot with text, which behaves closely to how the shapes can look like.</p>
  <p><a href="https://i.stack.imgur.com/r8us5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/r8us5.png" alt="google maps image" /></a></p>
  <p>Here's some examples of how my program would be delivered text:</p>
  <p><a href="https://i.stack.imgur.com/qTKLs.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qTKLs.png" alt="enter image description here" /></a> This one is the easiest. I was able to solve with just threshold.</p>
  <p><a href="https://i.stack.imgur.com/MW356.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MW356.png" alt="enter image description here" /></a>  &amp; <a href="https://i.stack.imgur.com/0QOsy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0QOsy.png" alt="enter image description here" /></a><br />
  With these two, I'm having a little more trouble with the yellow lines (highways in the map)</p>
  <p><a href="https://i.stack.imgur.com/m72wB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m72wB.png" alt="enter image description here" /></a> &amp; <a href="https://i.stack.imgur.com/wiKBl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wiKBl.png" alt="enter image description here" /></a></p>
  <p>These are harder to find because the text color is very close to the background color (even though it <em>is</em> darker)</p>
  <p>As a human, I know I can just execute threshold with google maps:</p>
  <p><a href="https://i.stack.imgur.com/5Zoua.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5Zoua.png" alt="enter image description here" /></a></p>
  <pre><code>img = cv2.imread(&quot;cut_429_798.png&quot;)
  gray = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
  _, threshold = cv2.threshold(gray, 200, 255, cv2.THRESH_BINARY)
  gray = cv2.cvtColor(threshold, cv2.COLOR_HSV2RGB)
  gray = cv2.cvtColor(gray, cv2.COLOR_RGB2GRAY)
  
  </code></pre>
  <p>... and then clean outside the letters.</p>
  <p>However, it's not the case with the documents I have. So, one way I tried solving, is by locating contours:</p>
  <pre><code>
  try:
      lower_white = np.array([0, 0, 0])
      upper_white = np.array([70, 70, 70])
      mask1 = cv2.inRange(img, lower_white, upper_white)
  except cv2.error:
      lower_white = np.array([0])
      upper_white = np.array([70])
      mask1 = cv2.inRange(img, lower_white, upper_white)
  
  
  try:
      lower_white = np.array([240, 240, 240])
      upper_white = np.array([255, 255, 255])
      mask2 = cv2.inRange(img, lower_white, upper_white)
  except cv2.error:
      lower_white = np.array([240])
      upper_white = np.array([255])
      mask2 = cv2.inRange(img, lower_white, upper_white)
  
  contours, hierarchy = cv2.findContours(mask1, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
  contours2, hierarchy2 = cv2.findContours(mask2, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
  
  if len(contours) &lt; 10:
      contours = contours2
      hierarchy = hierarchy2
  
  
  candidate_contours = []
  almost_childless_contours = []
  for i, contour in enumerate(contours):
      next, previous, first_child, parent = hierarchy[0, i]
  
      if first_child == -1:
          candidate_contours.append(contour)
          candidate_contours.append(contours[parent])
  
  </code></pre>
  <p>The steps I try to go through includes trying to get content from the whites or from the blacks. This (plus some small treatment of the mask) was able to push me to solve about 70-80% of the cases. However, I need a success rate of ~95% over thousands of individual images (the cut outs with the names, if you consider the google maps images).</p>
  <p>The code above is able to include, without failure a correct shape in <code>candidate_contours</code>. In total, <code>candidate_contours</code> contains:</p>
  <ol>
  <li>hole of shapes (not wanted)</li>
  <li>Shapes (wanted)</li>
  <li>Arbitrary content that was close to the shapes</li>
  </ol>
  <p>From manual tests and manual code manipulation, if I were able to distinguish whether a contour in the hierarchy is a shape without hole or is the inside of the hole of a shape, for the large majority (&gt;95%) I would be able to get rid of <code>1</code>. By exclusion, I could also get rid of problem <code>3</code>. The reason I do <code>contours[parent]</code> is due to <code>1</code> and causes <code>3</code> when there's no hole in the shape.</p>
  <p>A failed attempt was to use erode and dilate E.g.:</p>
  <pre><code># Failed!
  kernel = np.ones((3, 3), np.uint8)
  img = cv2.erode(img, kernel, iterations=1)
  </code></pre>
  <p>The shapes in the resulting image would often lose the contrast they had with the background.<br />
  So I tried to dilate</p>
  <pre><code># Failed!
  kernel = np.ones((3, 3), np.uint8)
  img = cv2.dilate(img, kernel, iterations=1)
  </code></pre>
  <p>Then I would suffer the opposite, where the shapes would just disappear and only the borders would remain, making it harder to find the shapes themselves by their contours.</p>
  <p>Here's examples of the intended outcome (in this case, with black background. It could have been white background too):</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: right;">Original</th>
  <th>Processed</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: right;"><a href="https://i.stack.imgur.com/wMo6u.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wMo6u.png" alt="enter image description here" /></a></td>
  <td><a href="https://i.stack.imgur.com/UnhKx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UnhKx.png" alt="enter image description here" /></a></td>
  </tr>
  <tr>
  <td style="text-align: right;"><a href="https://i.stack.imgur.com/n4y0n.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/n4y0n.png" alt="enter image description here" /></a></td>
  <td><a href="https://i.stack.imgur.com/NbI1j.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NbI1j.png" alt="enter image description here" /></a></td>
  </tr>
  <tr>
  <td style="text-align: right;"><a href="https://i.stack.imgur.com/qEgPP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qEgPP.png" alt="enter image description here" /></a></td>
  <td><a href="https://i.stack.imgur.com/BkGqw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BkGqw.png" alt="enter image description here" /></a></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>If that helps, you can think that the shapes are as varied as <a href="https://en.wikipedia.org/wiki/Wingdings" rel="nofollow noreferrer">wingdings 1,2 and 3</a> except there's also detached &quot;strokes&quot; such as how i has the &quot;.&quot; split from the vertical stroke.</p>