text: I have an application where I trained a model using Mask-R CNN to identify windows and doors in images of houses. The model predicts objects in unseen images online and on these predictions we try to fit new designs of windows and doors using a custom built front end. To fit new designs onto the detected masks, we try to detect four corners of the said mask. These new calculated points would be the corners for the uploaded designs using a front end called as configurator.
  Below is the code used to calculate the corners of all the instances (masks) predicted within an image.
  import cv2 import numpy as np r = results[0] # r now contains all the masks predicted mask = r['masks'].astype(np.float32) for i in range(len(r['rois'])): if r['scores'][i] > 0.95: find_corners(mask[:, :, i], mask.shape) def simplify_contour(contour, n_corners=4): """ Binary searches best `epsilon` value to force contour approximation contain exactly `n_corners` points. :param contour: OpenCV2 contour. :param n_corners: Number of corners (points) the contour must contain. :returns: Simplified contour in successful case. Otherwise returns initial contour. """ n_iter, max_iter = 0, 100 lb, ub = 0., 1. while True: n_iter += 1 if n_iter > max_iter: return contour k = (lb + ub)/2. eps = k*cv2.arcLength(contour, True) approx = cv2.approxPolyDP(contour, eps, True) if len(approx) > n_corners: lb = (lb + ub)/2. elif len(approx) < n_corners: ub = (lb + ub)/2. else: return approx def find_corners(mask, size): """ Returns the 4 corners for the given door or window mask The points are normalized to be between 0..1 The order must be d c a b -> [a, b, c, d] where the coordinate 0, 0 is the lower left corner. :param mask: float32 mask image :param size: tuple (x, y) the original image size :return: [{x: float, y: float}] point array """ m = np.uint8(mask) # thresh = cv2.Canny(m, 0, 1) contours, h = cv2.findContours(m, 1, 2) for cnt in contours: approx = simplify_contour(cnt, 4) if len(approx) == 4: p = [x[0] for x in approx] p_list.append(p) print(p) c = order_by_rows(p) return [{'x': float(x[0]) / size[1], 'y': float(x[1]) / size[0]} for x in c] return None def order_by_rows(points): """ Orders the given points counter clockwise around their center Only works because there are 4 entries :param points: 4 points :return: 4 points ordered """ c = sorted(points, key=cmp_to_key(compare)) if c[0][0] > c[1][0]: tmp = c[0] c[0] = c[1] c[1] = tmp if c[2][0] < c[3][0]: tmp = c[2] c[2] = c[3] c[3] = tmp return c def compare(a, b): if a[1] < b[1]: return -1 if a[1] > b[1]: return 1 return 0
  The problem is that when the new designs are fitted onto these corner points, majority of them are a bit skewed, leaving the object of interest partially uncovered. Following are the images of the mask predicted and the image from the configurator front end when trying to re-design windows.
  Source: Python Questions