title: How do I use the relationships between Flann matches to determine a sensible homography?
body: <p>I have a panorama image, and a smaller image of buildings seen within that panorama image. What I want to do is recognise if the buildings in that smaller image are in that panorama image, and how the 2 images line up.</p>
  
  <p>For this first example, I'm using a cropped version of my panorama image, so the pixels are identical.</p>
  
  <pre><code>import cv2
  import matplotlib.pyplot as plt
  import matplotlib.image as mpimg
  import numpy as np
  import math
  
  # Load images
  cwImage = cv2.imread('cw1.jpg',0)
  panImage = cv2.imread('pan1.jpg',0)
  
  # Prepare for SURF image analysis
  surf = cv2.xfeatures2d.SURF_create(4000)
  
  # Find keypoints and point descriptors for both images
  cwKeypoints, cwDescriptors = surf.detectAndCompute(cwImage, None)
  panKeypoints, panDescriptors = surf.detectAndCompute(panImage, None)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/XmBMX.png" rel="noreferrer"><img src="https://i.stack.imgur.com/XmBMX.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/SJgyc.png" rel="noreferrer"><img src="https://i.stack.imgur.com/SJgyc.png" alt="enter image description here"></a></p>
  
  <p>Then I use OpenCV's FlannBasedMatcher to find good matches between the two images:</p>
  
  <pre><code>FLANN_INDEX_KDTREE = 0
  index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
  search_params = dict(checks=50)
  flann = cv2.FlannBasedMatcher(index_params, search_params)
  
  # Find matches between the descriptors
  matches = flann.knnMatch(cwDescriptors, panDescriptors, k=2)
  
  good = []
  
  for m, n in matches:
    if m.distance &lt; 0.7 * n.distance:
      good.append(m)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/Haxn5.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Haxn5.png" alt="enter image description here"></a></p>
  
  <p>So you can see that in this example, it perfectly matches the points between images. So then I find the homography, and apply a perspective warp:</p>
  
  <pre><code>cwPoints = np.float32([cwKeypoints[m.queryIdx].pt for m in good
                            ]).reshape(-1, 1, 2)
  panPoints = np.float32([panKeypoints[m.trainIdx].pt for m in good
                            ]).reshape(-1, 1, 2)
  h, status = cv2.findHomography(cwPoints, panPoints)
  
  warpImage = cv2.warpPerspective(cwImage, h, (panImage.shape[1], panImage.shape[0]))
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/lPZmg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/lPZmg.png" alt="enter image description here"></a></p>
  
  <p>Result is that it perfectly places the smaller image within the larger image.</p>
  
  <p>Now, I want to do this where the smaller image isn't a pixel-perfect version of the larger image.</p>
  
  <p>For the new smaller image, the keypoints look like this:</p>
  
  <p><a href="https://i.stack.imgur.com/IJ4sM.png" rel="noreferrer"><img src="https://i.stack.imgur.com/IJ4sM.png" alt="enter image description here"></a></p>
  
  <p>You can see that in some cases, it matches correctly, and in some cases it doesn't.</p>
  
  <p>If I call <code>findHomography</code> with these matches, it's going to take all of these data points into account and come up with a non-sensical warp perspective, because it's basing it on the correct matches and the incorrect matches.</p>
  
  <p><a href="https://i.stack.imgur.com/eTmfs.png" rel="noreferrer"><img src="https://i.stack.imgur.com/eTmfs.png" alt="enter image description here"></a></p>
  
  <p>What I'm looking for is a missing step in between detecting the good matches, and calling <code>findHomography</code>, where I can look at the relationship between the matches, and determine which matches are therefore correct.</p>
  
  <p>I'm wondering if there's a function within OpenCV that I should be looking at for this step, or if this is something I'll need to work out on my own, and if so how I should go about doing that?</p>