title: distinguishing objects with opencv
body: <p>I want to identify lego bricks for building a lego sorting machine (I use c++ with opencv).
  That means I have to distinguish between objects which look very similar.</p>
  
  <p>The bricks are coming to my camera individually on a flat conveyer. But they might lay in any possible way: upside down, on the side or "normal". </p>
  
  <p>My approach is to teach the sorting machine the bricks by taping them with the camera in lots of different positions and rotations. Features of each and every view are calculated by surf-algorythm.</p>
  
  <pre><code>void calculateFeatures(const cv::Mat&amp; image,
          std::vector&lt;cv::KeyPoint&gt;&amp; keypoints,
          cv::Mat&amp; descriptors)
  {
    // detector == cv::SurfFeatureDetector(10)
    detector-&gt;detect(image,keypoints);
    // extractor == cv::SurfDescriptorExtractor()
    extractor-&gt;compute(image,keypoints,descriptors);
  }
  </code></pre>
  
  <p>If there is an unknown brick (the brick that i want to sort) its features also get calculated and matched with known ones.
  To find wrongly matched features I proceed as described in the book OpenCV 2 Cookbook:</p>
  
  <ol>
  <li><p>with the matcher (=cv::BFMatcher(cv::NORM_L2)) the two nearest neighbours  in both directions are searched</p>
  
  <pre><code>matcher.knnMatch(descriptorsImage1, descriptorsImage2,
    matches1,
        2);
  matcher.knnMatch(descriptorsImage2, descriptorsImage1,
    matches2,
    2);
  </code></pre></li>
  <li><p>I check the ratio between the distances of the found nearest neighbours. If the two distances are very similar it's likely that a false value is used.</p>
  
  <pre><code>// loop for matches1 and matches2
  for(iterator matchIterator over all matches)
    if( ((*matchIterator)[0].distance / (*matchIterator)[1].distance) &gt; 0.65 )
      throw away
  </code></pre></li>
  <li><p>Finally only symmatrical match-pairs are accepted. These are matches in which not only n1 is the nearest neighbour to feature f1, but also f1 is the nearest neighbour to n1.</p>
  
  <pre><code>for(iterator matchIterator1 over all matches)
    for(iterator matchIterator2 over all matches)
      if ((*matchIterator1)[0].queryIdx == (*matchIterator2)[0].trainIdx  &amp;&amp;
      (*matchIterator2)[0].queryIdx == (*matchIterator1)[0].trainIdx)
        // good Match
  </code></pre></li>
  </ol>
  
  <p>Now only pretty good matches remain. To filter out some more bad matches I check which matches fit the projection of img1 on img2 using the fundamental matrix.</p>
  
  <pre><code>std::vector&lt;uchar&gt; inliers(points1.size(),0);
  cv::findFundamentalMat(
      cv::Mat(points1),cv::Mat(points2), // matching points
      inliers,
      CV_FM_RANSAC,
      3,
      0.99);
  
  std::vector&lt;cv::DMatch&gt; goodMatches
  // extract the surviving (inliers) matches
  std::vector&lt;uchar&gt;::const_iterator itIn= inliers.begin();
  std::vector&lt;cv::DMatch&gt;::const_iterator itM= allMatches.begin();
  // for all matches
  for ( ;itIn!= inliers.end(); ++itIn, ++itM)
  
    if (*itIn)
      // it is a valid match
  </code></pre>
  
  <p><img src="https://i.stack.imgur.com/Nvozq.jpg" alt="good matches">
  The result is pretty good. But in cases of extreme alikeness faults still occur.<br>
  In the picture above you can see that a similar brick is recognized well.</p>
  
  <p><img src="https://i.stack.imgur.com/qymzR.jpg" alt="bad matches">
  However in the second picture a wrong brick is recognized just as well.</p>
  
  <p>Now the question is how I could improve the matching.</p>
  
  <p>I had two different ideas:</p>
  
  <p><img src="https://i.stack.imgur.com/iS6Ac.png" alt="All possible brick views"></p>
  
  <ul>
  <li><p>The matches in the second picture trace back to the features really fitting, but only if the visual field is intensely changed. To recognize a brick I have to compare it in many different positions anyway (at least as shown in figure three). This means I know that I am only allowed to minimally change the visual field. The information how intensely the visual field is changed should be hidden in the fundamental matrix. How can I read out of this matrix how far the position in the room has changed? Especially the rotation and strong scaling should be of interest; if the brick once is taped farer on the left side this shouldn't matter.</p></li>
  <li><p>Second idea:<br>
  I calculated the fundamental matrix out of 2 pictures and filtered out features that don't fit the projections - shouldn't there be a way to do the same using three or more pictures? (keyword Trifocal tensor). This way the matching should become more stable. But I neither know how to do this using OpenCV nor could I find any information on this on google.</p></li>
  </ul>