The BRIEF descriptor is one the simplest of the binary descriptors and also the first published. BRIEF operates by comparing the same set of smoothed pixel pairs for each local patch that it describes. For each pair, if the first smoothed pixel’s intensity is larger than that of the second BRIEF writes 1 in the final descriptor’s string, and 0 otherwise. The sampling pairs are chosen randomly, initialized only once and used for each image and local patch. As usual, the distance between two binary descriptors is computed as the number of different bits, and can be formally written as sum(XOR(descriptor1, descriptor2)).
Our method for adding rotation invariance is straightforward and uses the detector coupled with the descriptor. Many keypoint detectors can estimate the patch’s orientation (e.g. SIFT and SURF) and we can make use of that estimate to properly align the sampling pairs. For each patch, given the angle of the patch, we can rotate the sampling pairs according to the patch’s orientation and thus extract rotation invariant descriptors. The same principle is applied in the original implementation of the other rotation invariant binary descriptors (ORB, BRISK and FREAK), but as opposed to them we just take the orientation of the patch from the keypoint detector instead of devising some orientation measurement mechanism.
Now for the fun part – comparing rotation invariance BRIEF with BRIEF’s original version. I’ll also compare to SIFT to see how binary descriptors compete with some of the floating point descriptors.
Bark – zoom + rotation changes.
Boat – zoom + rotation changes.
Graffiti – view point changes.
Wall – view point changes.
Below are the images of each set in the benchmark. In each set, the images are ordered from left to right and top to bottom (the first row contains images 1-3, the second row contains images 4-6).
Bikes (blur): you can notice that image 6 is far more blurred than image 1.
The protocol for the benchmark is the following: in each set, we detect keypoints and extract descriptors from each of the images, compare the first image to each of the remaining five images and check for correspondences. The benchmark includes known ground truth transformations (homographies) between the images, thus we can compute the percent of the correct matches and display the performance of each descriptor using recall vs. 1-precision curves.
I used the public OpenCV implementation for our experiments. SIFT is used as a keypoint detector and I used the 512 bits version of BRIEF and rotation invariant BRIEF.
Below are tables summarizing the area under the recall vs. precision curve for each of the sets, averaged over the five image pairs – higher values means the descriptor performs better. For clarity, I also specified the type of image transformation introduced by each set.
Notice that for sets that depict orientation changes (Bark and Boat), the rotation invariant version of BRIEF performs much better than the original (not invariant) version. However, in sets that depict photometric changes (blur, illumination and JPEG compression) and do not depict orientation changes, the original version of BRIEF performs better than the rotation invariant one. It seems that when orientation changes are not present, trying to compensate for them introduces noise and reduces performance. Notice also that since the set Graffiti introduces some orientation changes (as can be seen from the images above), the rotation invariant version of BRIEF has an advantage over the original version of BRIEF. One can also see that although the “Wall” set exhibit view point changes, the images in the set have very much the same orientation, thus the rotation invariant version of BRIEF performs worse than the original one. On a side note, it is also very interesting to see that in some of the sets, BRIEF and Rotation Invariant BRIEF even outperform the SIFT descriptor (keep in mind that BRIEF is a lot faster to extract and match and also take much less storage space).
To further illustrate the difference in performance between the original and the rotation invariant version of BRIEF, below are recall vs. 1-precision curves for the sets Bikes, Graffiti and Boat, respectively.
Recall vs. Precision curves for the set Bikes – notice that the original, not rotation invariant, version of BRIEF outperforms the rotation invariant version.
Recall vs. Precision curves for the set Graffiti – notice that since the images depict orientation changes, the proposed rotation invariant version of BRIEF outperforms the original implementation.
Recall vs. Precision curves for the set Boat – notice that since the images depict orientation changes, the proposed rotation invariant version of BRIEF outperforms the original implementation.
Notice again that BRIEF outperforms it’s rotation invariant version in the Bikes sets, which depicts photometric changes (specifically, blur) while the rotation invariant version of BRIEF outperforms the original version in the sets Graffiti and Boat which depict rotation changes.
I’m in the process of contributing an implementation of the rotation invariant version of BRIEF to OpenCV. I’ve forked the GitHub OpenCV3.0 repository and implemented the changes under my forked repository.
I have presented a rotation invariant version of BRIEF that makes use of the detector’s estimation of the keypoint orientation in order to align the sampling point of the BRIEF descriptor, thus making it rotation invariant. I’ve demonstrated the advantage of the rotation invariant version of BRIEF in scenarios where orientation changes are present and also it’s disadvantage in dealing with photometric changes (blur, lightning and JPEG compression). Finally, I’ve published a C++ implementation of the proposed descriptor integrating it into OpenCV3.
 Rosten, Edward, and Tom Drummond. “Machine learning for high-speed corner detection.” Computer Vision–ECCV 2006. Springer Berlin Heidelberg, 2006. 430-443.
Cool stuff, glad you are writing again! Would there be any way of knowing in advance which version would perform better for , or would you have to run both and pick the better results?
Also, do you have any intention of posting the performance comparison that you have mentioned in the past?
It is always better to test more than one alternative and only then decide on an algorithm. Since the code I wrote follows OpenCV’s common interface for descriptors, it’s very easy to test both of BRIEF’s version in your application.
Having said that, you can estimate in advance in which of BRIEF’s versions you should use. In case that your application is likely to encounter orientation changes between image of the same scene/object – then you should use rotation invariant BRIEF. In case that it’s more likely to encounter photometric changes (e.g. blur and lightning changes), then you should use the original version of BRIEF (with no rotation invariance) since it copes better with those kind of image variations.
Regarding the performance comparison, I’ll actually planning to post it in the following month.
first of all thanks for your efforts you put in your blog posts.
If i remember correctly you also mentioned in some replies to previous blog posts that you experimented with rotation invariant BinBoost and tried to get this one into opencv too.
What are your experiences with rotation invariant BinBoost compared to BRIEF and BRISK so far?
Did your efforts on BinBoost also make it into your git repo you mentioned above?
I’m just curious, because BinBoosts 64bit length is a really nice property.
Thank you for your interesting questions!
Well, I compared the following descriptors and I hope to post my results very soon: BRIEF, ORB, BRISK, FREAK, LDA-Hash, DBRIEF, BinBoost, AKAZE, SIFT and SURF. To make a fair comparison, I enforced rotation invariance on all of the descriptors. I hope to post the results very soon.
By they way, if you like BinBoost for being 64bit length, keep in mind that DBRIEF is only 32 bit length.
The BinBoost code is licensed under GPL, while OpenCV is licensed under BSD.
I don’t know how the non_free stuff of openCV is licensed. (e.g. the SIFT code itself still could be BSD as BSD and patent protection are two completely unrelated things).
So if it would be possible at all to integrate BinBoost it may have to go into the non_free opencv section. Companies who help develop and use OpenCV really don’t want GPL code in there by default, otherwise they would have to make their source code public which is a no go. And even the danger that some GPL code could slip somewhere into a compiled binary is a nightmare.
In contrast the “loading from file” issue may probably not be that severe. I guess the .bin file contains some kind of precomputed tables so it may be possible to make this table part of the source code and compile it directly into the extractor related obj files?!
Licensing can indeed be problematic here, thanks for bringing that up. In any case, I’ll just make a pull request to OpenCV’s repository and let the moderators handle those issues.
Regarding the “loading from file” issue, your suggestion might solve that. I’ll look into it once I’ll make the necessary changes for integrating BinBoost into OpenCV3 (I integrated it into the earlier version – 2.4.9).
Which descriptor would you use for segmentation?
Namely it both patches are form the same object they will be “close” and if are not, on the contrary.
I’m not an expert on object segmentation and I’m not sure local descriptors are the best approach.
In any case, I don’t think that you need a rotation invariant descriptor if you are only comparing patches in a single image.
I want to thank you for your efforts you put in your blog posts.
I am a student who interested in doing research about feature descriptor.
really helps me a lot on research.
I recently wrote a program about descriptor evaluation with OpenCV3.0, and wanted to evaluate the difference in performance between different descriptor.
However, i encountered a problem is “how to calculate and draw recall vs 1-precision curve ?” just like the recall vs 1-precision curve in your post. Could you give me some information or teach me?
By the way, my English is not good enough. Sorry about that.
and i used the evaluation scripts successfully.
But, i have another question.
harris-affine detector and descriptors only updated to SIFT.
So, how should i do to evaluation detector (ex. FAST or ORB ) and descriptor (ex. SURF, BRIEF..etc) ?
In my experiments I extracted descriptors using OpenCV, wrote the results to text files and then used the Matlab evaluation scripts from that page.
I will try to find a correct way to write the results to evaluate descriptors.
Thanks for your reply and help in these days.
After a few days work, i still can’t get the correct data to evaluate descriptor with the scripts provided in the site by Mikolajczyk. I can get the keypoints from detector (e.g. SIFT, ORB…) with openCV, but i don’t know how to get the coefficients of the following equation of the ellipse : a(x-u)(x-u)+2b(x-u)(y-v)+c(y-v)(y-v)=1 with the keypoints i get.
However, I found that openCV2.4.* has a .cpp file call “detector_descriptor_evaluation” but the result i get is weird ( ORB is outperforms than BRIEF in “graf : viewpoint change database” ).
So, I think this evaluation file maybe isn’t suitable for binary descriptors or something i misunderstand.Do you have any suggestion?
Sorry for the late response and thank you for your interest in my blog.
Could you be kind to send me (guangyang89@gmail.com)the evaluation source code you used in LATCH to evaluate the performance of binary descriptors?
I’m working on object detection by using SIFT and SURF in Matlab, and I need to evaluate the precision of these algorithms. can you please help me with your source code?
Can i get matlab lab code for brief descriptor. i am facing lot of problem in writing code of brief. openCV codes cant be replicated due to functions used.
I’m afraid I don’t know of any Matlab implementation of BRIEF (other than the OpenCV code which you can use mex to run from Matlab).
Great tutorial. I want to implement the CUDA version of rotated BRIEF in OpenCV, I am planning to implement the three stages of the pipeline (detection, description and matching) in CUDA. Do you think this is a feasible project? We also want to parallelize the rotation of key point patch using CUDA.
Do you think this is a feasible project idea? I want to be sure this is not publicly implemented yet.
I’m afraid I don’t have any experience in CUDA programming so I can’t really give a good answer.
However, if you could mail me (gil.levi100@gmail.com) the details of the project, I’ll try to ask one of my colleagues who worked a lot on CUDA implementation of binary descriptors.
I also want to do some tests and benchmarks of descriptors but i have a problem if i want to use opencv and the oxford benchmarks.
How can i get from an opencv keypoint to the equation of the ellipse: a(x-u)(x-u)+2b(x-u)(y-v)+c(y-v)(y-v)=1 ?
I searched for it, but i didn’t find a good solution.