Sum of Squared Intensity Difference for block matching with ORB features, c++
Your results look reasonable. However, you're likely to get a somewhat unreliable result: the good matches are located in a small portion of the image, while a large spread would help in getting overall more accurate estimates (but that's probably due to your images).
Experimenting will work here! Otherwise, you can compute the expectation of the SSD between 2 blocks that are a rightful match under the noise level of your images. 


Eventually, bad matches should be eliminated by the RANSAC part, but you may have to do a trade-off between getting many but sometimes wrong matches (loose SSD threshold) or stricter ones (at the risk of not having enough matches o estimate the pose).