Motion tracking, scene doesn't line up with tracked data.
About the tracking points level ... maybe you have not set up the camera settings properly. Matching the cam settings (lens, sensor etc.) to the real camera, the footage was taken with, once did the thing for me - which you always have to do to get a proper result.

Hope that helps.