Camera Solving, the trackers in 3D space are inverted
I had this same problem. The points closer to the horizon were solved to be the closest point to the camera. That was when I was using auto detected points. I started over and manually selected a lot of high contrast point, only kept the good ones after looking at the graph clip editor, and then the scene solved perfectly