Why does camera this tracking fail completely even when the solution value is very low? The attached animated gif shows some frames of a video with a mesh box placed at each tracking marker used to successfully track (solution error &lt;.4) an 800 frame chunk of video. 

The point is to illustrate just how poorly the camera tracker calculated: when the box is at a marker placed at the top of a faraway building on the left, it is actually visually much larger than when it is at the tracking marker closest to where the real camera was. I went to great lengths to be sure that the camera's sensor size and focal length are correct, and manually placed and followed all markers to be sure they were accurately tracking. 

There is not a lot of parallax motion over the 800 frames, but there should have been enough to at least know that the point closest to the camera moved more than the point furthest away. Sometimes it seems to work, sometimes it doesn't. Does anybody know how to improve this?
For this kind of shot, where the camera hardly moves I don't think you're going to get good results using motion tracking. 

There is so little parallax information for the reconstruction.

For a detailed post on motion tracking read:

How can I get better results when doing camera motion tracking?

I'd suggest you stabilize the footage as if it was a fixed shot and then do the reconstruction using BLAM ( https://github.com/stuffmatic/blam ).


  NOTE: Blam is no longer mantained and has been replaced by Fspy: https://fspy.io/, but there is no importer for blender 2.79 or previous versions.


For a detailed post on how to use blam and Fspy read: How can I recreate geometry using a photograph?