Does anyone know how to fix random tracking errors in Blender?
Blender's trackers and 3D reconstruction are quite robust and advanced, but it is hard to pin down what is happening without access to the video you are trying to track. "Solid visually to me" can mean anything or nothing at all.

Solid reconstruction is only possible with accurate camera tracking.

Common mistakes are:

-Trying to track blurry or distorted video, or video where the tracked elements move to far from frame to frame, for that you need to enlarge the search box.

-Expecting 3d information out of shots where the camera is stationary (where the video was taken from a fixed place and the camera just pans and tilt without any displacement). For those shots you can use "Tripod" Solve, but know that there is no way to get 3d information out of such shots. It is considered 2D tracking as you will only get the camera movement but no real 3d reconstruction.

There is a very detailed post on this site that you should read to troubleshoot your problems. 

How can I get better results when doing camera motion tracking?..