Good Tracking Markers with BAD Object Tracking Solution
The error was using tripod solve, and not defining the correct span of frames to be used for the solution.

Using tripod solving does not result in accurate 3D information.

If the span of frames used for the solution is not set, blender might try to solve the object in the wrong segment of the video. Enabling the "Keyframe" option means that blender will choose what frames to use to find a solution automatically.

After turning off the tripod solve and allowing blender to define the keyframes to be used, the solve error was very low.