Motion Tracking + MatchMoving -- What am I doing wrong?
If the camera is static you need a different kind of solver then the default. Use tripod solver for the camera, and exclude the trackers for the object in the camera solution. 
Then track again the object only, and solve the object separately.