Solve error too much
Start by trying to gain as much tracks over the whole framerange as possible. Short tracks living only for a few frames are very inefficient and can be removed in the "Clean up" Panel. Also you might want to check out the "Graph" section of the Movie Clip Editor, where you can easily see and delete misbehaving tracks. Other tricks might be changing the keyframes in the "Solve" Panel or letting Blender guess the best keyframes itself. You can also try the "Refine" options beneath it. Good luck!