Camera tracking: Moving the camera too much
You should use weighted tracking; To do this, go to the Motion Tracking perspective, select a marker, and open the Properties Panel(N) and go to the subsection "Track"; There is a field labeled Weight, which controls the importance of that marker; 



Like most fields in Blender, this can be animated. What you need to do is make this field change, such that when it's almost out of view, the weight is 0, then, when it gets back in inside the Camera's frustum (Camera's View) make it go up again, 1 being the maximum; Keep in mind you shouldn't just make it 0 all of the sudden.