Tell Blender to ignore tracker for camera solve
No need to delete the tracker.

Each tracker can be assigned a weight, which determines the tracker's contribution for the final camera or object solve.

Just bring the value to zero.



More info: https://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.70/Motion_Tracker