Why is tracking getting less accurate after going past cached frames?
From the information you have provided it sounds like you are attempting to perform your tracking on your actual movie file.

If that is the case you should export the entire movie to a series of image frames, then import that image sequence into the movie clip editor and perform your tracking on the image sequence.

Most movie file formats store their data, not as a sequence of fixed images, but as a description of the changes in the image since the last frame, and depending on the compression level used will discard some of that information.
For an insight into how complex this issue can become you may like to look at the story of the recovery of the lost video from the first ocean landing of SpaceX's recoverable Falcon 9 Stage 1.

Skipping backwards and forwards through movie files can produce decoding differences in the frames being viewed, and can result in inconsistent output, which in turn is compounded in this case as the movie file is continually being re-read from disk.  

To break your movie file into an image sequence:  


Open a new instance of Blender
Set the render properties for the project (frame size, # of frames, frame rate etc. to match the movie file)
Set the output format to an image format (.png is usually OK) 
Set the output directory to a new directory to store these files
Switch to the video editing screen layout
With your mouse over the Video Sequence Editor window
Press Shift+A and select Movie to add a movie file (Navigate to it and select your movie file)
Make sure the strip is placed with the start frame on frame 1.  
Render the animation (switch back to the default view, or press Ctrl+F12).  


The image sequence produced can be used in place of the movie file in your tracking scene in exactly the same way - simply select all the frames in the directory, in place of the movie file.

Hopefully this solves your problem.