Prefetch maxing out 20+GB of RAM over a 355MB jpeg sequence
If you are tracking, you need the whole sequence and at full resolution (which conveniently you didn't mention... HD, 4k?).
Something to understand is that, even if your images are compressed as jpgs, blender will de-compress them to be able to use them, so multiply the number of horizontal pixels by the number of vertical pixels, then multiply that by 8(8 bit per pixel), and that number by 3 (Red, Green, Blue), and then by the number of frames, and that is the real memory usage.
Read: Why does Blender use so much memory for large textures?.
Also, not being able to cache the whole sequence is not a problem, as accessing jpegs is not very resource intensive. As long as you have enough resources to read the current frame you are fine.
What really matters is that you have enough RAM left to compute the solve information. So don't use all of your memory in the cache, and leave enough for whatever processing you need to do afterwards.