Movie clip editor playback problem Edit: It seems the problem is the format of the video file, Inter-frame and stuff, as converting it to an image sequence works perfectly (but awfully slow), I will try to convert it to another codec and report. Report: Apparently using intra-frame format makes the file big and slow as hell, same as image sequence. But at least it works.

I'm learning about motion tracking, and I'm having a weird problem. In the movie clip editor, the video is kind of stuttering and jumpy. At some frame it would freeze for 3 or 4 frames then continue, but skipping those frames ( sometimes continuing from the frame it stopped, or jumping back, resulting in the video losing some frames and becoming a complete wreckage ), while the trackers would act as if there was no stutter at all (their motions match the motion in the original video). All other video players haven't this problem. Using another program and played frame by frame aside each other, Blender is completely out of sync with the original.
An interesting thing is that when I reload and prefetch the clip again, the problem is gone, the clip plays smoothly and the trackers match up nicely, but after some time of working, it returns, and it always does when rendered. 

This happens at many points in my clip, and when rendered the 3D objects seem to be out of sync with the video. I have tried rendering the 3D elements as transparent and later putting in on the video, and it works much nicer. Any help will be appreciated.
I'm going to take a guess and say that your video is somehow corrupt, or you are dealing with issues with a delivery codec, meaning a highly compressed video files that use long GOP compression. Let me guess: h264, mp4, mts, AVC or AVCHD... or if we are talking about something recorded from a computer game... could be that you have one of those horrible videos that uses variable frame rate...

In any case there are a couple of solutions.

If your system has plenty of RAM, increase the cache size on the system preferences>System>Sequencer/Clip Editor>Memory Cache Limit.

Other thing to do is to transcode your video to an image sequence and use that to track instead.