Tracking forward/backward camera motion
Blender can track backwards and forward accurately with no problem. I've done it many times.
So there is your answer.
Most likely the issue is with the compression used on the video. Intraframe compression or image sequences works great. Video with long GOP does not and might lead to errors retrieving frames going backwards.