Fast approximate optical flow / image shift
This might be a slow terrible solution, but you could do a FFT-based cross-correlation of subsequent frames and then find the peak to identify the offset between frames.  Maybe only do it on a small subset of the image to save processor cycles.

It would not work with rotation or drastic scene changes from one frame to the next, and there are probably better methods.  This is kind of a "I have a hammer so everything looks like a nail" solution.  I guess this is just like your naive solution, except there's no need for edge detection and the FFT makes it a lot faster than explicitly shifting one pixel at a time.

This question is similar, and no one's suggesting anything other than cross-correlation, so maybe it's not so bad: Using MATLAB to calculate offset between successive images