Time shift between tracked camera motion and original movie
I think the problem was with the video settings not matching the render settings. The video was recorded with a cellphone camera, and came out as 24.856 frames per second. Strange indeed.
If I repeat the process on a list of PNGs, or a video shot with a decent camera that produces a decent frame rate, this works fine.

Thanks for all the helpful comments.