How do I match the audio and video streams in my video file with the audio I recorded separately?
You could try processing the audio first to get it to the same speed as the video playback. Using (for example) Audacity you can apply a speed change of -0.1% (1000/1001) which is very likely the amount you're drifting by. You can change the rate without changing pitch, but if you can tolerate the very slight pitch shift you'll get slightly better results by just changing speed.

If that isn't the correct rate, then you need to determine the change needed by measuring the exact length of the audio tracks between two points near the beginning and end. Once you know them you can compute the ratio and apply that.