OpenCV does some very odd things when saving grayscale videos. Specifically, it appears to covert them to RGB / BGR even if you have saving in a grayscale codec like Y800. This stack overflow post confirms, as does opening the files in a hex editor.
The real issue is that this conversion is lossy. When saving a grayscale image with pixel values of less than 10, they are converted to 0! Yikes!
The only reasonable solution I have found is to save all my movies at directories full of PGM files (similar to PPM but only grayscale).