Standalone ATMEGA328: Verification mismatch on random byte on -subsequent- uploads only
I got some help from someone today. I oversaw a small detail when using the Arduino IDE with an ISP programmer: Hold shift when uploading to target the MCU at the end of the line rather than the MCU in the programmer. 

So yeah, I hold shift when clicking upload and everything works great now. Sometimes it's the simplest things you'd never expect.