Operating an ATmega328P-PU at 2.5V
It just dawned on me that, in my research, I saw people mention brownout detection. I searched into it and this does seem to be the issue.

http://openenergymonitor.org/emon/node/342

EDIT: For those curious, change the extended fuses to 0x06 to drop the brownout voltage to 1.8V.