Arduino and AVR Serial Communication over USB
Turns out the problem was with my Makefile. avr-gcc was missing the -mmcu=atmega328p flag. Once I added that and changed F_CPU to 16000000UL everything works.