atmega1281 + Nokia 5110 LCD circuit works with USBtinyISP plugged in. Hangs without?
Seems that it was a power supply issue. I switched from the 9v regulated wall wart (that came with my Arduino from Adafruit) I was using to a 9v regulated supply that came with a piece of network gear. The wall wart put out 9v @ 2/3A. The other supply puts out 9v @ 1A. The other supply also has a proper ground plug to (it uses a IEC power cable).

So, it was either an issue of not enough amperage or unclean grounding.