Cheapest / easiest way to get started with AVR programming
My advice: Get a USBTinyISP programmer. It's a $20 upfront investment, but it's well worth it. (both types of programming cables included and will work with any modern computer). Also, while you're at it get a ZIF Socket for ease of programming.

Now all you need is a breadboard, wire, some of the basics(resistors, capacitors, AA batteries), an ATTiny, and you're good to go.