How to get started with microcontrollers? (ATTINY2313)
To get started with a bare ATTiny2313 you'll need an AVR ICSP, such as this USB AVR Programmer. You can breadboard the connection from the ICSP cable to the chip (see ATTiny2313 datasheet and ICSP pinout). Depending on the programmer used, it may be capable of providing power to the circuit (depending on your needs you wouldn't need to start with a 5V regulator). 

Also mentioned in the product page for the programmer is 'avrdude'; basically a free gcc based toolchain for developing and loading code onto Atmel AVR parts (including the attiny family). 

In theory an Arduino can also be used for ICSP if you have one laying about, but I'm less familiar with that process.