What can't the ATtiny do that the Arduino can?
Update: To better answer the question, ATTiny's can do anything that don't require hardware the ATTiny's are missing, which include the hardware serial port, hardware i2c, or multiple interrupts. Since this is hidden away in the Arduino Libraries, unless you have a good understanding of how they work or read the actual library files, things like the Wire library (hardware i2c) will not work.
From http://forum.arduino.cc/index.php/topic,5582.0.html:

For example a ATTINY84 has 1 Interrupt, no Hardware UART (Serial or i2c) and a small handful of GPIO pins, less program space and much less ram but comes in a 14 pin package.
A ATMEGA328 has 2 Interrupts, 1 Hardware UART, more ram more program space and more GPIO, which is great, if you really need it.

Additionally, most ATTiny's have no hardware multiplication, so a compiler would have to do non-base-2 multiplication in software.
Finally, Arduinos rely on a bootloader instead of spi programming, so you need an ATTiny with atleast 4~8kb flash for them to work with the Arduino bootloader.
ATTiny44/45/84/45/2313 are popular as micro-Arduinos. They have about a fourth or eighth of the ram and flash of a ATMega328 that the official Arduino Uno uses.
There are many ATTiny's, so no single statement can be said about them all.
Also see here: http://hlt.media.mit.edu/?p=1695