AVR startup time w/ external xtal/osc
So it turns out that using an external 32.768 crystal reduces the power consumption of an ATtiny85 a LOT.

Running with an internal 1 MHz clock source and having the AVR sleep most of the time in power down mode - only waking up every 32 ms - it consumes around 62 microamps. However, using a 32.768 crystal as the external clock source, the power consumption is only around 22 microamps, even when running at "full throttle" so to speak (32768 ticks per second with no sleep mode). 

So it turns out that running it at that low of a speed cuts down on power consumption so much that you don't actually need to sleep it.