My ATTiny84 circuit works fine until I connect a piezo buzzer
No voltage regulators or capacitors... could that be the problem?


Yes. With an unregulated power supply and no bypass capacitors the supply voltage will change whenever the current changes. CMOS logic gates draw brief spikes of high current when they change states, which interacts with power supply inductance and resistance to cause glitches that are likely to make the MCU misoperate.      

But why does it work OK without the buzzer connected? Probably because there is just enough stray capacitance in the chips and wiring to keep them working. But peizo buzzers have a relatively large capacitance which must be charged when voltage is applied to them, causing a massive current surge. The tiny amount of stray capacitance in your circuit is not nearly enough to hold the supply voltage stable while the piezo element charges, so Vcc sags and causes the MCU to freeze or do other weird stuff. Even when the buzzer is 'off' there may still be transitions when the I/O port is initialized or accessed, which produce pulses too short to hear but long enough to glitch the MCU.

The power supply pins of any digital logic IC should always be bypassed with a capacitor, even it seems to work fine without it. MCUs are full of sequential logic which often fails spectacularly if even a single glitch gets through, so having a stable noise-free power supply is essential.

You should install a 100nF ceramic capacitor directly across the supply pins of both the ATtiny84 and 74HC595, and a larger 'bulk' electrolytic capacitor >=10uF to handle longer current surges.