Why doesn't the Attiny85 work in this circuit?
As shown when the power switch is off the supply voltage to the ATTiny could reverse by being fed through the FET.
That probably stops the power on reset working properly and could possibly cause damage.
You need to put the switch in the positive lead to VDD. With the GND terminal of the MCU going direct to the battery.
When using a single cell the voltage to the MCU just goes to zero because of conduction through the FET reducing the problem enough for it to work.