Where to decouple battery in circuit with power switch?
Decoupling on the other side of the switch is not recommended as it will put the capacitor too far from the supply pin and add series resistance and inductance defeating the purpose of the capacitor.  10uF is OK, but in any case you should parallel the 10uF with a 0.1uF cap and maybe a .01uF cap as well for high frequency decoupling.

If you MUST put the 10uF on the far end of the switch, put a 0.1uF capacitor directly on the supply pins of the ATtiny and keep the larger capacitor on the opposite side of the switch.

Like the comment says you might be better off just sensing the switch with a GPIO.