Should I use a voltage regulator?
The safe voltage range for the ATtiny45 is between 2.7V and 5.5V, so you should be able to connect the battery directly to the MCU provided you don't need a different specific voltage for external components.

A switching power supply could allow you to get longer life out of the battery if you run it at about 3V or so the whole time, both because the MCU will use less current at the lower voltage and because the reduction of output voltage with a switching regulator translates to a reduction of input current (after taking the regulator efficiency into account).