Logic switch using logic gates
In electronics, the behavior of having a different voltage for a disconnect and a reconnect (or any particular usage similar) is called hysteresis. In you're case, you want to toggle off your circuit when the voltage is bellow 2.4v (this is done automatically by your circuit). However, you want to prevent a power up of the circuit unless the input voltage is greater than 4 volts (slightly above the rebound voltage that your battery is doing).

The specific circuit that you will be looking for is a Schmitt trigger. Depending on your choice of component around it, you will be able to enable and disable your circuit at will.

You can make one out of transistors, op-amps or simply buy a pre packaged Schmitt trigger IC that is ready to go.