Momentary switch on-off circuit
The following circuit is one that I modified from Mosaic Industries.



simulate this circuit &ndash; Schematic created using CircuitLab
I used a relay in the schematic for ease of testing, but a switch should work equally well.
As per your specs, it starts in the &quot;ON&quot; state. The voltage is near 5V when on, and near ground otherwise. The current draw is minimal. It is designed to deliver 1mA to RL. RL is necessary if your load draws less than 1mA.
There is somewhat of a trade-off between how often one can cycle toggle the circuit, and how immune it is to switch contact bounce. I cannot give an exact figure, for how long the switch needs to be closed for proper operation, but it is somewhere in the vicinity of 50mSec. That allows a toggle rate of about 10/sec. If you want more bounce immunity, C1 can be tweaked, but if tweaked too much other components may need to be tweaked as well.