Switch off this single cell lithium hi-power LED driver with protection circuit
Try this!

Put the red capacitor and the red resistor as shown below, and, instead of connecting \$SW_1\$ to ground, connect it to the junction between the new capactor \$C\$ and resistor \$R_{100}\$ (node \$V_x\$). I also suggest to short \$R_2\$. 

The values of \$R_1\$ and \$R_{T2c}\$ might need additional modifications.



How does it work 

When the circuit is off, \$C\$ is charged by \$R_{100}\$, \$R_3\$ and \$R_4\$ to \$V_{DD}\$. At the end of the transient \$V_x\$ will be 0V.

If you press the button, you'll give a pulse to the base of \$T_1\$, which will turn on, and the circuit will work as per your description.

Now, when you release the push button, \$C\$ will be discharge through \$R_{100}\$ and \$T_1\$, and at the end of the transient,  Vx will be \$V_{DD}-V_{SAT}\$ (let's approximate it to \$V_{DD}\$ for sake of simplicity).

Now, if you press again the push button, you'll force \$V_{DD}\$ in the junction between \$R_{T1}\$, \$R_{T2}\$, \$R_1\$. This will turn off \$T_1\$, and \$C\$ will be charged againg \$V_{DD}\$, i.e. \$V_x\$ will become 0 V at the end of the transient.

\$R_{100}\$ is choosen high to avoid an unwanted turn-on keep the pushbutton pressed, when you try to turn off the light. They will also act as a crude debouncer, together with \$C\$.

Note: the values are shown as an example. Tweaking/tuning might be necessary.