Push button to only turn off circuit
Like this







simulate this circuit &ndash; Schematic created using CircuitLab

Use the CMOS NAND, because the NAND4 output will rise completely up to VDD, which places ZERO volts on the base-emitter junction of the transistor, to ensure the transistor is OFF. 

How much current can this switch provide?  Depends on the current thru R2. 
And the current thru R2 depends on how close to Ground the output of NAND4 will go. Assume that output is not at 0volts, but at 0.4 volts. That places about  (5v - Veb - 0.4v ) or (5v - 0.6 - 0.4) = 4 volts across R2. Using Ohms Law, we have 4 milliAmps thru R2, and to achieve a well-saturated transistor you should keep the steady-state current below 10X that, or 40 milliAmps.

During turnon, you will be charging bypass capacitors in your circuit, and 100s of milliAmps will be flowing into the zero-voltage-initial capacitors. 
This "charging interval" will last microseconds or milliSeconds, as the bypass capacitors ramp up to about (5v - 0.2v) = 4.8 volts.

Is 4.8 volts (at 40 milliamp DC demand) adequate?

===========================================

When "unswitched power" initially rises,  the status of this latch will be uncertain because we do not know how the "external momentary low" behaves. The pullup of R1 is ideal, in that the "high" input will not alter LatchOut state. If "external momentary low" can become and remain LOW as Unswitched Power rises, then the Switched Power will be turned ON even as Unswitched Power ramps up.

You need to decide what behavior is needed. PowerOnReset circuits can be complex. Draw a block-diagram of your system: battery, stepup regulator, this switch, and your various loads and the currents needed.