How to fix an N-channel MOSFET that's always on, without using resistors
The problem is that when using the MOSFET, you cannot leave the GATE open.  It's a VERY high impedance input that controls a very high-gain device.  So leaving it open allows anything and everything to happen to it.
This pin is the GATE:

Normally you will tie it to Vcc or GND via a resistor to keep it in the state you want when you are not driving it.  In your case you have a simple switch that will drive it to 4.5V.  So a resistor to GND is appropriate to keep the MOSFET OFF until you want it to turn ON.
Alternatively, if for some strange reason you really don't want to use a resistor that costs a tiny amount, you could use a SPDT switch that connects the GATE to GND when it's not connected to your 4.5V source.