Trying to cut down to one power switch in a circuit with two power sources
You haven't mentioned using a resistor to jumper to the gate of your transistor.  As transistors are current controlled, you need a resistor to limit gate current.  There's a good chance that you're causing the volt drop by pulling too much current.  It would help if you added the theoretical transistor to the diagram to show us where you wired it and how.  Transistors and mosfets have to be connected correctly within the circuit, and it matters whether they are switching between the load and the positive rail or between the load and the ground rail.

One thing to consider is that you don't necessarily need to use two separate power sources.  If you want you can just use one battery and something like this to efficiently provide the other voltage.

If you're set on 2 power sources, if you want, you can also use a double pole single throw (DPST) switch, which switches two separate lines on a single switch.