Does it matter to which ground you connect a MOSFET?
Yes it matters. You must make sure that the gate is controlled in reference to the source.
Edit for the OP: You need to make sure that your gate is always connected to a part of the circuit that is connected to the source in such a way as to know the potential difference between the source and the gate. In you circuit, it is unclear that this is true. So I'm not going to say you can't make it work. But if your circuit block that you have labeled +16,8V has diodes or other components that isolates the source from the rest of the circuit when there is no power from the 20V supply, then any voltage you apply to the MOSFET is not going to turn it on (which should be fine since this is the desired state). But when you plug in your 20V power supply, the MOSFET will still be off so you won't charge the batteries. then when you turn on your switch with the 20V power supply attached, you're going to short that power supply. Maybe if you move the line from 20V- to the other side of the resistor, you might get a little closer without any smoke but your MOSFET is still off because it's tied to something that is negative relative to the source (off for an N-Channel MOSFET). Also, you probably don't want the MOSFET in the 20V power line, you want it to disconnect the Batteries when the 20V is plugged in.
Edit for everyone else: Let's just replace the imaginary circuit with one that is more likely to work with battery management and system power selection with well documented operation. Not quite the same learning experience though.



simulate this circuit &ndash; Schematic created using CircuitLab