Voltage regulator and loss of ground
Hot swap connectors needs to guarantee GND contact first. In your case, you can't have it, and if you don't have an extra connected ground, a diode won't work.

The solution is to use a pass MOSFET on Vdd side, between connector and your regulator. The PFET gate can be driven by a sinking BJT (which only operates with Vdd AND ground). A more robust design should use a hot swap controller, such as STPW12 or ADM1177.