MOSFET Floating Drain
Your circuit does not work as you are expecting.
In case you unplug 5Vusb: you cannot rely that source of mosfet is connected to 5V. 
MOSFET "has" diode between drain and source, and that is what plays role here.
This is scheme of standard P-MOSFET:


So, when I redraw your scheme, this is what is going on in your circuit in reality: