USB power/battery source switch
@AKR: Your suspicion about the body diode is right. Since the drain was higher than the source, the transistor was acting as a diode rather than an open transistor. So, fix #1 was to invert the transistor. In the battery only case, the transistor would initially act as a diode, so Vs = Vd - 0.7, then Vgs &lt; Vth and the transistor turns on, hence preventing the voltage drop. The other fix was to add a Schottky diode to prevent current from flowing back to the gate of the FET.

For folks who are interested, here is the final design:



Thanks for the tip! Moving on to other problems...