Switching between power sources using a P-mosfet. Problem with LDO
You need an LDO with reverse current protection such as the MIC5225 (example only).  See the reverse current rating on page 5 when EN is low.  Note that they show values for grounded and disconnected VIN.  You will need to find an LDO that matches your other specifications with the same reverse current protection feature.