P-Channel Enhancement MOSFET strange behaviour
Low side and high side switching: -



Note that for either FET type, the source connects to the power rail for optimum low volt-drop load switching.


  But there is voltage around 4.43V on the VBUS when OE is disabled


Re-wire the p channel MOSFET as shown above. At the moment the internal bulk diode will be always forward biased and drop about 0.7 volts to produce about 4.3 volts ALWAYS on the load. This is how you have your MOSFET connected at the moment - look at the body diode: -