Power glitch when switching between voltage sources
This answer was made before the OP corrected the MOSFET type!!

It's a poor choice of MOSFET the IRF9530 - it can have a gate-source threshold voltage as high as -4V and for a lot of circuits this is going to produce a vast spread of results. In fact the spread would be so large that a few percent of units will not manage to turn the MOSFET on at all. This could easily account for the differences seen between two PCBs.

As for the time taken to properly recover. the 10k resistor has to charge up the gate capacitance and this might be in the order of 1nF. This is a CR time of 10us so it doesn't look likely that this will contribute more than maybe a millisecond maximum. But there are other things to consider too: -


What do you have connected as a load across C1 (2.2uF)? 
How did you remove the presence of the USB supply? 
Did you remove the output from the LDO regulator or remove the input to the regulator? 
Were you consistent on both occasions?


Bottom line is the MOSFET choice is poor.