Switching between power supplies - USB dominant
If you don't mind sourcing and using a dedicated IC for the task and are willing to sacrifice a bit of board space, you can put a LTC4417 to work at this job, along the lines of the circuit below.  You'll have to tweak the values to fit, though: I did a "rough cut" at the datasheet procedure for applying this chip, but you'll want to go through the datasheet application procedure yourself to make sure you're getting things like inrush currents and voltage rail droop right.







simulate this circuit &ndash; Schematic created using CircuitLab

The good news with this solution is that you can't do something silly with it inadvertently, like backpower the USB port, and it takes care of all that prioritization business for you (the IC always prioritizes V1 over V2 when both are functioning).