Switching between two power sources
The simplest way that I can think of uses 3 parts. A CPC1117N, a resistor and a diode (eg. 1N5819). 







simulate this circuit &ndash; Schematic created using CircuitLab

Operation should be self-evident- the presence of the +5V USB source turns off the 9V battery source and D2 prevents back-feeding the USB +5. 

This circuit has no provisions against brown-out in the USB+5 input (for example, a 3V input could switch off the battery and leave only 2.5V at the output). If you need that, add a power supervisor chip to switch the SSR. An LM431 and two resistors would work too.