Backup power source from usb battery
I would suggest a PowerPath circuit topology. You could easily find many ICs for that job. 

Linear Technology has very affordable solutions. The most cheap common is the LTC4412, a tiny PowerPath controller. In page 10, Figure 2 and 3 show some solutions that might fit your demands.

Consider the "WALL ADAPTER INPUT" as your main 5V input. If you want always 5V output from a rechargeable battery even when the main 5V power is absence, just put a DC to DC step-up converter after the battery charger and before the Primary mosfet.