Use transistor to auto swap power source
This is often called "Power Path" and you use FETs connected as ideal diodes. Many chargers have this functionality built in, like the LTC4160 from Linear. TI also have a number of battery chargers with Power Path. (No direct link, you have to search the TI site). 

Microchip has an app note about load sharing if you want to roll your own simple version.

You can also get ideal diodes nicely packaged.