Portable Speaker Powered by Battery and Power Supply Separately but Can Charge the Battery From the Same Power Supply
As I understand the question, you want something like this:



This one is using a more powerfull amplifier than PAM8403, but works like you suggest.

This link has all the details and components used in the schematic presented above. 

I suggest you to study this, and in case of any doubts just ask.