When using USB and external power, how should I isolate the grounds?
I'm almost1 certain that in your scenario you can connect the ground of USB to the ground of the wall-wart and not worry about it.  Wall-warts2 have isolation transformers.  Its ground is floating.  When you connect the floating ground to the other ground, they will equalize, and there will be no current between the grounds because of the difference in ground potential.

Many of the self-powered USB devices have USB ground connected to the external power ground.

1 "Almost", because I have to go by a verbal description rather than a schematic or a block diagram.
2 Properly made wall-warts.  I'm not talking about the fake ones from China on eBay.