How to handle 2nd USB and Audio ground in a project powered from isolated 5V Wall Wart
I would recommend isolating the FTDI chip by using opto-isolators on the UART lines. It is much cheaper and simpler than isolating USB.

The FTDI ground can then be referenced to external ground without affecting the rest of the circuit.