Keep the current audio output device, when attaching a new one
I use a free app from Rogue Amoeba called SoundSource that lets you quickly pick the source for input, output and system sounds from the task bar. One of the side effects of running SoundSource is your source doesn't change when you connect something new to your Mac. You have to select the new source explicitly via SoundSource to have audio routed to or from it.

The source will switch automatically if you remove a source and that source happened to be the currently selected source for any of the channels.