Building a RS232 switch
If you don't need all the control lines on the serial port you could re-allocate some of the pins on a 1990s-style switch-box switch to your audio.

Switching or not switching the grounds may have a significant effect on your audio.
These multi-pole rotary switches are not all that common and if you need that many pins, taking them out of a box may be cheaper than buying the switch as a component, for a one-off.