What component to use for audio signal switching?
I'd consider using analogue switches to do this - they are used extensively on switching thermocouples in and out of circuits and thermocouples are very low level signals. For example, the DG409 is used in this type of application and is specified in the brochure for audio routing applications. They are low power devices too. They can operate with a single +5V supply or from +/-5V supplies to +/-20V. The DG409 has two 4:1 multiplexers meaning you can route your signal to one of four places or choose on of 4 signals to route to a single place. This type of analogue switch comes with different switch configurations such as changeover or normally open too.