Line level audio and muxing
You need something like the 4066.  Luckily, there are smaller versions with only one switch inside, like a tiny 6-pin SPDT, with 1/3 the circuitry of a 4053.


NLAS4599: Single SPDT Analog Switch Single Supply
NLASB3157: Single SPDT Analog Switch
FSA4157 Low Voltage 1 Ohm SPDT Analog Switch


(I was just looking at these parts for this question.)

You can find examples of how to bias and interface with these kinds of parts here:


Bypassing and Switching with the CD4053 CMOS Analog MUX  
Audio Switch Notes (also shows a JFET switch)
"The Juggler 4 is a CMOS based effects order switcher"