Tiny, cheap audio buffer with mute?
If you really want zero clicks, then you want to fade to zero instead of shutting off suddenly, because that is still introducing a step change to the signal.  Or maybe you could shut off on a zero crossing, but I can't think of an obvious circuit to do so.

I think I would feed the signal into a divider made of a resistor and FET to ground, and put an RC on the gate of the FET so it would switch slowly into conduction.  This will introduce a severe DC bias in the output, so it could be capacitively coupled into the opamp.  If the DC bias is applied and removed slowly, it won't cause a pop.  Remind me and I will sketch the circuit up.