Use transitor as a switch to ground audio input
If what you want is the ability to switch between audio and silence, you can use an analog mux to switch between your audio input and ground, instead of the amplifier output. As an example, you would ground S1, S2, and A1, hook up your low power audio source to A0, connect A to your amplifier, and use S0 as your switch input. 

If you want stereo, you'll need to use a chip that has two analog multiplexers, and repeat for each line.