How to workaround switch noise
In order to debug and try and solve this I'd suggest the following (requires oscilloscope):

1)Use an oscilloscope with FFT capability on the output. Then you can tell what is the spectrum of the interference. In fact in Audio system testing, this is done with automated equipment to ensure a high SNR. If the noise is rather narrow, then it's likely something is oscillating and that's easier to fix than if it is wideband.

2) Connect an oscilloscope to VCC and take a look. Do you see anything out of the ordinary like waves riding on it? If you do, an LDO or regulator can act as a low pass filter and filter a lot of noise out of it. Might be a good idea to use a clean supply as well.