How to make my sound reactive LEDs with Tip31C work?
PAM8403 is a Class D bridging amplifier so you are shorting out one of the drivers by grounding it, among other things. 

Try using just the + on each side and use an RC low-pass filter like this: 







simulate this circuit &ndash; Schematic created using CircuitLab

You may have to try other values of R and C to get what you want.