DC blocking capacitor
This is most likely because you haven't provided any impedance for C1 to work against.  If the signal at right is going into a high impedance input of a chip, which it sounds like could be the case, then any DC level injected at left will transfer to the right and stay there for quite a while.

For example, if whatever is to the right has 10 M&Omega; input impedance, then the time constant with C1 is 10 seconds.  Audio is generally considered to go down to 20 Hz at the lowest, so you can and should filter frequencies below that.  Let's say you want the high pass filter rolloff frequency to be 10 Hz to have little effect at 20 Hz.  That means you want a 16 k&Omega; resistor to ground immediately to the right of C1.  As long as the input impedance of the amplifier to the right is significantly higher than 16 k&Omega;, this will work fine and provide a nice and predictable DC rejection filter.