Microphone mute anti-pop circuit works but still picks up a little audio
Check to see if C2 is microphonic by listening while tapping directly on C2. If this is the problem, it can be fixed by replacing C2 with a Mylar capacitor. Ceramic capacitors are often microphonic. I don't expect the aluminum electrolytic to be microphonic, but it is worth checking. 

It could be that the effective series resistance (ESR) of C1 is too high. Check for this by adding another 100uF capacitor in parallel with it. If this improves the problem by about 6dB, ESR is probably the root cause.

Good aluminum electrolytic capacitors have a specification for ESR. Typical 100uF capacitor ESR is in the 0.3 to 2 Ohm range, with extremes from 0.07 to 14 Ohms. Choose a 100uF part with an ESR of about 0.08 Ohms to make a better short.