MATLAB Adaptive Filters
Reason for it might be the continuous adaptation, even in non speech regions. So, you can try Voice Activity Detector (VAD) for knowing the exact regions for adaptation of filter, and processing will be frame wise (20 ms).