Why does over-modelling an adaptive AR NLMS filter fix sharp spikes? I just simulated an auto-regressive second-order model fueled by white noise and estimated the parameters with normalized least-mean-square filters of orders 1-4.

As the first-order filter under-models the system, of course the estimations are weird. The second-order filter finds good estimates, although it has a couple of sharp jumps. This is to be expected from the nature of NLMS filters.

What confuses me is the third- and fourth-order filters. They seem to eliminate the sharp jumps, as seen in the figure below. I can't see what they would add, as the second-order filter is enough to model the system. The redundant parameters hover around $0$ anyway.

Could someone explain this phenomenon for me, qualitatively? What causes it, and is it desirable?

I used step size $\mu=0.01$, $10^4$ samples, and the AR model $x(t)=e(t)-0.9x(t-1)-0.2x(t-2)$ where $e(t)$ is white noise with variance 1.



The MATLAB code, for reference:

% ar_nlms.m
function th=ar_nlms(y,order,mu)
N=length(y);
th=zeros(order,N); % estimated parameters
for t=na+1:N
    phi = -y( t-1:-1:t-na, : );
    residue = phi*( y(t)-phi'*th(:,t-1) );
    th(:,t) = th(:,t-1) + (mu/(phi'*phi+eps)) * residue;
end

% main.m
y = filter( [1], [1 0.9 0.2], randn(1,10000) )';
plot( ar_nlms( y, 2, 0.01 )' );
What appears to be happening is, as you start over-modeling, the error signal becomes less and less white.

I modified your code to return the error signal (part of the residue term).

This plot shows the off-zero-lag coefficients of the xcorr of the error for order = 2 (blue), 3 (red), and 4 (green).  As you can see, the close-to-but-not-zero lag terms are getting larger in magnitude. 

If we look at the FFT (spectrum) of the xcorr of the error, then we see that the lower frequency terms (that cause the large drifts) are getting smaller (the error is containing more high frequencies).

So it seems the effect of over-modeling in this case is to high-pass filter the error, which (for this example) is beneficial.





function [th,err]=ar_nlms(y,order,mu)
eps = 0.000000001;
N=length(y);
th=zeros(order,N); // estimated parameters
err = zeros(1,N);
for t=order+1:N
    phi = -y( t-1:-1:t-order, : );
    err(t) = y(t)-phi'*th(:,t-1);
    residue = phi*( err(t) );
    th(:,t) = th(:,t-1) + (mu/(phi'*phi+eps)) * residue;
    size(residue)
end