Least-squares digital IIR filter design (with arbitrary responses)
Well, let's decode it:

om=pi*[linspace(0,0.2,20),linspace(0.4,1,60)];


om is a variable that ranges from 0 to pi with two different spacings (one from 0 to 0.2*pi and one from 0.4*pi to pi.

Then, to find D (the desired response), we do:

-1i*om(1:20)*tau


This takes the first twenty entries in om and multiplies them by tau and -i1 = $-\sqrt{-1}$.

That makes the frequency range (D in the above) from 0 to 0.2*pi*tau (i.e. pi since tau is 5). 

Checking page 171, we see:

om frequency grid 0&lt;=om&lt;= pi
D complex desired frequency response on the grid om


which means that om has to be between 0 and pi, and D is the desired response.

If you've changed tau, causing D to have a greater phase (more delay).

What I suspect is happening is that the phase (and, therefore, the delay) you've changed it to cannot easily be approximated by a $4^{\rm th}$ order filter (N or M in your script).