Stability varying with order
FDLS is similar to rational polynomial interpolation in that the interpolation quickly becomes unstable if the target polynomial degree is set too high.  (e.g. just a tiny amount of variation or noise in the spec at a sub-optimal grid spacing can cause extremely wild excursions in the graph of a high order polynomial).  Thus it is usually better if the FDLS order you choose is overdetermined by your specification points to some degree (usually selected by trial-and-error).

The jagged-ness of a frequency response is more related to how close a group of poles or zeros are to the unit circle and each other, rather than to their absolute number.