Berchin's FDLS arbitrary filter design algorithm
FDLS requires a causal frequency response. Your prototype frequency response has zero phase everywhere, which is most definitely not causal.

An IIR filter order of 50 is humongous. When FDLS has too many poles and zeroes available, it "tries" to cancel excess poles with excess zeroes. Unfortunately, due to numerical limitations, the cancellation is often pretty poor. For your notch filter, try 2nd or 4th order numerator and denominator, after making the response causal.

Though (number of measurements) = (number of coefficients) = [(numerator order) + (denominator order) + 1] is mathematically sufficient, I find that FDLS likes many more measurements than that. I recommend an absolute minimum of twice as many measurements as coefficients, and prefer 5x to 10x.

Greg Berchin