Filter Design near Nyquist frequency (complex curve fitting)
You can't get a good fit very near the Nyquist frequency (or near DC) because information is lost due to the imaginary components around that frequency being of (almost) opposite phase and thus (nearly) cancelling out.  Thus, there will be ambiguity errors due to trying to fit any curve to lost information.

If possible, you can try to gather more information near (but not at) the Nyquist frequency by sampling for a longer time span or at a higher sample rate.  This will move the measurement point slightly farther from where there is complete cancellation of the imaginary component.