How to Design Nonlinear Phase FIR Filter? I've recently encountered a rather clever low-pass filter that appears to have been designed to sacrifice uniform group delay in the pass-band to get better frequency response relative to the filter length. What methods are available to design FIR filters that maximize frequency performance relative to length while allowing phase to become nonlinear (but still continuous) in the pass-band?
Although not a comprehensive list, the following are filter algorithms available in Matlab for designing non-linear phase FIR filters:

firlpnorm: Nonlinear phase equiripple or least squares, uses Lp norm: for equiripple filters p = infinity, for least squares p =2.

cfirmpm: complex and nonlinear phase equiripple filters.

See the Matlab documentation for full details on how to design non-linear phase using these algorithms.