Using a Wiener Filter to Estimate a Transfer Function
The desired MSE is application dependent, so there can be no general
rule. If the approximation doesn't satisfy your needs you can
increase the filter length to obtain a better match.
There is no straightforward way to change the FIR Wiener filter solution to an IIR solution because the IIR formulation results in a set of nonlinear equations which have no closed-form solution. The IIR solution might also be unstable, so FIR filters are a much more practical choice when computing a Wiener filter.