High Order Filtering
So, first of all, we need to be clear that you mean FIR filtering not IIR.

By "high-order filtering", I presume you mean an FIR with a long length:  $L \gg 1$.  That is not unusual.  FIR filters have to be pretty long in order to get anything useful done.  The sharper the selectivity of the filter, the larger $L$ must be.

Most often, the best practice for using the "windowing method" of FIR design, is to use a Kaiser Window because it's optimized to give you, for a fixed $L$, the best stopband rejection or the sharpest transition-band region (what goes in between the passband and the stopband).  There is this $\beta$ parameter that let's you adjust that tradeoff.

So, if you're doing the windowing method (say, using MATLAB), first choose and FFT length, $N$, that is much much larger than your expected FIR length.  Like your FIR length might be $L=1024$, but then set $N=65536$ or something.  Then draw out your target frequency response and inverse FFT that to see what the impulse response might look like.  Remember the latter half is really negative frequency before the iFFT and negative time after the iFFT, so you may have to use fftshift() to swap the halves.  Then apply the window to that longer FIR to make it shorter, but for the purpose of analysis, keep the zeroed samples that became zero because of the windowing.  Then FFT back to see how bad the windowing affected your frequency response.  Adjust the Kaiser $\beta$ to see what your tradeoff looks like.

Rinse and repeat.  If you can't get the frequency response you want, you may have to increase the FIR length $L$.