IIR filtering vs IFFT filtering by zero binning and making bandpass IIR filters with high sample rates? I am having some issues making thin IIR filters that work at the sample rate of my data and was wondering if there is a way to improve my method? I am also curious about the deficiencies in using the method of filtering of applying and FFT, bin-zeroing and the IFFTing back to get the filtered data.

For example I use scipy.signal.iirdesign with scipy.signal.filtfilt to filter a 300KHz signal with a 5MHz sample frequency with bandwidth of 4KHz and a transition width of 1KHz. If I do this I find I get NaNs in the filtered signal, if I reduce the sample frequency by indexing every $N^{th}$ element of the data (e.g. by Signal[::N]) I get a correctly filtered signal if I use N=4 (i.e. sample frequency of 5/4MHz). However this is not ideal as I am having to reduce the sample rate of my data and can therefore get no higher temporal resolution. I can increase this frequency by widening the bandpass but this is undesirable as other frequencies are often present nearby. 

I have also found I can create very thin filters by FFTing, zeroing bins and then IFFTing and this works great for a perfect sine wave signal with some noise. However I am interested in extracting a sine wave with random instantaneous phase jumps, and this appears (in simulated tests I have done where i know the true sine wave a-priori) to result in ripples in the amplitude of the signal with the ripples extending furthur the larger the phase shift between the first half of the signal and the next but this is sometimes a preferable method if I can not construct a filter with scipy that will work with the original sample frequency of the data and requires down-sampling.

The following plots demonstrate what I observe.

The blue is the true signal, the red is the filtered signal.

The first plots are a perfect sine wave with no phase shifts or amplitude variation:

IIR:



IFFT:



The IFFT works much better, but this is not that surprising to me since this is a perfect sine wave and is not particularly realistic to observe.

The next plots are of sine wave with a phase shifts of pi/4 in the center with no amplitude variation:

IIR:



IFFT:



The IIR filter used for this has a bandwidth of 1KHz and a transition width of 0.5KHz, it also has a sample frequency that is 4 times less than the original data. The IFFT filter had a bandwidth of 500KHz and appears to fit what the data is doing much closer, it also benefits from having the same sample frequency as the original data.

It would appear from this that using the IFFT filtering method is vastly superior but I know that it should introduce a sinc function to the time signal and the reason FIR and IIR filters are used to approximate a perfect rectangular window in frequency space is that doing so is a bad idea due to the introduction of this sinc function. What is wrong with IFFT filtering in this way with zero binning and how might I improve my method of bandpass filtering such that I can retain a high sample rate and a reasonable thin frequency window?
An IFFT of a single non-zero bin (or conjugate mirrored pair) produces a single pure sinusoid (with no phase or amplitude changes).  An IFFT of just a few adjacent non-zero FFT bins produces something like a slowly amplitude modulated sinewave (a beat note) possibly very slowly modulated in frequency.  Note that none of those IFFT results resemble a sinewave that includes a sudden instantaneous phase shift.  The information about the location of a possible sudden phase change is located in all the other FFT bins that an ultra-narrow FFT filter throws away.  e.g. you would have to de-zero a vast number of FFT bins to get that change to reappear in the IFFT result.  That is why sharp changes have very wide-band spectrums.

This renders your proposed filter (zeroing all but a few FFT bins) nearly useless for determining the location of any sharp phase shift.