fft - spectral leakage from noise into target frequency?
I am assuming you are wanting to measure the phase of the signal, perhaps the amplitude, since you already know the frequency.

I don't think a window will help you at all for the reasons you have cited.  In addition, since you are only interested in a single bin value, that is all you have to calculate instead of doing a whole FFT.

A much better noise reduction approach is to smooth your signal in the time domain.  There are many ways to do this.  My favorite way is to run a simple exponential IIR in the forward direction, then in the backward direction, and average the results of the two passes.  This process leaves the phase unaffected and attenuates the amplitude by a known frequency dependent factor.  This takes out a lot of the noise since a DFT inherently does a least squares fit and outlier data points have a disproportionate effect.

Others may recommend other filters, such as symmetric FIRs, but they take a little bit more calculation.

You may also want to turn up the volume on your emitter.

Hope this helps.

Ced

=========================================

Followup:

I've looked your code over and there are some issues I see:

First, your calculation for reduced_len is very fragile/ineffecient and depends on good values for samplerate and target_freq.  You are better off using this code:


samples_per_cycle = samplerate / target_freq
cycles_per_frame  = 17166 / samples_per_cycle
whole_cycles      = int( cycles_per_frame )
new_length        = whole_cycles * samples_per_cycle
reduced_length    = int( new_length + .5 )


Second, it appears that you are constructing your window function for the full length of y, not reduced_len.

Third, your window correction calculation makes no sense to me.

Now, turning to your problem in general.  There are basically two ways to get rid of the influence of nearby tones in a DFT.  Windowing or estimating its parameters and removing its contribution from the DFT.  Obviously, the latter approach is more complicated.  I have three blog articles that give the math on how to do this and if you are interested I will elaborate further.

For windowing, the one I use for display purposes, and the simplest to understand is the VonHann window.  The window correction factor is a factor of 2 as $ Y[k] = -.25 X[k-1] + .5 X[k] - .25 X[k+1] $ is what happens in the DFT.  For calculation purposes, I don't do windows, they muddle values.

For your application, it doesn't seem to me that important that you calculate the actual amplitude.  If you are measuring changes in amplitude on a relative basis any proportional value will work.

If you want to contact me directly, you can find my email address in my bio in my blog articles.  You can find my blog link on my profile page.

Ced