Correct comparison of FFT noise with FFT signal
It's interesting that there are no slam-dunk answers here - I think that's good, it means that there is no right answer and what works is going to hinge on the specifics of my implementation. Here is what I did:


I switched to measuring the power spectrum normalized for the Hanning window. This seems to give me a more stable center frequency and reduces the noise when there is no signal
I measured the noise on a per-bin basis and recorded the power spectrum and then averaged the output over a series of periodigrams. Since this reduces the gaussian noise I then multiplied by sqrt(N) where N was the number of periodigrams that I collected. I realise that this will amplify the non-random noise, but it also gives me a reasonable noise floor for the gaussian noise
I compared the noise power with the signal noise, converted to dB and allowed this to be configured by the user. 25dB works well in my case. I special-cased the situation where the noise was &lt; 1 and made this the floor


This seems to work pretty well.