Frequency tracking with huge noise I'm working on a frequency tracking problem with noise, where the amplitude of the noise is orders of magnitude higher than the amplitude of my signal (~1000x). Some details:


The signal is roughly sinusoidal, less than 1 Hz with non-stationary frequency and amplitude
The noise is caused by motion, and is (in many cases) not periodic
I have a 3-channel accelerometer reference for the noise. My target signal has some correlation with motion, so adaptive filtering may have issues


Empirical Mode Decomposition and even simple windowed FFT work well in the no-motion case, but fall apart as soon as motion kicks in - the broadband effects of the noise mask the frequency-domain peak of the signal. I tried RLS adaptive filtering, but I believe the aperiodicity of the noise prevents the filter from converging to anything useful. Would something like Extended Kalman Filtering be able to handle this case? What other options are there when your noise overpowers your signal?

Update

Phase-locked Loops:

According to this article, PLLs...


  ... can demodulate FM signals with very high accuracy and reliability, or it can detect signals buried in noise, but it can't do both in a single configuration, because the two tasks require very different setups and assumptions. In its role as an FM detector, a PLL doesn't reject noise very efficiently, and as a weak-signal detector, it can't decode FM very efficiently. The reason should be obvious — to detect FM modulation of a given bandwidth, the PLL's feedback loop low-pass filter (Figure 2, green) must be opened up enough to allow the modulation's bandwidth to pass unimpeded, but this causes the PLL to become more susceptible to noise.




Kalman Filtering:

I've been testing out an Unscented Kalman Filter (UKF) for the problem. I found examples of people using Extended Kalman filters, which work ok for the simplest problem of tracking a sine wave because it's easy to derive the Jacobian of a sinusoid. I opted to try the UKF because of the ease of incorporating further non-linear constraints / future problems where the Jacobian isn't easily accessible.

I set up the filter using a state variable of $\begin{bmatrix} A &amp; \omega &amp; \phi &amp; b\end{bmatrix}^T$, where the measured signal associated with the state is $z = A \sin (\omega - \phi) + b$.

The process model assumes that $A$, $\omega$ and $b$ remain constant, and the phase advances according to the current frequency: $\phi[t+1] = \phi[t] + \omega[t] \times dt$ ($\phi$ is constrained to $[-\pi, \pi)$).

The problem I find with this model is that solutions are not unique. For instance, in the case pictured below, the filter has converged by sending the frequency to 0, and manipulating the phase variable to influence the signal:



In each subfigure, the black line represents the estimate from the filter. The top subfigure is the fit of the model against noisy measurements (blue dots). The remaining subfigures show each variable of the state vector - colored dots are the true value that we would like the filter to converge to.

Even if I could restrict the argument of the sin to one process variable, it would always be possible to set the frequency to 0 and use the amplitude or baseline shift to fit the signal.

My sense is there are clever ways of designing the process model to constrain the behavior of the filter. Can anyone suggest a better way to set up the problem?

For reference, I have attached a python 3 notebook with the code I am using.
might look at:


B. F. La Scala and R. R. Bitmead, "Design of an extended Kalman filter > frequency tracker," Acoustics, Speech, and Signal
Processing, 1994.



  Abstract: This paper discusses the problem of designing a frequency tracker using the extended Kalman filter (EKF). The design
  trade-off between balancing noise rejection and tracking at a maximal
  slew rate is considered, as are the effect of varying design noise
  covariance values. The performance penalties for over- and
  under-design of noise covariances are examined and theoretically
  supported design guidelines are suggested.


There's also the book :


Quinn, Barry G., and Edward James Hannan. The estimation and tracking of frequency. Vol. 9. Cambridge University Press, 2001.


As Peter noted, in SONAR, tracking low SNR tones is done but they typically run off a DFT.  If you want to go this route, you would have to do some experimentation on bin widths and short term averaging.  Fundamentally, you need to filter the signal down to a band that results in a positive SNR.  The band can contain more than a few bins, but the signal has to display a reliable peak when it occupies the bin.  A much more expensive approach would be a bank of chirps.  

I would also suggest, for your remark about process and measure noise being correlated.  One of the reasons I like the book is that they derive the KF directly from the case where measurement and process noise is correlated.


Kailath, Thomas, Ali H. Sayed, and Babak Hassibi. Linear estimation. Vol. 1. Upper Saddle River, NJ: Prentice Hall, 2000.