How to filter noise from continuous path in image Problem statement



In the dataset above, you can notice a continuous path and then a noise-band at around 320-400 on the x-axis. I want to be able to apply a filter that amplifies the continuous path while suppressing the noise-band. The data inside the noise-band is useless.

Attempt 1

I am new to signal processing, but I took a shot at designing my own filter. I tried designing a minimum filter. I defined it to make the output signal $ S $ the minimum of the next $ w $ terms in the input signal $ F $: $ S[n] = \min \{F[x] | x \in \{n, n+1, n+2, ..., n+w\}\} $

for i in range(len(original_signal) - window_len):
    filtered_signal[i:i+window_len] = min(original_signal[i:i+window_len])


It represents each pixel by the lowest pixel in the area which should make the noise go away because the noise band will have a lot of low values in between a lot of high values. It takes out too much of the signal as well.

This is an image showing what the filter took away (the difference between the original signal and the filtered signal)



Attempt 2

A median filter will not work, because the median of the noise-band is relatively high (1e4):



whereas the median of empty space is not that high (1e3):



So the median filter would not reduce the noise to be even with empty space.

Do you know any filters I could try?

Additional Data

Histogram of desired signal:



All histograms normalized and put on the same axes:
So entropy filter sounded good, but it didn't really work because the actual signal is also considered high entropy, even after adjusting the window.

What I did was take a derivative of all of the data in the time axis (left to right). Then I capped the maximum value. Then I integrated the derivative.

deriv = data[i, 1:] - data[i, :-1]
deriv[deriv &gt; max_] = max_


The derivitave is $ \frac{dy}{dx }$. In my mind dy is a vertical line segment, dx is a horizontal line segment. Putting them together makes a right triangle, where the function is the hypotenuse is f, the horizontal leg is dx and the vertical leg is dy. Thus $ tan^{-1} {\frac{dy}{dx}} $ is the angle of elevation for f. In my mind, putting a maximum on the derivative is like putting a maximum angle on f. It is like saying, "Cap the angle of elevation for f. If it goes above 150 degrees from the horizon, make it smaller."

It seems to work. It limits the maximum angle of elevation for the signal. You can see the original signal (in red) and the signal after derivative filter (in green).


Then I take original signal minus derivative filter applied to original signal. This leaves me with just the noise band, which I can amplify, blur, and then subtract out.