I monitor a variable in real-time where new value is generated every 1/25 second. Depending on the conditions, this variable may either be stable, i.e. fluctuate a little (+/-1%) around some value (for simplicity, I'll take real value of 40 from my recent experiments) or fluctuate widely from 0 up to 6x or 8x times bigger than stable value (200-370 for the example). My goal is to detect these stable periods.Right now, I solve this problem like follows: measure mean and deviation of the incoming variable since thebegginning of measurements; if at any given point the ratio ofdeviation/mean is greater than a threshold (0.15 right now) Iconsider that the variable has entered unstable period, otherwise - thevariable is considered stable.This approach generally worked, but not so good. Sometimes, detector fails to detect unstable periods. Here is the picture of recent experiment where ratio (in percentage) was less than 15%, detector considered period as a stable, however it can be clearly seen that variable becomes unstable towards the end of the graph (left axis is for the mean and delta - value of the variable; right axis for the deviation/mean ratio in percents):As you can see, variable (blue line, delta) starts to fluctuate wildly at the end of the graph, however ratio deviation/mean stays under threshold (the maximum is around 8%) and doesn't trigger detector.So I'm wondering, whether there are some existing oscillation detectors techniques/algorithms which can help to adjust the sensitivity of the detector? Currently, I'm thinking of using sliding average for the mean value and lowering the threshold up to 8%.High-level system descriptionThere is a producer which generates packets at constant rate, though it might get interrupted and inter-producing delay may not be exactly the one chosen (i.e. target is 25fps, but delay may vary between 30 and 50ms, very rarely it can be interrupted and delay may become twice as much). These packets are available on the network and cached. Consumer asks for these packets by issuing requests. It aims at exhausting the cache - by issuing large amount of requests. Those requests that ask for non-existent data, will become pending and answered as soon as corresponding packets are generated. Therefore, consumer will know that it has exhausted the cache once the data will arrive with stable period, i.e. the one that roughly corresponds to the target producer rate. As one can see, this inter-arrival delay is affected 1/ by producer (it's not exactly 40ms) and 2/ network disruptions.Here's the sample data; there are bunch of additional parameters, but the real observed value is in delta column.
Oscillations detector: how to effectively detect oscillations of a variable?
real time;statistics
I tested a solution that keeps a running average of the previous N samples, and N samples before that. This has the effect of smoothing the noise out, but you can compare the differences between the two averages to determine if any significant change has occurred between the two.If this solution doesn't work, I have other ideas, but this is simple and seems to work well with the sample data you posted.I did a mock up in Python. I'm not sure of your comfort level with that, so I didn't make it very pythonic, so that it could be easily transcribed to whatever language you're using.import numpy as npfrom collections import dequeimport pylab as pltN = 25windowA=deque(maxlen=N)windowB=deque(maxlen=N)avgA = []avgB = []with open('stab-estimator - Sheet1.tsv', 'r') as f:  f.readline() # skip header  for line in f:    delta = float( line.split()[1] )    windowB.append(delta)    windowA.append(windowB[0])    # I'll store the averages so we can plot them, but you can just use    # avgA/avgB as a metric for stability, without storing a history.    avgA.append(np.sum(windowA)/len(windowA))    avgB.append(np.sum(windowB)/len(windowB))# Plot the result:avgA = np.array(avgA)avgB = np.array(avgB)plt.figure()# plt.plot(avgA-avgB)# Or equally telling:plt.plot(avgA/avgB)plt.show()The plot of that ratio looks like this:As you can see, it presents a good metric for your observed instability. You could easily set a threshold of 10% and catch that instability at the end. You may also want to tune N. The size of N will be a compromise between smoothing and sensitivity to changes. If N is too large, then the signal will have to grow more quickly to overcome the large averaging window.Also, you will have to mute the alarm at the beginning while the averages converge to running values.HTH.