Cleaning time series data I have a time series data about daily usage of a computer program, here is an example


2017-11-10: 0
2017-11-09: 14
2017-11-08: 0
2017-11-07: 6
2017-11-06: 102
2017-11-05: 0
2017-11-04: 0


As you can see 11-06 has a spike at 102. Due to our way of gathering this data, we know that data is probably erroneous and we are sure that 102 is not correct according other values.

So we need to clean these dirty values.

Is there a mathematical way to do this?
Is there a python lib to help us?
I think you have a few options:


If you have a pre-set rule to exclude outliers, such as a hard-threshold at 100 which you know the data shouldn't exceed, then something as simple as x = [e for e in x if e &lt; 100] will do.
If you have a parametric belief, such as any observation that falls beyond so many standard deviations from mean, or quartiles, are outliers; then you can implement the other answers that have been mentioned.
Else, you can go for a clustering approach. Here I believe your first shot should be a k-means clustering. This is super easy to build and interpret. See my code below.

x = [0,14,0,6,102,0,0]
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2).fit(np.array(x).reshape(-1, 1))

#First cluster:
np.array(x)[np.where(kmeans.labels_ == 0)]

#Second cluster (outliers):
np.array(x)[np.where(kmeans.labels_ == 1)]
K-means is known to be sensitive to outliers, hence a more robust method such as MeanShift, which you tried, is a good rival to k-means. I would run both, and stick with the result that makes better sense to me.


Hope this helps!