Pre-processing irregular, high frequency time-series data in python ...posted originally in StackOverflow (might be better suited here)


Small Picture:


I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data points, as seen below in the timestamp field: 

"timestamp": "2018-06-03T12:27:54.253" 
"timestamp": "2018-06-03T12:27:54.409"
"timestamp": "2018-06-03T12:27:54.548"


I'm in the process of developing a sampling scheme for this time-series data so that I can reduce the number of data points, and standardize time steps without losing information or introducing any bias.

So far I've been using Pandas pd.resample() on just a small subset of our data (5 days ~ 2 million records) by using mean as the aggregation function and linear interpolation. I am on downsampling the data by seconds, minutes, and hours for experimental purposes which takes care of the irregular time steps of the original data.


Larger Picture:


I am working with many millions of records (data from April - today) that have been queried from elasticsearch and, ideally, want to pre-process/sample within this large pool of time series data to only obtain statistically significant data points. The purpose of this pre-processing step is for future data exploration/modeling. 


Question:


How can I modify my current pre-processing scheme to ensure optimal processing for Millions of records and still maintain their statistical properties (no bias introduced)? 

I'm aware that there is possibly a better way to approach the processing this kind of data. Any input is greatly appreciated...thanks!
Keep in mind, downsampling is always lossy. I will give you two hints.

Nyquist-Shannon Theorem. What it says in short is that the max frequency that you may observe in a sampled signal is half of the sampling frequency. This means that if you downsample the data every 1 hour, you can observe dynamic phenomena with period of 2 hours (or more). Another way of looking at the theorem is &quot;you have to sample the signal at least twice as fast as the maximum frequency that you expect to see in it&quot;.
Engineering rule of thumb: Sample the signal at least 10 times faster than the expected highest frequency, not just 2 times.

Histogram Representation. Since you have a huge database, it might make sense to convert your variables into histograms! In this way, you will greatly reduce their size. You discretize every real-valued variable into bins and then instead of storing the exact value of the variable you just increase the count of the appropriate bin by 1. There are a lot of Machine Learning algorithms that deal with histogram representations, such as this:
https://pdfs.semanticscholar.org/b8c8/347f9c33935b97703ecd35a67af5c5508487.pdf


Hope it helps :) !