Incrementally and statistically measuring samples with Python (Numpy, Pandas, etc) and performance
Question: Let's say I take a stream of incoming data (very fast) and I want to view
various stats for a window (std deviation, (say, the last N samples, N being
quite large). What's the most efficient way to do this with Python?

For example,

    
    
    df=ps.DataFrame(np.random.random_sample(200000000))
    df2 = df.append([5])
    

Is crashing my REPL environment in visual studio.

Is there a way to append to an array without this happening? Is there a way to
tell which operations on the dataframe are computed incrementally other than
by doing timeit on them?


Answer: I recommend building a circular buffer out of a numpy array.

This will involve keeping track of an index to your last updated point, and
incrementing that when you add a new value.

    
    
    import numpy as np
    
    circular_buffer = np.zeros(200000000, dtype=np.float64)
    head = 0
    
    # stream input
    
        circular_buffer[head] = new_value
    
        if head == len(circular_buffer) - 1:
            head = 0
        else:
            head += 1
    

Then you can compute the statistics normally on circular_buffer.

### If You Don't Have Enough RAM

Try implementing something similar in
[bquery](https://github.com/visualfabriq/bquery) and
[bcolz](https://github.com/Blosc/bcolz). These store your data more
efficiently than numpy (using compression) and offer similar performance.
Bquery now has mean and standard deviation. _Note: I'm a contributer to
bquery_

