Rotating Large Arrays, Fastest Possible
Question: I am relatively new to Python and looking for best optimized code for rotating
large multi-dimensional arrays. In the following code I have a 16X600000 32bit
floating point multi-dimensional array and according to the timer it takes
about 30ms to rotate the contents on my quad core acer windows 8 tablet. I was
considering using some Cython routines or something similar if it would be
possible to reduce the time required to rotate the array.

Eventually the code will be used to store y-axis values for a high speed data
plotting graph based around the VisPy package and the 32bit float array will
be passed to an OpenGL routine. I would like to achieve less than 1ms if
possible.

Any comments, recommendations or sample code would be much appreciated.

    
    
    import sys, timeit
    from threading import Thread
    from PyQt4 import  QtGui
    import numpy as np
    
    m = 16              # Number of signals.
    n = 600000          # Number of samples per signal.
    y = 0.0 * np.random.randn(m, n).astype(np.float32) 
    Running = False
    
    class Window(QtGui.QWidget):
        def __init__(self):
            QtGui.QWidget.__init__(self)
            self.button = QtGui.QPushButton('Start', self)
            self.button.clicked.connect(self.handleButton)
            layout = QtGui.QVBoxLayout(self)
            layout.addWidget(self.button)
    
        def handleButton(self):
            global Running, thread, thrTest
            if Running == True:
                Running = False
                self.button.setText('Start')
                thrTest.isRunning = False
                print ('stop')
            else:
                Running = True
                self.button.setText('Stop')
                thrTest = testThread()
                thread = Thread(target=thrTest.run, daemon=True )
                thread.start()
                print ("Start")   
    
    class testThread(Thread):
        def __init__(self):
            self.isRunning = True
        def run(self):
            print('Test: Thread Started')
            while self.isRunning == True:
                start_time = timeit.default_timer()
                y[:, :-1] = y[:, 1:]
                elapsed = timeit.default_timer() - start_time
                print ('Time (s)= ' + str(elapsed))
            print('Test: Closed Thread')
    
    
    if __name__ == '__main__':
        app = QtGui.QApplication(sys.argv)
        window = Window()
        window.show()
        sys.exit(app.exec_())
    

## Update

I guess there has been some confusion about exactly what I am trying to do so
I will try to explain a little better.

The ultimate goal is to have a fast real-time data logging device which draws
line on a graph representing the signal value. There will be multiple channels
and a sampling rate of at least 1ms and as much recording time as possible. I
have started with [this VisPy
example](http://vispy.readthedocs.org/en/latest/examples/demo/gloo/realtime_signals.html).
The code in the example which writes the new data into the arrays and sends it
to OpenGL is in the `On_Timer` function near the bottom. I have modified this
code slightly to integrate the OpenGL canvas into a Qt gui and added some code
to get data from an Arduino Mega through an ethernet socket.

Currently I can produce a real time graph of 16 lines with a sampling rate
right about 1ms and a frame rate of around 30Hz with a recording time of about
14 seconds. If I try to increase the channel count or the recording length any
more the program stops working as it cannot keep up with the flow of data
coming in through the Ethernet port at 1ms.

The biggest culprit I can find for this is the time it takes to complete the
data buffer shift using the `y[:, :-1] = y[:, 1:]` routine. Originally I
submitted benchmark code where this function was being timed in the hope that
someone knew of a way to do the same thing in a more efficient manner. The
purpose of this line is to shift the entire array one index to the left, and
then in my very next line of code I write new data to the first slot on the
right.

Below you can see my modified graph update routine. First it takes the new
data from the queue and unpacks into a temporary array, then it shifts the
contents of the main buffer array, and finally it copies the new data into the
last slot of the main array. Once the queue is empty it calls the update
function so that OpenGL updates the display.

    
    
    def on_timer(self, event):
        """Add some data at the end of each signal (real-time signals)."""
        k=1
        s = struct.Struct('>16H')
        AdrArray =  0.0 * np.random.randn(16,1).astype(np.float32)
        if not q.qsize() == 0:
            while q.qsize() > 0:
                print (q.qsize())
                print ('iin ' + str(datetime.datetime.now()))
                AdrArray[:,0]= s.unpack_from(q.get(), offset=4)
                y[:, :-1] = y[:, 1:]
                y[:, -1:] = .002*AdrArray 
                print ('out ' + str(datetime.datetime.now()))
            self.program['a_position'].set_data(y.ravel().astype(np.float32))
            self.update()
    


Answer: Do you really want this 'roll'? It's shifting the values left, filling with
the last column

    
    
    In [179]: y = np.arange(15).reshape(3,5)
    In [180]: y[:,:-1]=y[:,1:]
    In [181]: y
    Out[181]: 
    array([[ 1,  2,  3,  4,  4],
           [ 6,  7,  8,  9,  9],
           [11, 12, 13, 14, 14]])
    

For nonstandard 'roll' like this it is unlikely that there's anything faster.

`np.roll` has a different fill on the left

    
    
    In [190]: np.roll(y,-1,1)
    Out[190]: 
    array([[ 1,  2,  3,  4,  0],
           [ 6,  7,  8,  9,  5],
           [11, 12, 13, 14, 10]])
    

For what it's worth, the core of `roll` is:

    
    
    indexes = concatenate((arange(n - shift, n), arange(n - shift)))
    res = a.take(indexes, axis)
    

Your particular 'roll' could be reproduced with a similar 'take'

    
    
    In [204]: indexes=np.concatenate([np.arange(1,y.shape[1]),[y.shape[1]-1]])
    In [205]: y.take(indexes,1)
    

Your `y[:,:-1]...` is faster than `roll` because it does not create a new
array; instead it just overwrites part of the existing one.

`take` accepts an `out` parameter, so this is possible:

    
    
    y.take(indexes,1,y)
    

though speed wise this only helps with small arrays. For large ones your
overwriting assignment is faster.

I'd also suggest looking at using the transpose, and rolling on the `axis=0`.
For an `order='C'` array, the values of a row form a contiguous block.

The big time consumer is that you have to copy (nearly) all of the array from
one location to another, either in a new array, or onto itself. If the data
were in some sort of ring buffer, you could just change a pointer, without
having to copy any data.

