title: How to improve the efficiency of array slicing and manipulation in Python (Numpy)?
body: <p>I have a dataset carrying N arrays, each array may have a fixed or variable length. I am going to extract a window from the dataset and counts the occurrence of a given number. To make it simple, I assume each array contains 90 numbers. The model will pick a random position on each array and extract M consecutive numbers started from that position as a window strip, and thus total N strip will be obtained. I assume each strip has the same size so all N strips composing a view (window). I have a C++ code to perform the job and work very well, for a dataset of 5x90, repeating the window sampling 1000000 times, it takes roughly 0.002 seconds to finish every 10,000 iterations. I am trying to port the C++ to Python such that I could use the powerful data frame (Pandas) and other tools for related analysis. After some research, it seems that the Numpy array is a good representation of the data array but since the size of each array varied, I need to warp all arrays into a list. The code I have shown below (majority of irrelated code are removed)</p>
  <pre><code>import numpy as np
  import random
  import time
  
  class dataSource:
      _data = None
      _NCOLS = 0
      _NROWS = 0
  
      # NCOLS, NROWS is the dimension of a window (view) of the data
      # _data is a list of NCOLS Numpy array, each array carry 90 numbers
      def __init__(self, NCOLS, NROWS):
          # assuming each array has 90 numbers for this exmaple, it could be varied in actual case
          src = [[1]*90]*NCOLS # just assume all numbers in the array to be 1 for this example
          self._data = [np.array(cc) for cc in src]
          self._NCOLS = NCOLS
          self._NROWS = NROWS
  
      def extractView(self, view):
          pos = [random.randint(0, len(cc)-1-self._NROWS) for cc in self._data]
          for col in range(len(self._data)):
          view[:, col] = self._data[col][pos[col]:pos[col]+self._NROWS]
  
  class dataView:
      _NCOLS = 0
      _NROWS = 0
      _view = None
  
      def __init__(self, NCOLS, NROWS):
          self._NCOLS = NCOLS
          self._NROWS = NROWS
          self._view = np.zeros([self._NROWS, self._NCOLS], order='F')
  
      def __setitem__(self, key, value):
          self._view[key] = value
  
      def count(self, elem):
          return np.count_nonzero(self._view==elem)
  
  if __name__ == '__main__':
      ds = dataSource(5, 3)
      dv = dataView(5, 3)
      batchCount, totalTime, startTime = 1, 0, time.process_time()
      for n in range(10000000):
          if ((n+1)%10000==0):
              dt = time.process_time() - startTime
              totalTime += dt
              print(totalTime / batchCount, ' ', dt)
              batchCount += 1
              startTime = time.process_time()
  
          ds.extractView(dv)
          cnt = dv.count(1) # this could then be used for other analysis, dv may be used by other functions
  </code></pre>
  <p>Running this code in Python 3.8.8 showing that the code takes 0.23 seconds to finish every 10,000 iterations. The Python code is about 100 times slower than C++. In an actual case, the data source could have up to 50 arrays and each may have 50 to 500 numbers, so the run times in Python could be even longer. To apply the C++ code in the real data, it takes about 5 hours to finish all iterations, based on the above estimation, it takes 500 hours to do the same job in Python. I understand that there may be some limit in Python that makes it hard to boost the speed up, but if anything that I could do to optimize my code so to reduce the time (even 10%) will help a lot.</p>