title: How to improve peformance of extracting data code in Python and numpy
body: <p>I'm trying to implement missing point in DIC (Digital Image Correlation) results, which are missing due to destruction of material (photo). </p>
  
  <p><img src="https://i.stack.imgur.com/IcSBe.png" alt="enter image description here"></p>
  
  <p>I would like to put points (without values) for this area where it is missing. (native points from source data)</p>
  
  <p><img src="https://i.stack.imgur.com/sJWv2.png" alt="enter image description here"></p>
  
  <p>This code is part of my thesis and I would like to find those points which are missing due to destruction of material during tensile test. Data comming from DIC (Digital image correlation) method which measure strain on surface of specimen. When specimen suffer local failure DIC software cannot find pixel cluster and finaly missing points in this region.  I have over 30 specimens, each 50 frames. Total computation time for this data is about week.  Code shown bellow in version 1 - nothing to change on my computer is runing something about 4 minutes. Code in version 2 is shorten that: statement MARK1 from begin to end is commented, and Version 2 comment lines are uncommented, but still takes 3 minutes 45 seconds.</p>
  
  <p>Input data: 
  <a href="https://github.com/MarekSawicki/data/blob/master/022_0_29-03-2018_e11_45.csv" rel="nofollow noreferrer">https://github.com/MarekSawicki/data/blob/master/022_0_29-03-2018_e11_45.csv</a></p>
  
  <pre><code>import numpy as np
  import os
  # changing of folder
  os.chdir("D:/Marek/doktorat/Badania_obrobione/test")
  # load data from file
  data = np.genfromtxt('022_0_29-03-2018_e11_45.csv', delimiter=',',dtype='float64')
  # separation of coordintates (points) and values (both floats64)
  # data in format: list of points (X,Y) and list of values
  points = data[1:,1:3]
  values = data[1:,4]
  #shifting coordinates to zero (points coordinates might be negative or offset from 0) (-x0)
  points[:,0] -= min(points[:,0])
  points[:,1] -= min(points[:,1])
  #scale factor K_scale
  k_scale=2
  points[:,0:2] *= k_scale
  # vector reshape
  values=  np.reshape(values, [len(data)-1,1])
  # sort the points to keep order in X direction 
  # points X are assumed as points[:,0]
  # points Y are assumed as points[:,1]
  array1 = np.ascontiguousarray(points)
  a_view = array1.view(dtype=[('', array1.dtype)]*array1.shape[-1])
  a_view.sort(axis=0)
  points_sorted = array1
  # Start of processing points
  # a and b are respectively X and Y limits 
  a = np.int32(np.ceil(np.max(points[:,0])))+1
  b = np.int32(np.ceil(np.max(points[:,1])))+1
  # length 1 unit array cluster
  array2=np.empty((0,2))
  for m in range(0,3):
      for n in range(0,3):
          array2=np.append(array2,[[m*.5,n*.5]],axis=0)
  # initialization of variables
  k=0 # searching limits
  bool_array_del=np.zeros((9,1), dtype=bool) # determine which line should be deleted - bool type
  # array4 is a container of values which meets criteria
  array4=np.empty((0,2))
  # array7 is output container
  array7=np.empty((0,2))
  # main loop of concerned code:
  for i in range(0,a): # X wise loop, a is a X limit
      for i2 in range(0,b): # Y wise loop, a is a Y limit
          array3 = np.copy(array2) # creating a cluster in range (i:i+1,i2:i2+1, step=0.5)
          array3[:,0]+=i
          array3[:,1]+=i2
          # value container (each loop it should be cleaned)
          array4=np.empty((0,2))
          # container which determine data to delete (each loop it should be cleaned)
          bool_array_del = np.empty((0,1),dtype=bool)
          k=0 # set zero searching limits
  # loop for searching points which meet conditions.
  # I think it is the biggest time waster
  #To make it shorter I deal with sorted points which allows me 
  #to browse part of array insted of whole array 
  #(that is why I used k parameter and if-break condition )
          for i3 in range(k,points_sorted.shape[0]):
              valx = points_sorted[i3,0]
              valy = points_sorted[i3,1]
              if valx&gt;i-1:
                  k=i3
              if valx&gt;i+1.5:
                  break
  #this condition check does considered point has X and  coordinates is range : i-0.5:i+1.5
  # If yes then append this coordinate to empty container (array4)
              if np.abs(valx-(i+.5))&lt;=1:
                  if np.abs(valy-(i2+.5))&lt;=1:
                      array4=np.append(array4,[[valx,valy]],axis=0)
  # (version 2)       break
  # Then postprocessing of selected points container - array4. To determine - do all point out of array4 should are close enough to be deleted?
      if array4.shape[0]!=0:
  # (version 2) pass
  # begin(MARK1)
  # round the values from array4 to neares .5 value
          array5 = np.round(array4*2)/2
  # if value from array5 are out of bound for proper cluster values then shift it to the closest correct value
          for i4 in range(0,array5.shape[0]):
              if array5[i4,0]&gt;i+1:
                  array5[i4,0]= i+1
              elif array5[i4,0]&lt;i:
                  array5[i4,0]=i
              if array5[i4,1]&gt;i2+1:
                  array5[i4,1]=i2+1
              elif array5[i4,1]&lt;i2:
                  array5[i4,1]=i2
  # substract i,i2 vector and double from value of array5 to get indices which should be deleted
          array5[:,0]-=i
          array5[:,1]-=i2
          array5*=2
  # create empty container with bool values - True - delete this value, False - keep
          array_bool1=np.zeros((3,3), dtype=bool)
          for i5 in range(0,array5.shape[0]):
  # below condition doesn't work - it is too rough
              #array_bool1[int(array5[i5,0]),int(array5[i5,1])]=True
  # this approach works with correct results but I guess it is second the biggest time waster.
              try: 
                  array_bool1[int(array5[i5,0]),int(array5[i5,1])]=True
                  array_bool1[int(array5[i5,0]+1),int(array5[i5,1]-1)]=True
                      array_bool1[int(array5[i5,0]+1),int(array5[i5,1])+1]=True
                      array_bool1[int(array5[i5,0]+1),int(array5[i5,1])]=True
                      array_bool1[int(array5[i5,0]-1),int(array5[i5,1]+1)]=True
                      array_bool1[int(array5[i5,0]-1),int(array5[i5,1]-1)]=True
                      array_bool1[int(array5[i5,0]-1),int(array5[i5,1])]=True
                      array_bool1[int(array5[i5,0]),int(array5[i5,1]+1)]=True
                      array_bool1[int(array5[i5,0]),int(array5[i5,1]-1)]=True
                  except: 
                      pass
  # convert bool array to list
              for i6 in range(0,array_bool1.shape[0]):
                  for i7 in range(0,array_bool1.shape[1]):
                      bool_array_del=np.append(bool_array_del,    [[array_bool1[i6,i7]]],axis=0)
  # get indices where bool list (unfotunatelly called bool_array_del) is true
              result= np.argwhere(bool_array_del)
              array6=np.delete(array3,result[:,0],axis=0)
  # append it to output container
              array7=np.append(array7,array6,axis=0)
  # if nothing is found in loop for searching points which meet conditions append full cluster to output array
  # end(MARK1)
          else:
              array7=np.append(array7,array3,axis=0)
  </code></pre>
  
  <p>This code gives me satisfactionary results for version 1 (Fig 3) and acceptable results for version 2.(Fig 4)</p>
  
  <p><a href="https://i.stack.imgur.com/O3fjS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/O3fjS.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/HxzjQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HxzjQ.png" alt="enter image description here"></a></p>
  
  <p>I am new in python and numpy. Could you tell me what I can do better to speed up my code? I thought about switch to panda</p>