title: What is the fastest way to map group names of numpy array to indices?
body: <p>I'm working with 3D pointcloud of Lidar. The points are given by numpy array that looks like this:</p>
  
  <pre><code>points = np.array([[61651921, 416326074, 39805], [61605255, 416360555, 41124], [61664810, 416313743, 39900], [61664837, 416313749, 39910], [61674456, 416316663, 39503], [61651933, 416326074, 39802], [61679969, 416318049, 39500], [61674494, 416316677, 39508], [61651908, 416326079, 39800], [61651908, 416326087, 39802], [61664845, 416313738, 39913], [61674480, 416316668, 39503], [61679996, 416318047, 39510], [61605290, 416360572, 41118], [61605270, 416360565, 41122], [61683939, 416313004, 41052], [61683936, 416313033, 41060], [61679976, 416318044, 39509], [61605279, 416360555, 41109], [61664837, 416313739, 39915], [61674487, 416316666, 39505], [61679961, 416318035, 39503], [61683943, 416313004, 41054], [61683930, 416313042, 41059]])
  </code></pre>
  
  <p><strong>I'd like to keep my data grouped into cubes of size <code>50*50*50</code> so that every cube preserves some hashable index and numpy indices of my <code>points</code> it contains</strong>. In order to get splitting, I assign <code>cubes = points \\ 50</code> which outputs to:</p>
  
  <pre><code>cubes = np.array([[1233038, 8326521, 796], [1232105, 8327211, 822], [1233296, 8326274, 798], [1233296, 8326274, 798], [1233489, 8326333, 790], [1233038, 8326521, 796], [1233599, 8326360, 790], [1233489, 8326333, 790], [1233038, 8326521, 796], [1233038, 8326521, 796], [1233296, 8326274, 798], [1233489, 8326333, 790], [1233599, 8326360, 790], [1232105, 8327211, 822], [1232105, 8327211, 822], [1233678, 8326260, 821], [1233678, 8326260, 821], [1233599, 8326360, 790], [1232105, 8327211, 822], [1233296, 8326274, 798], [1233489, 8326333, 790], [1233599, 8326360, 790], [1233678, 8326260, 821], [1233678, 8326260, 821]])
  </code></pre>
  
  <h3>My desired output looks like this:</h3>
  
  <pre><code>{(1232105, 8327211, 822): [1, 13, 14, 18]), 
  (1233038, 8326521, 796): [0, 5, 8, 9], 
  (1233296, 8326274, 798): [2, 3, 10, 19], 
  (1233489, 8326333, 790): [4, 7, 11, 20], 
  (1233599, 8326360, 790): [6, 12, 17, 21], 
  (1233678, 8326260, 821): [15, 16, 22, 23]}
  </code></pre>
  
  <p>My real pointcloud contains up to few hundreds of millions of 3D points. What is the fastest way to do this kind of grouping?</p>
  
  <p>I've tried a majority of various solutions. Here is comparison of time compsumption assuming size of points is arround 20 millions and size of distinct cubes is arround 1 million:</p>
  
  <h3>Pandas [tuple(elem) -> np.array(dtype=int64)]</h3>
  
  <pre><code>import pandas as pd
  print(pd.DataFrame(cubes).groupby([0,1,2]).indices)
  #takes 9sec
  </code></pre>
  
  <h3>Defauldict [elem.tobytes() or tuple -> list]</h3>
  
  <pre><code>#thanks @abc:
  result = defaultdict(list)
  for idx, elem in enumerate(cubes):
      result[elem.tobytes()].append(idx) # takes 20.5sec
      # result[elem[0], elem[1], elem[2]].append(idx) #takes 27sec
      # result[tuple(elem)].append(idx) # takes 50sec
  </code></pre>
  
  <h3>numpy_indexed [int -> np.array]</h3>
  
  <pre><code># thanks @Eelco Hoogendoorn for his library
  values = npi.group_by(cubes).split(np.arange(len(cubes)))
  result = dict(enumerate(values))
  # takes 9.8sec
  </code></pre>
  
  <h3>Pandas + dimensionality reduction [int -> np.array(dtype=int64)]</h3>
  
  <pre><code># thanks @Divakar for showing numexpr library:
  import numexpr as ne
  def dimensionality_reduction(cubes):
      #cubes = cubes - np.min(cubes, axis=0) #in case some coords are negative 
      cubes = cubes.astype(np.int64)
      s0, s1 = cubes[:,0].max()+1, cubes[:,1].max()+1
      d = {'s0':s0,'s1':s1,'c0':cubes[:,0],'c1':cubes[:,1],'c2':cubes[:,2]}
      c1D = ne.evaluate('c0+c1*s0+c2*s0*s1',d)
      return c1D
  cubes = dimensionality_reduction(cubes)
  result = pd.DataFrame(cubes).groupby([0]).indices
  # takes 2.5 seconds
  </code></pre>
  
  <p>It's possible to download <code>cubes.npz</code> file <a href="https://github.com/loijord/lidar_home/blob/master/cubes.npz" rel="nofollow noreferrer">here</a> and use a command </p>
  
  <pre><code>cubes = np.load('cubes.npz')['array']
  </code></pre>
  
  <p>to check performance time.</p>