title: Fast duplicates removal in numpy and python
body: <p>Is there any fast way to obtain unique elements in numpy? I have code similar to this (the last line)</p>
  
  <pre><code>tab = numpy.arange(100000000)
  
  indices1 = numpy.random.permutation(10000)
  indices2 = indices1.copy()
  indices3 = indices1.copy()
  indices4 = indices1.copy()
  
  result = numpy.unique(numpy.array([tab[indices1], tab[indices2], tab[indices3], tab[indices4]]))
  </code></pre>
  
  <p>This is just an example and in my situation <code>indices1, indices2,...,indices4</code> contains different set of indices and have various size. The last line is executed many times and Inoticed that it's actually the bottleneck in my code (<code>{numpy.core.multiarray.arange}</code> to be precesive). Besides, ordering is not important and element in indices array are of <code>int32</code> type. I was thinking about using hashtable with element value as key and tried:</p>
  
  <pre><code>seq = itertools.chain(tab[indices1].flatten(), tab[indices2].flatten(), tab[indices3].flatten(), tab[indices4].flatten())
  myset = {}
  map(myset.__setitem__, seq, [])
  result = numpy.array(myset.keys())
  </code></pre>
  
  <p>but it was even worse.</p>
  
  <p>Is there any way to speed this up? I guess the performance penalty comes from 'fancy indexing' that copy the array but I need the resulting element only to read (I don't modify anything). </p>