title: doing better than numpy's in1d mask function: ordered arrays?
body: <p>This operation needs to be applied as fast as possible as the actual arrays which contain millions of elements. This is a simple version of the problem.</p>
  
  <p>So, I have a random array of <em>unique</em> integers (normally millions of elements).</p>
  
  <blockquote>
    <p>totalIDs = [5,4,3,1,2,9,7,6,8 ...]</p>
  </blockquote>
  
  <p>I have another array (normally a tens of thousands) of unique integers which I can create a mask.</p>
  
  <pre><code>subsampleIDs1 = [5,1,9]
  subsampleIDs2 = [3,7,8]
  subsampleIDs3 = [2,6,9]
  ...
  </code></pre>
  
  <p>I can use numpy to do </p>
  
  <blockquote>
    <p>mask = np.in1d(totalIDs,subsampleIDs,assume_unique=True)</p>
  </blockquote>
  
  <p>I can then extract the information I want of another array using the mask (say column 0 contains the one I want).</p>
  
  <blockquote>
    <p>variable = allvariables[mask][:,0]</p>
  </blockquote>
  
  <p>Now given that the IDs are unique in both arrays, is there any way to speed this up significantly. It takes a long time to construct the mask for a few thousand points (subsampleIDs) matching against millions of IDs (totalIDs).</p>
  
  <p>I thought of going through it once and writing out a binary file of an index (to speed up future searches).</p>
  
  <pre><code>for i in range(0,3):
      mask = np.in1d(totalIDs,subsampleIDs,assume_unique=True)
      index[mask] = i
  </code></pre>
  
  <p>where X is in subsampleIDsX. Then I can just do:</p>
  
  <pre><code>for i in range(0,3):
      if index[i] == i:
          rowmatch = i
          break
  
  variable = allvariables[rowmatch:len(subsampleIDs),0]
  </code></pre>
  
  <p>right? But this is also slow because there is a conditional in the loop to find when it first matches. Is there a faster way to find when a number first appears in an ordered array so the conditional doesn't slow the loop?</p>