title: Speeding up searching for indices within a Large R Data Frame
body: <p>This may look like an innocuously simple problem but it takes a very long time to execute. Any ideas for speeding it up or vectorization etc. would be greatly appreciated.</p>
  
  <p>I have a R data frame with 5 million rows and 50 columns : <code>OriginalDataFrame</code></p>
  
  <p>A list of Indices from that Frame : <code>IndexList</code> (55000 [<code>numIndex</code>] unique indices)</p>
  
  <p>Its a time series so there are ~ 5 Million rows for 55K unique indices.</p>
  
  <p>The <code>OriginalDataFrame</code> has been ordered by <code>dataIndex</code>. All the indices in <code>IndexList</code> are not present in <code>OriginalDataFrame</code>. The task is to find the indices that are present, and construct a new data frame : <code>FinalDataFrame</code> </p>
  
  <p>Currently I am running this code using <code>library(foreach)</code>:</p>
  
  <pre><code>FinalDataFrame &lt;- foreach (i=1:numIndex, .combine="rbind") %dopar% {
    OriginalDataFrame[(OriginalDataFrame$dataIndex == IndexList[i]),]
  }
  </code></pre>
  
  <p>I run this on a machine with 24 cores and 128GB RAM and yet this takes around 6 hours to complete. </p>
  
  <p>Am I doing something exceedingly silly or are there better ways in R to do this?</p>