title: inner join with huge dataframes (~2 million columns)
body: <p>I am trying to join two data frames (<code>df1</code> and <code>df2</code>) based on matching values from one column (called 'Names') that is found in each data frame. I have tried this using R's <code>inner_join</code> function as well as Python's pandas <code>merge</code> function, and have been able to get both to work successfully on smaller subsets of my data. I think my problem is with the size of my data frames. </p>
  
  <p>My data frames are as follows: </p>
  
  <ul>
  <li><code>df1</code> has the 'Names' column with 5 additional columns and has ~900 rows. </li>
  <li><code>df2</code> has the 'Names' column with ~2million additional columns and has ~900 rows.  </li>
  </ul>
  
  <p>I have tried (in R):</p>
  
  <pre><code>df3 &lt;- inner_join(x = df1, y = df2, by = 'Name') 
  </code></pre>
  
  <p>I have also tried (in Python where <code>df1</code> and <code>df2</code> are Pandas data frames):</p>
  
  <pre><code>df3 = df1.merge(right = df2, how = 'inner', left_on = 1, right_on = 0)   
  </code></pre>
  
  <p>(where the 'Name'  column is at index 1 of df1 and at index 0 of df2)</p>
  
  <p>When I apply the above to my full data frames, it runs for a very long time and eventually crashes.  Additionally, I suspect that the problem may be with the 2 million columns of my <code>df2</code>, so I tried sub-setting it (row-wise) into smaller data frames. My plan was to join the small subsets of <code>df2</code> with <code>df1</code> and then row bind the new data frames together at the end. However, joining even the smaller partitioned <code>df2</code>s was unsuccessful.  </p>
  
  <p>I would appreciate any suggestions anyone would be able to provide. </p>