title: Merge with allow.cartasian=TRUE results in too many observations
body: <p>I want to merge 2 data frames (<code>data1</code> and <code>data2</code>). Both initially contain around 35 million observations (around 2GB each).</p>
  
  <p>I removed the duplicates from <code>data2</code>. I would need to keep the duplicates in <code>data 1</code>, as I wish to use them for further calculations per observation in <code>data1</code>.</p>
  
  <p>I initially get the well documented error:</p>
  
  <blockquote>
    <p>Check for duplicate key values in <code>i</code>, each of which join to the same group in <code>x</code> over and over again. If that's ok, try including <code>j</code> and dropping <code>by</code> (by-without-by) so that <code>j</code> runs for each group to avoid the large allocation. If you are sure you wish to proceed, rerun with <code>allow.cartesian=TRUE</code>. Otherwise, please search for this error message in the FAQ, Wiki, Stack Overflow and datatable-help for advice.</p>
  </blockquote>
  
  <p>As a solution (I looked at several topics, such <a href="https://stackoverflow.com/questions/25204859/error-while-merging-data-frames-using-data-table-package">here</a>, <a href="https://stackoverflow.com/questions/11610562/merging-data-tables-uses-more-than-10-gb-ram">here</a>, and <a href="https://stackoverflow.com/questions/23087358/why-is-allow-cartesian-required-at-times-when-when-joining-data-tables-with-dupl">here</a>), I included <code>allow.cartesian=TRUE</code>, but now I run into memory issues. Also, for a subset it works, but it gives me more observations than I wish (<code>data1</code> now has 50 million observations, although is specify <code>all.x=TRUE</code>).</p>
  
  <p>My code is:</p>
  
  <pre><code>#Remove duplicates before merge
  data2 &lt;- unique(data2)
  
  #Merge
  require(data.table)
  data1 &lt;- merge(data1, data2, by="ID", all.x=TRUE, allow.cartesian=TRUE)
  </code></pre>
  
  <p>Any advice on how to merge this, is very welcome.</p>