title: Merging data.frames leads to out-of-memory error
body: <p>I have 4GB of RAM, and am having trouble pulling 147.6MB into memory in R in Linux, according to the error message I'm getting: <code>Error: cannot allocate vector of size 147.6 Mb</code>.</p>
  
  <p>How can I diagnose this?</p>
  
  <p>Here is my code:</p>
  
  <pre><code>IDs &lt;- read.csv('Set1.csv')  #   2 MB
  Set2 &lt;- read.csv('Set2.csv') # 240 MB 
  data &lt;- merge(IDs, Set2, by='MemberID')
  rm(IDs)                      # Remove junk! 
  rm(Set2)                     # Remove junk!
  gc()
  print('First merge complete')
  
  Set3 &lt;- read.csv('Set3.csv') # 25 MB
  data &lt;- merge(data, Set3, by='MemberID')
  rm(Set3)                     # Remove junk!
  gc()
  print('Second merge complete')
  </code></pre>
  
  <p>The execution halts after the first print statement.  I can't understand where the extra memory usage is coming from.  Looking around at documentation on memory() in R, it seems to be a contiguous memory issue?  Is there a way to address this in R on Ubuntu?</p>
  
  <p>Also looked at other people asking similar questions here, but the solutions proposed were Windows-specific.</p>
  
  <p><strong>EDIT 1</strong></p>
  
  <p>Some comments to address the comments below:</p>
  
  <pre><code>&gt; print(object.size(IDs), units="Mb")
  1.3 Mb
  &gt; print(object.size(Set2), units="Mb")
  142.6 Mb
  &gt; print(object.size(Set3), units="Mb")
  12.5 Mb
  </code></pre>
  
  <p>So, it doesn't look like the objects are changing size too much from being read in from CSV.  I'll check up on data.table() and the rest...</p>
  
  <p><strong>EDIT 2</strong></p>
  
  <p>I have updated my code to use data.table() and have the same error.  This makes me concerned that perhaps it is somehow particular to my machine? This just seems very strange for the size of the files involved.  <code>Error: cannot allocate vector of size 147.6 Mb</code></p>
  
  <pre><code>IDs &lt;- as.data.table(read.csv('Set1.csv'))  #   2 MB
  Set2 &lt;- as.data.table(read.csv('Set2.csv')) # 240 MB 
  data &lt;- merge(IDs, Set2, by='MemberID')
  rm(IDs)                      # Remove junk! 
  rm(Set2)                     # Remove junk!
  gc()
  print('First merge complete')
  
  Set3 &lt;- as.data.table(read.csv('Set3.csv')) # 25 MB
  data &lt;- merge(data, Set3, by='MemberID')
  rm(Set3)                     # Remove junk!
  gc()
  print('Second merge complete')
  </code></pre>
  
  <p><strong>EDIT 3</strong></p>
  
  <p>Checked through my data, I suspect the problem may be here.  There were some common field-names in Set3.csv, so I think it was doing nasty n x n joins or something.</p>