title: What is the fastest way and fastest format for loading large data sets into R
body: <p>I have a large dataset (about 13GB uncompressed) and I need to load it repeatedly. The first load (and save to a different format) can be very slow but every load after this should be as fast as possible.  What is the fastest way and fastest format from which to load a data set?</p>
  <p>My suspicion is that the optimal choice is something like</p>
  <pre><code> saveRDS(obj, file = 'bigdata.Rda', compress = FALSE)
   obj &lt;- loadRDS('bigdata.Rda)
  </code></pre>
  <p>But this seems slower than using <code>fread</code> function in the <code>data.table</code> package. This should not be the case because <code>fread</code> converts a file from CSV (although it is admittedly highly optimized).</p>
  <p>Some timings for a ~800MB dataset are:</p>
  <pre><code>&gt; system.time(tmp &lt;- fread(&quot;data.csv&quot;))
  Read 6135344 rows and 22 (of 22) columns from 0.795 GB file in 00:00:43
       user  system elapsed 
       36.94    0.44   42.71 
   saveRDS(tmp, file = 'tmp.Rda'))
  &gt; system.time(tmp &lt;- readRDS('tmp.Rda'))
       user  system elapsed 
       69.96    2.02   84.04
  </code></pre>
  <h3>Previous Questions</h3>
  <p><a href="https://stackoverflow.com/questions/4756989/how-to-load-data-quickly-into-r">This</a> question is related but does not reflect the current state of R, for example an answer suggests reading from a binary format will always be faster than a text format. The suggestion to use *SQL is also not helpful in my case as the entire data set is required, not just a subset of it.</p>
  <p>There are also related questions about the fastest way of loading data once (eg: <a href="https://stackoverflow.com/questions/1727772/quickly-reading-very-large-tables-as-dataframes-in-r">1</a>).</p>