title: R reading a huge csv
body: <p>I have a huge csv file. Its size is around 9 gb. I have 16 gb of ram. I followed the advises from the <a href="https://stackoverflow.com/questions/1395229/increasing-or-decreasing-the-memory-available-to-r-processes">page</a> and implemented them below. </p>
  
  <pre><code>If you get the error that R cannot allocate a vector of length x, close out of R and add the following line to the ``Target'' field: 
  --max-vsize=500M 
  </code></pre>
  
  <p>Still I am getting the error and warnings below. How should I read the file of 9 gb into my R? I have R 64 bit 3.3.1 and I am running below command in the rstudio 0.99.903. I have windows server 2012 r2 standard, 64 bit os.</p>
  
  <pre><code>&gt; memory.limit()
  [1] 16383
  &gt; answer=read.csv("C:/Users/a-vs/results_20160291.csv")
  Error: cannot allocate vector of size 500.0 Mb
  In addition: There were 12 warnings (use warnings() to see them)
  &gt; warnings()
  Warning messages:
  1: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  2: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  3: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  4: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  5: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  6: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  7: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  8: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  9: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  10: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  11: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  12: In scan(file = file, what = what, sep = sep, quote = quote,  ... :
    Reached total allocation of 16383Mb: see help(memory.size)
  </code></pre>
  
  <h1>------------------- Update1</h1>
  
  <p>My 1st try based upon suggested answer</p>
  
  <pre><code>&gt; thefile=fread("C:/Users/a-vs/results_20160291.csv", header = T)
  Read 44099243 rows and 36 (of 36) columns from 9.399 GB file in 00:13:34
  Warning messages:
  1: In fread("C:/Users/a-vsingh/results_tendo_20160201_20160215.csv",  :
    Reached total allocation of 16383Mb: see help(memory.size)
  2: In fread("C:/Users/a-vsingh/results_tendo_20160201_20160215.csv",  :
    Reached total allocation of 16383Mb: see help(memory.size)
  </code></pre>
  
  <h1>------------------- Update2</h1>
  
  <p>my 2nd try based upon suggested answer is as below</p>
  
  <pre><code>thefile2 &lt;- read.csv.ffdf(file="C:/Users/a-vs/results_20160291.csv", header=TRUE, VERBOSE=TRUE, 
  +                    first.rows=-1, next.rows=50000, colClasses=NA)
  read.table.ffdf 1..
  Error: cannot allocate vector of size 125.0 Mb
  In addition: There were 14 warnings (use warnings() to see them)
  </code></pre>
  
  <p>How could I read this file into a single object so that I can analyze the entire data in one go</p>
  
  <h1>------------------update 3</h1>
  
  <p>We bought an expensive machine. It has 10 cores and 256 gb ram. That is not the most efficient solution but it works at least in near future. I looked at below answers and I dont think they solve my problem :( I appreciate these answers. I want to perform the market basket analysis and I dont think there is no other way around rather than keeping my data in RAM</p>