title: Why is Unix/Terminal faster than R?
body: <p>I'm new to Unix, however, I have recently realized that very simple Unix commands can do very simple things to large data set very very quickly.  My question is why are these Unix commands so fast relative to R?   </p>
  
  <p>Let's begin by assuming that the data is big, but not larger than the amount of RAM on your computer. </p>
  
  <p>Computationally, I understand that Unix commands are likely faster than their R counterparts.  However, I can't imagine that this would explain the entire time difference.  After all basic R functions, like Unix commands, are written in low-level languages like C/C++.</p>
  
  <p>I therefore suspect that the speed gains have to do with I/O.  While I only have a basic understanding of how computers work,  I do understand that to manipulate data it most first be read from disk (assuming the data is local). This is slow.  However, regardless of whether you use R functions or Unix commands to manipulate data both most obtain the data from disk. </p>
  
  <p>Therefore I suspect that how data is read from disk, if that even makes sense, is what is driving the time difference.  Is that intuition correct?</p>
  
  <p>Thanks!</p>
  
  <p><strong>UPDATE:</strong>  Sorry for being vague.  This was done on purpose, I was hoping to discuss this idea in general, rather than focus on a specific example. </p>
  
  <p>Regardless, I'll generate an example of counting the number of rows</p>
  
  <p>First I'll generate a big data set.</p>
  
  <pre><code>row = 1e7
  col = 50
  df&lt;-matrix(rpois(row*col,1),row,col)
  write.csv(df,"df.csv")
  </code></pre>
  
  <p>Doing it with Unix</p>
  
  <pre><code>time wc -l df.csv
  real    0m12.261s
  user    0m1.668s
  sys     0m2.589s
  </code></pre>
  
  <p>Doing it with R</p>
  
  <pre><code>library(data.table)
  system.time({ nrow(fread("df.csv")) })
  ...
  user   system   elapsed
  26.77    1.67     47.07
  </code></pre>
  
  <p>Notice that elapsed/real > user + system.  This suggests that the CPU is waiting on the disk.</p>
  
  <p>I suspected the slow speed of R has to do with reading the data in.  It appears that I'm right:</p>
  
  <pre><code>system.time(fread("df.csv"))
  user  system  elapsed
  34.69   2.81    47.41
  </code></pre>
  
  <p>My question is how is the I/O different for Unix and R.  Why?</p>