title: Fast reading and combining several files using data.table (with fread)
body: <p>I have several different txt files with the same structure. Now I want to read them into R using fread, and then union them into a bigger dataset. </p>
  
  <pre><code>## First put all file names into a list 
  library(data.table)
  all.files &lt;- list.files(path = "C:/Users",pattern = ".txt")
  
  ## Read data using fread
  readdata &lt;- function(fn){
      dt_temp &lt;- fread(fn, sep=",")
      keycols &lt;- c("ID", "date")
      setkeyv(dt_temp,keycols)  # Notice there's a "v" after setkey with multiple keys
      return(dt_temp)
  
  }
  # then using 
  mylist &lt;- lapply(all.files, readdata)
  mydata &lt;- do.call('rbind',mylist)
  </code></pre>
  
  <p>The code works fine, but the speed is not satisfactory. Each txt file has 1M observations and 12 fields. </p>
  
  <p>If I use the <code>fread</code> to read a single file, it's fast. But using <code>apply</code>, then speed is extremely slow, and obviously take much time than reading files one by one. I wonder where went wrong here, is there any improvements for the speed gain?</p>
  
  <p>I tried the <code>llply</code> in <code>plyr</code> package, there're not much speed gains.</p>
  
  <p>Also, is there any syntax in <code>data.table</code> to achieve vertical join like <code>rbind</code> and <code>union</code> in <code>sql</code>? </p>
  
  <p>Thanks.</p>