title: More efficient means of creating a corpus and DTM with 4M rows
body: <p>My file has over 4M rows and I need a more efficient way of converting my data to a corpus and document term matrix such that I can pass it to a bayesian classifier.</p>
  
  <p>Consider the following code:</p>
  
  <pre><code>library(tm)
  
  GetCorpus &lt;-function(textVector)
  {
    doc.corpus &lt;- Corpus(VectorSource(textVector))
    doc.corpus &lt;- tm_map(doc.corpus, tolower)
    doc.corpus &lt;- tm_map(doc.corpus, removeNumbers)
    doc.corpus &lt;- tm_map(doc.corpus, removePunctuation)
    doc.corpus &lt;- tm_map(doc.corpus, removeWords, stopwords("english"))
    doc.corpus &lt;- tm_map(doc.corpus, stemDocument, "english")
    doc.corpus &lt;- tm_map(doc.corpus, stripWhitespace)
    doc.corpus &lt;- tm_map(doc.corpus, PlainTextDocument)
    return(doc.corpus)
  }
  
  data &lt;- data.frame(
    c("Let the big dogs hunt","No holds barred","My child is an honor student"), stringsAsFactors = F)
  
  corp &lt;- GetCorpus(data[,1])
  
  inspect(corp)
  
  dtm &lt;- DocumentTermMatrix(corp)
  
  inspect(dtm)
  </code></pre>
  
  <p>The output:</p>
  
  <pre><code>&gt; inspect(corp)
  &lt;&lt;VCorpus (documents: 3, metadata (corpus/indexed): 0/0)&gt;&gt;
  
  [[1]]
  &lt;&lt;PlainTextDocument (metadata: 7)&gt;&gt;
  let big dogs hunt
  
  [[2]]
  &lt;&lt;PlainTextDocument (metadata: 7)&gt;&gt;
   holds bar
  
  [[3]]
  &lt;&lt;PlainTextDocument (metadata: 7)&gt;&gt;
   child honor stud
  &gt; inspect(dtm)
  &lt;&lt;DocumentTermMatrix (documents: 3, terms: 9)&gt;&gt;
  Non-/sparse entries: 9/18
  Sparsity           : 67%
  Maximal term length: 5
  Weighting          : term frequency (tf)
  
                Terms
  Docs           bar big child dogs holds honor hunt let stud
    character(0)   0   1     0    1     0     0    1   1    0
    character(0)   1   0     0    0     1     0    0   0    0
    character(0)   0   0     1    0     0     1    0   0    1
  </code></pre>
  
  <p><strong>My question is,</strong> what can I use to create a corpus and DTM faster?  It seems to be extremely slow if I use over 300k rows.</p>
  
  <p>I have heard that I could use <code>data.table</code> but I am not sure how.</p>
  
  <p>I have also looked at the <code>qdap</code> package, but it gives me an error when trying to load the package, plus I don't even know if it will work.</p>
  
  <p>Ref. <a href="http://cran.r-project.org/web/packages/qdap/qdap.pdf" rel="noreferrer">http://cran.r-project.org/web/packages/qdap/qdap.pdf</a></p>