title: Memory-efficient ways to extract word nGrams in R
body: <p>I've been using the <code>ngram</code> function(<code>ngram</code> package, same name) to produce some simple ngrams on some text data.  The reason I am looking for alternatives is how the ngram class object is stored is killing my memory.</p>
  
  <p>What I have is a data frame that has two columns uniqueid, texttoparse</p>
  
  <p>Example</p>
  
  <p>1         a bit of text</p>
  
  <p>2         some more text</p>
  
  <p>3         even more to review</p>
  
  <pre><code>w6$ngram &lt;- sapply(w6$texttoparse,ngram)
  w6$actNgram &lt;- as.character(sapply(w6$ngram,get.ngrams))
  </code></pre>
  
  <p>gives me the following ngrams</p>
  
  <p>1  a bit of text // a bit // bit of // of text</p>
  
  <p>2 some more text // some more // more text</p>
  
  <p>3 even more to review // even more // more to // to review</p>
  
  <p>my problem is, the ngram and actNgram fields while displaying correctly appear to have pointers and references to every row in my data frame.  This makes them massive and unwieldy.</p>
  
  <p><a href="https://cran.r-project.org/web/packages/ngram/ngram.pdf" rel="nofollow">https://cran.r-project.org/web/packages/ngram/ngram.pdf</a></p>
  
  <p>even mentions the ngram container impacts the ability to save or load data.</p>
  
  <p>Is there another way I can get to this output?  </p>