title: Document-term matrix in R - bigram tokenizer not working
body: <p>I am trying to make 2 document-term matrices for a corpus, one with unigrams and one with bigrams. However, the bigram matrix is currently just identical to the unigram matrix, and I'm not sure why.</p>
  
  <p>The code:</p>
  
  <pre><code>docs&lt;-Corpus(DirSource("data", recursive=TRUE))
  
  # Get the document term matrices
  BigramTokenizer &lt;- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
  dtm_unigram &lt;- DocumentTermMatrix(docs, control = list(tokenize="words", 
      removePunctuation = TRUE, 
      stopwords = stopwords("english"), 
      stemming = TRUE))
  dtm_bigram &lt;- DocumentTermMatrix(docs, control = list(tokenize = BigramTokenizer,
      removePunctuation = TRUE,
      stopwords = stopwords("english"),
      stemming = TRUE))
  
  inspect(dtm_unigram)
  inspect(dtm_bigram)
  </code></pre>
  
  <p>I also tried using ngram(x, n=2) from the ngram package as the tokenizer, but that doesn't work either. How do I fix the bigram tokenization? </p>