title: Stemming function for text2vec
body: <p>I am using text2vec in R and having difficulty writing a stemming function that works with the itoken function in the text2vec package. The text2vec documentation suggests this stemming function: </p>
  
  <pre><code>stem_tokenizer1 =function(x) {
    word_tokenizer(x) %&gt;% lapply(SnowballC::wordStem(language='en'))
   }
  </code></pre>
  
  <p>However, this function does not work. This is the code I ran (borrowed from previous stackoverflow answers): </p>
  
  <pre><code>library(text2vec)
  library(data.table)
  library(SnowballC)
  data("movie_review")
  train_rows = 1:1000
  prepr = tolower
  stem_tokenizer1 =function(x) {
    word_tokenizer(x) %&gt;% lapply(SnowballC::wordStem(language='en'))
   }
  tok = stem_tokenizer1
  it &lt;- itoken(movie_review$review[train_rows], prepr, tok, ids = movie_review$id[train_rows])
  </code></pre>
  
  <p>This is the error it produces:  </p>
  
  <p><strong>Error in { : argument "words" is missing, with no default</strong></p>
  
  <p>I believe the issue is that wordStem needs a character vector, but word_tokenizer produces a list of character vectors.</p>
  
  <pre><code>mr&lt;-movie_review$review[1]
  stem_mr1&lt;-stem_tokenizer1(mr)
  </code></pre>
  
  <p><strong>Error in SnowballC::wordStem(language = "en") : 
    argument "words" is missing, with no default</strong></p>
  
  <p>To fix this issue, I wrote this stemming function: </p>
  
  <pre><code>stem_tokenizer2 = function(x)  {
    list(unlist(word_tokenizer(x)) %&gt;% SnowballC::wordStem(language='en') )
  }
  </code></pre>
  
  <p>However, this function does not work with the create_vocabulary function. </p>
  
  <pre><code>data("movie_review")
  train_rows = 1:1000
  prepr = tolower
  stem_tokenizer2 = function(x)  {
    list(unlist(word_tokenizer(x)) %&gt;% SnowballC::wordStem(language='en') )
  }
  tok = stem_tokenizer2
  it &lt;- itoken(movie_review$review[train_rows], prepr, tok, ids = movie_review$id[train_rows])
  v &lt;- create_vocabulary(it) %&gt;% prune_vocabulary(term_count_min = 5)
  </code></pre>
  
  <p>No error, but when you look at the document count,  the number of documents is different than the 1000 in the data, and so you cannot create a document term matrix or run an LDA.</p>
  
  <pre><code>v$document_count
  </code></pre>
  
  <p><strong>[1] 10</strong></p>
  
  <p>This code:</p>
  
  <pre><code>dtm_train &lt;- create_dtm(it, vectorizer)
  dtm_train
  </code></pre>
  
  <p>Producess this error: </p>
  
  <p><strong>10 x 3809 sparse Matrix of class "dgCMatrix"
  Error in validObject(x) : 
    invalid class “dgCMatrix” object: length(Dimnames[1]) differs from Dim[1] which is 10</strong></p>
  
  <p>My questions are: is there something wrong with the function I wrote, and why would the function I wrote produce this error with create_vocabulary? I suspect it is a problem with the format of the output of my function, but it looks identical to the word_tokenizer function's output format, and that works fine with itoken and create_vocabulary:</p>
  
  <pre><code>mr&lt;-movie_review$review[1]
  word_mr&lt;-word_tokenizer(mr)
  stem_mr&lt;-stem_tokenizer2(mr)
  str(word_mr)
  str(stem_mr)
  </code></pre>