Text mining of big data using R Server I have a big table (150 Million rows and ~ 70 columns).
In three of the columns in the table I have text input (3-20 words/column), which I need to use for a classification algorithm.

For smaller datasets, I have used the tm R package and created a DocumentTermMatrix, where I used the frequency of word (or word parts) as predictors in a SVM &amp; Decision Forest algorithm.

However, now my dataset is much bigger, which is why I converted it to a xdf file and used RevoScaleR packages for merging, joining etc. so far.  

However, I am unsure how to do text mining with very large datasets (e.g., create a document term matrix which I can use for classification). I did not find any pre-build functions. It may be possible to create a document term matrix in a chunk of data, and then somehow sum the frequencies. I am not sure if that is the best way, could you help me with that?

Thank you,
Lisa
Thank you for your reply.
By accident I found out that MicrosoftML (https://msdn.microsoft.com/en-us/microsoft-r/microsoftml-introduction) offers just what I need - by using the featurizeText functions offered within Microsoft R client and R server.
Maybe it helps others - I found their example  and could it translate to my data (featurizeText() help under Microsoft ML).

trainReviews &lt;- data.frame(review = c( 
        "This is great",
        "I hate it",
        "Love it",
        "Do not like it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I kind of hate it",
        "I do like it",
        "I really hate it",
        "It is very good",
        "I hate it a bunch",
        "I love it a bunch",
        "I hate it",
        "I like it very much",
        "I hate it very much.",
        "I really do love it",
        "I really do hate it",
        "Love it!",
        "Hate it!",
        "I love it",
        "I hate it",
        "I love it",
        "I hate it",
        "I love it"),
     like = c(TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, 
        FALSE, TRUE, FALSE, TRUE), stringsAsFactors = FALSE
    )

    testReviews &lt;- data.frame(review = c(
        "This is great",
        "I hate it",
        "Love it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I love it",
        "I do like it",
        "I really hate it",
        "I love it"), stringsAsFactors = FALSE)


outModel &lt;- rxLogisticRegression(like ~ reviewTran, data = trainReviews,
    mlTransforms = list(featurizeText(vars = c(reviewTran = "review"),
    stopwordsRemover = stopwordsDefault(), keepPunctuations = FALSE)))
# 'hate' and 'love' have non-zero weights summary(outModel)