title: How to use spark Naive Bayes classifier for text classification with IDF?
body: <p>I want to convert text documents into feature vectors using tf-idf, and then train a naive bayes algorithm to classify them.</p>
  
  <p>I can easily load my text files without the labels and use HashingTF() to convert it into a vector, and then use IDF() to weight the words according to how important they are. But if I do that I get rid of the labels and it seems to be impossible to recombine the label with the vector even though the order is the same.</p>
  
  <p>On the other hand, I can call HashingTF() on each individual document and keep the labels, but then I can't call IDF() on it since it requires the whole corpus of documents (and the labels would get in the way).</p>
  
  <p>The spark documentation for naive bayes only has one example where the points are already labeled and vectorized so that isn't much help.</p>
  
  <p>I also had a look at this guide: <a href="http://help.mortardata.com/technologies/spark/train_a_machine_learning_model" rel="nofollow">http://help.mortardata.com/technologies/spark/train_a_machine_learning_model</a>
  but here he only applies the hashing function on each document without idf.</p>
  
  <p>So my question is whether there is a way to not only vectorize but also weight the words using idf for the naive bayes classifier? The main problem seems to be sparks's insistence on only accepting rdds of labeledPoints as input to NaiveBayes.</p>
  
  <pre><code>def parseLine(line):
      label = row[1] # the label is the 2nd element of each row
      features = row[3] # the text is the 4th element of each row
      features = tokenize(features)
      features = hashingTF.transform(features)
      return LabeledPoint(label, features)
  labeledData = data1.map(parseLine)
  </code></pre>