title: Why does Spark ML NaiveBayes output labels that are different from the training data?
body: <p>I use the <a href="https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala" rel="nofollow">NaiveBayes</a> classifier in <a href="http://spark.apache.org/docs/latest/ml-guide.html" rel="nofollow">Apache Spark ML</a> (version 1.5.1) to predict some text categories. However, the classifier outputs labels that are different from the labels in my training set. Am I doing it wrong?</p>
  
  <p>Here is a small example that can be pasted into e.g. Zeppelin notebook:</p>
  
  <pre><code>import org.apache.spark.ml.Pipeline
  import org.apache.spark.ml.classification.NaiveBayes
  import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
  import org.apache.spark.mllib.linalg.Vector
  import org.apache.spark.sql.Row
  
  // Prepare training documents from a list of (id, text, label) tuples.
  val training = sqlContext.createDataFrame(Seq(
    (0L, "X totally sucks :-(", 100.0),
    (1L, "Today was kind of meh", 200.0),
    (2L, "I'm so happy :-)", 300.0)
  )).toDF("id", "text", "label")
  
  // Configure an ML pipeline, which consists of three stages: tokenizer, hashingTF, and lr.
  val tokenizer = new Tokenizer()
    .setInputCol("text")
    .setOutputCol("words")
  val hashingTF = new HashingTF()
    .setNumFeatures(1000)
    .setInputCol(tokenizer.getOutputCol)
    .setOutputCol("features")
  val nb = new NaiveBayes()
  
  val pipeline = new Pipeline()
    .setStages(Array(tokenizer, hashingTF, nb))
  
  // Fit the pipeline to training documents.
  val model = pipeline.fit(training)
  
  // Prepare test documents, which are unlabeled (id, text) tuples.
  val test = sqlContext.createDataFrame(Seq(
    (4L, "roller coasters are fun :-)"),
    (5L, "i burned my bacon :-("),
    (6L, "the movie is kind of meh")
  )).toDF("id", "text")
  
  // Make predictions on test documents.
  model.transform(test)
    .select("id", "text", "prediction")
    .collect()
    .foreach { case Row(id: Long, text: String, prediction: Double) =&gt;
      println(s"($id, $text) --&gt; prediction=$prediction")
    }
  </code></pre>
  
  <p>The output from the small program:</p>
  
  <pre><code>(4, roller coasters are fun :-)) --&gt; prediction=2.0
  (5, i burned my bacon :-() --&gt; prediction=0.0
  (6, the movie is kind of meh) --&gt; prediction=1.0
  </code></pre>
  
  <p>The set of predicted labels {0.0, 1.0, 2.0} are disjoint from my training set labels {100.0, 200.0, 300.0}. </p>
  
  <p>Question: How can I map these predicted labels back to my original training set labels?</p>
  
  <p>Bonus question: why do the training set labels have to be doubles, when any other type would work just as well as a label? Seems unnecessary.</p>