title: OneHotEncoder in Spark Dataframe in Pipeline
body: <p>I've been trying to get an example running in Spark and Scala with the <a href="http://mlr.cs.umass.edu/ml/datasets/Adult" rel="nofollow noreferrer">adult dataset</a> .</p>
  
  <p>Using Scala 2.11.8 and Spark 1.6.1.</p>
  
  <p>The problem (for now) lies in the amount of categorical features in that dataset that all need to be encoded to numbers before a Spark ML algorithm can do its job..</p>
  
  <p>So far I have this:</p>
  
  <pre><code>import org.apache.spark.ml.Pipeline
  import org.apache.spark.ml.classification.LogisticRegression
  import org.apache.spark.ml.feature.OneHotEncoder
  import org.apache.spark.sql.SQLContext
  import org.apache.spark.{SparkConf, SparkContext}
  
  object Adult {
    def main(args: Array[String]): Unit = {
      val conf = new SparkConf().setAppName("Adult example").setMaster("local[*]")
      val sparkContext = new SparkContext(conf)
      val sqlContext = new SQLContext(sparkContext)
  
      val data = sqlContext.read
        .format("com.databricks.spark.csv")
        .option("header", "true") // Use first line of all files as header
        .option("inferSchema", "true") // Automatically infer data types
        .load("src/main/resources/adult.data")
  
      val categoricals = data.dtypes filter (_._2 == "StringType")
      val encoders = categoricals map (cat =&gt; new OneHotEncoder().setInputCol(cat._1).setOutputCol(cat._1 + "_encoded"))
      val features = data.dtypes filterNot (_._1 == "label") map (tuple =&gt; if(tuple._2 == "StringType") tuple._1 + "_encoded" else tuple._1)
  
      val lr = new LogisticRegression()
        .setMaxIter(10)
        .setRegParam(0.01)
      val pipeline = new Pipeline()
        .setStages(encoders ++ Array(lr))
  
      val model = pipeline.fit(training)
    }
  }
  </code></pre>
  
  <p>However, this doesn't work. Calling <code>pipeline.fit</code> still contains the original string features and thus throws an exception.
  How can I remove these <code>"StringType"</code> columns in a pipeline?
  Or maybe I'm doing it completely wrong, so if someone has a different suggestion I'm happy to all input :).</p>
  
  <p>The reason why I choose to follow this flow is because I have an extensive background in Python and Pandas, but am trying to learn both Scala and Spark.</p>