title: PySpark, Decision Trees (Spark 2.0.0)
body: <p>I am new to Spark (using PySpark). I tried running the Decision Tree tutorial from <a href="https://spark.apache.org/docs/1.5.2/ml-decision-tree.html" rel="nofollow noreferrer">here (link)</a>. I execute the code:</p>
  <pre><code>from pyspark.ml import Pipeline
  from pyspark.ml.classification import DecisionTreeClassifier
  from pyspark.ml.feature import StringIndexer, VectorIndexer
  from pyspark.ml.evaluation import MulticlassClassificationEvaluator
  from pyspark.mllib.util import MLUtils
  
  # Load and parse the data file, converting it to a DataFrame.
  data = MLUtils.loadLibSVMFile(sc, &quot;data/mllib/sample_libsvm_data.txt&quot;).toDF()
  labelIndexer = StringIndexer(inputCol=&quot;label&quot;, outputCol=&quot;indexedLabel&quot;).fit(data)
  
  # Now this line fails
  featureIndexer =\
      VectorIndexer(inputCol=&quot;features&quot;, outputCol=&quot;indexedFeatures&quot;, maxCategories=4).fit(data)
  </code></pre>
  <p>I get the error message:</p>
  <blockquote>
  <p>IllegalArgumentException: u'requirement failed: Column features must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually org.apache.spark.mllib.linalg.VectorUDT@f71b0bce.'</p>
  </blockquote>
  <p>When searching the web for this error I found an answer that says:</p>
  <blockquote>
  <p>use<br />
  <code>from pyspark.ml.linalg import Vectors, VectorUDT</code><br />
  instead of<br />
  <code>from pyspark.mllib.linalg import Vectors, VectorUDT</code></p>
  </blockquote>
  <p>which is odd, since I haven't used it. Also, adding this import to my code solves nothing and I still get the same error.</p>
  <p>I am not quite clear on how to debug this situation. When looking into the raw data I see:</p>
  <pre><code>data.show()
  +--------------------+-----+
  |            features|label|
  +--------------------+-----+
  |(692,[127,128,129...|  0.0|
  |(692,[158,159,160...|  1.0|
  |(692,[124,125,126...|  1.0|
  |(692,[152,153,154...|  1.0|
  </code></pre>
  <p>This looks like a list, starts with '('.</p>
  <p>I am not sure how to solve this issue, or even debug.</p>