title: Scala java.lang.String cannot be cast to java.lang.Double error when converting double type dataframe to LabeledPoint in Spark
body: <p>I have a dataset of 2002 variables. All variables are numeric. I first read in the dataset to Spark 1.5.0 and created a Double Type dataframe following the instruction <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema" rel="nofollow noreferrer">here</a> . Then I converted the dataframe to LabeledPoint following instructions <a href="https://stackoverflow.com/questions/31638770/rdd-to-labeledpoint-conversion">here</a> and <a href="https://stackoverflow.com/questions/28785064/scala-convert-arraystring-to-arraydouble">here</a>. However, when I tried to print out sample rows in the generated LabeledPoint, I got the <code>"java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Double"</code> error. Below is the Scala code I used. Sorry for the long code but I hope that will help the debug.</p>
  
  <p>Could anyone please tell me where the error is coming from and how to resolve the problem? Thank you very much for your help!</p>
  
  <p>Below is the Scala code I used:</p>
  
  <pre><code>// Read in dataset but drop the header row
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  val trainRDD = sc.textFile("train.txt").filter(line =&gt; !line.contains("target"))
  // Read in header file to get column names. Store in an Array.
  val dictFile = "header.txt"
  var arrName = new Array[String](2002)
  for (line &lt;- Source.fromFile(dictFile).getLines) {
      arrName = line.split('\t').map(_.trim).toArray
  }
  
  // Create dataframe using programmatically specifying the schema method
  // Encode schema in a string
  var schemaString = arrName.mkString(" ")
  // Import Row
  import org.apache.spark.sql.Row
  // Import RDD
  import org.apache.spark.rdd.RDD
  // Import Spark SQL data types
  import org.apache.spark.sql.types.{StructType,StructField,StringType,IntegerType,LongType,FloatType,DoubleType}
  // Generate the Double Type schema based on the string of schema
  val schema = StructType(schemaString.split(" ").map(fieldName =&gt; StructField(fieldName, DoubleType, true)))
  // Create rowRDD and convert String type to Double type
  val arrVar = sc.broadcast(0 to 2001 toArray)
  def createRowRDD(rdd:RDD[String], anArray:org.apache.spark.broadcast.Broadcast[Array[Int]]) : org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = {
      val rowRDD = rdd.map(_.split("\t")).map(_.map({y =&gt; y.toDouble})).map(p =&gt; Row.fromSeq(anArray.value map p))
      return rowRDD
  }
  val rowRDDTrain = createRowRDD(trainRDD, arrVar)
  // Apply the schema to the RDD.
  val trainDF = sqlContext.createDataFrame(rowRDDTrain, schema)
  trainDF.printSchema
  // Verified all 2002 variables are in "double (nullable = true)" format
  
  // Define toLabeledPoint( ) to convert dataframe to LabeledPoint format
  // Reference: https://stackoverflow.com/questions/31638770/rdd-to-labeledpoint-conversion
  def toLabeledPoint(dataDF:org.apache.spark.sql.DataFrame) : org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = {
      import org.apache.spark.mllib.linalg.Vectors
      import org.apache.spark.mllib.regression.LabeledPoint
      val targetInd = dataDF.columns.indexOf("target")
      val ignored = List("target")
      val featInd = dataDF.columns.diff(ignored).map(dataDF.columns.indexOf(_))
      val dataLP = dataDF.rdd.map(r =&gt; LabeledPoint(r.getDouble(targetInd),
       Vectors.dense(featInd.map(r.getDouble(_)).toArray)))
      return dataLP
  }
  
  // Create LabeledPoint from dataframe
  val trainLP = toLabeledPoint(trainDF)
  // Print out sammple rows in the generated LabeledPoint
  trainLP.take(5).foreach(println)
  // Failed: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Double
  </code></pre>
  
  <p><strong>Update:</strong>  </p>
  
  <p>Thanks a lot for David Griffin's and zero323's comments below. David is correct. I find that exception is indeed caused by the <code>null</code> values in the data. I replaced the following original code:</p>
  
  <pre><code> def createRowRDD(rdd:RDD[String], anArray:org.apache.spark.broadcast.Broadcast[Array[Int]]) : org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = {
      val rowRDD = rdd.map(_.split("\t")).map(_.map({y =&gt; y.toDouble})).map(p =&gt; Row.fromSeq(anArray.value map p))
      return rowRDD
  }
  </code></pre>
  
  <p>with this one to impute <code>null</code> values to 0.0 and then the problem is gone:</p>
  
  <pre><code>def createRowRDD(rdd:RDD[String], anArray:org.apache.spark.broadcast.Broadcast[Array[Int]]) : org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = {
      val rowRDD = rdd.map(_.split("\t")).map(_.map({y =&gt; try {y.toDouble} catch {case _ : Throwable =&gt; 0.0}})).map(p =&gt; Row.fromSeq(anArray.value map p))
      return rowRDD
  }
  </code></pre>