title: Failed to execute user defined function in Apache Spark using Scala
body: <p>I have the following dataframe:</p>
  
  <pre><code>+---------------+-----------+-------------+--------+--------+--------+--------+------+-----+
  |   time_stamp_0|sender_ip_1|receiver_ip_2|s_port_3|r_port_4|acknum_5|winnum_6| len_7|count|
  +---------------+-----------+-------------+--------+--------+--------+--------+------+-----+
  |06:36:16.293711|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58| 65161|  130|
  |06:36:16.293729|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58| 65913|  130|
  |06:36:16.293743|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|131073|  130|
  |06:36:16.293765|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|196233|  130|
  |06:36:16.293783|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|196985|  130|
  |06:36:16.293798|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|262145|  130|
  |06:36:16.293820|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|327305|  130|
  |06:36:16.293837|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|328057|  130|
  |06:36:16.293851|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|393217|  130|
  |06:36:16.293873|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|458377|  130|
  |06:36:16.293890|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|459129|  130|
  |06:36:16.293904|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|524289|  130|
  |06:36:16.293926|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|589449|  130|
  |06:36:16.293942|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|590201|  130|
  |06:36:16.293956|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|655361|  130|
  |06:36:16.293977|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|720521|  130|
  |06:36:16.293994|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|721273|  130|
  |06:36:16.294007|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|786433|  130|
  |06:36:16.294028|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|851593|  130|
  |06:36:16.294045|   10.0.0.1|     10.0.0.2|   55518|    5001|       0|      58|852345|  130|
  +---------------+-----------+-------------+--------+--------+--------+--------+------+-----+
  only showing top 20 rows
  </code></pre>
  
  <p>I have to add features and label to my <code>dataframe</code> to predict the count value. However as I ran the code I will see the below error:</p>
  
  <pre><code>Failed to execute user defined function(anonfun$15: (int, int, string, string, int, int, int, int, int) =&gt; vector)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  </code></pre>
  
  <p>I also <code>cast(IntegerType)</code> all my features but again the error occurs. Here is my code:</p>
  
  <pre><code>val Frist_Dataframe = sqlContext.createDataFrame(Row_Dstream_Train, customSchema)
  
         val toVec9 = udf[Vector, Int, Int, String, String, Int, Int, Int, Int, Int] { (a, b, c, d, e, f, g, h, i) =&gt;
                val e3 = c match {
                  case "10.0.0.1" =&gt; 1
                  case "10.0.0.2" =&gt; 2
                  case "10.0.0.3" =&gt; 3
                }
  
                val e4 = d match {
                  case "10.0.0.1" =&gt; 1
                  case "10.0.0.2" =&gt; 2
                  case "10.0.0.3" =&gt; 3
                }
                Vectors.dense(a, b, e3, e4, e, f, g, h, i)
              }
  
              val final_df = Dataframe.withColumn(
                "features",
                toVec9(
                  // casting into Timestamp to parse the string, and then into Int
                  $"time_stamp_0".cast(TimestampType).cast(IntegerType),
                  $"count".cast(IntegerType),
                  $"sender_ip_1",
                  $"receiver_ip_2",
                  $"s_port_3".cast(IntegerType),
                  $"r_port_4".cast(IntegerType),
                  $"acknum_5".cast(IntegerType),
                  $"winnum_6".cast(IntegerType),
                  $"len_7".cast(IntegerType)
                )
              ).withColumn("label", (Dataframe("count"))).select("features", "label")
  </code></pre>
  
  <p>final_df.show()</p>
  
  <pre><code>val trainingTest = final_df.randomSplit(Array(0.8, 0.2))
  val TrainingDF = trainingTest(0).toDF()
  val TestingDF=trainingTest(1).toDF()
  TrainingDF.show()
  TestingDF.show()
  </code></pre>
  
  <p>My dependencies also are:</p>
  
  <pre><code>libraryDependencies ++= Seq(
    "co.theasi" %% "plotly" % "0.2.0",
    "org.apache.spark" %% "spark-core" % "2.1.1",
    "org.apache.spark" %% "spark-sql" % "2.1.1",
    "org.apache.spark" %% "spark-hive" % "2.1.1",
    "org.apache.spark" %% "spark-streaming" % "2.1.1",
    "org.apache.spark" %% "spark-mllib" % "2.1.1"
  )
  </code></pre>
  
  <p>The funnest point is that if I change all my <code>cast(IntegerType)</code> to <code>cast(TimestampType).cast(IntegerType)</code> in the last part of my code, the error disappear and the output will be something like this:</p>
  
  <pre><code>+--------+-----+
  |features|label|
  +--------+-----+
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  |    null|  130|
  +--------+-----+
  </code></pre>
  
  <p><strong>UPDATE</strong>: After applying @Ramesh Maharjan solution the result of my dataframe works well but, whenever I try to splitting my final_df dataframe into training and testing the result is something like below and I still have the same problem of having null rows.</p>
  
  <pre><code>+--------------------+-----+
  |            features|label|
  +--------------------+-----+
  |                null|  130|
  |                null|  130|
  |                null|  130|
  |                null|  130|
  |                null|  130|
  |                null|  130|
  |                null|  130|
  |                null|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  |[1.497587776E9,13...|  130|
  +--------------------+-----+
  </code></pre>
  
  <p>Can you help me?</p>