title: spark convert spark-SQL to RDD API
body: <p>Spark SQL is pretty clear to me. However, I am just getting started with spark's RDD API. As <a href="https://stackoverflow.com/questions/41425308/spark-apply-function-to-columns-in-parallel#41426792">spark apply function to columns in parallel</a> points out this should allow me to get rid of slow shuffles for</p>
  
  <pre><code>def handleBias(df: DataFrame, colName: String, target: String = this.target) = {
      val w1 = Window.partitionBy(colName)
      val w2 = Window.partitionBy(colName, target)
  
      df.withColumn("cnt_group", count("*").over(w2))
        .withColumn("pre2_" + colName, mean(target).over(w1))
        .withColumn("pre_" + colName, coalesce(min(col("cnt_group") / col("cnt_foo_eq_1")).over(w1), lit(0D)))
        .drop("cnt_group")
    }
  }
  </code></pre>
  
  <p>In pseudo code: <code>df foreach column (handleBias(column)</code>
  So a minimal data frame is loaded up</p>
  
  <pre><code>val input = Seq(
      (0, "A", "B", "C", "D"),
      (1, "A", "B", "C", "D"),
      (0, "d", "a", "jkl", "d"),
      (0, "d", "g", "C", "D"),
      (1, "A", "d", "t", "k"),
      (1, "d", "c", "C", "D"),
      (1, "c", "B", "C", "D")
    )
    val inputDf = input.toDF("TARGET", "col1", "col2", "col3TooMany", "col4")
  </code></pre>
  
  <p>but fails to map correctly</p>
  
  <pre><code>val rdd1_inputDf = inputDf.rdd.flatMap { x =&gt; {(0 until x.size).map(idx =&gt; (idx, x(idx)))}}
        rdd1_inputDf.toDF.show
  </code></pre>
  
  <p>It fails with</p>
  
  <pre><code>java.lang.ClassNotFoundException: scala.Any
  java.lang.ClassNotFoundException: scala.Any
  </code></pre>
  
  <p>An example can be found <a href="https://github.com/geoHeil/sparkContrastCoding" rel="nofollow noreferrer">https://github.com/geoHeil/sparkContrastCoding</a> respectively <a href="https://github.com/geoHeil/sparkContrastCoding/blob/master/src/main/scala/ColumnParallel.scala" rel="nofollow noreferrer">https://github.com/geoHeil/sparkContrastCoding/blob/master/src/main/scala/ColumnParallel.scala</a> for the problem outlined in this question.</p>