title: Pyspark apply parallel summary function to all columns
body: <p>I would like to apply summary and customized stats functions to all columns, independently and <em>in parallel</em>. </p>
  
  <pre><code>from pyspark.sql.functions import rand, randn
  df = sqlContext.range(0, 10).withColumn('uniform', rand(seed=10)).withColumn('normal', randn(seed=27))
  </code></pre>
  
  <p>I have tried to look for answers such as <a href="https://stackoverflow.com/questions/50414300/efficient-column-processing-in-pyspark">here</a> and <a href="https://changhsinlee.com/pyspark-udf/" rel="nofollow noreferrer">here</a>, but they seem to be giving one value for each row. It seems that <code>udf</code> and <code>withColumns</code> are solutions, but I am not sure how to put together to achieve something similar to this :  </p>
  
  <pre><code>df.describe().show()
  +-------+------------------+-------------------+--------------------+
  |summary|                id|            uniform|              normal|
  +-------+------------------+-------------------+--------------------+
  |  count|                10|                 10|                  10|
  |   mean|               4.5| 0.5215336029384192|-0.01309370117407197|
  | stddev|2.8722813232690143|  0.229328162820653|  0.5756058014772729|
  |    min|                 0|0.19657711634539565| -0.7195024130068081|
  |    max|                 9| 0.9970412477032209|  1.0900096472044518|
  +-------+------------------+-------------------+--------------------+
  </code></pre>
  
  <p>Let's say we have a couple of dummy functions for summary</p>
  
  <pre><code>def mySummary1(df_1_col_only):
     return mean(df_1_col_only)*second_largest(df_1_col_only)
  
  def mySummary2(df_1_col_only):
     return 10th_decile(df_1_col_only)/median(df_1_col_only)
  </code></pre>
  
  <p>so that when applying in parallel to all columns </p>
  
  <pre><code>df.something_that_allow_spark_parallel_processing_all_columns(column_list, [mean, mySummary1, mySummary2]) 
  </code></pre>
  
  <p>and produce (preferably the following, or the transposed)</p>
  
  <pre><code>+-------+------------------+-------------------+--------------------+
      |col_name|            mean|       mySummary1|            mySummary2|
      +-------+------------------+-------------------+--------------------+
      |  id|                10|                 10|                  10|
      |uniform|               4.5| 0.5215336029384192|-0.01309370117407197|
      | normal|2.8722813232690143|  0.229328162820653|  0.5756058014772729|
       +-------+------------------+-------------------+--------------------+
  </code></pre>
  
  <p>I would want to utilize the parallel power of Spark to parallelize in both columns list and inside each <em>mySummaryxxx</em> functions.</p>
  
  <p>Another question is what should my df_1_col_only be like for efficiency ? 1-column Spark dataframe ? Is there a way of not copying of duplicating the original dataframe into a bunch of 1-column dataframe ?</p>