title: PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe
body: <p>I would like to compute the maximum of a subset of columns for each row and add it as a new column for the existing <code>Dataframe</code>.</p>
  
  <p>I managed to do this in very awkward way:</p>
  
  <pre class="lang-py prettyprint-override"><code>def add_colmax(df,subset_columns,colnm):
       '''
       calculate the maximum of the selected "subset_columns" from dataframe df for each row, 
       new column containing row wise maximum is added to dataframe df. 
  
       df: dataframe. It must contain subset_columns as subset of columns
       colnm: Name of the new column containing row-wise maximum of subset_columns
       subset_columns: the subset of columns from w
       '''
       from pyspark.sql.functions import monotonicallyIncreasingId
       from pyspark.sql import Row
       def get_max_row_with_None(row):
           return float(np.max(row))
  
       df_subset = df.select(subset_columns)
       rdd = df_subset.map( get_max_row_with_None)
       df_rowsum = rdd.map(Row(colnm)).toDF()
       df_rowsum = df_rowsum.withColumn("id",monotonicallyIncreasingId())
       df = df.withColumn("id",monotonicallyIncreasingId())
       df = df.join(df_rowsum,df.id == df_rowsum.id).drop(df.id).drop(df_rowsum.id)
       return df
  </code></pre>
  
  <p>This function works as:</p>
  
  <pre class="lang-py prettyprint-override"><code>rdd1 =  sc.parallelize([("foo", 1.0,3.0,None), 
                      ("bar", 2.0,2.0,-10), 
                      ("baz", 3.3,1.2,10.0)])
  
  
  df1 = sqlContext.createDataFrame(rdd1, ('v1', 'v2','v3','v4'))
  df_new = add_colmax(df1,['v2','v3','v4'],"rowsum")   
  df_new.collect()
  </code></pre>
  
  <p>returns:</p>
  
  <pre><code> [Row(v1=u'bar', v2=2.0, v3=2.0, v4=-10, rowsum=2.0),
    Row(v1=u'baz', v2=3.3, v3=1.2, v4=None, rowsum=3.3),
    Row(v1=u'foo', v2=1.0, v3=3.0, v4=None, rowsum=3.0)]
  </code></pre>
  
  <p>I think that if I could use user defined functions with <code>withColumn</code>, this can be done much simpler. But I could not figure out how to do it.
  Please let me know if you have simpler way to achieve this.
  I am using Spark 1.6</p>