title: PySpark divide column by its sum
body: <p>I am trying to divide columns in PySpark by their respective sums. My dataframe(using only one column here) looks like this:</p>
  
  <pre><code>event_rates = [[1,10.461016949152542], [2, 10.38953488372093], [3, 10.609418282548477]]
  event_rates = spark.createDataFrame(event_rates, ['cluster_id','mean_encoded'])
  event_rates.show()
  
  +----------+------------------+
  |cluster_id|      mean_encoded|
  +----------+------------------+
  |         1|10.461016949152542|
  |         2| 10.38953488372093|
  |         3|10.609418282548477|
  +----------+------------------+
  </code></pre>
  
  <p>I tried two methods to do this but have failed in getting results</p>
  
  <pre><code>from pyspark.sql.functions import sum as spark_sum
  cols = event_rates.columns[1:]
  for each in cols:
      event_rates = event_rates.withColumn(each+"_scaled", event_rates[each]/spark_sum(event_rates[each]))
  </code></pre>
  
  <p>This gives me the following error</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: grouping expressions sequence is empty, and '`cluster_id`' is not an aggregate function. Wrap '((`mean_encoded` / sum(`mean_encoded`)) AS `mean_encoded_scaled`)' in windowing function(s) or wrap '`cluster_id`' in first() (or first_value) if you don't care which value you get.;;
  Aggregate [cluster_id#22356L, mean_encoded#22357, (mean_encoded#22357 / sum(mean_encoded#22357)) AS mean_encoded_scaled#2
  </code></pre>
  
  <p>and following the question <a href="https://stackoverflow.com/questions/39993879/normalize-values-of-multiple-columns-in-spark-dataframe-using-only-dataframe-ap?rq=1">here</a> I tried the following </p>
  
  <pre><code>stats = (event_rates.agg([spark_sum(x).alias(x + '_sum') for x in cols]))
  event_rates = event_rates.join(broadcast(stats))
  exprs = [event_rates[x] / event_rates[event_rates + '_sum'] for x in cols]
  event_rates.select(exprs)
  </code></pre>
  
  <p>But I get an error from the first line stating </p>
  
  <pre><code>AssertionError: all exprs should be Column
  </code></pre>
  
  <p>How do I get across this?</p>