title: Pyspark - create new column from operations of DataFrame columns gives error "Column is not iterable"
body: <p>I have a PySpark DataFrame and I have tried many examples showing how to create a new column based on operations with existing columns, but none of them seem to work.</p>
  
  <p>So I have t̶w̶o̶ one questions:</p>
  
  <p>1- Why doesn't this code work?</p>
  
  <pre><code>from pyspark import SparkContext, SparkConf
  from pyspark.sql import SQLContext
  import pyspark.sql.functions as F
  
  sc = SparkContext()
  sqlContext = SQLContext(sc)
  
  a = sqlContext.createDataFrame([(5, 5, 3)], ['A', 'B', 'C'])
  a.withColumn('my_sum', F.sum(a[col] for col in a.columns)).show()
  </code></pre>
  
  <p>I get the error:
  <code>TypeError: Column is not iterable</code> </p>
  
  <p><strong>EDIT: Answer 1</strong></p>
  
  <p>I found out how to make this work. I have to use the native Python <code>sum</code> function. <code>a.withColumn('my_sum', F.sum(a[col] for col in a.columns)).show()</code>. It works, but I have no idea why.</p>
  
  <p>2- If there is a way to make this sum work, <strong>how can I write a <code>udf</code> function to do this (and add the result to a new column of a DataFrame)?</strong></p>
  
  <pre><code>import numpy as np
  def my_dif(row):
      d = np.diff(row) # creates an array of differences element by element
      return d.mean() # returns the mean of the array
  </code></pre>
  
  <p>I am using Python 3.6.1 and Spark 2.1.1.</p>
  
  <p>Thank you!</p>