title: Pyspark: Using lambda function and .withColumn produces a none-type error I'm having trouble understanding
body: <p>I have the following code below. Essentially what I'm trying to do is to generate some new columns from the values in existing ones.  After I do that, I save the dataframe with the new columns as a table in the cluster.  Sorry I'm new to pyspark still.  </p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  from pyspark.sql.functions import udf, array
  from pyspark.sql.types import DecimalType
  import numpy as np
  import math
  
  df = sqlContext.sql('select * from db.mytable')
  
  angle_av = udf(lambda (x, y): -10 if x == 0 else math.atan2(y/x)*180/np.pi, DecimalType(20,10))
  
  df = df.withColumn('a_v_angle', angle_av(array('a_v_real', 'a_v_imag')))
  
  df.createOrReplaceTempView('temp')
  
  sqlContext.sql('create table new_table as select * from temp')
  </code></pre>
  
  <p>These operations actually don't produce any errors. I then attempt to store the df as a table and get the following error (i'm guessing since this is when the operations are actually executed):</p>
  
  <pre class="lang-python prettyprint-override"><code>File "/usr/hdp/current/spark2-client/python/pyspark/worker.py", line 171, in main
      process()
    File "/usr/hdp/current/spark2-client/python/pyspark/worker.py", line 166, in process
      serializer.dump_stream(func(split_index, iterator), outfile)
    File "/usr/hdp/current/spark2-client/python/pyspark/worker.py", line 103, in &lt;lambda&gt;
      func = lambda _, it: map(mapper, it)
    File "&lt;string&gt;", line 1, in &lt;lambda&gt;
    File "/usr/hdp/current/spark2-client/python/pyspark/worker.py", line 70, in &lt;lambda&gt;
      return lambda *a: f(*a)
    File "&lt;stdin&gt;", line 14, in &lt;lambda&gt;
  TypeError: unsupported operand type(s) for /: 'NoneType' and 'NoneType'
      at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
      at org.apache.spark.api.python.PythonRunner$$anon$1.&lt;init&gt;(PythonRDD.scala:234)
  </code></pre>