title: IF Statement Pyspark
body: <p>My data looks like the following: </p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>+----------+-------------+-------+--------------------+--------------+---+
  |purch_date|  purch_class|tot_amt|       serv-provider|purch_location| id|
  +----------+-------------+-------+--------------------+--------------+---+
  |03/11/2017|Uncategorized| -17.53|             HOVER  |              |  0|
  |02/11/2017|    Groceries| -70.05|1774 MAC'S CONVEN...|     BRAMPTON |  1|
  |31/10/2017|Gasoline/Fuel|    -20|              ESSO  |              |  2|
  |31/10/2017|       Travel|     -9|TORONTO PARKING A...|      TORONTO |  3|
  |30/10/2017|    Groceries|  -1.84|         LONGO'S # 2|              |  4|
  </code></pre>
  
  <p>I am attempting to create a binary column which will be defined by the value of the tot_amt column. I would like to add this column to the above data. 
  If tot_amt &lt;(-50) I would like it to return 0 and if tot_amt > (-50) I would like it to return 1 in a new column. </p>
  
  <p>My attempt so far: </p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.sql.types import IntegerType
  from pyspark.sql.functions import udf
  
  def y(row):
      if row['tot_amt'] &lt; (-50):
          val = 1
      else:
          val = 0
          return val
  
  y_udf = udf(y, IntegerType())
  df_7 = df_4.withColumn('Y',y_udf(df_4['tot_amt'], (df_4['purch_class'], 
  (df_4['purch_date'], (df_4['serv-provider'], (df_4['purch_location'])))
  display(df_7)
  </code></pre>
  
  <p>Error message I'm receiving: </p>
  
  <pre class="lang-python prettyprint-override"><code>SparkException: Job aborted due to stage failure: Task 0 in stage 67.0 failed 
  1 times, most recent failure: Lost task 0.0 in stage 67.0 (TID 85, localhost, 
  executor driver): org.apache.spark.api.python.PythonException: Traceback (most 
  recent call last):
  File "/databricks/spark/python/pyspark/worker.py", line 177, in main
  process()
  File "/databricks/spark/python/pyspark/worker.py", line 172, in process
  serializer.dump_stream(func(split_index, iterator), outfile)
  File "/databricks/spark/python/pyspark/worker.py", line 104, in &lt;lambda&gt;
  func = lambda _, it: map(mapper, it)
  File "&lt;string&gt;", line 1, in &lt;lambda&gt;
  File "/databricks/spark/python/pyspark/worker.py", line 71, in &lt;lambda&gt;
  return lambda *a: f(*a)
  TypeError: y() takes exactly 1 argument (2 given)
  </code></pre>