title: PySpark program is throwing error "TypeError: Invalid argument, not a string or column"
body: <p>I have a dataframe named 'new_emp_final_1'. When I try to derive a column 'difficulty' from cookTime and prepTime, by calling the function difficulty from a udf, it is giving me error.</p>
  
  <p>new_emp_final_1.dtypes is below - </p>
  
  <pre><code>[('name', 'string'), ('ingredients', 'string'), ('url', 'string'), ('image', 'string'), ('cookTime', 'string'), ('recipeYield', 'string'), ('datePublished', 'strin
  g'), ('prepTime', 'string'), ('description', 'string')]
  </code></pre>
  
  <p>Result of new_emp_final_1.schema is -</p>
  
  <pre><code>StructType(List(StructField(name,StringType,true),StructField(ingredients,StringType,true),StructField(url,StringType,true),StructField(image,StringType,true),StructField(cookTime,StringType,true),StructField(recipeYield,StringType,true),StructField(datePublished,StringType,true),StructField(prepTime,StringType,true),StructField(description,StringType,true)))
  </code></pre>
  
  <p>Code: </p>
  
  <pre><code>def difficulty(cookTime, prepTime):   
      if not cookTime or not prepTime:
          return "Unkown"
  
      total_duration = cookTime + prepTime
      if total_duration &gt; 3600:
          return "Hard"
      elif total_duration &gt; 1800 and total_duration &lt; 3600:
          return "Medium"
      elif total_duration &lt; 1800:
          return "Easy" 
      else: 
          return "Unkown"
  
  func_udf = udf(difficulty, IntegerType())
  new_emp_final_1 = new_emp_final_1.withColumn("difficulty", func_udf(new_emp_final_1.cookTime, new_emp_final_1.prepTime))
  new_emp_final_1.show(20,False)
  </code></pre>
  
  <p>Error is -</p>
  
  <p><code>File "/home/raghavcomp32915/mypycode.py", line 56, in &lt;module&gt;
      func_udf = udf(difficulty, IntegerType())
    File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/udf.py", line 186, in wrapper
    File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/udf.py", line 166, in __call__
    File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/column.py", line 66, in _to_seq
    File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/column.py", line 54, in _to_java_column
  TypeError: Invalid argument, not a string or column: &lt;function difficulty at 0x7f707e9750c8&gt; of type &lt;type 'function'&gt;. For column literals, use 'lit', 'array', 's
  truct' or 'create_map' function.</code></p>
  
  <p>I am expecting a column named difficulty in existing dataframe new_emp_final_1 with values as Hard, Medium, Easy or Unknown.</p>