title: Creating Pyspark DataFrame column that coalesces two other Columns, why am I getting error of 'unicode' object has no attribute isNull?
body: <p>I'm having some trouble with a Pyspark Dataframe.  Specifically, I'm trying to create a column for a dataframe, which is a result of coalescing two columns of the dataframe. </p>
  
  <p>E.g.</p>
  
  <pre><code>this_dataframe = this_dataframe.withColumn('new_max_price', coalesce(this_dataframe['max_price'],this_dataframe['avg(max_price)']).cast(FloatType()))
  </code></pre>
  
  <p>The problem with this code is that it still returns values of "null" in certain rows.  Specifically I'm running this code:</p>
  
  <pre><code>this_dataset.where(col("new_max_price").isNull()).count()
  </code></pre>
  
  <p>This code gives positive results.  So, while this code works, it does not produce intended results.</p>
  
  <p>I found some other questions (such as <a href="https://stackoverflow.com/questions/36179330/selecting-values-from-non-null-columns-in-a-pyspark-dataframe">Selecting values from non-null columns in a PySpark DataFrame</a>) that were asked that were similar, but for some reason I'm unable to replicate their results.  </p>
  
  <p>Here's some code that I have that is based on the aforementioned link:</p>
  
  <pre><code>def coalesce_columns(c1, c2):
      if c1 != None and c2 != None:
          return c1
      elif c1 == None:
          return c2
      else:
          return c1
  
  coalesceUDF = udf(coalesce_columns)
  max_price_col = [coalesceUDF(col("max_price"), col("avg(max_price)")).alias("competitive_max_price")]
  this_dataset.select(max_price_col).show()
  </code></pre>
  
  <p>When I try to execute the last line to test that my results are correct I receive an error.</p>
  
  <blockquote>
    <p>AttributeError: 'unicode' object has no attribute 'isNull'</p>
  </blockquote>
  
  <p>So basically the question is, how can I use a spark sql function to create a column that is the result of coalescing two pyspark dataframe columns?  If this is impossible, what kind of UDF can I use  in order to create some dataframe column that I can append to another dataframe?</p>