title: Is there a null-safe comparison operator for pyspark?
body: <p>When trying to create  boolean column that is True if two other column are equal and False otherwise, I noticed that Null == Null = False in spark.</p>
  <pre><code>df.withColumn('newCol', F.when(F.col('x')==F.col('y'), True).otherwise(False))
  </code></pre>
  <p>This: <a href="https://spark.apache.org/docs/3.0.0-preview/sql-ref-null-semantics.html" rel="nofollow noreferrer">https://spark.apache.org/docs/3.0.0-preview/sql-ref-null-semantics.html</a>
  Suggests that I could use &lt;=&gt; if I was using SQL syntax, but am looking to stick to pyspark sql api if I can.</p>
  <pre><code>df.withColumn('newCol', F.when(F.col('x')&lt;=&gt;F.col('y'), True).otherwise(False))
  </code></pre>
  <p>doesn't seem to work.</p>
  <p>Does anyone have any suggestions?</p>