title: Differences between null and NaN in spark? How to deal with it?
body: <p>In my DataFrame, there are columns including values of null and NaN respectively, such as: </p>
  
  <pre><code>df = spark.createDataFrame([(1, float('nan')), (None, 1.0)], ("a", "b"))
  df.show()
  
  +----+---+
  |   a|  b|
  +----+---+
  |   1|NaN|
  |null|1.0|
  +----+---+
  </code></pre>
  
  <p>Are there any difference between those? How can they be dealt with?</p>