title: null value and countDistinct with spark dataframe
body: <p>I have a very simple dataframe </p>
  
  <pre><code>  df = spark.createDataFrame([(None,1,3),(2,1,3),(2,1,3)], ['a','b','c'])
  
    +----+---+---+
    |   a|  b|  c|
    +----+---+---+
    |null|  1|  3|
    |   2|  1|  3|
    |   2|  1|  3|
    +----+---+---+
  </code></pre>
  
  <p>When I apply a <code>countDistinct</code> on this dataframe, I find different results depending on the method:</p>
  
  <h3>First method</h3>
  
  <pre><code>  df.distinct().count()
  </code></pre>
  
  <blockquote>
    <p>2</p>
  </blockquote>
  
  <p>It's the result I except, the 2 last rows are identical but the first one is distinct (because of the null value) from the 2 others</p>
  
  <h3>Second Method</h3>
  
  <pre><code>  import pyspark.sql.functions as F
    df.agg(F.countDistinct("a","b","c")).show()
  </code></pre>
  
  <blockquote>
    <p>1</p>
  </blockquote>
  
  <p>It seems that the way <code>F.countDistinct</code> deals with the <code>null</code> value is not intuitive for me.</p>
  
  <p>Does it looks a bug or normal for you ? And if it is normal, how I can write something that output exactly the result of the first approach but in the same spirit than the second Method. </p>