title: How do I calculate the percentage of None or NaN values in Pyspark?
body: <p>I have a larger data-set in PySpark and want to calculate the percentage of None/NaN values per column and store it in another dataframe called percentage_missing. For example if the following were the input dataframe: </p>
  
  <pre><code>df = sc.parallelize([
      (0.4, 0.3),
      (None, None),
      (9.7, None), 
      (None, None)
  ]).toDF(["A", "B"])
  </code></pre>
  
  <p>I would like the output to be a dataframe where column 'A' contains the value 0.5 and column 'B' contains the value 0.75.</p>
  
  <p>I am looking for something like this:</p>
  
  <pre><code>for column_ in my_columns:
    amount_missing = df[df[column_] == None].count().div(len(df)) * 100
  </code></pre>
  
  <p>If there is a library with a function that does this I would also be happy to use it. </p>