title: iterate over pyspark dataframe columns
body: <p>I have the following <code>pyspark.dataframe</code>:</p>
  
  <pre><code>age  state  name    income
  21    DC    john    30-50K
  NaN   VA    gerry   20-30K
  </code></pre>
  
  <p>I'm trying to achieve the equivalent of <code>df.isnull().sum()</code> (from pandas) which produces:</p>
  
  <pre><code>age      1
  state    0
  name     0
  income   0
  </code></pre>
  
  <p>At first I tried something along the lines of:</p>
  
  <pre><code>null_counter = [df[c].isNotNull().count() for c in df.columns]
  </code></pre>
  
  <p>but this produces the following error:</p>
  
  <pre><code>TypeError: Column is not iterable
  </code></pre>
  
  <p>Similarly, this is how I'm currently iterating over columns to get the minimum value:</p>
  
  <pre><code>class BaseAnalyzer:
      def __init__(self, report, struct):
          self.report = report
          self._struct = struct
          self.name = struct.name
          self.data_type = struct.dataType
          self.min = None
          self.max = None
  
      def __repr__(self):
          return '&lt;Column: %s&gt;' % self.name
  
  
  class BaseReport:
      def __init__(self, df):
          self.df = df
          self.columns_list = df.columns
          self.columns = {f.name: BaseAnalyzer(self, f) for f in df.schema.fields}
  
      def calculate_stats(self):
          find_min = self.df.select([fn.min(self.df[c]).alias(c) for c in self.df.columns]).collect()
          min_row = find_min[0]
          for column, min_value in min_row.asDict().items():
              self[column].min = min_value
  
      def __getitem__(self, name):
          return self.columns[name]
  
      def __repr__(self):
          return '&lt;Report&gt;'
  
  report = BaseReport(df)
  calc = report.calculate_stats()
  
  for column in report1.columns.values():
  if hasattr(column, 'min'):
      print("{}:{}".format(column, column.min))
  </code></pre>
  
  <p>which allows me to 'iterate over the columns'</p>
  
  <pre><code>&lt;Column: age&gt;:1
  &lt;Column: name&gt;: Alan
  &lt;Column: state&gt;:ALASKA
  &lt;Column: income&gt;:0-1k
  </code></pre>
  
  <p>I think this method has become way to complicated, how can I properly iterate over ALL columns to provide vaiour summary statistcs (min, max, isnull, notnull, etc..)  The distinction between <code>pyspark.sql.Row</code> and <code>pyspark.sql.Column</code> seems strange coming from pandas.</p>