title: Column is not iterable in pySpark
body: <p>So, we are a bit puzzled. In Jupyter Notebook we have the following data frame:</p>
  
  <pre><code>+--------------------+--------------+-------------+--------------------+--------+-------------------+ 
  |          created_at|created_at_int|  screen_name|            hashtags|ht_count|     single_hashtag|
  +--------------------+--------------+-------------+--------------------+--------+-------------------+
  |2017-03-05 00:00:...|    1488672001|     texanraj|  [containers, cool]|       1|         containers|
  |2017-03-05 00:00:...|    1488672001|     texanraj|  [containers, cool]|       1|               cool|
  |2017-03-05 00:00:...|    1488672002|   hubskihose|[automation, future]|       1|         automation|
  |2017-03-05 00:00:...|    1488672002|   hubskihose|[automation, future]|       1|             future|
  |2017-03-05 00:00:...|    1488672002|    IBMDevOps|            [DevOps]|       1|             devops|
  |2017-03-05 00:00:...|    1488672003|SoumitraKJana|[VoiceOfWipro, Cl...|       1|       voiceofwipro|
  |2017-03-05 00:00:...|    1488672003|SoumitraKJana|[VoiceOfWipro, Cl...|       1|              cloud|
  |2017-03-05 00:00:...|    1488672003|SoumitraKJana|[VoiceOfWipro, Cl...|       1|             leader|
  |2017-03-05 00:00:...|    1488672003|SoumitraKJana|      [Cloud, Cloud]|       1|              cloud|
  |2017-03-05 00:00:...|    1488672003|SoumitraKJana|      [Cloud, Cloud]|       1|              cloud|
  |2017-03-05 00:00:...|    1488672004|SoumitraKJana|[VoiceOfWipro, Cl...|       1|       voiceofwipro|
  |2017-03-05 00:00:...|    1488672004|SoumitraKJana|[VoiceOfWipro, Cl...|       1|              cloud|
  |2017-03-05 00:00:...|    1488672004|SoumitraKJana|[VoiceOfWipro, Cl...|       1|managedfiletransfer|
  |2017-03-05 00:00:...|    1488672004|SoumitraKJana|[VoiceOfWipro, Cl...|       1|         asaservice|
  |2017-03-05 00:00:...|    1488672004|SoumitraKJana|[VoiceOfWipro, Cl...|       1|   interconnect2017|
  |2017-03-05 00:00:...|    1488672004|SoumitraKJana|[VoiceOfWipro, Cl...|       1|                hmi|
  |2017-03-05 00:00:...|    1488672005|SoumitraKJana|[Cloud, ManagedFi...|       1|              cloud|
  |2017-03-05 00:00:...|    1488672005|SoumitraKJana|[Cloud, ManagedFi...|       1|managedfiletransfer|
  |2017-03-05 00:00:...|    1488672005|SoumitraKJana|[Cloud, ManagedFi...|       1|         asaservice|
  |2017-03-05 00:00:...|    1488672005|SoumitraKJana|[Cloud, ManagedFi...|       1|   interconnect2017|
  +--------------------+--------------+-------------+--------------------+--------+-------------------+
  only showing top 20 rows
  
  root
   |-- created_at: timestamp (nullable = true)
   |-- created_at_int: integer (nullable = true)
   |-- screen_name: string (nullable = true)
   |-- hashtags: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- ht_count: integer (nullable = true)
   |-- single_hashtag: string (nullable = true)
  </code></pre>
  
  <p>We are trying to get the count of hashtags per hour. The approach we are taking is to use Window to partition by <code>single_hashtag</code>. Something like this:</p>
  
  <pre><code># create WindowSpec                                 
  hashtags_24_winspec = Window.partitionBy(hashtags_24.single_hashtag). \  
              orderBy(hashtags_24.created_at_int).rangeBetween(-3600, 3600)
  </code></pre>
  
  <p>However, when we try to do the sum of the <code>ht_count</code> column using:</p>
  
  <pre><code>#sum_count_over_time = sum(hashtags_24.ht_count).over(hashtags_24_winspec)
  </code></pre>
  
  <p>we get the following error:</p>
  
  <pre><code>Column is not iterable
  Traceback (most recent call last):
    File "/usr/hdp/current/spark2-client/python/pyspark/sql/column.py", line 240, in __iter__
      raise TypeError("Column is not iterable")
  TypeError: Column is not iterable
  </code></pre>
  
  <p>The error message is not very informative and we are puzzled, which column exactly to investigate. Any ideas?</p>