title: TypeError: Column is not iterable - How to iterate over ArrayType()?
body: <p>Consider the following DataFrame:</p>
  
  <pre class="lang-none prettyprint-override"><code>+------+-----------------------+
  |type  |names                  |
  +------+-----------------------+
  |person|[john, sam, jane]      |
  |pet   |[whiskers, rover, fido]|
  +------+-----------------------+
  </code></pre>
  
  <p>Which can be created with the following code:</p>
  
  <pre class="lang-python prettyprint-override"><code>import pyspark.sql.functions as f
  data = [
      ('person', ['john', 'sam', 'jane']),
      ('pet', ['whiskers', 'rover', 'fido'])
  ]
  
  df = sqlCtx.createDataFrame(data, ["type", "names"])
  df.show(truncate=False)
  </code></pre>
  
  <p>Is there a way to directly modify the <code>ArrayType()</code> column <code>"names"</code> by applying a function to each element, without using a <code>udf</code>?</p>
  
  <p>For example, suppose I wanted to apply the function <code>foo</code> to the <code>"names"</code> column. (I will use the example where <code>foo</code> is <code>str.upper</code> just for illustrative purposes, but my question is regarding any valid function that can be applied to the elements of an iterable.)</p>
  
  <pre class="lang-python prettyprint-override"><code>foo = lambda x: x.upper()  # defining it as str.upper as an example
  df.withColumn('X', [foo(x) for x in f.col("names")]).show()
  </code></pre>
  
  <blockquote>
    <p>TypeError: Column is not iterable</p>
  </blockquote>
  
  <p>I could do this using a <code>udf</code>:</p>
  
  <pre class="lang-python prettyprint-override"><code>foo_udf = f.udf(lambda row: [foo(x) for x in row], ArrayType(StringType()))
  df.withColumn('names', foo_udf(f.col('names'))).show(truncate=False)
  #+------+-----------------------+
  #|type  |names                  |
  #+------+-----------------------+
  #|person|[JOHN, SAM, JANE]      |
  #|pet   |[WHISKERS, ROVER, FIDO]|
  #+------+-----------------------+
  </code></pre>
  
  <p>In this specific example, I <em>could</em> avoid the <code>udf</code> by exploding the column, call <code>pyspark.sql.functions.upper()</code>, and then <code>groupBy</code> and <code>collect_list</code>:</p>
  
  <pre class="lang-python prettyprint-override"><code>df.select('type', f.explode('names').alias('name'))\
      .withColumn('name', f.upper(f.col('name')))\
      .groupBy('type')\
      .agg(f.collect_list('name').alias('names'))\
      .show(truncate=False)
  #+------+-----------------------+
  #|type  |names                  |
  #+------+-----------------------+
  #|person|[JOHN, SAM, JANE]      |
  #|pet   |[WHISKERS, ROVER, FIDO]|
  #+------+-----------------------+
  </code></pre>
  
  <p>But this is a lot of code to do something simple. Is there is a more direct way to iterate over the elements of an <code>ArrayType()</code> using spark-dataframe functions?</p>