title: PySpark list() in withColumn() only works once, then AssertionError: col should be Column
body: <p>I have a DataFrame with 6 string columns named like 'Spclty1'...'Spclty6' and another 6 named like 'StartDt1'...'StartDt6'. I want to zip them and collapse into a columns that looks like this:
  <code>[[Spclty1, StartDt1]...[Spclty6, StartDt6]]</code></p>
  
  <p>I first tried collapsing just the 'Spclty' columns into a list like this:</p>
  
  <pre><code>DF = DF.withColumn('Spclty', list(DF.select('Spclty1', 'Spclty2', 'Spclty3', 'Spclty4', 'Spclty5', 'Spclty6')))
  </code></pre>
  
  <p>This worked the first time I executed it, giving me a new column called 'Spclty' containing rows such as <code>['014', '124', '547', '000', '000', '000']</code>, as expected.</p>
  
  <p>Then, I added a line to my script to do the same thing on a different set of 6 string columns, named 'StartDt1'...'StartDt6': </p>
  
  <pre><code>DF = DF.withColumn('StartDt', list(DF.select('StartDt1', 'StartDt2', 'StartDt3', 'StartDt4', 'StartDt5', 'StartDt6'))))
  </code></pre>
  
  <p>This caused <code>AssertionError: col should be Column</code>. </p>
  
  <p>After I ran out of things to try, I tried the original operation again (as a sanity check):</p>
  
  <pre><code>DF.withColumn('Spclty', list(DF.select('Spclty1', 'Spclty2', 'Spclty3', 'Spclty4', 'Spclty5', 'Spclty6'))).collect()
  </code></pre>
  
  <p>and got the assertion error as above.</p>
  
  <p>So, it would be good to understand why it only worked the first time (only), but the main question is: what is the correct way to zip columns into a collection of dict-like elements in Spark?</p>