title: How to zip two array columns in Spark SQL
body: <p>I have a Pandas dataframe. I have tried to join two columns containing string values into a list first and then using zip, I joined each element of the list with '_'. My data set is like below:</p>
  
  <pre class="lang-none prettyprint-override"><code>df['column_1']: 'abc, def, ghi'
  df['column_2']: '1.0, 2.0, 3.0'
  </code></pre>
  
  <p>I wanted to join these two columns in a third column like below for each row of my dataframe.</p>
  
  <pre class="lang-none prettyprint-override"><code>df['column_3']: [abc_1.0, def_2.0, ghi_3.0]
  </code></pre>
  
  <p>I have successfully done so in python using the code below but the dataframe is quite large and it takes a very long time to run it for the whole dataframe. I want to do the same thing in PySpark for efficiency. I have read the data in spark dataframe successfully but I'm having a hard time determining how to replicate Pandas functions with PySpark equivalent functions. How can I get my desired result in PySpark?</p>
  
  <pre><code>df['column_3'] = df['column_2']
  for index, row in df.iterrows():
    while index &lt; 3:
      if isinstance(row['column_1'], str):      
        row['column_1'] = list(row['column_1'].split(','))
        row['column_2'] = list(row['column_2'].split(','))
        row['column_3'] = ['_'.join(map(str, i)) for i in zip(list(row['column_1']), list(row['column_2']))]
  </code></pre>
  
  <p>I have converted the two columns to arrays in PySpark by using the below code</p>
  
  <pre><code>from pyspark.sql.types import ArrayType, IntegerType, StringType
  from pyspark.sql.functions import col, split
  
  crash.withColumn("column_1",
      split(col("column_1"), ",\s*").cast(ArrayType(StringType())).alias("column_1")
  )
  crash.withColumn("column_2",
      split(col("column_2"), ",\s*").cast(ArrayType(StringType())).alias("column_2")
  )
  </code></pre>
  
  <p>Now all I need is to zip each element of the arrays in the two columns using '_'. How can I use zip with this? Any help is appreciated.</p>