title: How to map/convert each element in the ArrayType in Apache Spark 2.3.1
body: <p><strong>Short version:</strong> How can I convert each entry in the nested array to something different (for example a struct) ?</p>
  
  <p>How to convert this:</p>
  
  <pre><code>+---------+
  |column_a |
  +---------+
  |[A, B, C]|
  |[D, E]   |
  |[F]      |
  |[]       |
  +---------+
  </code></pre>
  
  <p>To this:</p>
  
  <pre><code>+---------+---------------------------+
  |column_a |column_b                   |
  +---------+---------------------------+
  |[A, B, C]|[[A, aa], [B, bb], [C, cc]]|
  |[D, E]   |[[D, dd], [E, ee]]         |
  |[F]      |[[F, ff]]                  |
  |[]       |[]                         |
  +---------+---------------------------+
  </code></pre>
  
  <p><strong>Longer version:</strong></p>
  
  <p>Let's assume there is existing DataFrame, with a <code>column_A</code> containing Array of Strings:</p>
  
  <pre><code>val schema = Seq(
     StructField("column_a", ArrayType(StringType), true)
  )
  val data = Seq(
     Row(Array("A", "B", "C")),
     Row(Array("D", "E")),
     Row(Array("F")),
     Row(Array())
  )
  val df = spark.createDataFrame(
     spark.sparkContext.parallelize(data),
     StructType(schema)
  )
  df.show(false)
  
  +---------+
  |column_a |
  +---------+
  |[A, B, C]|
  |[D, E]   |
  |[F]      |
  |[]       |
  +---------+
  </code></pre>
  
  <p>I would like to end up with following DataFrame:</p>
  
  <pre><code>val schema2 = List(
    StructField("column_a", ArrayType(StringType)),
    StructField("column_b", ArrayType(StructType(
    Array(StructField("colA", StringType), StructField("colB", StringType))
    )))
  )
  
  val data2 = Seq(
    Row(Array("A", "B", "C"),
      Array(Row("A", "aa"), Row("B", "bb"), Row("C", "cc"))),
    Row(Array("D", "E"),
      Array(Row("D", "dd"), Row("E", "ee"))),
    Row(Array("F"),
      Array(Row("F", "ff"))),
    Row(Array(), Array())
  )
  
  val df2 = spark.createDataFrame(
    spark.sparkContext.parallelize(data2),
    StructType(schema2)
  )
  
  df2.show(false)
  
  +---------+---------------------------+
  |column_a |column_b                   |
  +---------+---------------------------+
  |[A, B, C]|[[A, aa], [B, bb], [C, cc]]|
  |[D, E]   |[[D, dd], [E, ee]]         |
  |[F]      |[[F, ff]]                  |
  |[]       |[]                         |
  +---------+---------------------------+
  </code></pre>
  
  <p>Values in <code>column_b</code> should be generated using <code>column_A</code> entries. For each entry in <code>column_A</code> there is a new struct in <code>column_B</code>.</p>
  
  <p>What is the valid way in Spark 2.3.1 to achieve this?</p>
  
  <p>(I would prefer to use DSL, although SQL is acceptable.)</p>