title: Add an element to an array of structs and merge the arrays of structs in Spark 2.3
body: <p>I've the below dataframe schema as df.currentSchema and need to obtain the expectedSchema as df.expectedSchema, is there a way i can achieve this in Spark 2.3</p>
  
  <p>df.currentSchema:</p>
  
  <pre><code> |-- enqueuedTime: timestamp (nullable = true)
   |-- VIN: string (nullable = true)
   |-- TT: long (nullable = true)
   |-- MSG_TYPE: string (nullable = true)
   |-- ADA: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- E: long (nullable = true)
   |    |    |-- V: double (nullable = true)
   |    |    |-- E: long (nullable = true)
   |    |    |-- V: double (nullable = true)
   |-- ADW: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- E: long (nullable = true)
   |    |    |-- V: double (nullable = true)
  </code></pre>
  
  <p>df.expectedSchema:</p>
  
  <pre><code> |-- enqueuedTime: timestamp (nullable = true)
   |-- VIN: string (nullable = true)
   |-- TT: long (nullable = true)
   |-- MSG_TYPE: string (nullable = true)
   |-- SIGNAL: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- SN: string (nullable = true)
   |    |    |-- E:  long (nullable = true)
   |    |    |-- V:  double (nullable = true)
   |    |    |-- SN: string (nullable = true) 
   |    |    |-- E:  long (nullable = true)
   |    |    |-- V:  double (nullable = true)
   |    |    |-- SN: string (nullable = true)
   |    |    |-- E: long (nullable = true)
   |    |    |-- V: double (nullable = true)
  </code></pre>
  
  <p>Sample data: </p>
  
  <pre><code> +----------------+---+---------+-----------------------------------------------------+--------------------------+
  |vin              |tt |msg_type |ada                                                  |adw                       |                      |
  +-----------------+---+---------+-----------------------------------------------------+--------------------------+
  |FU7XXXXXXXXXXXXXX|0  |SIGNAL   |[{"E":15XXXXXXXX,"V":2, {"E":15XXXXXXXX,"V":1}]      |null                      |                          
  |FU7XXXXXXXXXXXXXX|0  |SIGNAL   |null                                                 |[{"E":15XXXXXXXX,"V":3}]  |                      
  |FU7XXXXXXXXXXXXXX|0  |SIGNAL   |null                                                 |[{"E":15XXXXXXXX,"V":4.1}]|
  +-----------------+---+---------+--------------------------+--------------------------+--------------------------+
  </code></pre>
  
  <p>Note: Two things need to be achieved here:</p>
  
  <ol>
  <li>New field SN to be created for each E, V pair within the element and it's value should be a array name. ex: For the first array column(ADA), the value of SN = ADA.                                               </li>
  <li>Merge the arrays(ADA, ADW) into one single outer array(SIGNAL). </li>
  </ol>