title: Transpose dataframe columns generated from parsed XML in Pyspark
body: <p>As one of the problem statement I am working on parsing XML data using PySpark.</p>
  <p>Below is the sample data -</p>
  <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;no&quot; ?&gt;
  &lt;component&gt;Engineering
  
    &lt;headers&gt;
      &lt;header&gt;
        &lt;name&gt;Date&lt;/name&gt;
        &lt;value&gt;05/05/2021 14:50:03&lt;/value&gt;
      &lt;/header&gt;
      &lt;header&gt;
        &lt;name&gt;StdName&lt;/name&gt;
        &lt;value&gt;CoreEngineering&lt;/value&gt;
      &lt;/header&gt;
      &lt;header&gt;
        &lt;name&gt;ID&lt;/name&gt;
        &lt;value&gt;12432AA&lt;/value&gt;
      &lt;/header&gt;
      &lt;header&gt;
        &lt;name&gt;DeviceType&lt;/name&gt;
        &lt;value&gt;EngineGear&lt;/value&gt;
      &lt;/header&gt;
    &lt;/headers&gt;
    
     &lt;headers&gt;
      &lt;header&gt;
        &lt;name&gt;Date&lt;/name&gt;
        &lt;value&gt;05/05/2021 14:59:13&lt;/value&gt;
      &lt;/header&gt;
      &lt;header&gt;
        &lt;name&gt;StdName&lt;/name&gt;
        &lt;value&gt;CoreEngineering&lt;/value&gt;
      &lt;/header&gt;
      &lt;header&gt;
        &lt;name&gt;ID&lt;/name&gt;
        &lt;value&gt;98344AA&lt;/value&gt;
      &lt;/header&gt;
      &lt;header&gt;
        &lt;name&gt;DeviceType&lt;/name&gt;
        &lt;value&gt;EngineExhaust&lt;/value&gt;
      &lt;/header&gt;
    &lt;/headers&gt;
    
    
   &lt;/component&gt;
  </code></pre>
  <p>While parsing this xml file in databricks using pyspark I am using below logic -</p>
  <pre><code>timestamp = datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S')
  
  df = spark.read \
      .format(&quot;com.databricks.spark.xml&quot;) \
      .option(&quot;rootTag&quot;, &quot;headers&quot;) \
      .option(&quot;rowTag&quot;, &quot;header&quot;) \
      .load(&quot;/mnt/test/sourcedata/sample.xml&quot;) \
      .withColumn(&quot;processeddatetime&quot;,unix_timestamp(lit(timestamp),'yyyy-MM-dd HH:mm:ss').cast(&quot;timestamp&quot;))
  </code></pre>
  <p>And the output received is :</p>
  <p><a href="https://i.stack.imgur.com/w8bMT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/w8bMT.png" alt="enter image description here" /></a></p>
  <p>However, wondering how I can transpose the data received since that is required for further data transformations.
  Required dataset should have the schema like below-</p>
  <p><a href="https://i.stack.imgur.com/24fhX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/24fhX.png" alt="enter image description here" /></a></p>
  <p>How to transpose this data to above expected schema in PySpark?</p>